Proposing a Deep Learning-Based Method for Estimating the Absolute Position of Potholes in Urban Streets

Hadigol, Homayoon; Hosseininaveh, Ali

[Home ] [Archive]

[ فارسی ]

Journal of Geomatics Science and Technology

Main Menu

Home

Browse

Journal Info

Guide for Authors

Submit Manuscript

Articles archive

For Reviewers

Contact us

Site Facilities

Reviewers

Search in website

Receive site information

Volume 15, Issue 1 (9-2025)

JGST 2025, 15(1): 49-67

Back to browse issues page

Proposing a Deep Learning-Based Method for Estimating the Absolute Position of Potholes in Urban Streets

Homayoon Hadigol ^*

, Ali Hosseininaveh

Abstract: (128 Views)

Accurate identification and location of potholes on asphalt surfaces plays an important role in improving driver safety, reducing maintenance costs, and optimizing the management of urban transportation infrastructure. In this study, a deep learning-based method is presented that achieves these goals using affordable and readily available technologies. The proposed method uses a deep neural network to detect and segment potholes in images captured by smartphone cameras. Then, using a data fusion algorithm, the information extracted from the images is combined with Global Positioning System (GPS) and Inertial Measurement Unit (IMU) data received from the phone sensors to estimate the absolute location of the potholes in the Universal Coordinate System (UTM) with high accuracy. This innovative approach effectively overcomes the challenges associated with converting image coordinates to universal coordinates and errors caused by the low accuracy of GPS data. In order to evaluate the performance of the proposed method, a comprehensive dataset of urban streets was collected using a common smartphone. In this process, the location of potholes was determined in two ways: once using the proposed method presented in this study and again using a G1 Plus Sout multi-frequency receiver with an accuracy of 0.012 m, which was considered as an accurate reference. Then, the results of these two methods were compared with each other to evaluate the accuracy and efficiency of the proposed method. The experimental results show that this method is able to estimate the absolute location of potholes with an average error of less than 2 m in the UTM coordinate system. This approach allows for accurate and up-to-date mapping of the surface condition of urban streets using everyday and publicly available tools. In addition, the simplicity and cost-effectiveness of this method allows for widespread citizen participation in data collection and improvement of urban infrastructure.

Article number: 4

Keywords: Deep Learning, Neural Network, Object Detection, Visual Odometry, Data Fusion

Full-Text [PDF 875 kb] (56 Downloads)

Type of Study: Research | Subject: Photo&RS
Received: 2025/02/24

References

1. Nistér, D., O. Naroditsky, and J. Bergen. Visual odometry. in Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. 2004. Ieee

2. Davison, A.J., et al., MonoSLAM: Real-time single camera SLAM. IEEE transactions on pattern analysis and machine intelligence, 2007. 29(6): p. 1052-1067. [DOI:10.1109/TPAMI.2007.1049]

3. Klein, G. and D. Murray. Parallel tracking and mapping for small AR workspaces. in 2007 6th IEEE and ACM international symposium on mixed and augmented reality. 2007. IEEE. [DOI:10.1109/ISMAR.2007.4538852]

4. Comport, A.I., E. Malis, and P. Rives. Accurate quadrifocal tracking for robust 3d visual odometry. in Proceedings 2007 IEEE International Conference on Robotics and Automation. 2007. IEEE. [DOI:10.1109/ROBOT.2007.363762]

5. Newcombe, R.A., et al. KinectFusion: Real-time dense surface mapping and tracking. in 2011 10th IEEE International Symposium on Mixed and Augmented Reality. 2011. IEEE. [DOI:10.1109/ISMAR.2011.6162880]

6. Kerl, C., J. Sturm, and D. Cremers. Robust odometry estimation for RGB-D cameras. in 2013 IEEE international conference on robotics and automation. 2013. IEEE. [DOI:10.1109/ICRA.2013.6631104]

7. Newcombe, R.A., S.J. Lovegrove, and A.J. Davison. DTAM: Dense tracking and mapping in real-time. in 2011 international conference on computer vision. 2011. IEEE. [DOI:10.1109/ICCV.2011.6126513]

8. Engel, J., T. Schöps, and D. Cremers. LSD-SLAM: Large-scale direct monocular SLAM. in European conference on computer vision. 2014. Springer. [DOI:10.1007/978-3-319-10605-2_54]

9. Forster, C., M. Pizzoli, and D. Scaramuzza. SVO: Fast semi-direct monocular visual odometry. in 2014 IEEE international conference on robotics and automation (ICRA). 2014. IEEE. [DOI:10.1109/ICRA.2014.6906584]

10. Meier, L., et al. Pixhawk: A system for autonomous flight using onboard computer vision. in 2011 IEEE International Conference on Robotics and Automation. 2011. IEEE. [DOI:10.1109/ICRA.2011.5980229]

11. Engel, J., J. Sturm, and D. Cremers. Camera-based navigation of a low-cost quadrocopter. in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. 2012. IEEE. [DOI:10.1109/IROS.2012.6385458]

12. Comport, A.I., E. Malis, and P. Rives. Accurate quadrifocal tracking for robust 3d visual odometry. in Proceedings 2007 IEEE International Conference on Robotics and Automation. 2007. IEEE. [DOI:10.1109/ROBOT.2007.363762]

13. Zhang, W. and J. Kosecka. Image based localization in urban environments. in Third international symposium on 3D data processing, visualization, and transmission (3DPVT'06). 2006. IEEE. [DOI:10.1109/3DPVT.2006.80]

14. Bloesch, M., et al. Robust visual inertial odometry using a direct EKF-based approach. in 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS). 2015. IEEE. [DOI:10.1109/IROS.2015.7353389]

15. Li, M. and A.I. Mourikis, High-precision, consistent EKF-based visual-inertial odometry. The International Journal of Robotics Research, 2013. 32(6): p. 690-711. [DOI:10.1177/0278364913481251]

16. Leutenegger, S., et al., Keyframe-based visual-inertial odometry using nonlinear optimization. The International Journal of Robotics Research, 2015. 34(3): p. 314-334. [DOI:10.1177/0278364914554813]

17. Usenko, V., et al. Direct visual-inertial odometry with stereo cameras. in 2016 IEEE International Conference on Robotics and Automation (ICRA). 2016. IEEE. [DOI:10.1109/ICRA.2016.7487335]

18. Engel, J., V. Koltun, and D. Cremers, Direct sparse odometry. IEEE transactions on pattern analysis and machine intelligence, 2017. 40(3): p. 611-625. [DOI:10.1109/TPAMI.2017.2658577]

19. Mur-Artal, R. and J.D. Tardós, Visual-inertial monocular SLAM with map reuse. IEEE Robotics and Automation Letters, 2017. 2(2): p. 796-803. [DOI:10.1109/LRA.2017.2653359]

20. Leutenegger, S., et al., Keyframe-based visual-inertial odometry using nonlinear optimization. The International Journal of Robotics Research, 2015. 34(3): p. 314-334. [DOI:10.1177/0278364914554813]

21. Qin, T., P. Li, and S. Shen, Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics, 2018. 34(4): p. 1004-1020 [DOI:10.1109/TRO.2018.2853729]

22. Forster, C., M. Pizzoli, and D. Scaramuzza. SVO: Fast semi-direct monocular visual odometry. in 2014 IEEE international conference on robotics and automation (ICRA). 2014. IEEE. [DOI:10.1109/ICRA.2014.6906584]

23. Wang, X., et al. DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2017. [DOI:10.1109/ICRA.2017.7989236]

24. Zhou, T., et al. Unsupervised Learning of Depth and Ego-Motion from Video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. [DOI:10.1109/CVPR.2017.700]

25. Bloesch, M., et al. CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. [DOI:10.1109/CVPR.2018.00271]

26. Yang, Z., et al. CubeSLAM: Monocular 3D Object Detection and SLAM without Prior Knowledge. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019

27. Campos, R., et al. ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM. IEEE Transactions on Robotics, vol. 37, no. 6, pp. 1460-1476, Dec. 2021. [DOI:10.1109/TRO.2021.3075644]

28. Engel, J., V. Koltun, and D. Cremers. Direct Sparse Odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 3, pp. 611-625, March 2018. [DOI:10.1109/TPAMI.2017.2658577]

29. Koch, C., and M. P. Heinrich. "Deep learning-based pothole detection using smartphone images." Journal of Transportation Engineering, Part A: Systems, vol. 145, no. 11, 2019.

30. Jo, Y., and S. Ryu. "Pothole detection system using vehicle vibration signals and image processing." Sensors, vol. 19, no. 14, 2019.

31. Madli, R., et al. "Intelligent road surface monitoring system using ultrasonic sensors and GPS." IEEE Transactions on Intelligent Transportation Systems, vol. 20, no. 4, pp. 1046-1055, April 2019.

32. Wang, W., et al. "Road surface defect detection using deep convolutional neural networks." IEEE Transactions on Intelligent Transportation Systems, vol. 21, no. 5, pp. 1231-1242, May 2020.

33. Dhiman, C., and R. Klette. "Multiscale image processing and deep learning for pothole detection." IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 1, pp. 241-252, January 2021.

34. Chen, Y., et al. "Road surface monitoring system using SLAM and multi-objective optimization." IEEE Robotics & Automation Letters, vol. 6, no. 2, pp. 2530-2537, April 2021.

35. Jahanshahi, M. R., et al. "Automated measurement of pothole depth and volume using stereo images." Journal of Computing in Civil Engineering, vol. 35, no. 4, 2021.

36. Yu, X., and E. Salari. "Pothole detection using LiDAR and camera fusion." IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 11, pp. 7342-7353, November 2021.

37. Fox, D., et al. "Automated pothole mapping using drones and aerial imagery." IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-13, 2022.

38. Hiremath, R., et al. (2021). A Smart App for Pothole Detection Using Yolo Model. In ICT Analysis and Applications (pp. 1-13). Springer. [DOI:10.1007/978-981-15-8354-4_16]

39. Diwan, T., Anirudh, G., & Tembhurne, J. V. (2023). Object detection using YOLO: Challenges, architectural successors, datasets and applications. Multimedia Tools and Applications, 82(1), 1-23. [DOI:10.1007/s11042-022-13644-y]

40. Zhang, Y., Chen, Z., & Wei, B. (2020). A Sport Athlete Object Tracking Based on Deep Sort and Yolo V4 in Case of Camera Movement. In 2020 IEEE 6th International Conference on Computer and Communications (ICCC). [DOI:10.1109/ICCC51575.2020.9345010]

41. J. Chen et al., "Multi-View Triangulation: Systematic Comparison and an Improved Method," in IEEE Access, vol. 8, pp. 21017-21027, 2020, [DOI:10.1109/ACCESS.2020.2969082]

42. Rehder, J., Nikolic, J., Schneider, T., Hinzmann, T., & Siegwart, R. (2016). Extending kalibr: Calibrating the extrinsics of multiple IMUs and of individual axes. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 4304-4311. [DOI:10.1109/ICRA.2016.7487628]

43. Xu, G., et al. (2021). Position and orientation measurement adopting camera calibrated by projection geometry of Plücker matrices of three-dimensional lines. Scientific Reports, 7, 44092. [DOI:10.1038/srep44092]

44. Lam, S. K., et al. (2016). Improved Allan Variance Analysis for Inertial Measurement Units. Journal of Navigation, 75(4), 747-764.

45. Madry, M., et al. (2018). Visual-Inertial Odometry: A Comparative Study. IEEE Robotics and Automation Letters, 3(4), 3383-3390.

46. Poulose, A., & Han, D. S. (2020). Hybrid Indoor Localization Using IMU Sensors and Smartphone Camera. Sensors, 19(23), 5084. [DOI:10.3390/s19235084]

47. Kaplan, E. D., & Hegarty, C. J. (2018). Understanding GPS: Principles and Applications (Third Edition). Artech House.

48. Wang, X., et al. (2023). Temporal Feature Aggregation for Video Understanding: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11), 28345-28364.

49. Yan, Y., Zhang, B., Zhou, J., Zhang, Y., & Liu, X. (2022). Real-time localization and mapping utilizing multi-sensor fusion and visual-IMU-wheel odometry for agricultural robots in unstructured, dynamic and GPS-denied environments. Agronomy, 12(8), 1740. [DOI:10.3390/agronomy12081740]

50. Ma, H., Yan, L., Xia, Y., & Fu, M. (2020). Kalman filtering and information fusion. Springer. [DOI:10.1007/978-981-15-0806-6]

51. Chen, J., Li, W., & Chen, G. (2023). Multi-sensor data fusion with heterogeneous sampling rates: Challenges and solutions. IEEE Access, 11, 10501-10515.

52. Xu, B., Wang, D., Chen, H., & Li, M. (2023). Data augmentation for deep learning: A survey. IEEE Access, 11, 11353-11373.

53. https://github.com/umer0586/SensorServerv

54. Oliveira, M., Castro, A., Madeira, T., Pedrosa, E., et al. (2020). A ROS framework for the extrinsic calibration of intelligent vehicles: A multi-sensor, multi-modal approach. Robotics and Autonomous Systems, 134, 103509. [DOI:10.1016/j.robot.2020.103558]

55. Huai, J., Zhuang, Y., Shao, Y., Jozkow, G., & Wang, B. (2023). A Review and Comparative Study of Close-Range Geometric Camera Calibration Tools. arXiv preprint arXiv:2306.09014.

56. Zhang, Q., Li, Z., & Zhang, X. (2022). GPS data augmentation using UTM coordinate transformation and linear interpolation. Journal of Applied Geoinformatics, 18(2), 147-158.

57. Schneider, J., Förstner, W. (2015). Real-Time Accurate Geo-Localization of a MAV with Omnidirectional Visual Odometry and GPS. In L. Agapito, M. M. Bronstein, & C. Rother (Eds.), Computer Vision - ECCV 2014 Workshops (pp. 233-248). Springer International Publishing. [DOI:10.1007/978-3-319-16178-5_18]

58. Zhang, G., Liu, B., & Liang, Z. (2023). Implementation and optimization of ORB-SLAM2 algorithm based on ROS on mobile robots. Third International Conference on Robotics and Artificial Intelligence, 1-9. [DOI:10.1117/12.2675118]

59. Telceken, M., Akgun, D., Kacar, S., & Bingol, B. (2024). A New Approach for Super Resolution Object Detection Using an Image Slicing Algorithm and the Segment Anything Model. Sensors (Basel, Switzerland). [DOI:10.3390/s24144526]

60. Kaess, M., Johannsson, H., Roberts, R., Ila, V., Leonard, J. J., & Dellaert, F. (2012). iSAM2: Incremental Smoothing and Mapping Using the Bayes Tree. The International Journal of Robotics Research, 31(2), 216-235. [DOI:10.1177/0278364911430419]

61. Zhang, B., Li, Y., Liu, S., Pang, Z. and Zhao, H., 2022, July. Trajectory planning of upper limb exoskeleton rehabilitation Robot based on polynomial interpolation. In 2022 IEEE International Conference on Real-time Computing and Robotics (RCAR) (pp. 727-732). IEEE. [DOI:10.1109/RCAR54675.2022.9872300]

62. Hakim, I.M., Zakaria, H., Muslim, K. and Ihsani, S.I., 2023, February. 3D human pose estimation using blazepose and direct linear transform (DLT) for joint angle measurement. In 2023 International Conference on Artificial Intelligence in Information and Communication (ICAIIC) (pp. 236-241). IEEE. [DOI:10.1109/ICAIIC57133.2023.10066978]

63. Luhmann, T., Fraser, C. and Maas, H.G., 2016. Sensor modelling and camera calibration for close-range photogrammetry. ISPRS Journal of Photogrammetry and Remote Sensing, 115, pp.37-46. [DOI:10.1016/j.isprsjprs.2015.10.006]

64. Akyon, F.C., Altinuc, S.O. and Temizel, A., 2022. Slicing aided hyper inference and fine-tuning for small object detection. In 2022 IEEE International Conference on Image Processing (ICIP) (pp. 966-970). IEEE. [DOI:10.1109/ICIP46576.2022.9897990]

65. Sing S, Y., 2023. Custom Object Detection Using YOLO Integrated with a Segment Anything Model (SAM). International Research Journal of Engineering and Technology (IRJET), 10(10), pp.1

Send email to the article author

Add your comments about this article

Mendeley

Zotero

RefWorks

Hadigol H, Hosseininaveh A. Proposing a Deep Learning-Based Method for Estimating the Absolute Position of Potholes in Urban Streets. JGST 2025; 15 (1) : 4
URL: http://jgst.issgeac.ir/article-1-1216-en.html

Rights and permissions
	This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Volume 15, Issue 1 (9-2025)

Back to browse issues page