Abstract

Autonomous vehicles (AVs) have emerged as captivating engineering ventures in the twenty-first century, capturing the interest of numerous academics and engineers across multiple generations. The world looks forward to leveraging AVs for reducing accidents caused by human errors and optimizing parking space utilization, particularly in urban areas. Accurate localization is pivotal for effective AV navigation, enabling the vehicle to pinpoint its precise position. While global positioning system (GPS) coordinates are widely used, their inherent errors and limitations can render them inadequate for determining precise location information, particularly in urban settings. Furthermore, drifting errors can undermine the efficacy of simultaneous localization and mapping (SLAM) algorithms. The proposed approach involves the utilization of a deep neural network, specifically a modified AlexNet architecture, which is a convolutional neural network (CNN), for localizing AVs in well-lit urban driving environments. The CNN enhances accuracy while reducing computational complexity and training time. Instead of relying on costly light detection and ranging (LiDAR) or radar sensors, a more affordable red green blue (RGB) camera sensor is employed. During testing, depth images are combined with RGB images using the intensity hue saturation (IHS) algorithm to enhance precision. Simulation results demonstrate an impressive accuracy rate of 95.49%, affirming the effectiveness of the proposed strategy. This study introduces a lightweight, precise, and reliable CNN architecture that significantly improves the accuracy of AV localization, simultaneously reducing predicted position errors by a considerable margin. The network's superiority is evidenced by mean square error (MSE) values of 0.039, 0.0099, and 0.0047 for position x, y, and orientation predictions, respectively. To validate real-time performance, the trained CNN was implemented in Python and integrated into the car learning to act (CARLA) simulator, enabling the online localization of a self-driving vehicle. This application successfully showcases the feasibility and efficacy of the proposed method.

Keyword

Autonomous vehicle, Localization, Deep learning, Convolutional neural networks CNN, Intensity hue saturation IHS, K-mean algorithm.

Cite this article

Ghintab SS, Hassan MY

Refference

[1][1]Karur K, Sharma N, Dharmatti C, Siegel JE. A survey of path planning algorithms for mobile robots. Vehicles. 2021; 3(3):448-68.

[2][2]Chen S, Liu B, Feng C, Vallespi-gonzalez C, Wellington C. 3d point cloud processing and learning for autonomous driving: impacting map creation, localization, and perception. IEEE Signal Processing Magazine. 2020; 38(1):68-86.

[3][3]Fayyad J, Jaradat MA, Gruyer D, Najjaran H. Deep learning sensor fusion for autonomous vehicle perception and localization: a review. Sensors. 2020; 20(15):1-35.

[4][4]Reid TG, Houts SE, Cammarata R, Mills G, Agarwal S, Vora A, et al. Localization requirements for autonomous vehicles. SAE International Journal of Connected and Automated Vehicles. 2019:1-16.

[5][5]Rublee E, Rabaud V, Konolige K, Bradski G. ORB: an efficient alternative to SIFT or SURF. In international conference on computer vision 2011 (pp. 2564-71). IEEE.

[6][6]Lowe DG. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision. 2004; 60:91-110.

[7][7]Bay H, Tuytelaars T, Van GL. Surf: speeded up robust features. In computer vision–ECCV: 9th European conference on computer vision, Graz, Austria, 2006. Proceedings, Part I, 2006 (pp. 404-17). Springer Berlin Heidelberg.

[8][8]Fischler MA, Bolles RC. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM. 1981; 24(6):381-95.

[9][9]Mur-artal R, Tardós JD. Orb-slam2: an open-source slam system for monocular, stereo, and RGB-d cameras. IEEE Transactions on Robotics. 2017; 33(5):1255-62.

[10][10]Engel J, Koltun V, Cremers D. Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017; 40(3):611-25.

[11][11]Shotton J, Glocker B, Zach C, Izadi S, Criminisi A, Fitzgibbon A. Scene coordinate regression forests for camera relocalization in RGB-D images. In proceedings of the conference on computer vision and pattern recognition 2013 (pp. 2930-7).

[12][12]Kendall A, Gal Y, Cipolla R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In proceedings of the IEEE conference on computer vision and pattern recognition 2018 (pp. 7482-91).

[13][13]Ballardini AL, Fontana S, Cattaneo D, Matteucci M, Sorrenti DG. Vehicle localization using 3D building models and point cloud matching. Sensors. 2021; 21(16):1-19.

[14][14]Zghair NA, Al-araji AS. A one decade survey of autonomous mobile robot systems. International Journal of Electrical and Computer Engineering. 2021; 11(6):4891-906.

[15][15]Stenborg E, Toft C, Hammarstrand L. Long-term visual localization using semantically segmented images. In international conference on robotics and automation 2018 (pp. 6484-90). IEEE.

[16][16]Parisotto E, Chaplot D, Zhang J, Salakhutdinov R. Global pose estimation with an attention-based recurrent network. In proceedings of the conference on computer vision and pattern recognition workshops 2018 (pp. 237-46). IEEE.

[17][17]Heng L, Choi B, Cui Z, Geppert M, Hu S, Kuan B, et al. Project autovision: localization and 3d scene perception for an autonomous vehicle with a multi-camera system. In international conference on robotics and automation 2019 (pp. 4695-702). IEEE.

[18][18]Amini A, Rosman G, Karaman S, Rus D. Variational end-to-end navigation and localization. In international conference on robotics and automation 2019 (pp. 8958-64). IEEE.

[19][19]Ma WC, Tartavull I, Bârsan IA, Wang S, Bai M, Mattyus G, et al. Exploiting sparse semantic HD maps for self-driving vehicle localization. In IEEE/RSJ international conference on intelligent robots and systems 2019(pp. 5304-11). IEEE.

[20][20]Yin H, Wang Y, Ding X, Tang L, Huang S, Xiong R. 3D LiDAR-based global localization using siamese neural network. IEEE Transactions on Intelligent Transportation Systems. 2019; 21(4):1380-92.

[21][21]Wan G, Yang X, Cai R, Li H, Zhou Y, Wang H, et al. Robust and precise vehicle localization based on multi-sensor fusion in diverse city scenes. In IEEE international conference on robotics and automation 2018 (pp. 4670-77). IEEE.

[22][22]Chen X, Vizzo I, Läbe T, Behley J, Stachniss C. Range image-based LiDAR localization for autonomous vehicles. In international conference on robotics and automation 2021 (pp. 5802-8). IEEE.

[23][23]Héry E, Xu P, Bonnifait P. Consistent decentralized cooperative localization for autonomous vehicles using LiDAR, GNSS, and HD maps. Journal of Field Robotics. 2021; 38(4):552-71.

[24][24]Qin T, Zheng Y, Chen T, Chen Y, Su Q. A light-weight semantic map for visual localization towards autonomous driving. In international conference on robotics and automation 2021 (pp. 11248-54). IEEE.

[25][25]Chu X, Lu Z, Gesbert D, Wang L, Wen X. Vehicle localization via cooperative channel mapping. IEEE Transactions on Vehicular Technology. 2021; 70(6):5719-33.

[26][26]Li Y, Cai Y, Malekian R, Wang H, Sotelo MA, Li Z. Creating navigation map in semi-open scenarios for intelligent vehicle localization using multi-sensor fusion. Expert Systems with Applications. 2021; 184:115543.

[27][27]Liu J, Guo G. Vehicle localization during GPS outages with extended Kalman filter and deep learning. IEEE Transactions on Instrumentation and Measurement. 2021; 70:1-10.

[28][28]Guo C, Lin M, Guo H, Liang P, Cheng E. Coarse-to-fine semantic localization with HD map for autonomous driving in structural scenes. In IEEE/RSJ international conference on intelligent robots and systems 2021 (pp. 1146-53). IEEE.

[29][29]Ren R, Fu H, Xue H, Li X, Hu X, Wu M. LiDAR‐based robust localization for field autonomous vehicles in off‐road environments. Journal of Field Robotics. 2021; 38(8):1059-77.

[30][30]Yanase R, Hirano D, Aldibaja M, Yoneda K, Suganuma N. LiDAR-and radar-based robust vehicle localization with confidence estimation of matching results. Sensors. 2022; 22(9):1-20.

[31][31]Peng B, Xie H, Chen W. ROLL: long-term robust LiDAR-based localization with temporary mapping in changing environments. In IEEE/RSJ international conference on intelligent robots and systems 2022 (pp. 2841-7). IEEE.

[32][32]Dauptain X, Koné A, Grolleau D, Cerezo V, Gennesseaux M, Do MT. Conception of a high-level perception and localization system for autonomous driving. Sensors. 2022; 22(24):9661.

[33][33]Lee J, Back M, Hwang SS, Chun IY. Improved real-time monocular SLAM using semantic segmentation on selective frames. IEEE Transactions on Intelligent Transportation Systems. 2022; 24(3):2800-13.

[34][34]Kang MS, Ahn JH, Im JU, Won JH. LiDAR-and V2X-based cooperative localization technique for autonomous driving in a GNSS-denied environment. Remote Sensing. 2022; 14(22):1-16.

[35][35]Han L, Shi Z, Wang H. A localization and mapping algorithm based on improved LVI-SAM for vehicles in field environments. Sensors. 2023; 23(7):1-14.

[36][36]Raheem F, Abdulwahhab AA. Deep learning convolution neural networks analysis and comparative study for static alphabet ASL hand gesture recognition. Journal of Xidian University. 2020; 14(4):1871-81.

[37][37]He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In proceedings of the IEEE conference on computer vision and pattern recognition 2016 (pp. 770-8). IEEE.

[38][38]Fujiyoshi H, Hirakawa T, Yamashita T. Deep learning-based image recognition for autonomous driving. IATSS Research. 2019; 43(4):244-52.

[39][39]Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC. Mobilenetv2: inverted residuals and linear bottlenecks. In proceedings of the conference on computer vision and pattern recognition 2018 (pp. 4510-20). IEEE.

[40][40]Wani MA, Bhat FA, Afzal S, Khan AI. Advances in deep learning. Springer; 2020.

[41][41]Alzubaidi L, Zhang J, Humaidi AJ, Al-dujaili A, Duan Y, Al-shamma O, et al. Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. Journal of Big Data. 2021; 8:1-74.

[42][42]Abdulhussein AA, Raheem FA. Hand gesture recognition of static letters American sign language (ASL) using deep learning. Engineering and Technology Journal. 2020; 38(6):926-37.

[43][43]Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems. 2012:1-9.

[44][44]Gómez VV. LiDAR-based scene understanding for autonomous driving using deep learning (Doctoral Dissertation, Universitat Politècnica de Catalunya (UPC). 2020.

[45][45]Mishra D, Palkar B. Image fusion techniques: a review. International Journal of Computer Applications. 2015; 130(9):7-13.

[46][46]Zhao X. Image fusion based on IHS transform and principal component analysis (PCA) transform. In international conference on computer technology, electronics and communication (ICCTEC) 2017(pp. 304-7). IEEE.

[47][47]Atiyah HA, Hassan MY. Outdoor localization in mobile robot with 3D LiDAR based on principal component analysis and K-Nearest neighbors algorithm. Engineering and Technology Journal. 2021; 39(6):965-76.

[48][48]Hassan MY, Kothapalli G. Comparison between neural network based PI and PID controllers. In 7th international multi-conference on systems, signals and devices 2010 (pp. 1-6). IEEE.

[49][49]Zhou B, Liu J, Sun W, Chen R, Tomlin CJ, Yuan Y. PBSGD: powered stochastic gradient descent methods for accelerated non-convex optimization. In proceedings of the twenty-ninth international joint conference on artificial intelligence 2020 (pp. 3258-66).

[50][50]Dosovitskiy A, Ros G, Codevilla F, Lopez A, Koltun V. CARLA: an open urban driving simulator. In conference on robot learning 2017 (pp. 1-16). PMLR.

[51][51]Ghintab SS, Hassan MY. CNN-based visual localization for autonomous vehicles under different weather conditions. Engineering and Technology Journal. 2023; 41(2):375-86.

[52][52]Wu Y, Li Y, Ge X, Gao Y, Qian W. An efficient method for calculating the error statistics of block-based approximate adders. IEEE Transactions on Computers. 2018; 68(1):21-38.