Abstract

In the past decade, dietary assessment has been one of the most popular topics of research in the food industry, which has resulted in developing several automatic or semi-automatic dietary assessment systems using visible spectrum images for food recognition. However, the main shortcoming of visible spectrum image-based systems is its inability to differentiate foods of similar color. Researchers have added additional features such as shape, size and texture to the color model to improve the overall accuracy. However, the shape and size features are rendered inefficient when recognizing food in the mixed or cooked form. The aim of this research is to show the capability of hyperspectral bands for accurate food recognition based on individual spectral bands. In this work we use a hyperspectral imaging system of 240 spectral bands with the wavelength range between 400 nm to 900 nm. The ReliefF and PCA methods select/extract less, but the most informative features which are important to learn Logistic Regression and Support Vector Machine (SVM) as binary and multiple classifiers, respectively. A total of 20 different food samples in various forms (uncut and cut), shapes, and sizes were used in this study. The prediction results indicate that the hyperspectral images have the advantages of being able to recognize different food items from a mixed form with similar color and similar food types with different colors. In our experiments the highest classification accuracy of 0.6874 with 20 different food samples is produced by SVM multi-classification of ReliefF data with the top 110 hyperspectral features. We are able to obtain approximately 0.90 accuracy, using binary classification on a specific subset of food samples.

Keyword

Food recognition, Hyperspectral, Logistic regression, PCA, ReliefF, RGB, SVM.

Cite this article

Esfahani SN, Muthukumar V, Regentova EE, Taghva K, Trabia M

Refference

[1][1]Esfahani SN, Muthukumar V, Regentova EE, Taghva K, Trabia M. Complex food recognition using hyper-spectral imagery. In annual computing and communication workshop and conference 2020 (pp. 662-7). IEEE.

[2][2]Khanna N, Boushey CJ, Kerr D, Okos M, Ebert DS, Delp EJ. An overview of the technology assisted dietary assessment project at Purdue University. In international symposium on multimedia 2010 (pp. 290-5). IEEE.

[3][3]Zhu F, Bosch M, Boushey CJ, Delp EJ. An image analysis system for dietary assessment and evaluation. In international conference on image processing 2010 (pp. 1853-6). IEEE.

[4][4]Zhu F, Bosch M, Woo I, Kim S, Boushey CJ, Ebert DS, Delp EJ. The use of mobile devices in aiding dietary assessment and evaluation. IEEE Journal of Selected Topics in Signal Processing. 2010; 4(4):756-66.

[5][5]Kong F, Tan J. Dietcam: regular shape food recognition with a camera phone. In international conference on body sensor networks 2011 (pp. 127-32). IEEE.

[6][6]Bosch M, Zhu F, Khanna N, Boushey CJ, Delp EJ. Combining global and local features for food identification in dietary assessment. In 18th international conference on image processing 2011 (pp. 1789-92). IEEE.

[7][7]Xu C, Khanna N, Boushey CJ, Delp EJ. Low complexity image quality measures for dietary assessment using mobile devices. In international symposium on multimedia 2011 (pp. 351-6). IEEE.

[8][8]Rahman MH, Li Q, Pickering M, Frater M, Kerr D, Bouchey C, et al. Food volume estimation in a mobile phone based dietary assessment system. In international conference on signal image technology and internet based systems 2012 (pp. 988-95). IEEE.

[9][9]Zhu F, Bosch M, Khanna N, Boushey CJ, Delp EJ. Multiple hypotheses image segmentation and classification with application to dietary assessment. IEEE Journal of Biomedical and Health Informatics. 2014; 19(1):377-88.

[10][10]Liu C, Cao Y, Luo Y, Chen G, Vokkarane V, Yunsheng M, et al. A new deep learning-based food recognition system for dietary assessment on an edge computing service infrastructure. IEEE Transactions on Services Computing. 2017; 11(2):249-61.

[11][11]Jiang L, Qiu B, Liu X, Huang C, Lin K. DeepFood: food image analysis and dietary assessment via deep model. IEEE Access. 2020; 8:47477-89.

[12][12]Tan RZ, Chew X, Khaw KW. Quantized deep residual convolutional neural network for image-based dietary assessment. IEEE Access. 2020; 8:111875-88.

[13][13]Chen M, Dhingra K, Wu W, Yang L, Sukthankar R, Yang J. PFID: pittsburgh fast-food image dataset. In international conference on image processing 2009 (pp. 289-92). IEEE.

[14][14]Chen HC, Jia W, Li Z, Sun YN, Sun M. 3D/2D model-to-image registration for quantitative dietary assessment. In annual northeast bioengineering conference 2012 (pp. 95-6). IEEE.

[15][15]He J, Shao Z, Wright J, Kerr D, Boushey C, Zhu F. Multi-task image-based dietary assessment for food recognition and portion size estimation. In conference on multimedia information processing and retrieval 2020 (pp. 49-54). IEEE.

[16][16]Hafiz R, Islam S, Khanom R, Uddin MS. Image based drinks identification for dietary assessment. In international workshop on computational intelligence 2016 (pp. 192-7). IEEE.

[17][17]Lo FP, Sun Y, Qiu J, Lo B. A novel vision-based approach for dietary assessment using deep learning view synthesis. In 16th international conference on wearable and implantable body sensor networks 2019 (pp. 1-4). IEEE.

[18][18]He Y, Xu C, Khanna N, Boushey CJ, Delp EJ. Analysis of food images: Features and classification. In international conference on image processing 2014 (pp. 2744-8). IEEE.

[19][19]Lo FP, Sun Y, Qiu J, Lo BP. Point2volume: a vision-based dietary assessment approach using view synthesis. IEEE Transactions on Industrial Informatics. 2019; 16(1):577-86.

[20][20]Subhi MA, Ali SH, Mohammed MA. Vision-based approaches for automatic food recognition and dietary assessment: a survey. IEEE Access. 2019; 7:35370-81.

[21][21]Park B, Lu R, editors. Hyperspectral imaging technology in food and agriculture. New York: Springer; 2015.

[22][22]Sun DW. Hyperspectral imaging for food quality analysis and control. Elsevier; 2010.

[23][23]Bhunia AK, Kim MS, Taitt CR, editors. High throughput screening for food safety assessment: Biosensor technologies, hyperspectral imaging and practical applications. Elsevier; 2014.

[24][24]Haboudane D, Miller JR, Pattey E, Zarco-Tejada PJ, Strachan IB. Hyperspectral vegetation indices and novel algorithms for predicting green LAI of crop canopies: modeling and validation in the context of precision agriculture. Remote Sensing of Environment. 2004; 90(3):337-52.

[25][25]Haboudane D, Miller JR, Tremblay N, Zarco-Tejada PJ, Dextraze L. Integrated narrow-band vegetation indices for prediction of crop chlorophyll content for application to precision agriculture. Remote Sensing of Environment. 2002; 81(2-3):416-26.

[26][26]Rodarmel C, Shan J. Principal component analysis for hyperspectral image classification. Surveying and Land Information Science. 2002; 62(2):115-22.

[27][27]Chang CI, Du Q, Sun TL, Althouse ML. A joint band prioritization and band-decorrelation approach to band selection for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing. 1999; 37(6):2631-41.

[28][28]Guo B, Gunn SR, Damper RI, Nelson JD. Band selection for hyperspectral image classification using mutual information. IEEE Geoscience and Remote Sensing Letters. 2006; 3(4):522-6.

[29][29]MartÍnez-UsÓMartinez-Uso A, Pla F, Sotoca JM, García-Sevilla P. Clustering-based hyperspectral band selection using information measures. IEEE Transactions on Geoscience and Remote Sensing. 2007; 45(12):4158-71.

[30][30]Du Q, Yang H. Similarity-based unsupervised band selection for hyperspectral image analysis. IEEE Geoscience and Remote Sensing Letters. 2008; 5(4):564-8.

[31][31]Melgani F, Bruzzone L. Classification of hyperspectral remote sensing images with support vector machines. IEEE Transactions on Geoscience and Remote Sensing. 2004; 42(8):1778-90.

[32][32]Gualtieri JA, Chettri S. Support vector machines for classification of hyperspectral data. IEEE 2000 international geoscience and remote sensing symposium. Taking the pulse of the planet: the role of remote sensing in managing the environment. Proceedings 2000 (pp. 813-5). IEEE.

[33][33]Puri M, Zhu Z, Yu Q, Divakaran A, Sawhney H. Recognition and volume estimation of food intake using a mobile device. In workshop on applications of computer vision 2009 (pp. 1-8). IEEE.

[34][34]Rahmana MH, Pickering MR, Kerr D, Boushey CJ, Delp EJ. A new texture feature for improved food recognition accuracy in a mobile phone based dietary assessment system. In international conference on multimedia and expo workshops 2012 (pp. 418-23). IEEE.

[35][35]Pouladzadeh P, Villalobos G, Almaghrabi R, Shirmohammadi S. A novel SVM based food recognition method for calorie measurement applications. In international conference on multimedia and expo workshops 2012 (pp. 495-8). IEEE.

[36][36]Duan P, Wang W, Zhang W, Gong F, Zhang P, Rao Y. Food image recognition using pervasive cloud computing. In international conference on green computing and communications and IEEE internet of things and IEEE cyber, physical and social computing 2013 (pp. 1631-7). IEEE.

[37][37]Pham C, Thanh TN. Fresh food recognition using feature fusion. In international conference on advanced technologies for communications 2014 (pp. 298-302). IEEE.

[38][38]Zhang W, Zhao D, Gong W, Li Z, Lu Q, Yang S. Food image recognition with convolutional neural networks. In international conference on Ubiquitous intelligence and computing and international conference on autonomic and trusted computing and 2015 IEEE 15th international conference on Scalable Computing and communications and its associated workshops (UIC-ATC-ScalCom) 2015 (pp. 690-3). IEEE.

[39][39]Pouladzadeh P, Kuhad P, Peddi SV, Yassine A, Shirmohammadi S. Food calorie measurement using deep learning neural network. In international instrumentation and measurement technology conference proceedings 2016 (pp. 1-6). IEEE.

[40][40]He H, Kong F, Tan J. DietCam: multiview food recognition using a multikernel SVM. IEEE Journal of Biomedical and Health Informatics. 2015; 20(3):848-55.

[41][41]Chen J, Ngo CW. Deep-based ingredient recognition for cooking recipe retrieval. In proceedings of the ACM international conference on Multimedia 2016 (pp. 32-41). ACM.

[42][42]Lu Y. Food image recognition by using convolutional neural networks (cnns). arXiv preprint arXiv:1612.00983. 2016.

[43][43]Ciocca G, Napoletano P, Schettini R. Food recognition: a new dataset, experiments, and results. IEEE Journal of Biomedical and Health Informatics. 2016; 21(3):588-98.

[44][44]Chu WT, Lin JH. Food image description based on deep-based joint food category, ingredient, and cooking method recognition. In international conference on multimedia & expo workshops 2017 (pp. 109-14). IEEE.

[45][45]Deng L, Chen J, Sun Q, He X, Tang S, Ming Z, et al. Mixed-dish recognition with contextual relation networks. In proceedings of the ACM international conference on multimedia 2019 (pp. 112-20).

[46][46]Zhang L, Zhao J, Li S, Shi B, Duan LY. From market to dish: multi-ingredient image recognition for personalized recipe recommendation. In international conference on multimedia and expo 2019 (pp. 1252-7). IEEE.

[47][47]Liu C, Liang Y, Xue Y, Qian X, Fu J. Food and ingredient joint learning for fine-grained recognition. IEEE Transactions on Circuits and Systems for Video Technology. 2020.

[48][48]Liang H, Wen G, Hu Y, Luo M, Yang P, Xu Y. MVANet: multi-tasks guided multi-view attention network for Chinese food recognition. IEEE Transactions on Multimedia. 2020:1-12.

[49][49]Freitas CN, Cordeiro FR, Macario V. MyFood: a food segmentation and classification system to Aid nutritional monitoring. In SIBGRAPI conference on graphics, patterns and images 2020 (pp. 234-9). IEEE.

[50][50]Chen J, Zhu B, Ngo CW, Chua TS, Jiang YG. A study of multi-task and region-wise deep learning for food ingredient recognition. IEEE Transactions on Image Processing. 2020; 30:1514-26.

[51][51]Barbin D, Elmasry G, Sun DW, Allen P. Near-infrared hyperspectral imaging for grading and classification of pork. Meat Science. 2012; 90(1):259-68.

[52][52]Cluff K, Naganathan GK, Subbiah J, Samal A, Calkins CR. Optical scattering with hyperspectral imaging to classify longissimus dorsi muscle based on beef tenderness using multivariate modeling. Meat Science. 2013; 95(1):42-50.

[53][53]Marshall S, Kelman T, Qiao T, Murray P, Zabalza J. Hyperspectral imaging for food applications. In European signal processing conference 2015 (pp. 2854-8). IEEE.

[54][54]Wang L, Liu D, Pu H, Sun DW, Gao W, Xiong Z. Use of hyperspectral imaging to discriminate the variety and quality of rice. Food Analytical Methods. 2015; 8(2):515-23.

[55][55]Al-Sarayreh M, Reis MM, Yan WQ, Klette R. Deep spectral-spatial features of snapshot hyperspectral images for red-meat classification. In international conference on image and vision computing 2018 (pp. 1-6). IEEE.

[56][56]Qiu Z, Chen J, Zhao Y, Zhu S, He Y, Zhang C. Variety identification of single rice seed using hyperspectral imaging combined with convolutional neural network. Applied Sciences. 2018; 8(2):1-12.

[57][57]Liu Y, Wu T, Yang J, Tan K, Wang S. Hyperspectral band selection for soybean classification based on information measure in FRS theory. Biosystems Engineering. 2019; 178:219-32.

[58][58]Steinbrener J, Posch K, Leitner R. Hyperspectral fruit and vegetable classification using convolutional neural networks. Computers and Electronics in Agriculture. 2019; 162:364-72.

[59][59]Xia C, Yang S, Huang M, Zhu Q, Guo Y, Qin J. Maize seed classification using hyperspectral image coupled with multi-linear discriminant analysis. Infrared Physics & Technology. 2019; 103:103077.

[60][60]Xiao H, Guo P, Dong X, Xing S, Sun M. Research on the method of hyperspectral and image deep features for bacon classification. In Chinese control and decision conference 2019 (pp. 4682-6). IEEE.

[61][61]Weng S, Tang P, Yuan H, Guo B, Yu S, Huang L, Xu C. Hyperspectral imaging for accurate determination of rice variety using a deep learning network with multi-feature fusion. Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy. 2020; 234:118237.

[62][62]Fabiyi SD, Vu H, Tachtatzis C, Murray P, Harle D, Dao TK, et al. Varietal classification of rice seeds using RGB and hyperspectral images. IEEE Access. 2020; 8:22493-505.

[63][63]Zhang C, Zhao Y, Yan T, Bai X, Xiao Q, Gao P, et al. Application of near-infrared hyperspectral imaging for variety identification of coated maize kernels with deep learning. Infrared Physics & Technology. 2020; 111:103550.

[64][64]Furlanetto RH, Moriwaki T, Falcioni R, Pattaro M, Vollmann A, Junior AC, et al. Hyperspectral reflectance imaging to classify lettuce varieties by optimum selected wavelengths and linear discriminant analysis. Remote Sensing Applications: Society and Environment. 2020; 20:100400.

[65][65]Kononenko I, Robnik-Sikonja M, Pompe U. ReliefF for estimation and discretization of attributes in classification, regression, and ILP problems. Artificial Intelligence: Methodology, Systems, Applications. 1996:31-40.

[66][66]Kira K, Rendell LA. A practical approach to feature selection. In machine learning proceedings 1992 (pp. 249-56). Morgan Kaufmann.

[67][67]Qian Y, Ye M, Zhou J. Hyperspectral image classification based on structured sparse logistic regression and three-dimensional wavelet texture features. IEEE Transactions on Geoscience and Remote Sensing. 2012; 51(4):2276-91.

[68][68]Yang Y, Liu X. A re-examination of text categorization methods. In proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval 1999 (pp. 42-9).

[69][69]Zahavi A, Palshin A, Liyanage DC, Tamre M. Influence of illumination sources on hyperspectral imaging. In international conference on research and education in mechatronics 2019 (pp. 1-5). IEEE.

[70][70]Ben-Hur A, Weston J. A user s guide to support vector machines. In data mining techniques for the life sciences 2010 (pp. 223-39). Humana Press.