Abstract

This study aimed to develop an emotion recognition model using brain signals. The brain computer interface (BCI) focuses on creating technology that enables direct brain-to-external device connections. There are two forms of BCI: invasive and non-invasive. In BCI, electroencephalography (EEG) is essential. EEG is a non-invasive method that involves applying electrodes to the scalp to capture electrical activity in the brain. EEG data is utilized to decode the user's intended emotions, activities, and thoughts. Emotions are important for human interaction, communication, and overall well-being. Many paralyzed people worldwide are unable to express their emotions or meet their needs, making it difficult to understand them, which leads to feelings of isolation. However, it is possible to detect emotions using BCI. Emotions are reflected in electrical brain activity and can be analyzed using EEG signals. The EEG signals are then decoded to detect a person's respective emotions. The decoding process mainly includes three steps. First, the signals are pre-processed to remove noise, and data is encoded. Second, the relevant features are extracted using the spectral power method. Third, emotions are classified using long short-term memory (LSTM), gated recurrent unit (GRU), and bidirectional long short-term memory (BiLSTM) algorithms. New EEG data is given to the model, and then emotions are displayed. The model developed using BiLSTM achieved an accuracy of 93.97%. A comparison was made with existing classification techniques that have used many three-dimensional (3D) models and the arousal-valence ratio to identify a person's emotion. The model's generalization will improve further by testing it on different types of datasets. The model's generalization improves further by testing it on different types of datasets.

Keyword

EEG data, Brain computer interface, Pre-processing, Classification, Spectral power, BiLSTM, Emotions.

Cite this article

Sowmya M, Varma PS, Deepika K.Emotion recognition from EEG signal data of the brain using bidirectional long short-term memory. International Journal of Advanced Technology and Engineering Exploration. 2024;11(115):930-942. DOI:10.19101/IJATEE.2023.10102441

Refference

[1]Huang X, Lajoie SP. Social emotional interaction in collaborative learning: why it matters and how can we measure it? Social Sciences & Humanities Open. 2023; 7(1):100447.

[2]Sowmya B, Alex SA, Kanavalli A, Supreeth S, Shruthi G, Rohith S. Machine learning model for emotion detection and recognition using an enhanced convolutional neural network. Journal of Integrated Science and Technology. 2024; 12(4):1-10.

[3]Kim H, Hong T. Enhancing emotion recognition using multimodal fusion of physiological, environmental, personal data. Expert Systems with Applications. 2024; 249:123723.

[4]Kim H, Hong T. Enhancing emotion recognition using multimodal fusion of physiological, environmental, personal data. Expert Systems with Applications. 2024; 249:123723.

[5]Shu L, Yu Y, Chen W, Hua H, Li Q, Jin J, et al. Wearable emotion recognition using heart rate data from a smart bracelet. Sensors. 2020; 20(3):1-19.

[6]Liu H, Zhang Y, Li Y, Kong X. Review on emotion recognition based on electroencephalography. Frontiers in Computational Neuroscience. 2021; 15:1-15.

[7]Guger C, Ince NF, Korostenskaja M, Allison BZ. Brain-computer interface research: a state-of-the-art summary 11. Brain-Computer Interface Research 2024 (pp. 1-11). Cham: Springer Nature Switzerland.

[8]Kawala-sterniuk A, Browarska N, Al-bakri A, Pelc M, Zygarlicki J, Sidikova M, et al. Summary of over fifty years with brain-computer interfaces-a review. Brain Sciences. 2021; 11(1):1-41.

[9]Maria MA, Akhand MA, Hossain AA, Kamal MA, Yamada K. A comparative study on prominent connectivity features for emotion recognition from EEG. IEEE Access. 2023; 11:37809-31.

[10]Longo C, Shankar A, Nuttall P. “It’s not easy living a sustainable lifestyle”: how greater knowledge leads to dilemmas, tensions and paralysis. Journal of Business Ethics. 2019; 154:759-79.

[11]Xu X, Zuo Y, Chen S, Hatami A, Gu H. Advancements in brain research: the in vivo/in vitro electrochemical detection of neurochemicals. Biosensors. 2024; 14(3):1-32.

[12]Vansteensel MJ, Jarosiewicz B. Brain-computer interfaces for communication. Handbook of Clinical Neurology. 2020; 168:67-85.

[13]Haas LF. Hans berger (1873–1941), richard caton (1842–1926), and electroencephalography. Journal of Neurology, Neurosurgery & Psychiatry. 2003; 74(1).

[14]Khan AA, Laghari AA, Shaikh AA, Dootio MA, Estrela VV, Lopes RT. A blockchain security module for brain-computer interface (BCI) with multimedia life cycle framework (MLCF). Neuroscience Informatics. 2022; 2(1):100030.

[15]Pandarinath C, Bensmaia SJ. The science and engineering behind sensitized brain-controlled bionic hands. Physiological Reviews. 2022; 102(2):551-604.

[16]Li D, Xu J, Wang J, Fang X, Ji Y. A multi-scale fusion convolutional neural network based on attention mechanism for the visualization analysis of EEG signals decoding. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2020; 28(12):2615-26.

[17]Song R, Zhang X, Chen X, Chen X, Chen X, Yang S, et al. Decoding silent speech from high-density surface electromyographic data using transformer. Biomedical Signal Processing and Control. 2023; 80:104298.

[18]Mak JN, Wolpaw JR. Clinical applications of brain-computer interfaces: current state and future prospects. IEEE Reviews in Biomedical Engineering. 2009; 2:187-99.

[19]Bockbrader MA, Francisco G, Lee R, Olson J, Solinsky R, Boninger ML. Brain computer interfaces in rehabilitation medicine. PM&R. 2018; 10(9): S233-43.

[20]Klaes C. Invasive brain-computer interfaces and neural recordings from humans. In Handbook of Behavioral Neuroscience 2018 (pp. 527-39). Elsevier.

[21]Islam MK, Rastegarnia A. Recent advances in EEG (non-invasive) based BCI applications. Frontiers in Computational Neuroscience. 2023; 17:1151852.

[22]Guo Y, Nejati H, Cheung NM. Deep neural networks on graph signals for brain imaging analysis. In international conference on image processing 2017 (pp. 3295-9). IEEE.

[23]Koelsch S, Fritz T, V.camon DY, Müller K, Friederici AD. Investigating emotion with music: an fMRI study. Human Brain Mapping. 2006; 27(3):239-50.

[24]Al-nafjan A, Alharthi K, Kurdi H. Lightweight building of an electroencephalogram-based emotion detection system. Brain Sciences. 2020; 10(11):1-17.

[25]Zhang J, Hao Y, Wen X, Zhang C, Deng H, Zhao J, et al. Subject-independent emotion recognition based on EEG frequency band features and self-adaptive graph construction. Brain Sciences. 2024; 14(3):1-19.

[26]Wei Z, Li H, Ma L, Li H. Emotion recognition based on microstate analysis from temporal and spatial patterns of electroencephalogram. Frontiers in Neuroscience. 2024; 18:1355512.

[27]Mutawa AM, Hassouneh A. Multimodal real-time patient emotion recognition system using facial expressions and brain EEG signals based on machine learning and log-sync methods. Biomedical Signal Processing and Control. 2024; 91:105942.

[28]Wei C, Chen LL, Song ZZ, Lou XG, Li DD. EEG-based emotion recognition using simple recurrent units network and ensemble learning. Biomedical signal processing and control. 2020; 58:101756.

[29]Pandey P, Seeja KR. Subject independent emotion recognition from EEG using VMD and deep learning. Journal of King Saud University-Computer and Information Sciences. 2022; 34(5):1730-8.

[30]Li J, Hua H, Xu Z, Shu L, Xu X, Kuang F, et al. Cross-subject EEG emotion recognition combined with connectivity features and meta-transfer learning. Computers in Biology and Medicine. 2022; 145:105519.

[31]Peng Y, Wang W, Kong W, Nie F, Lu BL, Cichocki A. Joint feature adaptation and graph adaptive label propagation for cross-subject emotion recognition from EEG signals. IEEE Transactions on Affective Computing. 2022; 13(4):1941-58.

[32]Bano KS, Bhuyan P, Ray A. EEG-based brain computer interface for emotion recognition. In 5th international conference on computational intelligence and networks 2022 (pp. 1-6). IEEE.

[33]Tao W, Li C, Song R, Cheng J, Liu Y, Wan F, et al. EEG-based emotion recognition via channel-wise attention and self-attention. IEEE Transactions on Affective Computing. 2020; 14(1):382-93.

[34]Salankar N, Mishra P, Garg L. Emotion recognition from EEG signals using empirical mode decomposition and second-order difference plot. Biomedical Signal Processing and Control. 2021; 65:102389.

[35]Wang F, Wu S, Zhang W, Xu Z, Zhang Y, Wu C, et al. Emotion recognition with convolutional neural network and EEG-based EFDMs. Neuropsychologia. 2020; 146:107506.

[36]Dharia SY, Valderrama CE, Camorlinga SG. Multimodal deep learning model for subject-independent EEG-based emotion recognition. In Canadian conference on electrical and computer engineering 2023 (pp. 105-10). IEEE.

[37]Farokhah L, Sarno R, Fatichah C. Cross-subject channel selection using modified relief and simplified CNN-based deep learning for EEG-based emotion recognition. IEEE Access. 2023; 11:110136-50.

[38]Hwang S, Ki M, Hong K, Byun H. Subject-independent EEG-based emotion recognition using adversarial learning. In 8th international winter conference on brain-computer interface 2020 (pp. 1-4). IEEE.

[39]Dhara T, Singh PK, Mahmud M. A fuzzy ensemble-based deep learning model for EEG-based emotion recognition. Cognitive Computation. 2023:1-5.

[40]Keelawat P, Thammasan N, Kijsirikul B, Numao M. Subject-independent emotion recognition during music listening based on EEG using deep convolutional neural networks. In 15th international colloquium on signal processing & its applications 2019 (pp. 21-6). IEEE.

[41]Kanuboyina V, Shankar T, Penmetsa RR. Electroencephalography based human emotion state classification using principal component analysis and artificial neural network. Multiagent and Grid Systems. 2022; 18(3-4):263-78.

[42]Hancer E, Subasi A. EEG-based emotion recognition using dual tree complex wavelet transform and random subspace ensemble classifier. Computer Methods in Biomechanics and Biomedical Engineering. 2023; 26(14):1772-84.

[43]Sharma R, Pachori RB, Sircar P. Automated emotion recognition based on higher order statistics and deep learning algorithm. Biomedical Signal Processing and Control. 2020; 58:101867.

[44]Liu R, Chao Y, Ma X, Sha X, Sun L, Li S, Chang S. ERTNet: an interpretable transformer-based framework for EEG emotion recognition. Frontiers in Neuroscience. 2024; 18:1320645.

[45]Li G, Yuan B, Ouyang D, Li W, Pan Y, Guo Z, et al. Emotion recognition based on selected EEG signals by common spatial pattern. IEEE Sensors Journal. 2024.

[46]Liu W, Qiu JL, Zheng WL, Lu BL. Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition. IEEE Transactions on Cognitive and Developmental Systems. 2021; 14(2):715-29.