Abstract

Globally, the usage of social media has significantly increased and has become the most common way for people to deplete news. The easy sharing of multimedia content on social media has caused the fake news dimension, which threatens the stability as well as security of the society. Fake news detection (FND) in social media becomes challenging, because of which various tools are developed to detect them. Multimodal FND aims to determine fake data by text as well as images. Most commonly, researchers identify fake news only as text, but not as images. This research proposes a multimodal method for the detection and classification of fake news into real or fake. The proposed multimodal-based convolutional neural network (CNN) combines the designs of both text and image of fake news. This method utilizes two classification methods named hyperparameter tuning based bidirectional encoder representations from transformers (HTBERT) for text, and ResNet110 for images. Fakeddit dataset has used to estimate and evaluate the performance. The experimental results of the proposed ResNet110+HTBERT model achieves respective accuracy, precision, recall and F1-score values of about 0.931, 0.944, 0.942, and 0.946, which is superior when compared to the existing methods, recurrent CNN (RCNN) and fine-grained multimodal fusion network (FMFN). From the analysis, it is evident that the proposed method ResNet110+HTBERT achieves an accuracy of 0.931, and hence shows better results for overall metrics when compared to the existing methods of RCNN and FMFN.

Keyword

Convolutional neural network, Hyperparameter tuning based bidirectional encoder representations from transformers, Multimodal fake news detection, ResNet110, Social media.

Cite this article

Agrawal C, Pandey A, Goyal S.Multimodal fake news detection using hyperparameter-tuned BERT and ResNet110. International Journal of Advanced Technology and Engineering Exploration. 2024;11(114):759-772. DOI:10.19101/IJATEE.2023.10102314

Refference

[1]Katariya P, Gupta V, Arora R, Kumar A, Dhingra S, Xin Q, et al. A deep neural network-based approach for fake news detection in regional language. International Journal of Web Information Systems. 2022; 18(5/6):286-309.

[2]Fu L, Peng H, Liu S. KG-MFEND: an efficient knowledge graph-based model for multi-domain fake news detection. The Journal of Supercomputing. 2023; 79(16):18417-44.

[3]Dixit DK, Bhagat A, Dangi D. An accurate fake news detection approach based on a levy flight honey badger optimized convolutional neural network model. Concurrency and Computation: Practice and Experience. 2023; 35(1):e7382.

[4]Katarya R, Dahiya D, Checker S. Fake news detection system using featured-based optimized msvm classification. IEEE Access. 2022; 10:113184-99.

[5]Lai CM, Chen MH, Kristiani E, Verma VK, Yang CT. Fake news classification based on content level features. Applied Sciences. 2022; 12(3):1-21.

[6]Liao Q, Chai H, Han H, Zhang X, Wang X, Xia W, et al. An integrated multi-task model for fake news detection. IEEE Transactions on Knowledge and Data Engineering. 2021; 34(11):5154-65.

[7]Meel P, Vishwakarma DK. HAN, image captioning, and forensics ensemble multimodal fake news detection. Information Sciences. 2021; 567:23-41.

[8]Singhal S, Pandey T, Mrig S, Shah RR, Kumaraguru P. Leveraging intra and inter modality relationship for multimodal fake news detection. In companion proceedings of the web conference 2022 (pp. 726-34). ACM.

[9]Papadopoulos SI, Koutlis C, Papadopoulos S, Petrantonakis PC. VERITE: a robust benchmark for multimodal misinformation detection accounting for unimodal bias. International Journal of Multimedia Information Retrieval. 2024; 13(1):1-15.

[10]Raza S, Ding C. Fake news detection based on news content and social contexts: a transformer-based approach. International Journal of Data Science and Analytics. 2022; 13(4):335-62.

[11]Cui X, Yang L. Fake news detection in social media based on multi-modal multi-task learning. International Journal of Advanced Computer Science and Applications. 2022; 13(7):1-7.

[12]Dong D, Lin F, Li G, Liu B. Similarity-aware attention network for multimodal fake news detection. Third international conference on big data & artificial intelligence & software engineering 2022 (pp.76-85).

[13]Ying Q, Hu X, Zhou Y, Qian Z, Zeng D, Ge S. Bootstrapping multi-view representations for fake news detection. In proceedings of the AAAI conference on artificial intelligence 2023 (pp. 5384-92).

[14]Hakak S, Alazab M, Khan S, Gadekallu TR, Maddikunta PK, Khan WZ. An ensemble machine learning approach through effective feature extraction to classify fake news. Future Generation Computer Systems. 2021; 117:47-58.

[15]Meel P, Vishwakarma DK. A temporal ensembling based semi-supervised ConvNet for the detection of fake news articles. Expert Systems with Applications. 2021; 177:115002.

[16]Mughaid A, Al-zu’bi S, Al AA, Al-amrat R, Alajmi R, Zitar RA, et al. An intelligent cybersecurity system for detecting fake news in social media websites. Soft Computing. 2022; 26(12):5577-91.

[17]Zhou Y, Yang Y, Ying Q, Qian Z, Zhang X. Multi-modal fake news detection on social media via multi-grained information fusion. In proceedings of the international conference on multimedia retrieval 2023 (pp. 343-52). ACM.

[18]Raj C, Meel P. ConvNet frameworks for multi-modal fake news detection. Applied Intelligence. 2021; 51(11):8132-48.

[19]Kaliyar RK, Goswami A, Narang P. EchoFakeD: improving fake news detection in social media with an efficient deep neural network. Neural Computing and Applications. 2021; 33:8597-613.

[20]Song C, Ning N, Zhang Y, Wu B. A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks. Information Processing & Management. 2021; 58(1):102437.

[21]Liu P, Qian W, Xu D, Ren B, Cao J. Multi-modal fake news detection via bridging the gap between Modals. Entropy. 2023; 25(4):1-17.

[22]Segura-bedmar I, Alonso-bartolome S. Multimodal fake news detection. Information. 2022; 13(6):1-16.

[23]Ying L, Yu H, Wang J, Ji Y, Qian S. Multi-level multi-modal cross-attention network for fake news detection. IEEE Access. 2021; 9:132363-73.

[24]Guo Y, Song W. A temporal-and-spatial flow based multimodal fake news detection by pooling and attention blocks. IEEE Access. 2022; 10:131498-508.

[25]Wang J, Mao H, Li H. FMFN: fine-grained multimodal fusion networks for fake news detection. Applied Sciences. 2022; 12(3):1-12.

[26]Jing J, Wu H, Sun J, Fang X, Zhang H. Multimodal fake news detection via progressive fusion networks. Information Processing & Management. 2023; 60(1):103120.

[27]Kumari R, Ekbal A. Amfb: attention based multimodal factorized bilinear pooling for multimodal fake news detection. Expert Systems with Applications. 2021; 184:115412.

[28]Rai N, Kumar D, Kaushik N, Raj C, Ali A. Fake news classification using transformer based enhanced LSTM and BERT. International Journal of Cognitive Computing in Engineering. 2022; 3:98-105.

[29]Li S, Yao T, Li S, Yan L. Semantic‐enhanced multimodal fusion network for fake news detection. International Journal of Intelligent Systems. 2022; 37(12):12235-51.

[30]Mehta D, Dwivedi A, Patra A, Anand KM. A transformer-based architecture for fake news classification. Social Network Analysis and Mining. 2021; 11:1-2.

[31]Palani B, Elango S, Viswanathan KV. CB-fake: a multimodal deep learning framework for automatic fake news detection using capsule neural network and BERT. Multimedia Tools and Applications. 2022; 81(4):5587-620.

[32]Kaliyar RK, Goswami A, Narang P. FakeBERT: fake news detection in social media with a BERT-based deep learning approach. Multimedia Tools and Applications. 2021; 80(8):11765-88.

[33]Aslam N, Ullah KI, Alotaibi FS, Aldaej LA, Aldubaikil AK. Fake detect: a deep learning ensemble model for fake news detection. Complexity. 2021; 2021:1-8.

[34]Choudhary A, Arora A. Linguistic feature based learning model for fake news detection and classification. Expert Systems with Applications. 2021; 169:114171.

[35]Xue J, Wang Y, Tian Y, Li Y, Shi L, Wei L. Detecting fake news by exploring the consistency of multimodal data. Information Processing & Management. 2021; 58(5):102610.

[36]Guo Y. A mutual attention based multimodal fusion for fake news detection on social network. Applied Intelligence. 2023; 53(12):15311-20.

[37]Yang P, Ma J, Liu Y, Liu M. Multi-modal transformer for fake news detection. Mathematical Biosciences and Engineering: MBE. 2023; 20(8):14699-717.

[38]Sastrawan IK, Bayupati IP, Arsa DM. Detection of fake news using deep learning CNN–RNN based methods. ICT Express. 2022; 8(3):396-408.

[39]Al OA, Khotanlou H, Mansoorizadeh M, Zabihzadeh D. Multimodal fake-news recognition using ensemble of deep learners. Entropy. 2022; 24(9):1-16.

[40]Uppada SK, Patel P. An image and text-based multimodal model for detecting fake news in OSN’s. Journal of Intelligent Information Systems. 2023; 61(2):367-93.

[41]https://paperswithcodecom/dataset/fakeddit.. Accessed 23 April 2024.

[42]Keya AJ, Wadud MA, Mridha MF, Alatiyyah M, Hamid MA. AugFake-BERT: handling imbalance through augmentation of fake news using BERT to enhance the performance of fake news classification. Applied Sciences. 2022; 12(17):1-21.

[43]Shishah W. Fake news detection using BERT model with joint learning. Arabian Journal for Science and Engineering. 2021; 46(9):9115-27.

[44]Liu Y, Pu H, Sun DW. Efficient extraction of deep image features using convolutional neural network (CNN) for applications in detecting and analysing complex food matrices. Trends in Food Science & Technology. 2021; 113:193-204.

[45]Barbhuiya AA, Karsh RK, Jain R. CNN based feature extraction and classification for sign language. Multimedia Tools and Applications. 2021; 80(2):3051-69.

[46]Guo N, Gu K, Qiao J, Bi J. Improved deep CNNs based on nonlinear hybrid attention module for image classification. Neural Networks. 2021; 140:158-66.

[47]Zhao W, Zhu L, Wang M, Zhang X, Zhang J. WTL-CNN: a news text classification method of convolutional neural network based on weighted word embedding. Connection Science. 2022; 34(1):2291-312.

[48]Zhang F. A hybrid structured deep neural network with Word2Vec for construction accident causes classification. International Journal of Construction Management. 2022; 22(6):1120-40.

[49]Zhang Y, Shi T, Lu ZM. Image splicing detection scheme based on error level analysis and local binary pattern. Journal of Network Intelligence. 2021; 6:303-12.

[50]Singh R, Singh S. Text similarity measures in news articles by vector space model using NLP. Journal of The Institution of Engineers (India): Series B. 2021; 102:329-38.