Performance of Generative AIs in Detecting Dengue-Related Misinformation: ChatGPT and Google Bard
Abstract
Dengue outbreaks have become a common occurrence in South Asian countries, including Bangladesh. It has caused widespread concern among people from all walks of life. Various misinformation about dengue proliferates among people, mainly through social media platforms. The project was designed to investigate the performance of generative AIs in detecting dengue-related disinformation. In this study, two famous generative AIs were chosen to explore the performance of generative AI in detecting dengue-related misinformation: ChatGPT and Google Bard. These AI platforms were given widely distributed misinformation about dengue and asked to determine whether it was accurate or untrue. False information was identified through content analysis of various stories about the dengue outbreak, particularly those circulating on social media platforms. After getting responses from generative AIs, the answers were cross-checked using fact-checkers and public health databases such as WHO and CDC to determine whether the answers were correct or not. This study examined the performance of three AI systems (ChatGPT and Google Bard) in reacting to ten regularly disseminated misconceptions about dengue, particularly on social networking sites. Based on public health database statements (e.g., WHO, CDC) and fact-checker comments, ChatGPT and Google BARD demonstrated promising outcomes in detecting disinformation and presenting factual information. Dengue outbreaks have become common in developing nations such as Bangladesh, and spreading dengue-related misinformation has become commonplace. While it is known that generative AI systems have inherent limitations and may not always excel at dealing with complex real-world circumstances, they have shown promise in terms of consistent answers and performance in the public health sector. More studies in this field are needed to realize the full promise of AI chatbots in these sectors.
Keywords
Full Text:
PDFReferences
Agley, J. (2020) Assessing changes in US public trust in science amid the COVID-19 pandemic Public health, 183, 122-125 https://doi.org/10.1016/j.puhe.2020.05.004. pmidArticle Id (pmcid)
Ali, R., Tang, O. Y., Connolly, I. D., Fridley, J. S., Shin, J. H., Sullivan, P. L. Z., & Asaad, W. F. (2022). Performance of ChatGPT, GPT-4, and Google bard on a neurosurgery oral boards preparation question bank. Neurosurgery, 10-1227.
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023). Evaluating the performance of chatgpt in ophthalmology: An analysis of its successes and shortcomings. Ophthalmology Science, 100324.
Atia, Sharmin, Bonna., Shahed, Rafi, Pavel., Mohammad, Azmat, Ali. (2023). Dengue in Bangladesh.. International Journal of Infectious Diseases, doi: 10.1016/j.ijid.2023.06.020
Austin, L., Liu, B. F., & Jin, Y. (2012) How audiences seek out crisis information: Exploring the social-mediated crisis communication model Journal of Applied Communication Research, 40(2), 188-207 https://doi.org/10.1080/00909882.2012.654498.
Bhowmik, K. K., Ferdous, J., Baral, P. K., & Islam, M. S. (2023). Recent outbreak of dengue in Bangladesh: A threat to public health. Health Science Reports, 6(4), e1210. https://doi.org/10.1002/hsr2.1210
Brennen, J. S., Simon, F. M., Howard, P. N., & Nielsen, R. K. (2020) Types, sources, and claims of COVID-19 misinformation https://reutersinstitute.politics.ox.ac.uk/types-sources-and-claims-covid-19-misinformation
CDC. (2023, August 15). Dengue home | CDC. Centers for Disease Control and Prevention. https://www.cdc.gov/dengue/index.html
Choi, J. H., Hickman, K. E., Monahan, A. B., & Schwarcz, D. (2021). Chatgpt goes to law school. J. Legal Educ., 71, 387.
Chowdhury, N., Khalid, A., & Turin, T. C. (2023). Understanding misinformation infodemic during public health emergencies due to large-scale disease outbreaks: a rapid review. Journal of Public Health, 31(4), 553-573.
Deng, J. and Lin, Y. (2022) ‘The Benefits and Challenges of ChatGPT: An Overview’, Frontiers in Computing and Intelligent Systems, 2(2), pp. 81–83. Available at: https://doi.org/10.54097/fcis.v2i2.4465.
Douglas, K. M., Uscinski, J. E., Sutton, R. M., Cichocka, A., Nefes, T., Ang, C. S., & Deravi, F. (2019) Understanding conspiracy theories Advances in Political Psychology, 40(S1), 3-35 https://doi.org/10.1111/pops.12568.
Elish, M., Hwang, T. (2017). An AI pattern language. 1st Edition. New York: Data & Society.
Elson, W. H., Reiner, R. C., Siles, C., Bazan, I., Vilcarromero, S., Riley-Powell, A. R., ... & Paz-Soldan, V. A. (2020). Heterogeneity of dengue illness in community-based prospective study, Iquitos, Peru. Emerging infectious diseases, 26(9), 2077.
Freeman, D., & Bentall, R. P. (2017) The concomitants of conspiracy concerns Social Psychiatry and Psychiatric Epidemiology, 52(5), 595-604 https://doi.org/10.1007/s00127-017-1354-4. pmidArticle Id (pmcid)
Gagnon-Dufresne, M. C., Azevedo Dantas, M., Abreu Silva, K., Souza dos Anjos, J., Pessoa Carneiro Barbosa, D., Porto Rosa, R., ... & Zinszer, K. (2023). Social Media and the Influence of Fake News on
Global Health Interventions: Implications for a Study on Dengue in Brazil. International Journal of Environmental Research and Public Health, 20(7), 5299.
Galliford, N., & Furnham, A. (2017) Individual difference factors and beliefs in medical and political conspiracy theories Scandinavian Journal of Psychology, 58(5), 422-428 https://doi.org/10.1111/sjop.12382. Pmid
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023). How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Medical Education, 9(1), e45312.
Guess, A. M., Nyhan, B., & Reifler, J. (2020) Exposure to untrustworthy websites in the 2016 US election Nature Human Behaviour, 4(5), 472-480 https://doi.org/10.1038/s41562-020-0833-x. pmidArticle Id (pmcid)
Hall, M. J., Norwood, A. E., Ursano, R. J., & Fullerton, C. S. (2003) The psychological impacts of bioterrorism Biosecurity and Bioterrorism, 1(2), 139-144 https://doi.org/10.1089/153871303766275817.
Hsan, K., Hossain, M. M., Sarwar, M. S., Wilder-Smith, A., & Gozal, D. (2019). Unprecedented rise in dengue outbreaks in Bangladesh. The Lancet Infectious Diseases, 19(12), 1287.
Huang, X., Ruan, W., Huang, W., Jin, G., Dong, Y., Wu, C., ... & Mustafa, M. A. (2023). A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation. arXiv preprint arXiv:2305.11391.Available at: https://doi.org/10.48550/arXiv.2305.11391.
Ifukor, M. O. (2013) Channels of information acquisition and dissemination among rural dwellers International Journal of Library and Information Science, 5(10), 306-312 https://doi.org/10.5897/IJLIS11.036.
Kok, J. N., Boers, E. J., Kosters, W. A., Van der Putten, P., & Poel, M. (2009). Artificial intelligence: definition, trends, techniques, and cases. Artificial Intelligence, 1, 270- 299.
Koga S, Martin NB, Dickson DW. Evaluating the performance of large language models: ChatGPT and Google Bard in generating differential diagnoses in clinicopathological conferences of neurodegenerative disorders. Brain Pathol Zurich Switz. 2023 Aug 8;e13207.
Kudchadkar, S. R., & Carroll, C. L. (2020) Using social media for rapid information dissemination in a pandemic: #PedsICU and Coronavirus disease 2019 Pediatric Critical Care Medicine, 21(8), e538-e546 https://doi.org/10.1097/PCC.0000000000002474.
Lobato, E. J. C., Powell, M., Padilla, L. M. K., & Holbrook, C. (2020) Factors predicting willingness to share COVID-19 misinformation Frontiers in Psychology, 11, 566108 https://doi.org/10.3389/fpsyg.2020.566108. pmidArticle Id (pmcid)
Lobato, E. J. C., & Zimmerman, C. (2019) Examining how people reason about controversial scientific topics Thinking & Reasoning, 25(2), 231-255 https://doi.org/10.1080/13546783.2018.1521870.
McCarthy, J. (2007). What is Artificial Intelligence? Available online: http://wwwformal.stanford.edu/jmc/ (Accessed 9 July 2023)
Mahmud, M. R., Bin Reza, R., & Ahmed, S. Z. (2023). The effects of misinformation on COVID-19 vaccine hesitancy in Bangladesh. Global Knowledge, Memory and Communication, 72(1/2), 82-97.
Malik, T., Dwivedi, Y., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., ... & Wright, R. (2023). “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, 102642.
Mariani, M. (2022) ‘Generative Artificial Intelligence and Innovation: Conceptual Foundations’. Rochester, NY. Available at: https://papers.ssrn.com/abstract=4249382 (Accessed: 14 December 2023).
Miller, T., 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, 267, pp.1-38.
Mitchell, M. (2019). Artificial Intelligence: A Guide for Thinking Humans. Penguin UK. Ouyang, F., & Jiao, P. (2021). Artificial Intelligence in Education: The Three Paradigms. Computers and Education: Artificial Intelligence, 2, 100020.
Okike, B. I. (2020) Information dissemination in an era of a pandemic (COVID-19): Librarians' role Library Hi Tech News, 37(9), 1-4 https://doi.org/10.1108/LHTN-04-2020-0040.
Patil, M., Yadav, H., Gawali, M., Suryawanshi, J., Patil, J., Yeole, A., & Potlabattini, J. (2024). A Novel Approach to Fake News Detection Using Generative AI. International Journal of Intelligent Systems and Applications in Engineering, 12(4s), 343-354.
Princeton Survey Research Associates International for the Pew Internet & American Life Project (2006) Blogger callback survey https://www.pewresearch.org/internet/wp-content/uploads/sites/9/media/Files/Questionnaire/Old/PIP_Bloggers_Topline_2006.pdf
Procopio, C. H., & Procopio, S. T. (2007) Do you know what it means to miss New Orleans? Internet communication, geographic community, and social capital in crisis Journal of Applied Communication Research, 35(1), 67-87 https://doi.org/10.1080/00909880601065722.
Puri, N., Coomes, E. A., Haghbayan, H., & Gunaratne, K. (2020) Social media and vaccine hesitancy: New updates for the era of COVID-19 and globalized infectious diseases Human Vaccines & Immunotherapeutics, 16(11), 2586-2593 https://doi.org/10.1080/21645515.2020.1780846.
Russell, S. J., & Norvig, P. (2010). Artificial Intelligence: A Modern Approach. Prentice Hall.
Sharmin, S., Viennet, E., Glass, K., & Harley, D. (2015). The emergence of dengue in Bangladesh: epidemiology, challenges and future disease risk. Transactions of The Royal Society of Tropical Medicine and Hygiene, 109(10), 619-627.
Shen, X., Chen, Z., Backes, M., & Zhang, Y. (2023). In chatgpt we trust? measuring and characterizing the reliability of chatgpt. arXiv preprint arXiv:2304.08979.
Singh, B., Goswami, A., Chawla, N., & Shyam, S. (2007). Role of helplines for dissemination of information during an outbreak of dengue fever in Delhi, India, in 2006: An expericence.
The Daily Star. (2023, October 1). Dengue deaths cross 1,000 mark. The Daily Star. https://www.thedailystar.net/health/disease/news/dengue-deaths-cross-1000-mark-3432161
Van Prooijen, J. W., & Acker, M. (2015) The influence of control on belief in conspiracy theories: Conceptual and applied extensions Applied Cognitive Psychology, 29(5), 753-761 https://doi.org/10.1002/acp.3161.
Vaswani, A. et al. (2017) ‘Attention is All you Need’, in Advances in Neural Information Processing Systems. Curran Associates, Inc. Available at: https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html (Accessed: 14 July 2023).
Voeten, H. A., de Zwart, O., Veldhuijzen, I. K., Yuen, C., Jiang, X.,
Elam, G., Abraham, T., & Brug, J. (2009) Sources of information and health beliefs related to SARS and avian influenza among Chinese communities in the United Kingdom and The Netherlands, compared to the general population in these countries International Journal of Behavioral Medicine, 16(1), 49-57 https://doi.org/10.1007/s12529-008-9006-4. pmid
WHO. (2023.). Dengue and severe dengue. World Health Organization. Retrieved October 11, 2023, from https://www.who.int/news-room/fact-sheets/detail/dengue-and-severe-dengue
Wilder-Smith, A., Ooi, E.E., Horstick, O. and Wills, B., 2019. Dengue. The Lancet, 393(10169), pp.350-363.
Wilson, S. L., & Wiysonge, C. (2020) Social media and vaccine hesitancy BMJ Global Health, 5(10), e004206 https://doi.org/10.1136/bmjgh-2020-004206.
Wu, L., Morstatter, F., Hu, X., & Liu, H. (2016). Mining misinformation in social media. Big data in complex and social networks, 123-152.
DOI: https://doi.org/10.31315/ijcs.v17i1.11768
Refbacks
- There are currently no refbacks.
Published by :
Department of Communication Studies
Faculty of Social and Political Sciences
Universitas Pembangunan Nasional “Veteran” Yogyakarta, Indonesia
