Accuracy and concordance of DMIND AI application with a renowned depression assessment tool in Thai adults

Authors

  • Solaphat Hemrungroj Center of Excellence in Digital and AI for Mental Health, Faculty of Engineering, Chulalongkorn University, Bangkok, Thailand
  • Kittipoch Saengsai Center of Excellence in Digital and AI for Mental Health, Faculty of Engineering, Chulalongkorn University, Bangkok, Thailand
  • Pasit Jakkrawankul Center of Excellence in Digital and AI for Mental Health, Faculty of Engineering, Chulalongkorn University, Bangkok, Thailand
  • Chanyanart Kiattiporn-Opas Center of Excellence in Digital and AI for Mental Health, Faculty of Engineering, Chulalongkorn University, Bangkok, Thailand
  • Kantapat Chaichareenon Center of Excellence in Digital and AI for Mental Health, Faculty of Engineering, Chulalongkorn University, Bangkok, Thailand
  • Arisara Amrapala Center of Excellence in Digital and AI for Mental Health, Faculty of Engineering, Chulalongkorn University, Bangkok, Thailand
  • Kulvara Lapanan Neuroscience Research Australia, Randwick, New South Wales, Australia and School of Population Health, Faculty of Medicine and Health, University of New South Wales, Sydney, New South Wales, Australia
  • Titipat Achakulvisut School of Population Health, Faculty of Medicine and Health, University of New South Wales, Sydney, New South Wales, Australia
  • Peerapol Vateekul Faculty of Engineering, Mahidol University, Bangkok, Thailand
  • Natawut Nupairoj Faculty of Engineering, Chulalongkorn University, Bangkok, Thailand
  • Phanupong Phutrakool Faculty of Medicine, Chulalongkorn University, Bangkok, Thailand
  • Rapinpat Yodlorchai Center of Excellence in Digital and AI for Mental Health, Faculty of Engineering, Chulalongkorn University, Bangkok, Thailand
  • Narin Hiransuthikul Faculty of Medicine, Chulalongkorn University, Bangkok, Thailand
  • Sarunya Hengprapom Faculty of Medicine, Chulalongkorn University, Bangkok, Thailand

Keywords:

Artificial intelligence, depression, detection

Abstract

Background: In Thailand, the high prevalence of depression, particularly in rural areas with limited mental health services, poses significant challenges. Advanced technologies like digital phenotyping and artificial intelligence (AI), particularly natural language processing (NLP) and Large Language Models (LLMs), offer promising solutions by analyzing digital data to detect early signs of depression.

Objectives: This study evaluated the performance of the "Detection and Monitoring Intelligence Network of Depression (DMIND)" application, an AI-powered screening tool for detecting depression in Thai adults that analyzes behavioral data from participant responses using machine learning algorithms, such as NLP models and LLMs. The study aimed to determine the agreement of the DMIND AI model and the Thai version of the Hamilton depression rating scale (HDRS-17 Thai) in classifying depression severity.

Methods: This cross-sectional study recruited 388 participants from one tertiary care hospital and two psychiatric hospitals. Initially, participants used the DMIND application, where they were asked to answer a series of questions. Their response in the application was recorded and a pre-trained AI model predicted their depression severity. Subsequently, a trained nurse or psychologist then assessed participants using the HDRS-17 Thai to establish a baseline measure of depression severity. Statistical analysis involved comparing the depression severity classifications from the DMIND AI model with the HDRS-17 Thai. Cohen's kappa coefficient, sensitivity, specificity, and predictive values were used to evaluate the agreement between the two assessments.

Results: Our DMIND application demonstrated moderate agreement with the HDRS-17 Thai, indicating substantial consistency in depression severity classification. The tool showed high sensitivity (87.3%) and moderate specificity (59.5%), with strong negative predictive values for detecting depression.

Conclusion: The AI-powered DMIND application effectively screens for depression by analyzing digital data from participants' responses. Its moderate agreement with a traditional clinical assessment and strong diagnostic performance highlights its potential as a scalable, accessible tool for mental health management in Thailand. Integrating AI tools like the DMIND into the public health infrastructure could significantly enhance the accessibility, accuracy, and responsiveness of mental health services, particularly in underserved regions, potentially revolutionizing the management and treatment of depression across the country.

Downloads

Download data is not yet available.

References

Prasartpornsirichoke J, Pityaratstian N, Poolvoralaks C, Sirinimnualkul N, Ormtavesub T, Hiranwattana N,et al. The prevalence and economic burden of treatment-resistant depression in Thailand. BMC Public Health 2023;23:1541.

https://doi.org/10.1186/s12889-023-16477-y

Nochaiwong S, Ruengorn C, Thavorn K, Hutton B,Awiphan R, Phosuya C, et al. Global prevalence of mental health issues among the general population during the coronavirus disease 2019 pandemic:a systematic review and meta-analysis. Sci Rep 2021;11:10173.

https://doi.org/10.1038/s41598-021-89700-8

Department of Mental Health's recent surveys and studies December 2022.

Moreno MA, Jelenchick LA, Egan KG, Cox E, Young H, Gannon KE, et al. Feeling bad on Facebook:depression disclosures by college students on a social networking site. Depress Anxiety 2011;28:447-55.

https://doi.org/10.1002/da.20805

De Choudhury M, Gamon M, Counts S, Horvitz E.Predicting depression via social media. Proceedings of the International AAAI Conference on Web and Social Media 2021;7:128-37.

https://doi.org/10.1609/icwsm.v7i1.14432

Cummins N, Dineley J, Conde P, Matcham F, Siddi S,Lamers F, et al. Multilingual markers of depression in remotely collected speech samples: a preliminary analysis. J Affect Disord 2023;341:128-36.

https://doi.org/10.1016/j.jad.2023.08.097

Hemrungrojn S, Saengsai K, Jakkrawankul P ,Kiattiporn-Opas C, Chaichareenon K, Amrapala A,et al. Development and evaluation of the DMIND questionnaire: Preparing for AI integration into an effective depression screening tool. medRxiv 2024:2024.06.07.24308625.

https://doi.org/10.1101/2024.06.07.24308625

Obuchowski NA. Sample size calculations in studies of test accuracy. Stat Methods Med Res 1998;7:371-92.

https://doi.org/10.1177/096228029800700405

Serengil SI, Ozpinar A, editors. LightFace: A hybrid deep face recognition framework. 2020 Innovations in intelligent systems and applications conference (ASYU); 2020 Oct 15-17; Istanbul, Turkey: IEEE; 2020.

https://doi.org/10.1109/ASYU50717.2020.9259802

Zhang X, Malkov Y, Florez O, Park S, McWilliams B,Han J, et al, editors. TwHIN-BERT: a socially-enriched pre-trained language model for multilingual tweetrepresentations at twitter. 29th ACM SIGKDD conference on knowledge discovery and data mining,KDD 2023; 2023 Aug 6; Long beach, United States:Association for computing machinery; 2023.

https://doi.org/10.1145/3580305.3599921

Lotraku M, Sukanich P, Sukying C. The reliability and validity of Thai version of Hamilton rating scale for depression. J Psychiatry Assoc Thailand 1996;41;235-246.

Landis JR, Koch GG. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics 1977;33:363-74.

https://doi.org/10.2307/2529786

Milintsevich K, Sirts K, Dias G. Towards automatic text-based estimation of depression through symptom prediction. Brain Inform 2023;10:4.

https://doi.org/10.1186/s40708-023-00185-9

Bauer B, Norel R, Leow A, Rached ZA, Wen B, Cecchi G. Using large language models to understand suicidality in a social media-based taxonomy of mental health disorders: linguistic analysis of Reddit posts.JMIR Ment Health 2024;11:e57234.

https://doi.org/10.2196/57234

Squires M, Tao X, Elangovan S, Gururajan R, Zhou X,Acharya UR, et al. Deep learning and machine learn ing in psychiatry: a survey of current progress in depression detection, diagnosis and treatment. Brain Inform 2023;10:10.

https://doi.org/10.1186/s40708-023-00188-6

Downloads

Published

2024-10-01

How to Cite

1.
Hemrungroj S, Saengsai K, Jakkrawankul P, Kiattiporn-Opas C, Chaichareenon K, Amrapala A, Lapanan K, Achakulvisut T, Vateekul P, Nupairoj N, Phutrakool P, Yodlorchai R, Hiransuthikul N, Hengprapom S. Accuracy and concordance of DMIND AI application with a renowned depression assessment tool in Thai adults. Chula Med J [Internet]. 2024 Oct. 1 [cited 2024 Dec. 11];68(4). Available from: https://he05.tci-thaijo.org/index.php/CMJ/article/view/3259