Early Detection Depression Based On Action Unit and Eye Gaze Features Using a Multi-Input CNN-WoPL Framework

Sugiyanto Sugiyanto; I Ketut Eddy  Purnama; Eko Mulyanto  Yuniarno; Mauridhi Hery Purnomo

doi:10.23887/janapati.v13i3.84674

Authors

Sugiyanto Sugiyanto Department of Electrical Engineering, Institut Teknologi Sepuluh Nopember https://orcid.org/0000-0003-1829-8944
I Ketut Eddy Purnama Department of Electrical Engineering, Institut Teknologi Sepuluh Nopember https://orcid.org/0000-0002-7438-7880
Eko Mulyanto Yuniarno Department of Electrical Engineering, Institut Teknologi Sepuluh Nopember https://orcid.org/0000-0003-1243-3025
Mauridhi Hery Purnomo Institut Teknologi Sepuluh Nopember

DOI:

https://doi.org/10.23887/janapati.v13i3.84674

Keywords:

Depression, Multi-Input, CNN, action unit, eye gaze

Abstract

Depression is a common mental disorder with significant life impact, including a high risk of suicide. Patients with depression attempt suicide five times more often than the general population. Self-reporting, subjective judgement and clinician expertise influence conventional diagnostic methods. For timely intervention and effective treatment, early and accurate diagnosis of depression is essential. This study proposes a framework called Multi-Input CNN-WoPL, a CNN-based method without a pooling layer that combines two features - action units and gaze - to improve accuracy and robustness in automatic depression detection. Pooling layer reduces spatial dimension of feature map, resulting in loss of information related to expression data, affecting depression detection result. The performance of the proposed method results in an accuracy of 0.994 and F1 score = 0.993, the F1 score value close to 1.0 indicates that the proposed method has good precision, recall and performance.

References

World Health Organization, Depression and Other Common Mental Disorders: Global Health Estimates. Geneva: World Health Organization, Licence: CCBY-NC-SA 3.0 IGO, 2017.

M. Nordentoft, P. B. Mortensen, and C. B. Pedersen, ‘Absolute Risk of Suicide After First Hospital Contact in Mental Disorder’, ARCH GEN PSYCHIATRY, vol. 68, no. 10, 2011.

M. Gili et al., ‘Mental disorders as risk factors for suicidal behavior in young people: A meta-analysis and systematic review of longitudinal studies’, Journal of Affective Disorders, vol. 245, pp. 152–162, Feb. 2019, doi: 10.1016/j.jad.2018.10.115.

M. Dong et al., ‘Prevalence of suicide attempt in individuals with major depressive disorder: a meta-analysis of observational surveys’, Psychol. Med., vol. 49, no. 10, pp. 1691–1704, Jul. 2019, doi: 10.1017/S0033291718002301.

H. Cai et al., ‘Prevalence of suicidal ideation and planning in patients with major depressive disorder: A meta-analysis of observation studies’, Journal of Affective Disorders, vol. 293, pp. 148–158, Oct. 2021, doi: 10.1016/j.jad.2021.05.115.

M. K. Nock, I. Hwang, N. A. Sampson, and R. C. Kessler, ‘Mental disorders, comorbidity and suicidal behavior: Results from the National Comorbidity Survey Replication’, Mol Psychiatry, vol. 15, no. 8, pp. 868–876, Aug. 2010, doi: 10.1038/mp.2009.29.

J. K. Hohls, H.-H. König, E. Quirke, and A. Hajek, ‘Anxiety, Depression and Quality of Life—A Systematic Review of Evidence from Longitudinal Observational Studies’, Int. J. Environ. Res. Public Health, 2021.

H. Minnis, R. Gajwani, and D. Ougrin, ‘Editorial: Early intervention and prevention of severe mental illness: A child and adolescent psychiatry perspective’, Front. Psychiatry, vol. 13, p. 963602, Jul. 2022, doi: 10.3389/fpsyt.2022.963602.

S. Saldivia, F. Cova, C. Inostroza, J. Aslan, and M. Farhang, ‘Preventive and Early Treatment of Depression in Older Adults’, in Prevention and Early Treatment of Depression Through the Life Course, V. Martínez and C. Miranda-Castillo, Eds., Cham: Springer International Publishing, 2023, pp. 167–187. doi: 10.1007/978-3-031-13029-8_9.

B. Bohman, ‘Clinicians’ perceptions and practices of diagnostic assessment in psychiatric services’, BMC Psychiatry, vol. 23, no. 1, p. 191, Mar. 2023, doi: 10.1186/s12888-023-04689-w.

J. M. Aultman, ‘Psychiatric Diagnostic Uncertainty: Challenges to Patient-Centered Care’, AMA Journal of Ethics, vol. 18, no. 6, pp. 579–586, Jun. 2016, doi: 10.1001/journalofethics.2016.18.6.ecas2-1606.

S. Song, S. Jaiswal, L. Shen, and M. Valstar, ‘Spectral Representation of Behaviour Primitives for Depression Analysis’, IEEE Trans. Affective Comput., vol. 13, no. 2, pp. 829–844, Apr. 2022, doi: 10.1109/TAFFC.2020.2970712.

K. C. D. Lacerda et al., ‘High depressive symptomatology reduces emotional reactions to pictures of social interaction’, Sci Rep, vol. 14, no. 1, p. 1266, Jan. 2024, doi: 10.1038/s41598-024-51813-1.

J. Joshi, R. Goecke, G. Parker, and M. Breakspear, ‘Can body expressions contribute to automatic depression analysis?’, in 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China: IEEE, Apr. 2013, pp. 1–7. doi: 10.1109/FG.2013.6553796.

J. M. Girard, J. F. Cohn, M. H. Mahoor, S. M. Mavadati, Z. Hammal, and D. P. Rosenwald, ‘Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses’, Image and Vision Computing, vol. 32, no. 10, pp. 641–647, Oct. 2014, doi: 10.1016/j.imavis.2013.12.007.

P. Ekman, W. V. Friesen, and J. Hager, Facial action coding system: A technique for the measurement of facial movement. The Manual on CDROM, 2002.

D. Venkataraman, ‘Extraction of Facial Features for Depression Detection among Students’.

A. Savran, B. Sankur, and M. T. Bilge, ‘Regression-based intensity estimation of facial action unit’, Image and Vision Computing, vol. 30, no. 10, pp. 774–784, Oct. 2012, doi: doi.org/10.1016/j.imavis.2011.11.008.

Z. Ming, A. Bugeau, J.-L. Rouas, and T. Shochi, ‘Facial Action Units intensity estimation by the fusion of features with multi-kernel Support Vector Machine’, in : Proc. of the 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) 2015, Ljubljana: IEEE, May 2015, pp. 1–6. doi: 10.1109/FG.2015.7284870.

Y. Li, S. M. Mavadati, M. H. Mahoor, Y. Zhao, and Q. Ji, ‘Measuring the intensity of spontaneous facial action units with dynamic Bayesian network’, Pattern Recognition, vol. 48, no. 11, pp. 3417–3427, Nov. 2015, doi: 10.1016/j.patcog.2015.04.022.

H. Akbar, S. Dewi, Y. A. Rozali, L. P. Lunanta, N. Anwar, and D. Anwar, ‘Exploiting Facial Action Unit in Video for Recognizing Depression using Metaheuristic and Neural Networks’, in : Proc. of the 1st International Conference on Computer Science and Artificial Intelligence (ICCSAI) 2021, Jakarta, Indonesia: IEEE, Oct. 2021, pp. 438–443. doi: 10.1109/ICCSAI53272.2021.9609747.

Z. Jiang, S. Harati, A. Crowell, H. S. Mayberg, S. Nemati, and G. D. Clifford, ‘Classifying Major Depressive Disorder and Response to Deep Brain Stimulation Over Time by Analyzing Facial Expressions’, IEEE Trans. Biomed. Eng., vol. 68, no. 2, pp. 664–672, Feb. 2021, doi: 10.1109/TBME.2020.3010472.

C. Yu, ‘Non-verbal Facial Action Units-based Automatic Depression Classification’, Nov. 20, 2022, arXiv: arXiv:2211.10911. Accessed: Jun. 15, 2024. [Online]. Available: http://arxiv.org/abs/2211.10911

M. Wen, Z. Dong, L. Zhang, B. Li, Y. Zhang, and K. Li, ‘Depression and Cognitive Impairment: Current Understanding of Its Neurobiology and Diagnosis’, NDT, vol. Volume 18, pp. 2783–2794, Nov. 2022, doi: 10.2147/NDT.S383093.

T. Shu, F. Zhang, and X. Sun, ‘Gaze Behavior based Depression Severity Estimation’, in 2023 IEEE 4th International Conference on Pattern Recognition and Machine Learning (PRML), Urumqi, China: IEEE, Aug. 2023, pp. 313–319. doi: 10.1109/PRML59573.2023.10348319.

O. Kobo, A. Meltzer-Asscher, J. Berant, and T. Schonberg, ‘Classification of depression tendency from gaze patterns during sentence reading’, Biomedical Signal Processing and Control, vol. 93, p. 106015, Jul. 2024, doi: 10.1016/j.bspc.2024.106015.

J. F. Cohn et al., ‘Detecting depression from facial actions and vocal prosody’, in 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, Amsterdam, Netherlands: IEEE, Sep. 2009, pp. 1–7. doi: 10.1109/ACII.2009.5349358.

Q. Wang, H. Yang, and Y. Yu, ‘Facial expression video analysis for depression detection in Chinese patients’, Journal of Visual Communication and Image Representation, vol. 57, pp. 228–233, Nov. 2018, doi: 10.1016/j.jvcir.2018.11.003.

N. Ghadiri, R. Samani, and F. Shahrokh, ‘Integration of Text and Graph-Based Features for Depression Detection Using Visibility Graph’, in Intelligent Systems Design and Applications, A. Abraham, S. Pllana, G. Casalino, K. Ma, and A. Bajaj, Eds., Cham: Springer Nature Switzerland, 2023, pp. 332–341.

Z. Li, Z. An, W. Cheng, J. Zhou, F. Zheng, and B. Hu, ‘MHA: a multimodal hierarchical attention model for depression detection in social media’, Health Inf Sci Syst, vol. 11, no. 1, p. 6, Jan. 2023, doi: 10.1007/s13755-022-00197-5.

S. Song, L. Shen, and M. Valstar, ‘Human Behaviour-Based Automatic Depression Analysis Using Hand-Crafted Statistics and Deep Learned Spectral Features’, in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi’an: IEEE, May 2018, pp. 158–165. doi: 10.1109/FG.2018.00032.

C. Chakraborty, M. Bhattacharya, S. Pal, and S.-S. Lee, ‘From machine learning to deep learning: Advances of the recent data-driven paradigm shift in medicine and healthcare’, Current Research in Biotechnology, vol. 7, p. 100164, 2024, doi: 10.1016/j.crbiot.2023.100164.

L. Koushik, ‘Interns@LT-EDI : Detecting Signs of Depression from Social Media Text’.

X. Kong, Y. Yao, C. Wang, Y. Wang, J. Teng, and X. Qi, ‘Automatic Identification of Depression Using Facial Images with Deep Convolutional Neural Network’, Med Sci Monit, vol. 28, Jun. 2022, doi: 10.12659/MSM.936409.

Z. Wang, C. Hu, W. Liu, X. Zhou, and X. Zhao, ‘EEG-based high-performance depression state recognition’, Front. Neurosci., vol. 17, p. 1301214, Jan. 2024, doi: 10.3389/fnins.2023.1301214.

A. Hassan and S. Bernadin, ‘A Comprehensive Analysis of Speech Depression Recognition Systems’, in SoutheastCon 2024, Atlanta, GA, USA: IEEE, Mar. 2024, pp. 1509–1518. doi: 10.1109/SoutheastCon52093.2024.10500078.

N. Ignatiev, I. Smirnov, and M. Stankevich, ‘Predicting Depression with Text, Image, and Profile Data from Social Media’:, in Proceedings of the 11th International Conference on Pattern Recognition Applications and Methods, Online Streaming, --- Select a Country ---: SCITEPRESS - Science and Technology Publications, 2022, pp. 753–760. doi: 10.5220/0010986100003122.

M. Patil, P. Mukherji, and V. Wadhai, ‘A novel hybrid optimization algorithm for depression detection using MRI and speech signal’, Biomedical Signal Processing and Control, vol. 86, p. 105046, Sep. 2023, doi: 10.1016/j.bspc.2023.105046.

W. Zhang, K. Mao, and J. Chen, ‘A Multimodal Approach for Detection and Assessment of Depression Using Text, Audio and Video’, Phenomics, May 2024, doi: 10.1007/s43657-023-00152-8.

P. Singh, P. Raj, and V. P. Namboodiri, ‘EDS pooling layer’, Image and Vision Computing, vol. 98, p. 103923, Jun. 2020, doi: 10.1016/j.imavis.2020.103923.

T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L.-P. Morency, ‘OpenFace 2.0: Facial Behavior Analysis Toolkit’, in : Proc. of the 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018), Xi’an, China: IEEE Explore, 2018, pp. 59–66. doi: doi: 10.1109/FG.2018.00019.

B. Arroll et al., ‘Validation of PHQ-2 and PHQ-9 to Screen for Major Depression in the Primary Care Population’, The Annals of Family Medicine, vol. 8, no. 4, pp. 348–353, Jul. 2010, doi: 10.1370/afm.1139.