TYPE Original Research

PUBLISHED 04 January 2024

DOI 10.3389/fpsyt.2023.1256571

Development of depression
OPEN ACCESS detection algorithm using text
scripts of routine psychiatric
EDITED BY
Ann John,
Swansea University Medical School,
United Kingdom

REVIEWED BY
interview
Saeid Komasi,
Mind GPS Institute, Iran
Ebrahim Ghaderpour,
Jihoon Oh 1†, Taekgyu Lee 2†, Eun Su Chung 2, Hyonsoo Kim 3,
Sapienza University of Rome, Italy Kyongchul Cho 3, Hyunkyu Kim 3, Jihye Choi 1,
*CORRESPONDENCE Hyeon-Hee Sim 1, Jongseo Lee 2, In Young Choi 4 and
Dai-Jin Kim
kdj922@catholic.ac.kr Dai-Jin Kim 1,4*
†
These authors have contributed equally to 1
Department of Psychiatry, College of Medicine, Seoul St. Mary’s Hospital, The Catholic
this work and share first authorship University of Korea, Seoul, Republic of Korea, 2 College of Medicine, The Catholic University of
RECEIVED 11July 2023 Korea, Seoul, Republic of Korea, 3 Acryl, Seoul, Republic of Korea, 4 Department of Medical
ACCEPTED 13 December 2023 Informatics, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea
PUBLISHED 04 January 2024

CITATION
Oh J, Lee T, Chung ES, Kim H, Cho K, Kim H,
Choi J, Sim HH, Lee J, Choi IY and Background: A psychiatric interview is one of the important procedures in
Kim DJ (2024) Development of depression diagnosing psychiatric disorders. Through this interview, psychiatrists listen
detection algorithm using text scripts of
routine psychiatric interview.
to the patient’s medical history and major complaints, check their emotional
Front. Psychiatry 14:1256571. state, and obtain clues for clinical diagnosis. Although there have been
doi: 10.3389/fpsyt.2023.1256571 attempts to diagnose a specific mental disorder from a short doctor-patient
COPYRIGHT conversation, there has been no attempt to classify the patient’s emotional
© 2024 Oh, Lee, Chung, Kim, Cho, Kim, Choi, state based on the text scripts from a formal interview of more than 30 min
Sim, Lee, Choi and Kim. This is an open-
access article distributed under the terms of and use it to diagnose depression. This study aimed to utilize the existing
the Creative Commons Attribution License machine learning algorithm in diagnosing depression using the transcripts
(CC BY). The use, distribution or reproduction of one-on-one interviews between psychiatrists and depressed patients.
in other forums is permitted, provided the
original author(s) and the copyright owner(s) Methods: Seventy-seven clinical patients [with depression (n = 60); without
are credited and that the original publication depression (n = 17)] with a prior psychiatric diagnosis history participated in
in this journal is cited, in accordance with
accepted academic practice. No use, this study. The study was conducted with 24 male and 53 female subjects
distribution or reproduction is permitted with the mean age of 33.8 (± 3.0). Psychiatrists conducted a conversational
which does not comply with these terms. interview with each patient that lasted at least 30 min. All interviews with
the subjects between August 2021 and November 2022 were recorded and
transcribed into text scripts, and a text emotion recognition module was
used to indicate the subject’s representative emotions of each sentence.
A machine learning algorithm discriminates patients with depression and
those without depression based on text scripts.
Results: A machine learning model classified text scripts from depressive
patients with non-depressive ones with an acceptable accuracy rate (AUC
of 0.85). The distribution of emotions (surprise, fear, anger, love, sadness,
disgust, neutral, and happiness) was significantly different between patients
with depression and those without depression (p < 0.001), and the most
contributing emotion in classifying the two groups was disgust (p < 0.001).
Conclusion: This is a qualitative and retrospective study to develop a tool
to detect depression against patients without depression based on the text
scripts of psychiatric interview, suggesting a novel and practical approach
to understand the emotional characteristics of depression patients and to
use them to detect the diagnosis of depression based on machine learning
methods. This model could assist psychiatrists in clinical settings who

Frontiers in Psychiatry 01 frontiersin.org

Oh et al. 10.3389/fpsyt.2023.1256571

conduct routine conversations with patients using text transcripts of the

interviews.

KEYWORDS

machine learning, depression, emotions, psychological interview, sentiment

analysis

Introduction audio features were used to estimate depression severity scores and
detect depression (17).
Depression is the most prevalent mental health issue that affects A series of studies have focused mainly on the acoustic and text
hundreds of millions of people and is considered one of the leading features from the conversations (18, 19). Acoustic features in
causes of burden globally (1, 2). It is estimated that the lifetime spontaneous speech were used to recognize depression against the
prevalence of depression among adults was 10.8% from 1994 to 2014 normal control, with improvement was reported in the performance
(3), and the burden due to mental disorders has not been reduced using the first few sentences (18). Indirect text features from the
despite evidence-based interventions (1). In addition, the prevalence patients, such as the total number of sentences, average words
of depression in South Korea shows an increasing trend (4). spoken in each sentence, frequency of laughter, and depression-
The diagnosis and evaluation of major depressive disorder related words, were fed into the model in addition to audio and
(MDD) are based on diagnostic criteria based on DSM-5, which visual features (19). However, the nature of audio and video data
requires a clinical judgment of a trained clinician on listed requires much preparation for consistent recording quality across
symptoms, including depressed mood, markedly diminished interest the samples (20), and even the laboratory setup to collect audiovisual
or pleasure, significant weight loss, slowing down of thought, a data still requires extensive pre-processing to guarantee the quality
reduction of physical movement, fatigue or loss of energy, reduced of input into the model (21).
ability to think or concentrate, and recurrent thoughts of death (5). In addition, there have not been many attempts to measure
The screening for these symptoms mainly depends on diagnostic symptom severity or identify depression by directly collecting data
questionnaires such as Patient Health Questionnaire-9 (PHQ-9), from the psychiatric interviews between the psychiatrists and the
Beck Depression Inventory (BDI) (6), and the Hamilton Depression patients, where structured psychiatric interviews are essential in
Rating Scale (HDRS) (7). This questionnaire-based diagnostic making an accurate diagnosis to satisfy the categorical conditions
approach necessitates an interview with clinicians, but it can listed in DSM-5. The interviews are still often encouraged to induce
be prone to biases as they are either self-reported by patients or free-of-context, unstructured conversations that can illicit subjective
administered by clinicians (8). experiences from the patients (22), as such interviews are often the
It is important to start treatment earlier for patients with MDD single most important source of information in obtaining clinical
because the time to treatment is correlated with the prognosis (9). A cues for psychiatrists.
diverse range of barriers, such as education, income, and accessibility, In this study, we utilized XGBoost algorithm to identify
contribute to the underdiagnosis of depression (10). As an early depression based on the actual psychiatric interviews between the
diagnosis of depression may reduce the severe depressive symptoms psychiatrists and the patients. We aimed to identify patients with
and improve the prognosis, there is a need for an objective method depression against the psychiatric patients without depression based
that can diagnose patients’ emotional and depressive states. on the text scripts of routine psychotherapy sessions to overcome
Recent AI-based approaches have gained attraction to provide burdensome requirements in collecting and pre-processing the
additional information on diagnosing depression. Physiological audiovisual data that have been widely used to analyze the depression
signals such as electroencephalogram (11, 12) and features from patients with machine learning methods. We classified emotional
eye-blinking (13) were captured upon audio-visual stimuli to classify characteristics of the text scripts from the interviews on the back of
emotions by utilizing deep neural networks. More common the improved accuracy of text emotion recognition applications
approaches include applying deep learning models on audio and (23–25). Transcripts from psychiatric interviews are easy to collect
visual data from clinical patients and public datasets (14, 15), where and require minimal pre-processing, whereas audio and visual data
widely used datasets classified facial expressions into emotional are more complex in nature and data processing perspective. It is one
labels such as anger, disgust, fear, happiness, sadness, surprise, and of the first attempts to identify depression using text emotion
neutral (16). Symptom severity of depression was measured based recognition based on routine psychiatric interviews in the
on the speech and 3D facial scan data in DAIC-WOZ dataset, and clinical setting.
the convolutional neural network (CNN) model was reported to The rest of this paper is organized as follows: the data
demonstrate reliable results in detecting MDD (14). Potential acquisition process from the clinical patients and the machine
depression risk was tried to be identified on the video recordings of learning model were presented in Materials and Methods; results
depression patients in China conducting structured tasks with a of depression classification is presented in Results; summary,
deep belief network (DBN) based model (15). There was also an future works, and limitations are discussed in Discussion; and
audio-focused approach where patients’ low-level and high-level lastly Conclusions.

Frontiers in Psychiatry 02 frontiersin.org

Oh et al. 10.3389/fpsyt.2023.1256571

Materials and methods were recorded under the subjects’ consent, and text scripts were
produced by a separate scripter for the first 15 to 20 min of the
Participants voice recordings after each interview.
Then, sentences from psychiatrist were removed from the text
Seventy-seven clinical patients (24 male, 53 female) between scripts so that only the sentences from the subjects could be left
20 and 65 years old participated in this qualitative and retrospective in the scripts. Emotional classification of each sentence was
study to develop a tool to detect depression. The dialogue data were conducted by Emotional Analysis Module patented by Acryl Inc.
acquired in a consecutive manner from all inpatients and at the Republic of Korea Intellectual Property Office (26), where
outpatients who agreed to record their interview during the the input is a single sentence, and the output is a list of
treatment. Participants were diagnosed with depression or anxiety, probabilities of 8 emotions of the corresponding sentence, namely
with or without a current episode, established through DSM-5. The surprise, fear, anger, love, disgust, sadness, neutral, and
clinical diagnosis was provided by the agreement of two or more happiness. For each transcript, probabilities of eight emotions
psychiatrists at Seoul St. Mary’s Hospital by assessing the patients were derived for the first 250 sentences, resulting in 2,000
in person. Interviews with the participants were conducted from probability data. The average probability value for each emotion
August 2021 to November 2022. All participants were required to was calculated and appended as statistics in front of the 2,000
provide informed consent forms to be considered as the subjects, data. As a result, 2,008 probability data were formed as vectors
and the Institutional Review Board of Seoul St. Mary’s Hospital and became the input vector for the machine learning model.
approved this study (KC21ONSI0387). The transcription and feeding of the input vectors into the
Inclusion criteria included (1) adults aged 18–65 years; (2) machine learning model was conducted until the model to detect
individuals who have received a primary diagnosis of depression depression was believed to perform with adequate accuracy.
(ICD codes: F32, F33, F34) from the Department of Psychiatry
and have undergone treatment; (3) for the control group,
individuals who have not received the diagnoses or treatment Machine learning model to detect
mentioned in (2); and (4) individuals who have received sufficient depression
explanation of this clinical trial, have understood it, voluntarily
decided to participate, and provided written consent to adhere Boosting is an ensemble method to create a strong learner by
to precautions. combining multiple weak learners. A weak learner indicates a
Exclusion criteria included any current or lifetime axis model that performs slightly better than a randomized prediction.
I psychiatric disorders, such as schizophrenia, schizoaffective In contrast, a strong learner suggests a model that performs well,
disorder, other psychotic and substance-related disorders, organic significantly better than a randomized prediction. A model is
mental disorders, neurological disorders (e.g., epilepsy, dementia), iteratively modified to minimize a loss function by evaluating
and cardiovascular disorders. A total of 10 people were excluded errors from the previous model and adjust the weights to “boost”
due to intake of prohibited substances such as alcohol and the accuracy, but overfitting can remain as a problem (27).
psychostimulant (n = 3), change in diagnosis (n = 5), and voluntary XGBoost is an algorithm that combines multiple decision trees
withdrawal of consent (n = 2). to make predictions (28) based on Gradient Boosting Model
(GBM) to overcome the overfitting problem by adopting
Classification and Regression Tree (CART) model for regression.
Patient characteristics It also makes predictions extremely fast by parallel processi3ng of
the data. In addition, a weighted quantile sketch was used to handle
Among the 77 participants, 60 subjects were diagnosed with missing data.
depression, and 17 subjects had other psychiatric illnesses The 166 scripts were split in training and test sets using scikit-
(Table 1). The with-depression group included 16 males (26.7%) learn package, which uses the stratified random sampling method,
and 44 females (73.3%), whereas the without-depression group into an 80/20 ratio. 4-fold cross-validation was conducted on the
consisted of 8 males (47.1%) and nine females (52.9%). The mean training set to prevent overfitting (29). Hyperparameters, including
age was 33.2 (±3.3) for the with-depression group and 35.9 (±6.9) learning rate, maximum depth, regularization factor (lambda),
for the control group. There were no significant differences in early stopping, and evaluation metric, were optimized using grid
gender and age between the two groups (p > 0.05, Table 1). search (30).
The performance of a model was evaluated with Accuracy and
F1 score. Accuracy is the percentage of correct predictions made,
Data acquisition but it can sometimes be misleading when the dataset is unbalanced.
The F1 score is a harmonic mean of precision and recall, reflecting
A psychiatrist performed a psychiatric interview with each the imbalance of the dataset. In addition, Area Under the Curve
subject in a quiet psychiatric consultation room. The interviews (AUC) was also evaluated, where in general, AUC under 0.7
were conducted as part of psychotherapy, in the form of semi- indicates less reliable, AUC between 0.7 and 0.8 shows somewhat
structured format which included typical attributes such as daily reliable, and more than 0.8 means highly reliable.
lives, chief complaints, thought contents, cognitions, judgments, RStudio 2022.12.0 + 353 was used for the statistical analysis of
and insights. The interviews lasted 30 min or longer. All interviews the data collected.

Frontiers in Psychiatry 03 frontiersin.org

Oh et al. 10.3389/fpsyt.2023.1256571

TABLE 1 Patient characteristics.

No. (%) of patients

Total (n = 77) With- depression Without- p-value
(n = 60) depression (n = 17)
Sexa 0.192

Male 24 (31.2) 16 (26.7) 8 (47.1)

Female 53 (68.8) 44 (73.3) 9 (52.9)

Ageb 0.947

Mean 33.8 33.2 35.9

(95% CI) (30.8–36.8) (29.9–36.5) (29.0–42.7)

20–29 42 (54.5) 33 (55.0) 9 (52.9)

30–39 16 (20.8) 13 (21.7) 3 (17.6)

40–49 5 (6.5) 4 (6.7) 1 (5.9)

50–59 9 (11.7) 6 (10.0) 3 (17.6)

60+ 5 (6.5) 4 (6.7) 1 (5.9)

Minimum 20 20 20

Maximum 64 64 63

Diagnosis

Adjustment disorder with

3 (3.9) 3 (5.0)
depressed mood

Bipolar disorder (currently

11 (14.3) 11 (18.3)
depression)

Major depressive disorder 22 (28.6) 22 (36.7)

Persistent depressive disorder 22 (28.6) 22 (36.7)

Other specified depressive

2 (2.6) 2 (3.3)
disorder

Anxiety disorder 1 (1.3) 1 (5.9)

Anorexia nervosa 1 (1.3) 1 (5.9)

Acute stress disorder 1 (1.3) 1 (5.9)

Alochol use disorder 3 (3.9) 3 (17.6)

Bipolar and related disorder 1 (1.3) 1 (5.9)

Intermittent explosive
1 (1.3) 1 (5.9)
disorder

Post-traumatic stress disorder 6 (7.8) 6 (35.3)

Somatic symptom disorder 1 (1.3) 1 (5.9)

Substance use disorder 1 (1.3) 1 (5.9)

Trichotillomania 1 (1.3) 1 (5.9)

a
Chi-squared test on with-depression group vs. without-depression group.
b
Fisher’s exact test on with-depression group vs. without-depression group.

Results the subjects was 5.8. As a result, 166 scripts were eventually fed into
the model to detect depression.
Characteristics of extracted sentences As a result, a total of 20,405 sentences were split from the 166
scripts, and an emotion with the highest probability was considered
A total of 451 scripts were originally collected from the 77 as the representative emotion of each sentence in comparing
subjects. The scripts were pre-processed in the form appropriate for emotional characteristics of the two groups. In the with-depression
learning the model. To avoid overweighting a particular diagnosis or group, there were 15,223 sentences with an average of 2,184 words
subject, the emotion vectors collected from the first five scripts from consisting of 8,072 characters on each script. There were 5,182
each subject were selected in the sequential order and used for analysis sentences with an average of 2,171 words and 8,156 characters on each
to avoid oversampling, as the average number of scripts collected from script in the without-depression group.

Frontiers in Psychiatry 04 frontiersin.org

Oh et al. 10.3389/fpsyt.2023.1256571

TABLE 2 Emotions counts from the scripts.

No. (%) of sentences

Total With- depression Without- depression p-valuea
Emotions classified <0.001

Surprise 146 (0.7) 113 (0.7) 33 (0.6)

Fear 1,519 (7.4) 1,112 (7.3) 407 (7.9)

Anger 153 (0.7) 107 (0.7) 46 (0.9)

Love 45 (0.2) 39 (0.3) 6 (0.1)

Sadness 3,304 (16.2) 2,487 (16.3) 817 (15.8)

Disgust*** 2,357 (11.6) 1,626 (10.7) 731 (14.1)

Neutral** 12,069 (59.1) 9,110 (59.8) 2,959 (57.1)

Happiness 812 (4.0) 629 (4.1) 183 (3.5)

Total 20,405 (100.0) 15,223 (100.0) 5,182 (100.0)

a
Chi-squared test on with-depression group vs. without-depression group.
**p < 0.01 based on post-hoc analysis of Pearson’s Chi-squared test.
***p < 0.001 based on post-hoc analysis of Pearson’s Chi-squared test.

Distribution of emotions

The frequently represented emotions in the with-depression

group were neutral (59.8%), sadness (16.3%), disgust (10.7%), fear
(7.3%), and happiness (4.1%). The without-depression group had a
similar order of the frequently represented emotions, namely neutral
(57.1%), sadness (15.8%), disgust (14.1%), fear (7.9%), and happiness
(3.5%). The distribution of eight emotions represented by the
sentences significantly differed between the two groups based on the
Chi-squared test of homogeneity (p < 0.001, Table 2).
Disgust (p < 0.001) and neutral (p < 0.01) were identified as the
emotions that contributed to the significant difference in the
distributions between the two groups based on the post hoc analysis of
the residuals of the chi-squared test (31).

FIGURE 1

Classification results of with-depression ROC curve on the test set.

and without-depression groups

The ROC curve of the machine learning model which used the TABLE 3 Confusion matrix on the test set.
original probability vectors showed an AUC of 0.85 (Figure 1) upon
the hyperparameters optimized with grid search (32). The model Ground truth
classified patients with depression against those without depression With- Without-
with a sensitivity of 0.96, specificity of 0.25, an accuracy of 0.79, and depression depression
an F1 score of 0.88 (Table 3). Model With-depression 73.5% (25) 17.6% (6)
output Without-depression 2.9% (1) 5.9% (2)

Discussion * Key metrics of classification include accuracy of 79.4%, F1 score of 87.7%, sensitivity of
96.2%, and specificity of 25.0%. F1 score is defined by 2 * Precision * Recall /
(Precision + Recall).
Our text emotion recognition algorithm revealed the difference in
emotion distributions between the patients with depression and the
control group. The distribution of emotions extracted from the Among eight emotional labels (surprise, fear, anger, love, disgust,
sentences showed significant differences between the two groups, sadness, neutral, and happiness), the most contributing emotion that
mainly due to less frequent expressions of disgust in the with- discriminates between depression and the control was disgust. Patients
depression group. The machine learning model could classify patients with depression were known to have problems recognizing facial
with depression against the without-depression control with good expressions showing disgust (33, 34). Functional MRI signals
reliability based on the emotional profiles extracted from responded in higher intensity among patients with depression to
the transcripts. disgust (33), suggesting impaired functioning in the basal ganglia (34).

Frontiers in Psychiatry 05 frontiersin.org

Oh et al. 10.3389/fpsyt.2023.1256571

TABLE 4 Feature importance analysis.

Depression is associated with self-disgust (35), presumably due to
altered emotion regulation strategies (36). In addition to the Importance
recognition of and response to external stimuli, the findings of this
Neutral 0.187
study concur with the association of expression of disgust and
Love 0.155
depression. Neutral was one of the contributing factors in
discriminating the two groups. Fear 0.153

Depression is typically characterized by a depressed mood or Surprise 0.134

By emotions
sadness, but its contribution in discriminating the two groups in Anger 0.120
this study was not significant. The mood or the sadness is generally
Disgust 0.094
determined by physicians from the general atmosphere throughout
the conversation. Meanwhile in this study, “sadness” as an emotion Happiness 0.079

was derived from specific sentences, indicating sentiments at Sadness 0.077

certain moments during the conversation. Sadness was not 1–20 0.317
significant in our study due to the difference of the time-interval
81–100 0.186
between the clinical cue and our method. In addition, we compared
21–40 0.151
the with-depression group against the patients without depression,
not against the normal control group. Some patients in the control By location of 61–80 0.110
group such as those with PTSD and somatic symptoms disorder, sentences 41–60 0.103
might have expressed sadness as much as depression group under (nth sentence) 101–120 0.054
the influence of accompanying symptoms.
141–160 0.043
The probability vectors of emotions derived from sentences
were fed to train the machine learning model, and the model 121–140 0.036

discriminated depression from the control group with an AUC of 161 and later 0.000
0.85, indicating a high reliability of the model. Feature importance
analysis revealed that the model did not depend solely on any single
emotion in detecting the depression, and the probability vectors of and emotions brought up by the patients depending on the flow of
the sentences from the early part of the interviews were considered the conversation. Such less standardized interviews were thus
more important by the model compared to the latter part of the considered more suitable for this study. However, psychotherapy
interviews (Table 4). Feature importance represents the contribution sessions are less standardized and more difficult to quantify, and the
of each input feature in making branches in the decision tree. It is questions and contents may vary depending on the interviewers.
evaluated by the change in the model performance given the Structured interviews could have improved the credibility of the
exclusion of a certain input feature. probability vectors of the emotions derived from the interviews.
Previous studies have normally used audio and visual dataset as Also, the random split of input data by scikit learn package
inputs to detect depression and its severity (14, 15, 17, 18), but the might have resulted in the scripts from the same person being put
nature of audiovisual data poses hurdles in contemplating clinical into both the training and test set, considering the dataset size for
applications for psychiatrists (20, 21). In contrast, text data in the this study. The model could have been trained in a way that classifies
form of transcripts of conversations based on the recordings of depression based on the person’s traits rather than the traits of the
routine psychiatric interviews, as collected in this study, is depression itself. A larger dataset could improve the model, not
incomparably easier to obtain upon the subject’s consent. An only in terms of the overall performance, including sensitivity, but
ordinary voice recorder in the office and a mean to transcribe of the also by minimizing the possibility of learning any individual’s trait
conversation would suffice the setting for the data collection and so that the model ultimately identifies the depression solely based
the audio-to-text pre-processing. Such a simple requirement to on the emotional features of depression.
generate the model input suggests a great advantage in applying to There are a couple of factors that might have affected the
clinical situations. external validity of this study. The number of data is limited due to
Considering the objective of this study to assist psychiatrists in the retrospective nature of the study, and the model’s performance
the actual clinical situations, the model should be able to detect along with statistical power could have improved further by feeding
subtlety of depression that psychiatrists might have missed. model inputs. Also, the control group consisted of psychiatric
Currently, the model provides relatively low specificity compared patients without depression, rather than non-clinical samples
to its very high sensitivity. While we recognize the need to without any psychiatric diagnosis. It would have been valuable if
demonstrate improved overall performance of the model, we also such non-clinical samples were also recruited to compare against
believe that the advantage of high sensitivity outweighs any the with-depression group. However, we believe that it is more
disadvantage posed by the low specificity, as early recognition and difficult to detect patients with depression against the patients with
proper intervention are important in treating depression with better other psychiatric diagnosis, as conducted in this study. In addition,
outcomes (37). the subjects in the with-depression group and the without-
There are several limitations to this study. First, psychotherapy depression control group were not exactly matched due to the
sessions are semi-structured and conducted by multiple retrospective nature of this study. We plan to test the detection
psychiatrists of the hospital depending on the availability. This algorithm on non-clinical subjects in the future in a
would have allowed flexibility to explore deeper into the thoughts prospective manner.

Frontiers in Psychiatry 06 frontiersin.org

Oh et al. 10.3389/fpsyt.2023.1256571

The number of scripts collected for this study was originally Author contributions
much larger than that of the input scripts fed into the model.
We decided to use a maximum of 5 scripts for each subject to avoid JO: Conceptualization, Data curation, Formal analysis,
potential bias due to oversampling. For example, we collected more Methodology, Writing – original draft, Writing – review & editing,
than 40 scripts from five subjects, three from the with-depression Supervision, Investigation. TL: Formal analysis, Validation, Writing
group and the rest from the without-depression control group. It – original draft, Writing – review & editing, Investigation,
could have improved the performance metrics of the model when the Methodology. EC: Writing – original draft, Investigation. HyoK:
entire data collection was used, but the risk associated with depending Formal analysis, Visualization, Writing – review & editing. KC:
on a few subjects should be avoided. Collecting an evenly distributed Formal analysis, Visualization, Writing – review & editing. HyuK:
number of scripts from the subjects would improve the model’s Formal analysis, Visualization, Writing – review & editing. JC: Data
performance and avoid bias arising from the oversampling. curation, Writing – review & editing. H-HS: Data curation, Writing
Acryl’s Emotional Analysis Module, which was used to derive – review & editing. JL: Writing – review & editing. IC: Writing –
probability vectors assigned to the sentences of the text scripts, did review & editing. D-JK: Conceptualization, Data curation, Funding
not consider any context or meanings of the sentence. Large acquisition, Supervision, Writing – review & editing.
Language Models (LLM) has been increasingly used recently in
many applications which can consider textual contexts based on the
parameters and datasets much larger than the conventional models Funding
in analyzing text data. It remains as a future work to incorporate
LLM in the process of classifying emotions from the text scripts. The author(s) declare financial support was received for the
research, authorship, and/or publication of this article. This work was
supported by the National IT Industry Promotion Agency (NIPA)
Conclusion grant funded by the Ministry of Science and ICT (No. S0252-21-
1001). The funder had no role in the design, data collection, analysis,
This study suggests a novel approach to detect depression with interpretation of results, and manuscript drafting.
conversational scripts with patients based on text emotion recognition
and a machine learning model. Emotional distribution significantly
differed between the depression and the control group, and the model Conflict of interest
showed a reliable performance in classifying patients with depression
from those without depression. Our results could assist clinicians in HyoK, KC, and HyuK were employed by Acryl.
the initial diagnosis and follow-up of depressive patients with The remaining authors declare that the research was conducted in
conventional diagnostic tools. Further studies would improve the the absence of any commercial or financial relationships that could
performance, potentially detecting depression alongside the be construed as a potential conflict of interest.
psychiatrists in the clinics and hospitals.

Publisher’s note
Data availability statement
All claims expressed in this article are solely those of the authors
The raw data supporting the conclusions of this article will and do not necessarily represent those of their affiliated organizations,
be made available by the authors, without undue reservation. or those of the publisher, the editors and the reviewers. Any product
that may be evaluated in this article, or claim that may be made by its
manufacturer, is not guaranteed or endorsed by the publisher.
Ethics statement
The studies involving humans were approved by the Institutional Supplementary material
Review Board of Seoul St. Mary’s Hospital. The studies were conducted
in accordance with the local legislation and institutional requirements. The Supplementary material for this article can be found online
The participants provided their written informed consent to at: https://www.frontiersin.org/articles/10.3389/fpsyt.2023.1256571/
participate in this study. full#supplementary-material

References
1. GBD 2019 Mental Disorders Collaborators. Global, regional, and national burden 4. Kim GE, Jo M-W, Shin Y-W. Increased prevalence of depression in South Korea
of 12 mental disorders in 204 countries and territories, 1990-2019: a systematic analysis from 2002 to 2013. Sci Rep. (2020) 10:16979. doi: 10.1038/s41598-020-74119-4
for the global burden of disease study 2019. Lancet Psychiatry. (2022) 9:137–50. doi:
5. American Psychiatric Association. Diagnostic and statistical manual of mental
10.1016/S2215-0366(21)00395-3
disorders. 5th ed. Washington, D.C: American Psychiatric Association (2013).
2. Sinyor M, Rezmovitz J, Zaretsky A. Screen all for depression. BMJ. (2016) 352:i1617. 6. Maurer DM, Raymond TJ, Davis BN. Depression: screening and diagnosis. Am Fam
doi: 10.1136/bmj.i1617 Physician. (2018) 98:508–15.
3. Lim GY, Tam WW, Lu Y, Ho CS, Zhang MW, Ho RC. Prevalence of depression in 7. Bagby RM, Ryder AG, Schuller DR, Marshall MB. The Hamilton depression rating
the community from 30 countries between 1994 and 2014. Sci Rep. (2018) 8:2861. doi: scale: has the gold standard become a lead weight? AJP. (2004) 161:2163–77. doi:
10.1038/s41598-018-21243-x 10.1176/appi.ajp.161.12.2163

Frontiers in Psychiatry 07 frontiersin.org

Oh et al. 10.3389/fpsyt.2023.1256571

8. Thombs BD, Kwakkenbos L, Levis AW, Benedetti A. Addressing overestimation of 23. Kratzwald B, Ilic S, Kraus M, Feuerriegel S, Prendinger H. Deep learning for
the prevalence of depression based on self-report screening questionnaires. CMAJ. affective computing: text-based emotion recognition in decision support. Decis Support
(2018) 190:E44–9. doi: 10.1503/cmaj.170691 Syst. (2018) 115:24–35. doi: 10.1016/j.dss.2018.09.002
9. Ghio L, Gotelli S, Cervetti A, Respino M, Natta W, Marcenaro M, et al. Duration of 24. Calefato F, Lanubile F, Novielli N. EmoTxt: a toolkit for emotion recognition from
untreated depression influences clinical outcomes and disability. J Affect Disord. (2015) text. Proceedings of the 2017 Seventh International Conference on Affective Computing
175:224–8. doi: 10.1016/j.jad.2015.01.014 and Intelligent Interaction Workshops and Demos (ACIIW). San Antonio, TX: IEEE
10. Faisal-Cury A, Ziebold C, Rodrigues DMO, Matijasevich A. Depression (2017). p. 79–80
underdiagnosis: prevalence and associated factors. A population-based study. J Psychiatr 25. Batbaatar E, Li M, Ryu KH. Semantic-emotion neural network for emotion
Res. (2022) 151:157–65. doi: 10.1016/j.jpsychires.2022.04.025 recognition from text. IEEE Access. (2019) 7:111866–78. doi: 10.1109/
11. Liu J, Wu G, Luo Y, Qiu S, Yang S, Li W, et al. EEG-based emotion classification ACCESS.2019.2934529
using a deep neural network and sparse autoencoder. Front Syst Neurosci. (2020) 14:43. 26. Oh SS, Lee HH, Park WJinventors; Acryl Inc., assignee. Emotion recognition
doi: 10.3389/fnsys.2020.00043 method and computer program for executing the method, emotion recognizer
12. Ahmed MZI, Sinha N, Ghaderpour E, Phadikar S, Ghosh R. A novel baseline generation method and computer program for executing the method. Republic of Korea
removal paradigm for subject-independent features in emotion classification using EEG. Patent 10-2110393-0000. (2020).
Bioengineering. (2023) 10:54. doi: 10.3390/bioengineering10010054 27. Natekin A, Knoll A. Gradient boosting machines, a tutorial. Front Neurorobot.
13. Korda AI, Giannakakis G, Ventouras E, Asvestas PA, Smyrnis N, Marias K, et al. (2013) 7:21. doi: 10.3389/fnbot.2013.00021
Recognition of blinks activity patterns during stress conditions using CNN and 28. Chen T, Guestrin C. XGBoost: a scalable tree boosting system. Proceedings of the
Markovian analysis. Signals. (2021) 2:55–71. doi: 10.3390/signals2010006 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data
14. Haque A, Guo M, Miner AS, Fei-Fei L. Measuring depression symptom severity Mining. San Francisco California USA: ACM (2016). p. 785–794.
from spoken language and 3D facial expressions. arXiv [Preprint]. (2018). doi: 10.48550/ 29. Rodriguez JD, Perez A, Lozano JA. Sensitivity analysis of k-fold cross validation
arXiv.1811.08592 in prediction error estimation. IEEE Trans Pattern Anal Mach Intell. (2010) 32:569–75.
15. Guo W, Yang H, Liu Z, Xu Y, Hu B. Deep neural networks for depression doi: 10.1109/TPAMI.2009.187
recognition based on 2D and 3D facial expressions under emotional stimulus tasks.
30. Hutter F, Kotthoff L, Vanschoren J. Automated machine learning: methods, systems,
Front Neurosci. (2021) 15:609760. doi: 10.3389/fnins.2021.609760
challenges. Cham: Springer International Publishing (2019).
16. Almeida J, Vilaça L, Teixeira IN, Viana P. Emotion identification in movies
through facial expression recognition. Appl Sci. (2021) 11:6827. doi: 10.3390/ 31. Beasley TM, Schumacker RE. Multiple regression approach to analyzing
app11156827 contingency tables: post hoc and planned comparison procedures. J Exp Educ. (1995)
64:79–93. doi: 10.1080/00220973.1995.9943797
17. Rejaibi E, Komaty A, Meriaudeau F, Agrebi S, Othmani A. MFCC-based recurrent
neural network for automatic clinical depression recognition and assessment from 32. Fawcett T. An introduction to ROC analysis. Pattern Recogn Lett. (2006)
speech. Biomed Signal Process Control. (2022) 71:103107. doi: 10.1016/j.bspc.2021.103107 27:861–74. doi: 10.1016/j.patrec.2005.10.010

18. Alghowinem S, Goecke R, Wagner M, Epps J, Breakspear M, Parker G. Detecting 33. Surguladze SA, El-Hage W, Dalgleish T, Radua J, Gohier B, Phillips ML. Depression
depression: a comparison between spontaneous and read speech. Proceedings of the is associated with increased sensitivity to signals of disgust: a functional magnetic
2013 IEEE International Conference on Acoustics, Speech and Signal Processing. resonance imaging study. J Psychiatr Res. (2010) 44:894–902. doi: 10.1016/j.
Vancouver, BC, Canada: IEEE (2013). p. 7547–7551. jpsychires.2010.02.010

19. Dham S, Sharma A, Dhall A. Depression scale recognition from audio, visual and 34. Douglas KM, Porter RJ. Recognition of disgusted facial expressions in severe
text analysis. arXiv [Preprint]. (2017). Available at: http://arxiv.org/abs/1709.05865 depression. Br J Psychiatry. (2010) 197:156–7. doi: 10.1192/bjp.bp.110.078113
20. Kumar A, Kaur A, Kumar M. Face detection techniques: a review. Artif Intell Rev. 35. Gao S, Zhang L, Yao X, Lin J, Meng X. Associations between self-disgust,
(2019) 52:927–48. doi: 10.1007/s10462-018-9650-2 depression, and anxiety: a three-level meta-analytic review. Acta Psychol. (2022)
228:103658. doi: 10.1016/j.actpsy.2022.103658
21. Albahra S, Gorbett T, Robertson S, D’Aleo G, Kumar SVS, Ockunzzi S, et al.
Artificial intelligence and machine learning overview in pathology & laboratory 36. Ypsilanti A, Lazuras L, Powell P, Overton P. Self-disgust as a potential mechanism
medicine: a general review of data preprocessing and basic supervised concepts. Semin explaining the association between loneliness and depression. J Affect Disord. (2019)
Diagn Pathol. (2023) 40:71–87. doi: 10.1053/j.semdp.2023.02.002 243:108–15. doi: 10.1016/j.jad.2018.09.056
22. Nordgaard J, Sass LA, Parnas J. The psychiatric interview: validity, structure, and 37. Kraus C, Kadriu B, Lanzenberger R, Zarate CA Jr, Kasper S. Prognosis and
subjectivity. Eur Arch Psychiatry Clin Neurosci. (2013) 263:353–64. doi: 10.1007/ improved outcomes in major depression: a review. Transl Psychiatry. (2019) 9:127. doi:
s00406-012-0366-z 10.1038/s41398-019-0460-3

Frontiers in Psychiatry 08 frontiersin.org