PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

Assessing the Predictive Validity and Reliability of the DASS-21, PHQ-9 and GAD-7 in an Indonesian

Sample

Sandersan Onie PhD12, Amelia Citra Kirana3, Adisya Alfian3, Ninette Putri Mustika3, Veronica Adesla3 &

Ratih Ibrahim3

1
Black Dog Institute, Sydney, Australia
2
School of Psychology, UNSW Sydney, Sydney, Australia
3
Personal Growth, Jakarta, Indonesia

Corresponding Author:
Sandersan Onie
Black Dog Institute, Hospital Road, NSW 2031
s.onie@blackdog.org.au
Ph: +61 432 359 134
PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

Abstract

Anxiety and depression continue to be major issues in developing countries. Despite this, anxiety and

depression research are still lacking. A necessary tool to conduct quality research is validated and

reliable measurements. In this study, we assess the predictive validity and reliability of three frequently

used tools in the literature in an Indonesian population: the Participant Health Questionnaire 9,

Generalized Anxiety Disorder 7, and the depression and anxiety subscales of the Depression, Anxiety

and Stress Scale 21. In the study, 409 participants completed a questionnaire containing these three

measurements. McDonald’s Omega reliability analyses found that all three questionnaires had good

internal reliability (ω ≥ 0.785) and using the Hopkins Self-Checklist 25 (a previously validated

questionnaire in an Indonesian population), there was extreme evidence that each tool predicted the

corresponding HSCL subscale (BF10 ≥ 1.191 x 1026, R2 ≥ 0.268). Therefore, we provide initial evidence for

the validity and reliability of these questionnaires in an Indonesian population.

Keywords: Questionnaire Validation, Validity, Reliability, Depression, Anxiety

PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

Mental disorders constitute a significant burden of diseases globally. In 2016, more than 16% of

the population were reported to have been affected by mental or addictive disorders with depressive

and anxiety disorders being the most common (Rehm & Shield, 2019). According to the World Health

Organization (2017a), 27% of global depression cases and 23% of global anxiety cases occur in the

Southeast Asia region, which represent the biggest concentration of mental illness in a region. In

Indonesia, the high rates of mental health problems also became a major issue with more than nine

million and eight million individuals were reported to have depressive and anxiety disorders respectively

(WHO, 2017). Despite this, research on depression and anxiety in Indonesia is sparse (Kinzie, Blake,

Alvares, & McCormick-Ricket, 2016). Reliable research studies are needed to screen mental health issues

in large populations, of which results can be used to improve people’s mental health condition.

However, one barrier to good research in developing countries is the lack of validated and reliable

questionnaires. In this study, we seek to address that issue for the Indonesian population.

Three commonly used assessment tools are the Patient Health Questionnaire (PHQ-9),

Generalised Anxiety Disorder (GAD-7), and Depression, Anxiety and Stress Scale (DASS 21) depression

and anxiety subscales. These questionnaires have been used to assess various interventions (Dear et al.,

2011), neuroscience studies (Cohen, Holdnack, Kisala, & Tulsky, 2018; Neuman, Boyle, & Chan, 2013),

and in other novel treatments e.g. the assessment and modification of attentional biases (Onie & Most,

2017; Onie, Notebaert, Clarke & Most, 2019). Below we briefly discuss the various measures.

The Patient Health Questionnaire (PHQ-9) is often used due to its brevity (Kroenke, 2012). PHQ

was developed as a fully self-administered version of the original Primary Care Evaluation of Mental

Disorders (PRIME-MD) by Spitzer, Kroenke, Williams, and Patient Health Questionnaire Primary Care

Study Group (1999). The PHQ has been studied in several settings, such as medical setting (Arroll et al.,

2010; Huang, Chung, Kroenke, Delucchi, & Spitzer 2006), general setting (Kocalevent, Hinz, & Brahler,
PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

2013; Martin et al., 2006), using adolescents (Richardson et al., 2010), university students (Adewuya,

Ola, & Afolabi, 2006) to elderly (Han et al., 2008; Phelan et al., 2010) as the study population. Despite its

robustness as a tool for diagnosis and management of depression in several settings, the validation of

PHQ in Indonesia has not been done.

Generalised Anxiety Disorder-7 (GAD-7) was developed to overcome the shortcomings of

previous anxiety measures in the clinical setting that were usually long, have proprietary nature, lacking

in usefulness as a diagnostic and severity measure, and require clinician administration rather than self-

report by patient (Spitzer, Kroenke, Williams, & Lowe, 2006). GAD-7 consists of short questions that can

be completed in less than three minutes and considered to be an easy self-report measure, particularly

if compared to other screening measures such as Mini International Neuropsychiatric Interview-

International Classification of Diseases 10 (MINI ICD-10; Budikayanti et al., 2019). Items in GAD-7

describe the most prominent diagnostic features of the DSM-IV diagnostic criteria A, B, and C for

generalized anxiety disorder (Lowe et al., 2008), but can also be used for other anxiety disorders

(Johnson, Ulvenes, Oktedalen, & Hoffart, 2019). The GAD-7 has been studied in a wide variety of

settings, including medical setting (Seo & Park, 2015), occupational setting (Pavicic Zezelj et al., 2019), as

well as school setting (Osborn, Venturo-Conerly, Wasil, & Schleider, 2019) and adapted to various

languages including Dutch (Donker, van Straten, Marks, & Cuijpers, 2011), Arabic (Sawaya, Atoui,

Hamadeh, & Zeinoun, 2016), Filipino (Garabiles et al., 2019). A study using the Indonesian version of

GAD-7 for patients with epilepsy found that this measure has good reliability, sensitivity, and specificity

(Budikayanti et al., 2019). However, the validity of this measure for the general population remains

unknown.

Beside PHQ-9 and GAD-7, one of the most used instruments for assessing depression and

anxiety is Depression, Anxiety, and Stress Scale (DASS). This measure was developed by Lovibond and
PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

Lovibond (1983) as an instrument that has the full range of core symptoms of anxiety and depression

while providing maximum discrimination between the scales of anxiety and depression (Brown,

Chorpita, Korotitsch, & Barlow, 1997). During its development, a third factor (stress) emerged from the

analysis of scale structure consisting items related to difficulty relaxing, irritability, and agitation (Brown

et al., 1997). DASS-21 is a short version of the original DASS-42 and has been argued to have several

advantages relative to the DASS-42, namely fewer items, a cleaner factor structure, and smaller

interfactor correlations (Antony, Bieling, Cox, Enns, & Swinson, 1998). Several studies yielded a good

reliability and validity of this measure across various cultures in both clinical and non-clinical samples

(Antony et al., 1998; Asghari, Saed, & Dibajna, 2008; Jun, Johnston, Kim, & O’Leary, 2018; Sinclair et al.,

2012; Wood, Nicholas, Blyth, Asghari, & Gibson, 2010). Nonetheless the psychometric properties of

DASS-21 in Indonesian general population are still unclear.

In this study, we sought to investigate the predictive validity and reliability of the three

commonly used questionnaires: Depression, Anxiety and Stress Scale 21 (Lovibond & Lovibond, 1983),

Participant Health Questionnaire (Spitzer, Kroenke, & Williams, 1999), as well as the Generalized Anxiety

Disorder questionnaire (Spitzer et al., 2006), in an Indonesian sample. This is to further much needed

research in anxiety and depression by providing validated and reliable tools. In a previous study, the

Hopkins Symptom Checklist Scale-25 (HSCL-25) was validated in an Indonesian population (Turnip &

Hauff, 2007). Therefore, we will use the HSCL - 25 to assess the predictive validity of the unvalidated

questionnaires.

Methods

Participants

A total of 409 Indonesians aged between 18 and 39 years old were recruited to participate in

the study through non-probability, accidental sampling method (female = 53%; age, M = 27.1, SD =
PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

5.08). Participants were visitors of public primary health care centers in the West Jakarta. Participants

gave consent and were given a small notebook as a token for their participation. All data collection was

conducted in line with the Indonesian Psychological Code of Ethics, Research.

Materials

All data was collected through a paper and pen questionnaire and part of a larger data collection

which includes demographic data, as well as questionnaires assessing mental health and stigma. For

brevity, we will only discuss the components of the questionnaire relevant to this research question. The

translated versions of the PHQ -9, GAD – 7 and DASS – 21 can be found at https://osf.io/q2gk6/files/.

Demographics. Demographic variables included age, gender, marital status, dominant ethnic

background, religious belief, and educational level.

Mental Health Questionnaires. All the questionnaires were translated from English to

Indonesian by the authors, aiming to have the closest semantic approximate while not sacrificing the

closest word choice. This was due to certain language nuances that did not translate. The translations

were then back translated by a non-psychologist which closely matched the original. The questionnaires

will be available upon request.

Mental health was assessed by Hopkins Self-checklist-25 (HSCL-25) and Depression, Anxiety, and

Stress Scale-21 (DASS-21). Psychopathology symptoms were assessed by DSM-5 Self-rated Level 1 Cross-

cutting Symptom Measure for Adult (DSM-5 CCSM). Depression was assessed by Patient Health

Questionnaire-9 (PHQ-9). Lastly, Generalized Anxiety Disorder-7 (GAD-7) was used to assess anxiety

symptoms.

HSCL-25 consists of 25 items, including 15 items and 10 items to assess depression and anxiety

symptoms, respectively. Each item of HSCL-25 uses four-point Likert scale, with 1 corresponding not at
PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

all bothered by the symptoms and 4 corresponding extremely bothered by the symptoms (i.e trembling,

feeling tense or keyed up, crying easily). The lowest possible score is 25 and the highest is 100. The final

score is then divided by 25, so the final score is ranged between 1 and 4. HSCL-25 score lower than 1.75

indicates low level of psychological distress, whereas score higher than 1.75 indicates high level of

psychological distress. The Indonesian version of HSCL-25 was previously used by Turnip and Hauff

(2007).

DASS-21 consists of 21 items designed to measure emotional distress in three subcategories of

depression, anxiety, and stress (Lovibond & Lovibond, 1983), with seven items for each subcategory.

DASS-21 is a self-report questionnaire based on four-point rating scale, ranging from 0 (did not apply to

me at all/ never) to 3 (applied to me very much, or most of the time/ almost always). Participants were

asked how much the statements in DASS-21 applied to them over the past week. Score of DASS-21 is

obtained by adding up the rating given by participants for each item in each subcategory.

PHQ-9 is a nine-items self-report questionnaire. Participants were asked to rate how they felt in

the last two weeks, in a four-points Likert scale varied from 0 (not at all) to 3 (nearly every day). The final

score is obtained by adding up the score on each item, thus generate a score ranging from 0 to 27. Score

of 10 or higher is considered to indicate mild major depression, 15 or higher indicates moderate major

depression, and 20 or higher indicates severe major depression. Some settings use the threshold score

of 15 or higher to consider initiating treatment with antidepressants (Arroll et al., 2010). A sample item

is “Trouble concentrating on things, such as reading the newspaper or watching television.”

GAD-7 contains seven items to assess anxiety symptoms. Participants were asked to rate how

many days they have been bothered by the symptoms portrayed in each item over the last two weeks.

Each item is scored 0 (not at all) to 3 (nearly every day). A sample item is “Not being able to stop or

worrying too much.” The Indonesian version of this scale was previously used in a study conducted by
PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

Budikayanti et al. (2019) and generate an internal consistency coefficient of 0.87 (Alpha) whereas the

present study found an internal consistency coefficient of 0.80. These findings indicate the GAD-7 have

an adequate reliability to assess anxiety symptoms.

Procedure

Data was collected by 18 enumerators across eight public primary health care centers in West

Jakarta from May to September 2019. Enumerators would approach individuals leaving the health

centre and ask if they would volunteer to fill out a questionnaire. Following informed consent, the

participant filled out the questionnaire and the enumerator would sit by their side to answer any

questions. Through previous studies we found this the most effective way to collect data in this context,

as the population are prone to ask many questions or not understand what’s being asked. A pocket

notebook was given to participants as a token of thanks after they completed the questionnaires.

Results

Where available, we opted to perform Bayesian analyses due to its ability to quantify evidence

as opposed to a dichotomous outcome, as well as also provide evidence for the null (Onie, Notebaert,

Clarke & Most, 2019, but also see. Tendeiro & Kiers, 2019). Rather than using p-values to make

inferences, Bayesian analyses use Bayes Factors which is interpreted as the ratio between evidence for

the alternative hypothesis and evidence for the null hypothesis. For example, a Bayes Factor of 10,

would indicate that given the data, the alternative model is 10 times more likely than the null model.

Jeffrey’s scale was used to interpret Bayes Factors (Jeffreys, 1961), which places Bayes Factors into

labelled categories (e.g., BF = 1–3 is anecdotal evidence, BF = 3–10 is moderate evidence and BF = 10–30

is strong evidence). Furthermore, while unlike a Neymar-Pearson Frequentist approach where a critical

value is placed, a Bayes Factor of 10 is often used as a cutoff for strong evidence (Wagenmakers et al.,
PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

2015; Aczel et al., 2018). All analyses were conducted in JASP (JASP Core Team, 2019) and using default

priors (Rouder & Morey, 2012).

Reliability

To assess internal reliability, we will use McDonald’s Omega (Zinbarg, Revelle, Yovel, & Li, 2005).

Unlike the more commonly used split half and Cronbach’s Alpha, McDonald’s Omega has been shown to

be more robust when assumptions are violated. A critical assumption that is often violated is tau-

equivalence, which is the assumption that each and every item or trial measures the underlying

construct to the same degree. However, when studying a multi-faceted construct such as depression

with different diagnostic components (loss of pleasure, suicidal ideation, sleep patterns etc.), it is

unreasonable to expect that each component contributes or indexes the underlying construct to the

same degree. Thus, tau-equivalence is almost always violated. The analyses can be found below in Table

1.

Table 1.
Scale Reliability Statistics

Measures Internal reliability (McDonald’s ω)

GAD-7 0.809
PHQ-9 0.801
DASS - depression subscale 0.794
DASS - anxiety subscale 0.785
DASS - stress subscale 0.800
DASS - total 0.910
HSCL - anxiety 0.911
HSCL - depression 0.938
Note: Reliability estimates were calculated using JASP.

Validity
PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

To assess predictive validity, we will use the HSCL - 25 as the benchmark since it has been

validated in an Indonesian sample and see whether other questionnaires predict relevant subscales of

the HSCL - 25. Thus, we will assess whether the PHQ-9 and DASS-21 Depression subscale predict the

HSCL - Depression subscale and assess whether the GAD-7 and DASS-21 stress subscale predict the HSCL

- Anxiety subscale. If the scale has predictive validity, there will be strong evidence for a relationship the

relevant HSCL subscale. Unlike the reliability analyses, all the validity analyses used sum scores.

Depression Scales. To investigate the predictive validity of the PHQ-9 and DASS-21 Depression

Subscale, we investigated whether each scale predicted the HSCL - Depression subscale. The analysis

revealed extremely strong evidence that the PHQ-9 (BF 10 = 2.665 x 1029, R2 = 0.295, B = 1.046, 95% [0.898

1.212]) and DASS-21 Depression subscale (BF 10 = 2.970 x 1046, R2 = 0.420, B = 2.079, 95% [1.830 2.314])

predicted the HSCL - Depression subscale, providing extremely strong evidence for the predictive validity

of both.

Anxiety Scales. To investigate the predictive validity of the GAD-7 and DASS-21 Anxiety

Subscale, we investigated whether each scale predicted the HSCL - Anxiety subscale. The analysis

revealed extremely strong evidence that the GAD-7 (BF 10 = 1.191 x 1026, R2 = 0.268, B = 0.738, 95% [0.615

0.863]) and DASS-21 Anxiety subscale (BF 10 = 8.743 x 1047, R2 = 0.430, B = 1.167, 95% [1.027 1.288])

predicted the HSCL - Anxiety subscale, providing extremely strong evidence for the predictive value of

both scales.

Discussion

In this study, we investigated the internal reliability and predictive validity of the PHQ-9, GAD-7,

DASS-21 subscales of Depression and Anxiety. We investigated reliability by submitting to a McDonald’s

Omega analysis, and we assessed predictive validity by assessing whether each scale predicted the

relevant subscale of the HSCL - 25, a previously validated questionnaire. The results revealed that all the
PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

scales had good reliability (ω ≥ 0.785) and there was extremely strong evidence that each scale

predicted the corresponding HSCL subscale. Therefore, our study provides initial evidence that the

Indonesian version of the PHQ-9, GAD-7, DASS-21 subscales of depression and anxiety are internally

consistent and have predictive value.

One finding is that the HSCL subscales had higher internal reliability than the GAD-7, PHQ-9 or

DASS-21 subscales. Hedge, Powell, and Sumner (2017) found that reliability affected correlations in that

less reliable measures required greater sample sizes. Therefore, the HSCL may be more appropriate in

individual difference research where participants are limited; however, the GAD-7, PHQ-9 and DASS-21

may be preferable for comparing to existing studies with the same questionnaires.

One limitation to this study is that subjects in the present study were not assessed with a

following clinical interview to assess the sensitivity and specificity. This part due to the lack of a

structured clinical interview in Indonesia. Previous findings compared the PHQ diagnosis with diagnosis

based on clinical interview showed good sensitivity and specificity of the PHQ (Chen et al., 2013; Gelaye

et al., 2013; Martin et al., 2006), and we recommend that until this is established, to use recommended

cutoffs. In addition, we urge the reader to take the results with caution, as one area of research that is

much needed, but lacking is the conceptualization of psychopathology in diverse cultures.

In conclusion, this study provides initial evidence for the validity and reliability of three

important scales in the study of anxiety and depression, in a general Indonesian population.
PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

References

Aczel, B., Palfi, B., Szollosi, A., Kovacs, M., Szaszi, B., Szecsi, P., et al. (2018). Quantifying support for the

null hypothesis in psychology: an empirical investigation. Adv. Methods Pract. Psychol. Sci. 1, 357–

366. doi: 10.1177/2515245918773742

Adewuya, A. O., Ola, B. A., & Afolabi, O. O. (2006). Validity of the patient health questionnaire (PHQ-9)

as a screening tool for depression amongst Nigerian university students. Journal of affective

disorders, 96(1-2), 89-93.

Antony, M. M., Bieling, P. J., Cox, B. J., Enns, M. W., & Swinson, R. P. (1998). Psychometric properties of

the 42-item and 21-item versions of the Depression Anxiety Stress Scales in clinical groups and a

community sample. Psychological assessment, 10(2), 176.

Arroll, B., Goodyear-Smith, F., Crengle, S., Gunn, J., Kerse, N., Fishman, T., ... & Hatcher, S. (2010).

Validation of PHQ-2 and PHQ-9 to screen for major depression in the primary care population. The

Annals of Family Medicine, 8(4), 348-353.

Asghari, A., Saed, F., & Dibajnia, P. (2008). Psychometric properties of the Depression Anxiety Stress

Scales-21 (DASS-21) in a non-clinical Iranian sample. Int J psychol, 2(2), 82-102.

Brown, T. A., Chorpita, B. F., Korotitsch, W., & Barlow, D. H. (1997). Psychometric properties of the

Depression Anxiety Stress Scales (DASS) in clinical samples. Behaviour research and therapy,

35(1), 79-89.

Budikayanti, A., Larasari, A., Malik, K., Syeban, Z., Indrawati, L. A., & Octaviana, F. (2019). Screening of

Generalized Anxiety Disorder in Patients with Epilepsy: Using a Valid and Reliable Indonesian

Version of Generalized Anxiety Disorder-7 (GAD-7). Neurology Research International, 2019.

Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment (Vol. 17). Sage publications.
PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

Chen, S., Fang, Y., Chiu, H., Fan, H., Jin, T., & Conwell, Y. (2013). Validation of the nine ‐item Patient

Health Questionnaire to screen for major depression in a Chinese primary care population. Asia‐

Pacific Psychiatry, 5(2), 61-68.

Cohen, M. L., Holdnack, J. A., Kisala, P. A., & Tulsky, D. S. (2018). A comparison of PHQ-9 and TBI-QOL

depression measures among individuals with traumatic brain injury. Rehabilitation psychology,

63(3), 365.

Cohen, R. J., & Swerdlik, M. E. (2009). Psychological testing and assessment: An introduction to tests

and measurement. Boston: McGraw-Hill.

Dear, B. F., Titov, N., Schwencke, G., Andrews, G., Johnston, L., Craske, M. G., & McEvoy, P. (2011). An

open trial of a brief transdiagnostic internet treatment for anxiety and depression. Behaviour

research and therapy, 49(12), 830-837.

Donker, T., van Straten, A., Marks, I., & Cuijpers, P. (2011). Quick and easy self-rating of Generalized

Anxiety Disorder: validity of the Dutch web-based GAD-7, GAD-2 and GAD-SI. Psychiatry

research, 188(1), 58-64.

Fischer, E. H., & Farina, A. (1995). Attitudes toward seeking professional psychological help: A shortened

form and consideration for research. Journal of College Student Development, 36, 368-373.

Garabiles, M. R., Lao, C. K., Yip, P., Chan, E. W., Mordeno, I., & Hall, B. J. (2019). Psychometric

Validation of PHQ–9 and GAD–7 in Filipino Migrant Domestic Workers in Macao (SAR), China.

Journal of personality assessment, 1-12.

Gelaye, B., Williams, M. A., Lemma, S., Deyessa, N., Bahretibeb, Y., Shibre, T., ... & Zhou, X. H. A.

(2013). Validity of the patient health questionnaire-9 for depression screening and diagnosis in East

Africa. Psychiatry research, 210(2), 653-661.

Han, C., Jo, S. A., Kwak, J. H., Pae, C. U., Steffens, D., Jo, I., & Park, M. H. (2008). Validation of the

Patient Health Questionnaire-9 Korean version in the elderly population: the Ansan Geriatric study.

Comprehensive psychiatry, 49(2), 218-223.

PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

Hedge, C., Powell, G., & Sumner, P. (2018). The reliability paradox: Why robust cognitive tasks do not produce

reliable individual differences. Behavior Research Methods, 50(3), 1166-1186.

Huang, F. Y., Chung, H., Kroenke, K., Delucchi, K. L., & Spitzer, R. L. (2006). Using the Patient Health

Questionnaire‐9 to measure depression among racially and ethnically diverse primary care

patients. Journal of general internal medicine, 21(6), 547-552.

Jeffreys, H. (1961). Theory of Probability, 3rd Edn. Oxford: Clarendon Press.

Johnson, S. U., Ulvenes, P. G., Øktedalen, T., & Hoffart, A. (2019). Psychometric properties of the GAD-7

in a heterogeneous psychiatric sample. Frontiers in psychology, 10, 1713.

Jun, D., Johnston, V., Kim, J. M., & O’Leary, S. (2018). Cross-cultural adaptation and validation of the

Depression, Anxiety and Stress Scale-21 (DASS-21) in the Korean working population. Work,

59(1), 93-102.

Kaplan, R. M., & Saccuzzo, D. P. (2009). Psychological testing: Principles, applications, and issues.

Nelson Education.

Kinzie, E., Blake, A., Alvares, R., & McCormick‐Ricket, I. (2016). Mental Health Screening in North

Sulawesi, Indonesia: Kessler 6 pilot data and needs assessment results from the LearnToLive

Indonesian Health Initiative. Asia‐Pacific Psychiatry, 8(2), 118-126.

Kocalevent, R. D., Hinz, A., & Brähler, E. (2013). Standardization of the depression screener patient

health questionnaire (PHQ-9) in the general population. General hospital psychiatry, 35(5), 551-

555.

Kroenke, K. (2012). Enhancing the clinical utility of depression screening. CMAJ, 184(3), 281-282.

Löwe, B., Decker, O., Müller, S., Brähler, E., Schellberg, D., Herzog, W., & Herzberg, P. Y. (2008).

Validation and standardization of the Generalized Anxiety Disorder Screener (GAD-7) in the

general population. Medical care, 266-274.

PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

Martin, A., Rief, W., Klaiberg, A., & Braehler, E. (2006). Validity of the brief patient health questionnaire

mood scale (PHQ-9) in the general population. General hospital psychiatry, 28(1), 71-77.

Neumann, D. L., Boyle, G. J., & Chan, R. C. (2013). Empathy towards individuals of the same and

different ethnicity when depicted in negative and positive contexts. Personality and Individual

Differences, 55(1), 8-13.

Oei, T. P. S., Sawang, S., Goh, Y. W., & Mukhtar, F. (2013). Using the Depression Anxiety Stress Scale 21

(DASS-21) across cultures. International Journal of Psychology, 48(6), 1018–1029.

Onie, S., & Most, S. B. (2017). Two roads diverged: Distinct mechanisms of attentional bias differentially

predict negative affect and persistent negative thought. Emotion, 17(5), 884.

Onie, S., Notebaert, L., Clarke, P., & Most, S. B. (2019). Investigating the effects of Inhibition Training on

Attentional Bias change: a simple Bayesian Approach. Frontiers in psychology, 9, 2782.

Osborn, T. L., Venturo-Conerly, K. E., Wasil, A. R., Schleider, J. L., & Weisz, J. R. (2019). Depression

and anxiety symptoms, social support, and demographic factors among Kenyan high school

students. Journal of Child and Family Studies, 1-12.

Pavičić Žeželj, S., Cvijanović Peloza, O., Mika, F., Stamenković, S., Mahmutović Vranić, S., & Šabanagić

Hajrić, S. (2019). Anxiety and depression symptoms among gas and oil industry workers.

Occupational Medicine, 69(1), 22-27.

Phelan, E., Williams, B., Meeker, K., Bonn, K., Frederick, J., LoGerfo, J., & Snowden, M. (2010). A study

of the diagnostic accuracy of the PHQ-9 in primary care elderly. BMC family practice, 11(1), 63.

Rehm, J., & Shield, K. D. (2019). Global Burden of Disease and the Impact of Mental and Addictive

Disorders. Current psychiatry reports, 21(2), 10.

Richardson, L. P., McCauley, E., Grossman, D. C., McCarty, C. A., Richards, J., Russo, J. E., ... & Katon,

W. (2010). Evaluation of the Patient Health Questionnaire-9 Item for detecting major depression

among adolescents. Pediatrics, 126(6), 1117-1123.

PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

Rouder, J., and Morey, R. (2012). Default bayes factors for model selection in regression. Multivariate

Behav. Res. 47, 877–903. doi: 10.1080/00273171.2012.734737

Sawaya, H., Atoui, M., Hamadeh, A., Zeinoun, P., & Nahas, Z. (2016). Adaptation and initial validation of

the Patient Health Questionnaire–9 (PHQ-9) and the Generalized Anxiety Disorder–7

Questionnaire (GAD-7) in an Arabic speaking Lebanese psychiatric outpatient sample. Psychiatry

research, 239, 245-252.

Sinclair, S. J., Siefert, C. J., Slavin-Mulford, J. M., Stein, M. B., Renna, M., & Blais, M. A. (2012).

Psychometric evaluation and normative data for the depression, anxiety, and stress scales-21

(DASS-21) in a nonclinical sample of US adults. Evaluation & the Health Professions, 35(3), 259-

279.

Seo, J. G., & Park, S. P. (2015). Validation of the Generalized Anxiety Disorder-7 (GAD-7) and GAD-2 in

patients with migraine. The journal of headache and pain, 16(1), 97.

Spitzer, R. L., Kroenke, K., Williams, J. B., & Löwe, B. (2006). A brief measure for assessing generalized

anxiety disorder: the GAD-7. Archives of internal medicine, 166(10), 1092-1097.

Spitzer, R. L., Kroenke, K., Williams, J. B., & Patient Health Questionnaire Primary Care Study Group.

(1999). Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study.

Jama, 282(18), 1737-1744.

Sukantarat, K. T., Williamson, R. C. N., & Brett, S. J. (2007). Psychological assessment of ICU survivors:

a comparison between the Hospital Anxiety and Depression scale and the Depression, Anxiety and

Stress scale. Anaesthesia, 62(3), 239-243.

Tendeiro, J. N., & Kiers, H. A. (2019). A review of issues about null hypothesis Bayesian testing.

Psychological methods.

Turnip, S. S., & Hauff, E. (2007). Household roles, poverty and psychological distress in internally

displaced persons affected by violent conflicts in Indonesia. Social psychiatry and psychiatric

epidemiology, 42(12), 997-1004.

PREDICTIVE VALIDITY AND RELIABILITY IN AN INDONESIAN SAMPLE

Wagenmakers, E. J., Beek, T. F., Rotteveel, M., Gierholz, A., Matzke, D., Steingroever, H., et al., (2015).

Turning the hands of time again: a purely confirmatory replication study and a bayesian

analysis. Front. Psychol. 6:494. doi: 10.3389/fpsyg.2015.00494

Wood, B. M., Nicholas, M. K., Blyth, F., Asghari, A., & Gibson, S. (2010). The utility of the short version of

the Depression Anxiety Stress Scales (DASS-21) in elderly patients with persistent pain: does age

make a difference?. Pain Medicine, 11(12), 1780-1790.

World Health Organization. (2017). Depression and other common mental disorders. Retrieved from

https://apps.who.int/iris/bitstream/handle/10665/254610/WHO-MSD-MER-2017.2-eng.pdf

World Health Organization. (2019). Global health observatory data repository. Retrieved from

http://apps.who.int/gho/data/view.main.HWF11v

Zinbarg, R. E., Revelle, W., Yovel, I., and Li, W. (2005). Cronbach’s α, Revelle’s β, and McDonald’s ωH :

their relations with each other and two alternative conceptualizations of reliability. Psychometrika

70, 123–133. doi: 10.1007/s11336-003-0974-7