The Role of Human Support on Engagement in an Online Depression Prevention Program for Youth

Abstract

Background: Depression is a significant public health problem for adolescents. The goal of this study was to evaluate the moderating role of human support in an online depression prevention program on both depression outcomes and overall engagement with the intervention. CATCH-IT is an Internet-based depression prevention program that has been shown to reduce symptoms for adolescents who report elevated depression symptom scores, compared to a health education (HE) control group. Participants in the CATCH-IT arm received human support (e.g., motivational interviewing, completed contacts). This study analyzes the moderating role of human support on depressive outcomes and engagement, and examines if engagement predicts depression outcomes. Methods: This secondary analysis consists of a randomized controlled trial for adolescents assigned to the CATCH-IT group. Mixed effects modeling, general linear models, and an exploratory multiple linear regression were used to explore the moderating relationship of human support between intervention and overall engagement. Study variables included depression outcomes (e.g., Center for Epidemiological Studies Depression Scale (CESD)), engagement components (e.g., modules completed, time on the site, and characters typed) and human support (e.g., motivational interviews and completed contacts.) Results: Results showed no significant relationship between contacts, motivational interviews, and depression scores. However, motivational interviews increased engagement with the intervention, such that those who received more motivational interviews completed significantly more modules, spent more time on the site, and typed more characters (p < 0.05). The number of contacts increased engagement with the intervention, and those who received more contacts spent more time on the site and typed more characters (p < 0.05). Exploratory multiple linear regression modeling demonstrated that male, African American/Black, and Hispanic/Latinx users were less engaged compared to other users. Lastly, engagement was not a significant predictor of depression outcomes (p > 0.05). Conclusions: The efficacy of CATCH-IT is not better explained by the degree to which participants received doses of human support from providers during the use of this online intervention. This may reveal the high potential of effective online interventions without the blended integration of human support for adolescents. To increase engagement of adolescents with an online depression prevention program, human support may be more efficient when utilizing MI rather than technical support.

1. Introduction

Major Depressive Disorder is a highly prevalent mental health disorder that impacts youth at a significant rate. An estimated 3.2 million adolescents aged 12 to 17 in the United States have had at least one major depressive episode in the previous 12 months (National Center for Health Statistics and Centers for Disease Control and Prevention 2020). Depression typically functions as a life-course disorder, emerging mid-adolescence, and recurring every five to seven years in 80% of individuals (Kennard et al. 2018). Given the high prevalence of MDD in youth, the risk of harmful behaviors and lifelong illness, and the serious associated functional impairments, it is imperative that prevention and intervention programs are available to address this problem (Gladstone et al. 2015).

Research has shown that prevention efforts create significant, small effects in reducing depression, although significant benefits may not be sustained over time (National Academies of Sciences, Engineering, and Medicine 2019). Newer research shows promise for treating adolescent depression using websites and technology for treatment (Andersson and Cuijpers 2009; Hoek et al. 2012). When comparing the efficacy of different types of prevention programs, there are challenges with implementing effective programs that are both useful and generalizable for youth of all ages (Abuwalla et al. 2018). Although many prevention programs are delivered in person, there has been an increase in the creation and use of online prevention efforts via behavioral intervention technologies (BITs), which can combat high treatment cost, increase accessibility, and provide effective treatment.

The presence of online interventions has increased to reach populations that experience many barriers to receiving and benefiting from mental health care (e.g., stigma, waiting times, physical distance to services) (Gladstone et al. 2015). This could be an effective modality because adolescents may feel more comfortable with the Internet, have increased access to computers or tablets, and possibly view Internet-based programs as more acceptable than face-to-face interventions (Gladstone et al. 2015). Furthermore, online interventions are significantly more affordable and feasible than traditional in vivo interventions. Although there is evidence that face-to-face and BIT programs for adolescents show no differences in outcome, and that both modalities have small effect sizes, there is conflicting research about BITs, which warrants further research and discussion (Rasing et al. 2017). For example, Ebert et al. (2015) conducted a meta-analysis on a series of randomized controlled trials for Internet and computer-based cognitive behavioral therapy programs for youth with anxiety and depression, examining whether computerized-CBT (cCBT) is effective for treating symptoms of depression and anxiety in adolescents. cCBT proved to be efficacious compared to controls for interventions targeting anxiety (g = 0.68), depression (g = 0.76), and for transdiagnostic interventions (g = 0.94) (Ebert et al. 2015). Results indicated that there is strong evidence for the efficacy of cCBT in treating depression and anxiety in youth. Online interventions may be a promising treatment alternative when face-to-face treatment is not plausible. The current research provides promise for the utilization of BITs in the context of preventing depression in youth. CATCH-IT (Competent Adulthood Transition with Cognitive-behavioral, Humanistic, and Interpersonal Training) is a primary-care, Internet-based, “behavioral vaccine” designed to target adolescents at an elevated risk of depression (Gladstone et al. 2015). CATCH-IT utilizes motivational interviewing, goal setting, and telephone coaching to guide participants through the stages of change to reduce vulnerability and increase protective factors (Gladstone et al. 2015).

Although BITs can be efficacious, there are issues of sustained engagement and proper use. This lack of engagement with the intervention can ultimately hinder users from obtaining the maximum benefit of BITs (Schueller et al. 2017). To combat this, researchers utilize supported interventions. Supported interventions integrate the use of human support to bolster engagement and provide technical and clinical troubleshooting. Supported interventions tend to be more efficacious than non-supported, self-guided interventions (Schueller et al. 2017).

Human support is defined as contact with a human in the context of a BIT’s treatment, aimed to increase the client’s ability to use the BIT to improve efficacy (Schueller et al. 2017). Support may vary in type, including in terms of content of the support, quantity, timing, quality, and cost. Researchers found larger effects on adherence and overall clinical outcomes in supported interventions (Schueller et al. 2017), relative to interventions without human support. Human support has been integrated into BITs through numerous avenues, including text messaging, emails, and phone calls provided by supporters (Schueller et al. 2017). Andersson and Cuijpers (2009) summarized the effects of Internet-based treatments for adult depression, and results demonstrated that there was a significant difference between supported (d = 0.61) and unsupported (d = 0.25) treatments (Andersson and Cuijpers 2009). Although more studies are needed to explore this relationship, other computerized treatments show promise for online depression prevention programs. However, the role of human support in online depression prevention programs has not been clearly defined or vastly explored, especially within the adolescent population. In general, human support improves adherence and outcomes when using Internet-based treatments, although this finding may change based on patient population, disorder, and usability of the program (Mohr et al. 2013).

More recently, a new version of the CATCH-IT program was developed and tested by Van Voorhees, Gladstone, and colleagues (Gladstone et al. 2018). This clinical trial compared CATCH-IT plus MI versus Health Education (HE) for 369 adolescents aged 13-18 (Gladstone et al. 2018). Adolescents with higher baseline CES-D10 scores showed a significantly stronger effect of CATCH-IT on sub-threshold or probable episodes of depression (Gladstone et al. 2018). However, there were no significant differences in depressive symptoms between the CATCH-IT and HE group. Researchers hypothesized that it is possible that human support, such as MI and coaching calls, may improve the efficacy of the intervention; however, this effect was not examined. Therefore, more research on the effect of human support variables on engagement and outcomes may help lead future researchers to properly utilize human support elements in online interventions.

The current study sought to understand the moderating relationship between human support on depression outcomes and engagement with CATCH-IT. This study evaluates how human support (e.g., number of contacts and motivational interviewing) impacts engagement with the intervention (e.g., modules completed, time on the site, characters typed), how user demographic characteristics impact engagement, how human support (e.g., number of contacts) impacts depression outcomes (e.g., CES-D), and if engagement is a significant predictor between intervention and depression outcomes.

2. Methods

The current study comprised a secondary analysis of data collected from a randomized controlled trial comparing two groups of adolescents: (1) adolescents assigned to a health education (HE) control condition; and (2) adolescents assigned to the CATCH-IT group (Gladstone et al. 2018). All analyses looked at the adolescents assigned only to the CATCH-IT group to determine how human support influenced the efficacy of the CATCH-IT intervention on depression scores and level of engagement, and if engagement impacted depression scores.

2.1. Participants

Participants were recruited between 2012 and 2016 during doctor appointments, via flyers, and through recruitment letters sent to patients (Gladstone et al. 2018). Risk was screened on the phone or in person. After obtaining parental consent, adolescents participated in an eligibility assessment over the phone. The parent and adolescents attended an enrollment assessment at their primary care provider’s office, where written informed consent and assent were obtained, and assessments were given to confirm study eligibility (Gladstone et al. 2018). Adolescents aged 13 to 18 years with elevated depressive symptoms on the CES-D10 scale at screening or at baseline, and/or a history of depression or dysthymia, were eligible to be included in the original study (Gladstone et al. 2018). Exclusion criteria included the following: current MDE diagnosis or treatment; past cognitive behavioral therapy; CES-D10 scores of more than 17; schizophrenia, psychosis, or bipolar disorder; serious medical condition (e.g., causing serious disability or dysfunction); significant reading impairment or developmental disability; imminent suicidal risk; and current drug or alcohol abuse. These criteria were selected to avoid confounding factors in depression etiology, consistent with using CATCH-IT as a preventative intervention (Gladstone et al. 2018). There were 369 participants aged 13 to 18 in the original study. Demographic characteristics are described in

Table 1. Participant demographics.

Demographic Category		n	%
Age	13	73	19.78%
	14	93	25.2%
	15	64	17.34%
	16	78	21.14%
	17	40	10.84%
	18	21	5.69%
Sex	Female	251	68%
	Male	118	32%
Ethnicity	Hispanic/Latinx	77	20.9%
	non-Hispanic/Latinx	286	77.5%
	Unknown/No response	6	1.6%
Race	Black/African American	99	26.8%
	American Indian/Alaska Native	3	0.8%
	Asian	14	3.8%
	Caucasian	201	54.5%
	Native Hawaiian/Pacific Islander	1	0.3%
	Multiracial	31	8.4%
	No response	20	5.4%

.

2.2. Materials

CATCH-IT is a primary care, public health, Internet-based depression prevention program made for adolescents (Van Voorhees et al. 2005, 2008). It was developed at the University of Illinois-Chicago by a team of psychologists, psychiatrists, primary care providers, and youth. The program comprises 15 self-guided online modules for adolescents that teach skills from evidenced-based, face-to-face interventions such as Cognitive Behavioral Therapy (CBT), Interpersonal Psychotherapy (IPT), and Behavioral Activation (BA) (Van Voorhees et al. 2005, 2008). CATCH-IT modules were designed to teach resiliency skills to at-risk adolescents through user-friendly, interactive web-based modules, and utilized motivational support by primary care professionals and an Internet-based parental behavioral change course (Van Voorhees et al. 2005, 2008). The skills targeted included: effective relationship skills, behavioral activation, and decreasing procrastination, rumination, and avoidance (Van Voorhees et al. 2008, 2009a, 2009b). The CATCH-IT program incorporates culturally relevant lessons, stories, and graphics to increase personal relevance and multi-modal learning opportunities (Gladstone et al. 2015). The intervention intends to reduce negative cognitions and unhealthy behaviors, and challenge interpersonal interactions that can be associated with increased risk of depression (Gladstone et al. 2015). Furthermore, it seeks to strengthen positive behaviors, increase effective thoughts, and encourage interpersonal relations, all of which are protective variables against depression (Gladstone et al. 2015). The CATCH-IT intervention includes the Internet component which is the primary focus of the current study, a brief motivational component (3 physician motivational interviews at 0, 2, and 12 months), and 1 to 3 staff coaching phone calls to facilitate website use.

2.3. Measures

Center for Epidemiological Studies Depression Scale (CES-D10, CES-D20)

The CES-D (Radloff 1991) is a 20 item, or shortened 10 item, assessment that looks at depression across multiple subscales (depressed mood, somatization, happiness, and interpersonal). The CES-D measures the frequency of 10 depressive symptoms during the previous week, using a 4 point scale. It was completed at baseline, 2, 6, and 12 months. The CES-D has been validated on male and female adolescents with depression with high sensitivity and specificity (Garrison et al. 1991), and has also been found to be reliable in high school and college age youth (α = 0.86 and α = 0.87, respectively; Radloff 1991). Please refer to

Table 2. Descriptive statistics—CESD10.

	Mean	Std. Deviation	Minimum	Maximum
Baseline	9.41	4.57	0	21
2 Months	8.50	4.81	0	24
6 Months	8.01	4.78	0	21
12 Months	7.48	5.04	0	25

and

Table 3. Descriptive statistics—CESD20.

	Mean	Std. Deviation	Minimum	Maximum
Baseline	16.94	8.72	0	45
2 Months	15.47	9.34	0	46
6 Months	13.89	9.06	0	42
12 Months	12.73	9.13	0	42

for descriptive information on the depression instruments.

2.4. Engagement

Level of engagement was measured by the following components: total amount of time spent on the site, total number of modules completed by participants, and total number of characters typed into the site. These components are consistent with prior CATCH-IT research on engagement, which included the same variables (Van Voorhees et al. 2009a, 2009b). For the current analysis, a unit-weighted composite score was created to assess the level of engagement. The components related to engagement (time on site, modules completed, and characters typed) were standardized and averaged to create an overall level of engagement variable. Furthermore, individual models were also separately run to assess how each component of engagement was impacted in significant models.

2.5. Human Support Factors

Human support factors were assessed using the completed number of contacts made and total number of motivational interviews for youth in the CATCH-IT arm. These two variables were used to determine if human support moderates the relationship between the CATCH-IT intervention and depression outcomes, and to evaluate whether the level of engagement with the online intervention increases as a result of contacts or motivational interviews.

2.6. Contacts

Completed contacts indicates the number of contacts that were made, at different time points from baseline to 12 months at the Chicago site only. The completed contacts variables include contact made with a participant, an email received, or a call from a participant from the Chicago site. The contacts log included many merged components, including scheduling calls, administering motivational interviews and KSADs, calls about incentives, and other clerical purposes. There were 1 to 3 staff coaching calls during weeks 1 through 3 to facilitate website use and troubleshoot technology support.

2.7. Motivational Interviews

All participants in the CATCH-IT group were intended to receive motivational interviews; however, not all participants received a motivational interview at the intended time points (0, 2, and 12 months). The motivational interviewing variable represents the total count of motivational interviews each participant received from an interviewer within the CATCH-IT group. In the adolescent motivational interview, which lasted 10 to 15 min, the primary care clinician sought to help the adolescent weigh the balance of positives and negatives of engaging with the online intervention. Participants came in for their motivational interviews, which were administered by the physicians. The motivational interviews were solely designed to encourage the completion of the intervention and ultimately promote behavior change (Gladstone et al. 2015).

2.8. Contacts and Motivational Interviews

Completed contacts included reminders, administering MIs and other measures, and technological troubleshooting, whereas MIs were administered by providers aiming to increase engagement with the intervention. Descriptive statistics for human support variables are presented in

Table 4. Descriptive statistics of contacts and count motivational interviews (MIs).

	Mean	Std. Deviation	Minimum	Maximum
Completed Contacts	10.35	5.213	0	30
Count MI	1.96	0.909	0	3

.

2.9. Statistical Analysis

All analyses were conducted in SAS version 9.3. To assess how the number of contacts impacted the relationship between intervention and depression outcomes, mixed effects modeling was utilized. Study hypotheses examining the effect of human support on engagement components were conducted using several general linear regression models. The effect of demographic factors on engagement with CATCH-IT was examined using an exploratory multiple linear regression analysis with four demographic variables that were dummy coded and used as predictors: adolescent age at baseline, sex (male or female), race, and ethnicity. Lastly, mixed effects modeling was used to examine if engagement is a significant predictor of the efficacy of the intervention on depression scores. Depression severity at baseline was controlled throughout each analysis.

3. Results

3.1. MI on Modules Completed, Minutes, and Characters Typed

A general linear regression model was used to evaluate if the total number of motivational interviews administered increased the number of modules completed, minutes on the site, and characters typed. Results indicated that the total number of motivational interviews significantly increased the number of modules completed, such that for each one unit increase in MI, participants completed 2.23 more modules (p < 0.0001). Furthermore, severity of baseline depression was significant, such that those with higher baseline depression completed 0.091 fewer modules (p < 0.0001). The second model demonstrated that the total number of motivational interviews significantly increased the number of minutes on the site, such that for each extra MI, participants spent 67.896 more minutes on the site (p < 0.0001). Furthermore, severity of baseline depression was significant, such that those with higher baseline depression spent −1.225 fewer minutes on the site (p = 0.0212). Lastly, the total number of motivational interviews also significantly increased the number of characters typed on the site, such that for each extra MI, participants typed 2054.201 more characters on the site (p < 0.0001). Severity of baseline depression was significant, such that those with higher baseline depression typed −57.446 fewer characters on the site (p = 0.0011). The interaction terms between visit and total number of motivational interviews were not significant in any of the three models (p > 0.05).

Table 5. MI on Engagement Variables.

Modules Completed
Parameter	Estimate	SE	t
Intercept	0.63	0.63	1.00
Time	0.00	0.08	0.00
CountMI	2.23	0.24	9.15 **
Baseline CES−D	−0.09	0.02	−5.22 **
Time CountMI	0.00	0.04	0.00
Minutes on Site
Parameter	Estimate	SE	t
Intercept	−10.15	19.77	−0.51
Time	0.00	2.42	0.00
CountMI	67.90	7.61	8.93 **
Baseline CES−D	−1.26	0.54	−2.31 *
Time CountMI	0.00	1.12	0.00
Characters Typed
Parameter	Estimate	SE	t
Intercept	69.61	635.59	0.11
Time	0.00	77.77	0.00
CountMI	2054.20	244.57	8.40 **
Baseline CES−D	−57.45	17.51	−3.28 **
Time CountMI	0.00	35.85	0.00

Notes: * p < 0.05; ** p < 0.01.

outlines how MI predicts engagement with the intervention.

3.2. Contacts on Modules Completed, Minutes, and Characters

A general linear regression model was used to evaluate if the total number of contacts increased the number of modules completed, minutes on the site, and characters typed. Results indicated that the total number of contacts did not significantly increase the number of modules completed. In contrast, the total number of contacts did significantly increase the total number of minutes on the site, such that for each extra contact, participants spent an additional 3.825 more minutes on the site (p < 0.0001). Lastly, the total number of contacts also significantly increased the characters typed on the site, such that for each extra contact, participants typed an additional 109.120 characters on the site (p = 0.0161). The interaction terms between visit and total number of contacts were not significant in any of the three models (p > 0.05).

Table 6. Contacts on engagement variables.

Modules Completed
Parameter	Estimate	SE	t
Intercept	3.47	0.67	5.14 **
Time	0.00	0.09	0.00
Contacts	0.08	0.05	1.55
Baseline CES−D	−0.02	0.02	−0.79
Time Contacts	0.00	0.01	0.00
Minutes on Site
Parameter	Estimate	SE	t
Intercept	11.32	11.57	0.98
Time	0.00	1.48	0.00
Contacts	3.82	0.88	4.34 **
Baseline CES−D	−0.07	0.35	−0.20
Time Contacts	0.00	0.13	0.00
Characters Typed
Parameter	Estimate	SE	t
Intercept	914.72	666.61	1.37
Time	0.00	83.60	0.00
Contacts	109.12	45.18	2.41 **
Baseline CES−D	−0.20	19.71	−0.01
Time Contacts	0.00	6.66	0.00

Notes: ** p < 0.01.

depicts how the number of contacts impact engagement with the intervention.

3.3. Demographics on Engagement

An exploratory multiple linear regression was conducted to analyze how levels of engagement change based on demographics. The engagement composite variable included minutes spent on the website, number of modules completed, and number of characters typed into response boxes, creating a unit-weighted composite variable, which was used as the outcome variable. The four demographic variables used as predictors were age at baseline, sex (male or female), race, and ethnicity (Hispanic/Latinx or non-Hispanic/Latinx), which were dummy coded. The reference group for sex was male, and the reference group for ethnicity was non-Hispanic/Latinx. White was the reference group for all racial category variables. The categories of Pacific Islander and American Indian were combined due to their small numbers of participants, which prevented estimates from being generated for these individual categories in the analyses. Age, Asian participants, and American Indian/Pacific Islander identifying participants were not significant predictors of engagement. Results suggest that males are less engaged than female participants. Regarding ethnicity, non-Hispanic/Latinx participants were more engaged with CATCH-IT than Hispanic-Latinx participants. With respect to race, African American and Black participants were significantly less engaged with CATCH-IT than White participants. Lastly, adolescents who self-identified as multiracial were significantly more engaged than White participants.

Table 7. Coefficients for demographics.

Model	β	Coefficients Standard Error	t
Age	−0.04	0.02	−1.02
Sex	−0.12	0.07	−3.31 **
Ethnicity	0.22	0.09	5.87 **
African American/Black	−0.19	0.09	−5.09 **
Asian	0.03	0.17	0.87
American Indian/Pacific Islander	−0.05	0.34	−1.31
Multiracial	0.08	0.14	2.21 **

Note: All variables are dummy coded with Caucasian used as the reference race for the model; Ethnicity refers to participants self-identifying as either Hispanic/Latinx or non-Hispanic/Latinx; ** p < 0.01.

reveals the relationship between demographic user characteristics and engagement with CATCH-IT.

3.4. Contacts on Depression (CESD)

Mixed effects modeling was used to assess if the number of contacts made moderated the efficacy of the intervention on participants’ depression scores over time (0, 6, and 12 months) when controlling for baseline depression. The model was not significant for scores on the CES-D10 (p = 0.77) or the CES-D20 (p = 0.74). Time did not have a significant effect on depression scores on the CES-D10 or the CES-D20. The interaction terms between time and number of contacts were not significant (p > 0.05). Severity of depression at baseline had a significant effect on depression scores, such that participants with a higher baseline depression score scored higher on the CES-D10 (0.36 point increase) and on the CES-D20 (0.89 point increase) over time.

Table 8. Contacts on depression.

Contacts on CES−D 10
Parameter	Estimate	SE	t
Intercept	3.22	0.51	6.31 **
Time	−0.01	0.10	0.89
Contacts	−0.01	0.03	−0.30
Baseline CES−D	0.36	0.02	19.16 **
Time Contacts	−0.01	0.01	−1.44
Contacts on CES−D 20
Parameter	Estimate	SE	t
Intercept	1.90	0.81	2.34 *
Time	−0.08	0.27	−0.30
Contacts	−0.02	0.05	−0.33
Baseline CES−D	0.89	0.03	28.95 **
Time Contacts	−0.02	0.02	−0.94

Notes: * p < 0.05; ** p < 0.01.

demonstrates the impact of contacts on depression outcomes.

3.5. Engagement on Depression

Mixed effects modeling was used to assess if the level of engagement moderated the efficacy of the intervention on participants’ depression scores over time (0, 6, and 12 months) when controlling for baseline depression. The model was not significant for scores on the CES-D10 (p = 0.92) or the CES-D20 (p = 0.88). Time had a significant effect on scores, such that over time participants had significantly lower scores on the CES-D10 (−0.10 decrease, p < 0.01) and the CES-D20 (−0.28 decrease, p < 0.01). The interaction terms between time and overall level of engagement were not significant (p > 0.05). Severity of depression at baseline had a significant effect on depression scores, such that participants with a higher baseline depression score scored higher on the CES-D10 (0.36 point increase, p < 0.001) and the CES-D20 (0.84 point increase, p < 0.001) over time.

Table 9. Engagement on depression.

Parameter	Estimate	SE	t
Intercept	2.71	0.60	4.51 **
Time	−0.28	0.10	−2.89 **
Engagement	−0.04	0.26	−0.15
Baseline CES-D	0.84	0.03	27.22 **
Time Engagement	0.01	0.08	0.18

Notes: ** p < 0.01.

shows whether engagement is a significant predictor of depression outcomes.

See Figure 1 for a summary of the overall outcomes and results from this study.

4. Discussion

BITs may be a feasible and scalable means to address depression in youth. However, there are some challenges associated with BITs related to attrition, engagement, and retention of users (Schueller et al. 2017). Research has found that integrating aspects of human support into BITs may lead to stronger treatment effects, relative to non-supported interventions (Andersson and Cuijpers 2009; Mohr et al. 2013; Schueller et al. 2017). The current study aimed to examine the impact of human support for CATCH-IT participants. Contrary to prior research, participants who received more human support did not demonstrate a decrease in depressive symptom scores over time.

The lack of a significant effect of human support could be explained by the lack of distinction within the human support contacts variable. More specifically, human support included different types of contacts (e.g., call for incentives, scheduling, conducting motivational interviews, reminders, encouraging use of the site, administering other assessment measures), and it was difficult to discern which type of contact was given to each user, and thus conclude which components of these contacts were most helpful for users. Human support has been shown to yield better treatment response and outcomes in adults (Mohr et al. 2013; Schueller et al. 2017), but may not be as efficacious or necessary for adolescents, specifically for contacts or reminders. This may be because adolescents are highly familiarized with digital platforms and interventions, and may not benefit from the same degree of human contact or technological troubleshooting as adults.

Interestingly, a general linear model indicated that participants who received more motivational interviews were more likely to engage with the intervention. This may indicate that the more motivational interviews a participant received in the CATCH-IT condition, the more likely they were to engage with the intervention as measured by number of modules completed, number of minutes on the site, and total characters typed. This is consistent with previous literature that demonstrates that motivational interviewing yields more positive outcomes and increases engagement with online interventions (Hoek et al. 2011; Saulsberry et al. 2013b; Van Voorhees et al. 2009a, 2009b). This is important because when adolescents engage more heavily with an intervention, they have better and longer-lasting treatment gains in studies with follow-up data (Dickter et al. 2019; Van Voorhees et al. 2009a, 2009b). However, in this study, results did not show that users who engaged more had more positive treatment outcomes. It is also possible that motivational interviewing was more effective in increasing all aspects of engagement (e.g., module completion, time on the site, characters typed) compared to contacts on engagement because it increased intrinsic motivation to fully engage with all components of the intervention. MIs demonstrated an effect on behavior change, but to show a significant treatment effect, users may need a higher dose.

When examining contacts, a general linear model showed that participants who received more contacts were more likely to engage with the intervention. Specifically, results indicated that the total number of contacts did not significantly increase the number of modules completed. However, the total number of contacts significantly increased the total number of minutes on the site and characters typed on the site. Although contacts partially increased engagement with the intervention, it did not significantly facilitate more module completion. It is possible that more regular follow-up and interaction with participants may remind or motivate users to generally engage with the intervention, but may not promote meaningful intervention engagement. However, research has shown that the rate of response to follow-up when using phone calls is low and raises concern about the utility of adding phone calls (Muñoz et al. 2017). It is possible that completed contacts facilitated mild engagement, but not a strong enough change to see an effect on depression. Furthermore, it may also be likely that these contacts were seen as more of a nuisance, especially when compared to other types of reminders such as push notifications.

In contrast, the mixed effects model demonstrated that participants who engaged more with the intervention did not report a decrease on the CES-D over time. A Chinese language version of CATCH-IT demonstrated a dose response curve between module completion and changes in internalizing symptoms across time (Ip et al. 2016; Patel et al. 2016; Sobowale et al. 2013). Although it is not clear why the findings from these two CATCH-IT studies diverge, it is possible that the Chinese study had fewer confounding variables which warrants further research on the relationship between engagement and depression outcomes.

The finding of this study that higher levels of engagement did not improve depression outcomes may be understood through a number of factors. It is possible that the quality and fidelity of the motivational interviews may mediate the connection between engagement and outcome. Furthermore, it is also likely that some modules were more related to positive outcomes than others and, although MI increases engagement, this engagement may have not been with the modules that are more effective in yielding stronger treatment outcomes. As previously discussed, it is especially hard to demonstrate large treatment effects in youth and may be even more challenging to do so with depression prevention programs (McCarty and Weisz 2007). An earlier study of CATCH-IT demonstrated a borderline finding that adolescent ratings of module experience and training were borderline predictors of depressed mood changes across time (Hoek et al. 2011). Taken together, there may be other moderators that improve the relationship between engagement and depression outcomes. It is possible that higher doses of the intervention, or more effective utilization of human support, could yield better treatment outcomes. Perhaps human engagement (e.g., “motivational framework”), combined with certain experiential aspects of online learning (e.g., “effective components” and “structured implementation strategy”) may, somehow in combination, drive clinical improvement. These may represent complex interactions that we simply cannot measure with relatively small samples (Abuwalla et al. 2018; Van Voorhees et al. 2011). When considering the quality of engagement, it is likely that, although there were higher quantities of engagement with human support, the quality of engagement was not meaningful enough to produce treatment effects.

Lastly, the regression model revealed that user demographic variables influenced the overall level of engagement. Specifically, results indicate that males (32% of users), Hispanic/Latinx (20.9% of users), African American and Black (26.8% of users), and non-multiracial users (86.2% of users) were significantly less engaged than other users. This is important information given that nearly half of CATCH-IT users identified as non-White, and may provide context around engagement and treatment effect. Regarding gender, it is possible that it is more socially acceptable for females to verbally express their emotions, and it may be more stigmatizing for males to express feelings of depression (Oliffe et al. 2016). When examining engagement levels by ethnicity and race, these findings may highlight the disparities that minoritized individuals face financially and socially. Some of the communities mentioned may be generally more distrustful of providers and of seeking mental health services (Shoff and Yang 2012; Williams 2018). These specific groups may have more barriers to care and less experience with mental health interventions, and generally tend to have higher rates of dropout in face-to-face therapy (de Haan et al. 2018; Leong and Kalibatseva 2011; Meyer and Zane 2013). Similarly, it proved difficult to recruit and engage youth with an African American–Latinx adaptation of CATCH-IT, Chicago Urban Resiliency Building (CURB), despite cultural adaptation (Bansa et al. 2018; Saulsberry et al. 2013a). Although efforts were made to include cultural representation in the content, it is also possible that more efforts may be warranted. A share of 8.4% of participants who identify as multiracial were more engaged than White participants. This may be attributed to the different cultural examples included in CATCH-IT, which may not have felt relevant to specific groups, but were more generally appropriate for those identifying as multiracial. Furthermore, users who identified as multiracial comprised other races and ethnicities that perhaps yielded higher engagement scores than African American and Black users and Hispanic/Latinx participants. Some of these cultural groups may prefer more human interaction-based interventions (e.g., valuing or favoring interpersonal relationships) and may warrant more human support to prime users for the online intervention. In CURB, there was some suggestion that clinics that were more engaged with the intervention, or more cohesive, were more successful in engaging youth with the intervention (Bansa et al. 2018; Saulsberry et al. 2013a). Given the small sample size in this study, and the varying demographic backgrounds of users, more culturally applicable studies are needed to examine engagement, treatment, and demographics when using BITs.

Limitations and Future Directions

The present study has several limitations. Firstly, the coded variable for completed contacts consisted of many different contacts for incentives, reminders, motivational interviews, and the administration of other baseline measures. Thus, it is hard to identify the components of the contacts variable that account for the increases in engagement with the intervention. Furthermore, the completed contacts variable was only coded for the Chicago site, and not for Boston participants, further limiting the generalizability of these findings. In the future, documenting contact efforts, including the reason for the call, and coding these factors separately within the dataset, may provide more clarity about whether the contacts are effective. In turn, stronger conclusions and findings could be made about the aspects of the contacts variable that are most useful in yielding positive treatment outcomes or engagement. Secondly, prior research studies concluded that CATCH-IT may be useful for adolescents with subsyndromal depression, and found group differences for adolescents with high symptoms and depressive episodes. However, because there was not a significant effect between intervention and depression outcomes in the current analysis, it may be useful to use depressive episodes as an outcome variable in future analyses to look beyond symptomology and examine diagnoses.

Future CATCH-IT studies may consider dividing the construct of human support into two components: clerical human support such as contacts, and therapeutic human support including motivational interviews. Clerical human support would focus on technological troubleshooting and administering tests, whereas therapeutic human support would aim to increase engagement with the intervention. If clerical human support is not related to treatment outcome, as shown in this study, research efforts could focus more on therapeutic human support, such as MI, to increase engagement with the aim of increasing therapeutic outcomes.

It may prove useful to redesign the content of CATCH-IT, which could be especially helpful when considering the demographic groups that engaged less with the intervention, to target and tailor content to less-adherent users. As many users only engaged with the first few modules, targeting improvement in the first few modules may increase chances of participants returning for the following modules. Consideration should be given to the research of this topic because it is unclear if lower levels of engagement were a result of the content or other extraneous factors.

5. Conclusions

Given the increase in utilizing online platforms and BITs to provide treatment and prevention efforts, especially during the COVID-19 global pandemic, future CATCH-IT studies would need to continue to update the design of the platform to increase engagement of ethnic minority participants that tend to engage less with the site. This may include making a mobile-friendly version of the intervention, which utilizes push-notifications, reminders, automated text messages to users, or interactive chatbots. Furthermore, researchers could weigh the benefit of utilizing live chat or a chatbot throughout the intervention. A future trial of CATCH-IT may benefit from relying less on human contacts as reminders, and targeting their efforts to mediate intrinsic motivation to improve engagement and treatment gains from CATCH-IT. To increase the engagement of adolescents with an online depression prevention program, human support may be more efficient when utilizing MI rather than technical support.