Next Article in Journal / Special Issue
Adaptation of an Evidence-Based Online Depression Prevention Intervention for College Students: Intervention Development and Pilot Study Results
Previous Article in Journal
Is There a Real Need for the Preparatory Years in Higher Education? An Educational Data Analysis for College and Future Career Readiness
Previous Article in Special Issue
Randomized Clinical Trial of Primary Care Based Online Depression Prevention Intervention: Impact on Adolescent Modifiable Factors and Behaviors
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Brief Report

Adolescent Perspectives on How an Adjunctive Mobile App for Social Anxiety Treatment Impacts Treatment Engagement in Telehealth Group Therapy

Department of Psychology, University of Pittsburgh, Pittsburgh, PA 15213, USA
Department of Psychology, University of South Carolina, Columbia, SC 29208, USA
Department of Health Information Management, University of Pittsburgh, Pittsburgh, PA 15213, USA
Author to whom correspondence should be addressed.
Soc. Sci. 2021, 10(10), 397;
Submission received: 31 August 2021 / Revised: 11 October 2021 / Accepted: 12 October 2021 / Published: 16 October 2021
(This article belongs to the Special Issue Technological Approaches for the Treatment of Mental Health in Youth)


Adjunctive mobile mental health apps to supplement mental health treatment have been growing in recent years given their ability to address treatment engagement barriers. However, few studies have explicitly examined how these mobile apps impact treatment engagement, and even fewer have investigated this topic through adolescents’ perspectives. To this end, we conducted semi-structured interviews with five adolescents who used an adjunctive mobile mental health app in combination with telehealth cognitive behavioral group therapy for social anxiety. Using a multidimensional framework of treatment engagement, we elicited their perspectives on how the app impacted their engagement in telehealth group therapy and gathered their suggestions for improving the app. Using a consensual qualitative research approach, we learned that adolescents felt the app increased their comfort with others in therapy and their expectations about the effectiveness of mental health apps. They also indicated that the app prepared them for in-session participation and facilitated out-of-session skills practice. Adolescents had valuable suggestions such as adding app features to facilitate social connectedness between group members and adding appointment reminders in the app. This preliminary study highlights implications for future adjunctive mobile mental health app developers and researchers to increase adolescents’ treatment engagement in mental health services.

1. Introduction

Mobile mental health apps have great potential to supplement traditional adolescent mental health services and facilitate treatment engagement by addressing common barriers to treatment engagement (Silk et al. 2020). Some of these barriers include low application of therapeutic skills outside of session or poor knowledge of treatment concepts (Becker et al. 2018). For example, mobile mental health apps can utilize push-notifications to encourage adolescents to practice therapeutic skills throughout the day, and gamification features can make practicing therapeutic skills more exciting to adolescents compared to traditional paper-pencil approaches (Silk et al. 2020). Furthermore, adjunctive mobile mental health apps can be especially useful in keeping clients engaged in telehealth therapy, in which therapy is conducted through video conferencing, since previous research suggests that telehealth therapy can hinder therapeutic alliance (Farabee et al. 2016). Allowing an additional form of interaction with therapists through an app may improve therapeutic alliance by increasing clients’ perception of therapist availability (Lopez et al. 2019). Given that higher treatment engagement is associated with better treatment outcomes (Clarke et al. 2015; Danko et al. 2016), understanding how mobile mental health apps can be leveraged to support treatment engagement may improve adolescent mental health treatment outcomes as well.
Despite the growing number of adjunctive mobile mental health apps (Bolinski et al. 2018; Broglia et al. 2019; Neal-Barnett et al. 2019; Newton et al. 2020; Pramana et al. 2018; Whiteside et al. 2019), only a few studies have investigated how these apps may impact treatment engagement. Moreover, preliminary findings on mobile apps’ influence on treatment engagement have been mixed. For example, Paquette (2020) found that a mobile app for completing therapeutic homework had no impact on attendance, knowledge comprehension, or working alliance with adult clients. Interestingly, adult clients who used the mobile app had lower homework completion compared to those who did not have access to the app as part of therapy. In contrast, a study by Patterson Silver Wolf et al. (2020) found that adolescent clients who used a mobile app designed to provide psychoeducation and support skill acquisition, most had the highest attendance, were the most engaged, and were the most likely to successfully complete treatment. Furthermore, research on engagement in digital interventions and treatment outcome have been mixed, with some studies suggesting a positive association between engagement in the digital intervention and treatment outcome (Ip et al. 2016; Wu et al. 2021), while others did not (Rogers et al. 2021). These mixed findings in the literature illustrate the need to investigate the potential for adjunctive mobile mental health apps to impact treatment engagement more closely. Furthermore, previous studies investigating mobile mental health apps’ impact on treatment engagement collected information from providers rather than the clients. Eliciting feedback from clients themselves may reveal unique experiences and barriers that may be overlooked by providers.
There has been a shift in the field in recent years towards measuring treatment engagement as a multidimensional construct, rather than treating treatment engagement synonymously with treatment attendance or app use. Relatedly, Becker and colleagues (2018) proposed the REACH conceptual framework, which describes five domains of treatment engagement: Relationship (e.g., therapeutic alliance), Expectancy (e.g., beliefs about treatment efficacy), Attendance (e.g., presence at therapy sessions), Clarity (e.g., understanding of treatment), and Homework (e.g., in- and out-of-session participation). The REACH framework covers broad dimensions of engagement that span across in person and telehealth services. Using a multidimensional framework to measure treatment engagement allows for a comprehensive examination of how an adjunctive mobile app might impact treatment engagement. Furthermore, designing adjunctive mobile apps that target engagement barriers in various domains may lead to greater improvements in treatment engagement and consequently treatment outcomes. We briefly describe one adjunctive mobile mental health app that was designed to target homework and supplement cognitive behavioral therapy (CBT) for anxiety.

1.1. SmartCAT

SmartCAT (Smartphone-Enhanced Child Anxiety Treatment; Pramana et al. 2018) is a mobile health platform for child and adolescent anxiety treatment that consists of a gamified client app with in-game activities and rewards, as well as an integrated clinician portal connected to the app. The SmartCAT app is designed as an adjunctive resource to supplement CBT for anxiety. The app was designed specifically to target clients’ learning and application of therapeutic skills, since previous studies have found that youth report low interest in CBT homework, which results in lower homework completion (Bunnell et al. 2020). The SmartCAT app includes four interactive mini games (What’s the Feeling, Problem Solver, Thought Buster, and Thought Swapper) that support acquisition of CBT skills such as emotion identification, problem solving, and thought challenging. The app also includes three in-depth modules: (1) Skills Coach, which guides clients through coping and problem-solving steps, (2) Challenger, which allows clients to complete “challenges” or home-based exposures assigned by a clinician, and (3) Chillax, which provides guided instructions for progressive muscle relaxation and deep breathing. Additionally, the SmartCAT app utilizes daily smartphone push notifications to remind clients to use the app. A key feature of the SmartCAT app is its connection to a clinician portal. This secure web-based portal allows clinicians to engage with clients by viewing their progress in completing app activities and assigning clients activities to complete for the following week. A full description of SmartCAT can be found in Pramana et al. (2018).
Although SmartCAT has demonstrated strong feasibility, usability, acceptability, and preliminary efficacy in reducing anxiety symptoms in a pilot trial (Silk et al. 2020), there remain opportunities to fully explore its impact on adolescent treatment engagement, including in the context of telehealth group therapy. Additionally, gathering adolescents’ perspectives of the adjunctive mobile mental health app makes a unique contribution to the literature, as there are few studies that consider adolescents experiences with evidence-informed mobile mental health apps. Focusing on the adolescent or user perspective provides mental health app developers the most proximal information on how apps can be improved.

1.2. Current Study

This qualitative study investigated adolescents’ perceptions of how the SmartCAT app impacted their treatment engagement in telehealth cognitive behavioral group therapy for social anxiety. This study was part of an ongoing Quality Improvement (QI) project at an adolescent outpatient specialty clinic located at the University of Pittsburgh Medical Center (UPMC). The QI project’s goals were to implement SmartCAT into the center to improve treatment outcomes for adolescents receiving mental health services through video conferencing. The main aims of this study were to use a multidimensional framework to: (1) elicit adolescents’ perspectives on how SmartCAT impacted treatment engagement and (2) gather adolescents’ suggestions on how to improve SmartCAT to enhance its impact on treatment engagement.

2. Materials and Method

2.1. Participants

Participants were recruited from a social anxiety therapy group conducted in the spring of 2021 at an adolescent outpatient specialty clinic. All adolescents were receiving individual therapy for depression or anxiety in addition to group therapy. Out of the six adolescents who attended the group, five (83.3%) agreed to participate in the qualitative interview, and one (16.7%) declined given time conflicts. Participants were between the ages of 14 and 17 (M = 15.40; SD = 1.34), and all identified as White. Two participants (40%) identified as male, and three participants (60%) identified as female.

2.2. Intervention

Adolescents received the Stand Up, Speak Out program, a 10-session CBT group treatment for social anxiety (Albano and DiBartolo 2007). Sessions were held weekly for one hour and conducted through Microsoft Teams Video Conferencing Platform due to social distancing measures as a result of the COVID-19 pandemic. The group was led by five group facilitators and attended by six adolescents. At the start of the program, adolescents were introduced to SmartCAT as a supplemental app to group therapy. Of note, SmartCAT was designed to supplement any child and adolescent anxiety CBT manual and thus is not specific to the Stand Up, Speak Out program. Adolescents were instructed to download the free app onto their personal Android or iOS smartphone. SmartCAT is currently not publicly available for download.
Each group therapy session started with group members voluntarily sharing a success from the past week, such as if they completed an assigned home-based exposure. The remainder of the session was devoted to covering material from the Stand Up, Speak Out program. After each session, the primary group facilitator assigned adolescents homework activities in the app, such as completing mini games or modules, through the SmartCAT clinician app portal. Adolescents were instructed to complete the homework before the next session. The primary group facilitator then reviewed adolescents’ progress in completing assigned activities prior to the next session.

2.3. Procedure

All procedures were approved by the Qualitative Review Committee at UPMC. Adolescent assent and parent consent were digitally obtained through REDCap prior to data collection. The first author conducted all interviews via Microsoft Teams using a semi-structured interview script (see Appendix A). The script was developed by the first and second authors and was organized by the REACH domains of treatment engagement (Becker et al. 2018). The script was designed to elicit adolescents’ perspectives on how using the SmartCAT app impacted their engagement in group therapy, and how the app could be improved to increase treatment engagement. Adolescents did not receive monetary compensation for their participation. Interviews were audio-recorded, and later transcribed and checked by the third author and a research assistant.

2.4. Coding

To guide our qualitative coding process, we utilized a combination of inductive or “bottom-up” and deductive or “top-down” methods (Elo and Kyngäs 2008) to highlight adolescents’ perspectives while integrating existing conceptual theories or frameworks into our coding scheme. These methods have been used extensively in a broad range of disciplines, including social sciences (Saldaña 2013). All coding was conducted in Microsoft Word and Microsoft Excel.
Two coders, the first (post-baccalaureate student) and second (doctoral student) author, developed a qualitative codebook to support the coding process. Using an open coding method (Strauss and Corbin 1997), the two coders carefully read all transcripts and independently identified themes within the text of the transcripts until saturation was met, in which there were no new themes. Coders then met to agree upon an initial set of codes from these themes. Afterwards, in alignment with axial coding, the coders established connections between the codes. At this step, coders used a deductive approach and drew connections between codes and larger categories that reflected the REACH conceptual framework of treatment engagement. A preliminary codebook that consisted of codes and categories was developed and then tested with one transcript after this process of open and axial coding.
Later, a third coder (third author; post-baccalaureate student) was trained to use the codebook. Each transcript was coded by two coders, and the first coder of each transcript was responsible for segmenting the transcript into excerpts. Coders were instructed to apply only one code to each excerpt. After all transcripts were double-coded, a consensual qualitative research approach (Hill et al. 2005) was used to guide the next phase. All three coders met to discuss and resolve any coding discrepancies until consensus on a final code was reached. Some of these discussions led to changes to the definitions of codes or their relations to categories, which aligns with selective coding. This coding process allowed for an iterative and thorough examination of the concepts from the interview transcripts.

3. Results

Transcripts were segmented into 106 excerpts. The codebook consisted of 25 codes, which were organized into five categories according to the REACH domains of treatment engagement. Interrater reliability, as measured by Cohen’s kappa (Landis and Koch 1977), was high across all codes, ranging from substantial agreement to perfect agreement, (range = 0.65–1.00; pooled average = 0.93). Table 1 displays the frequency of codes and their respective kappa scores.

3.1. Relationship

Adolescents described how the app content and features helped them feel more comfortable sharing in group therapy. Specifically, adolescents expressed that developing a coping plan for an anxiety-provoking situation in the Skills Coach module helped them feel more comfortable discussing the same situation with others in session. One adolescent appreciated the feature in Skills Coach that allows clients to indicate if a situation could be discussed in group, rather than having to “start conversation in group about [the situation]”. Additionally, one adolescent expressed that they felt comforted knowing that other group members were using the app as well:
It gave something else to relate to [the other group members] … It was just nice to know “Oh, I’m not the only person who is doing all these”. … It was more comforting ‘cause although it’s something you haven’t done before, you’re not the only person doing it.
Adolescents also described how the app impacted their relationship with group facilitators by creating positive, bi-directional interactions. Specifically, one adolescent appreciated that they “weren’t just yelling at a brick wall” in reference to the primary group facilitator’s ability to view their progress through the clinician app portal. Adolescents also shared that completing app achievements allowed for opportunities to receive praise from group facilitators. Moreover, adolescents liked that the app enabled the primary group facilitator to individualize their assigned home-based exposures.
Adolescents had a variety of different suggestions on how SmartCAT could be improved to enhance clients’ relationship with group members and facilitators. One adolescent suggested to add more gamification features such as a scoreboard for games to bolster friendly competition between group members. Furthermore, another adolescent suggested to include a feature in the app that would allow for group members to interact and provide feedback to one another. Lastly, one adolescent suggested to have the SmartCAT app be more integrated with group therapy so it could lead to more “in-depth questions” about the app during group.

3.2. Expectancy

All adolescents reported having little to no prior experiences with mobile mental health apps, though some had previously participated in group therapy. Some adolescents felt nervous about starting group therapy or thought that group therapy would just be “a bunch of people sitting around, not doing anything”. One adolescent believed group therapy would not be helpful to them, stating that their “anxiety would kinda stay the same through [therapy]”. Similarly, some adolescents also had low expectations for how helpful the SmartCAT app would be for addressing their social anxiety: “At first I thought it was kinda stupid just how like it’s an app, like what is it gonna do”?
Despite having hesitancies, almost all adolescents reported positive shifts in their beliefs surrounding group therapy or adjunctive apps as a result of using SmartCAT. Adolescents shared how their positive experiences using the deep breathing module from the app led them to have more positive expectations of the effectiveness of group therapy. One adolescent described how the app exceeded their expectation of an adjunctive app since SmartCAT both built on content discussed in group therapy and offered new content, so that they never felt behind in therapy:
[SmartCAT] was a little more than just a companion app because it’s almost a standalone thing… They compound on each other… but they would be different from what we were doing in group therapy, so it was all—it was fresh… It was really easy to stay up with what was going on in the group even if you missed bits.
A common suggestion to improve SmartCAT to increase adolescents’ positive expectations of mobile mental health apps was to add a feature to allow adolescents to “get to know” or “interact with” other group members before and during group therapy. Several adolescents also suggested including more detailed descriptions about SmartCAT in the app store and enabling a preview of app features so adolescents can “know what to expect”.

3.3. Attendance

Adolescents primarily reported that the SmartCAT app did not impact their behavioral attendance to group therapy. Of note, the group was conducted virtually due to the COVID-19 pandemic, so adolescents could access sessions anywhere with a stable internet connection. Instead, adolescents shared that the app increased their motivation to attend group therapy. For example, one adolescent said that seeing the app on their phone reminded them of the other group members, which made them more motivated to attend sessions. Similarly, some adolescents felt “excited to go to group” so they could share their successes in completing challenges in the app with group members and receive praise from the group facilitators.
It affected [attendance] in a good way because whenever I would think about the app, I would think of group therapy… If I did something with the app, then I would be like “Ah, this I can share…” It motivated me a lot to attend.
Most adolescents suggested adding appointment reminders for group sessions in the app. Though unrelated to the app, a few adolescents shared that barriers such as an unreliable internet connection or access to a private, confidential location interfered with their ability to attend sessions.

3.4. Clarity

Adolescents stated that the app helped them understand the roles in group therapy. For example, they felt that using the app reinforced the idea that other group members in therapy were working with them through similar difficulties: “Everybody in the group is going through the same things, so they’re there going with you to like help get better at skills to get over it”. Additionally, some adolescents expressed that the app cemented the idea that the purpose of group therapy was “to learn how to relax and calm down when our anxiety is bad”.
Furthermore, adolescents reported that activities in the SmartCAT app taught them how social anxiety can manifest in themselves as well as others. Adolescents particularly liked the coping and problem-solving module, Skills Coach, because it helped them “understand how [they] were feeling,” and “untangle what happened in those [anxiety-provoking] situations”. Additionally, some adolescents felt that SmartCAT had “widened [their] view of social anxiety,” showing them different symptoms associated with anxiety and coping thoughts that others use when feeling anxious.
Adolescents had several different suggestions on how the app could be improved to enhance clarity about therapy or social anxiety. For example, some adolescents suggested to include informational pages in the app about group therapy and social anxiety. One adolescent specifically wanted to have more examples of coping thoughts to help them better respond to their social anxiety.

3.5. Homework

Almost all adolescents reported that the app positively impacted their in-session participation during group therapy. Not only did adolescents mention having increased motivation to share accomplishments with challenges or activities during group, but adolescents also reported feeling more prepared to speak during session. Specifically, adolescents stated the app provided opportunities to process, understand, or rehearse their thoughts so that they could “talk about it more accurately” and “convey it… clearly” in session. Another adolescent shared how using strategies they learned from the deep breathing module during sessions helped them “be prepared for what [they were] going to say” during session.
Several adolescents recounted specific instances of how the SmartCAT app impacted their out-of-session participation, namely in applying coping skills into their daily lives. For example, one adolescent shared a story of when they were able to use the deep breathing module from the app before an anxiety-provoking event:
I had like a band practice, first one, and I was very anxious for that. I did the deep breathing [module] and it calmed [me] down a lot. So, before I go, I normally do some deep breathing. I calm down a lot more, and then I go, and I have fun.
One adolescent noted that SmartCAT reinforced their knowledge of skills taught in group, stating that “SmartCAT really hit the nail on it afterwards”. Additionally, adolescents indicated that the accessibility and convenience of the mobile app made it easy to practice skills, as SmartCAT was “a thing that we always have in our pockets” and was “on a device that you use all the time”. However, there were mixed reactions to the built-in reminders to use the app. One adolescent shared that the reminders induced feelings of guilt for not using the app, while another found the reminders helpful to practice their therapy skills throughout the day.
Adolescents had several suggestions on how SmartCAT could be adapted to improve in- and out-of-session participation. One adolescent suggested adding an additional module that reviewed content discussed in the prior therapy session and content to be discussed in the next session, allowing adolescents to easily access information and to prepare for upcoming sessions:
I think it would allow them to recall a whole lot easier so they can keep it in their mind like when they practice. And when they get a preview… they can just prepare for it… Since they’re more prepared, they might be more willing to participate.
Another adolescent disclosed wanting more content that “wasn’t so surface-level”, such that more specific content would increase their understanding and willingness to participate. Some adolescents wanted more gamification features or “fun” designs” to make them more likely to use the app to practice skills. For example, one adolescent suggested adding features from one of their favorite mobile games including “competitions, missions, color… [and] setting new records”.

4. Discussion

This study examined adolescents’ perceptions of how an adjunctive mobile mental health app impacted their treatment engagement in telehealth group therapy. We interviewed five adolescents who used the SmartCAT app in conjunction with group therapy treatment for social anxiety. Due to the COVID-19 pandemic, therapy was conducted virtually through video conferencing. Through a qualitative approach, we coded qualitative interviews that investigated adolescents’ perspectives using a multidimensional framework of treatment engagement. Moreover, we elicited suggestions from adolescents who had first-hand experience using the app to guide future changes to the app. Although the SmartCAT app was developed to primarily target out-of-session skills practice, these qualitative interviews illuminated that adjunctive mobile apps can actually impact multiple domains of treatment engagement. We highlight the main findings and discuss the future implications below.
The SmartCAT app supported adolescents’ relationship with those in group by creating opportunities to receive praise from group facilitators for completing assigned app activities and a shared experience of using the app among group members. Still, adolescents wanted features to form stronger social connections with group members, such as ways of interacting with other group members through a chat function or scoreboards to celebrate others’ successes. Adding networking and communication features, similar to those of internet-based games, to mobile mental health apps can help enhance feelings of connectedness between group members by providing opportunities to interact with one another outside of sessions. Adolescents’ feedback demonstrates that adjunctive mobile apps can be a useful tool for forging social connections in therapy, which is particularly beneficial in the context of group therapy when relationships with other group members can facilitate other areas of engagement. Moreover, strengthening connectedness may be especially beneficial in virtual therapy, when clients’ inability to be physically present with their therapist or other group members can be a barrier to engagement in itself (Farabee et al. 2016). Thus, future app developers may consider adding features that allow app users to communicate with others while protecting confidentiality.
SmartCAT seemed to have a greater impact on adolescents’ motivation to attend group therapy sessions compared to their actual attendance to sessions. However, it is worth noting that adolescents attended group therapy virtually. Thus, it is still unknown how SmartCAT or adjunctive mobile mental health apps may affect attendance to in-person group therapy. Nevertheless, addressing attendance is important for ensuring optimal treatment outcomes. Given that appointment reminders are highly effective in increasing attendance (Branson et al. 2013; Oruche et al. 2014), developers should incorporate adolescents’ suggestion of including appointment reminders in the app as a simple way to improve attendance to sessions.
Adolescents had initial hesitancies about the effectiveness of SmartCAT in reducing their anxiety. Strategies that increase expectations about new technologies include promoting trust in the new technology (Thatcher et al. 2007). For instance, when introducing a new therapy app, therapists should provide a thorough explanation of the app and answer clients’ questions to reduce hesitancies surrounding the new app (Philippi et al. 2021). Despite these hesitancies, adolescents had positive shifts in belief surrounding adjunctive mobile apps or group therapy. Moreover, SmartCAT provided adolescents with psychoeducation about mental health services and social anxiety, with some adolescents desiring more psychoeducation through the SmartCAT app. In conjunction, these findings are promising as they highlight that SmartCAT can not only foster positive cognitive shifts in expectancies but can also efficiently disseminate information to adolescents. Though the number of evidence-based mobile mental health apps is currently small (Radovic et al. 2016; Torous et al. 2017), researchers looking to develop evidence-based adjunctive mobile mental health apps in the future should consider that adolescents are not only responsive to, but desire evidence-based tools and information through mobile apps.
As expected, many of the adolescents reported that SmartCAT was helpful for practicing skills out of session. Adolescents mainly enjoyed using features of the app that guided them in creating a coping plan or practicing deep breathing strategies. Less expected was the app’s impact on adolescents’ in-session participation by helping them prepare for group discussion beforehand. Additionally, adolescents were motivated to share app accomplishments during session to receive praise from group facilitators. As such, we recommend future adjunctive mobile mental health app developers to consider how their app might include content that prepares and motivates users for in- and out-of-session activities.

Strengths and Limitations

This study is among the first qualitative studies to examine the impacts of adjunctive mobile mental health apps on treatment engagement in telehealth group therapy from adolescents’ perspectives. One limitation of the current study is that engagement was assessed retrospectively after participants completed treatment. Future research should assess treatment engagement at different times throughout the course of treatment to capture real-time perceptions. Additionally, the current study did not examine the effects of adolescents’ comorbid depressive symptoms, which has previously been shown to be a predictor of treatment engagement in anxious youth (Gonzalez et al. 2011). Given high co-occurrence between anxiety and depression (Axelson and Birmaher 2001), future research should consider how the severity and complexity of presenting problems can impact treatment engagement. Furthermore, this study has contextual limitations in that it only investigated adolescents’ experiences with the app while enrolled in telehealth services, when the COVID-19 pandemic necessitated virtual mental health services. Telehealth and in person therapy can be different experiences for clients. Thus, these findings may be limited in their generalizability to treatment engagement outside of telehealth contexts. As both telehealth and adjunctive mobile apps become more popular, future research should consider how different therapeutic contexts can help or hinder engagement in mental health apps and therapy.
Although the sample size was small, the qualitative approach used provided a rich and in-depth examination of adolescents’ experiences that would not have been possible with only quantitative data. Still, findings from this study are preliminary, and more research needs to be conducted on the impact of adjunctive mobile mental health apps on treatment engagement. Moreover, since the first author was one of the group facilitators and had developed a prior working relationship with adolescents, this may have biased responses to the semi-structured interview questions. However, adolescents still expressed a variety of responses, and the familiarity with the interviewer may have made adolescents more willing to disclose their perspectives. Furthermore, since all adolescents were also receiving individual mental health services in addition to group therapy, the current sample may represent adolescents who had higher treatment engagement than adolescents who had no prior experiences in therapy. Lastly, though taking both a top-down and bottom-up approach may stray from a true grounded theory approach to qualitative analysis, we assert that this combined approach enabled us to gain the benefits of two methods, that is, centering our understanding to the adolescents’ perspectives while supplementing it with knowledge from the empirical literature.

5. Conclusions

Across adolescents’ responses, SmartCAT impacted not only one, but multiple areas of treatment engagement as assessed through the REACH framework (Relationship, Expectancy, Attendance, Clarity, Homework). It also became evident that an impact in one area of engagement subsequently led to an impact in another area of engagement. For instance, some adolescents shared that completing homework activities outside of session made them feel excited to attend group, or that receiving praise from group facilitators for completing activities in the app supported their relationship with group facilitators. Thus, future efforts should continue to investigate the impacts of mobile mental health apps on the multifaceted and interconnected nature of treatment engagement using a multidimensional framework. As app developers and researchers continue to develop more evidence-based mobile mental health apps, it is important to incorporate adolescents’ feedback and suggestions into the app design to increase the use and adoption of these apps in an effort to improve mental health outcomes for adolescents.

Author Contributions

Conceptualization, C.L., W.C., B.P., and J.S.S.; methodology, C.L., W.C., B.P., and J.S.S.; formal analysis, C.L., W.C., and S.M.; writing, C.L., W.C., S.M., B.P., and J.S.S. All authors have read and agreed to the published version of the manuscript.


This study was supported by a grant from the National Institute of Mental Health (NIMH), “The Center for Enhancing Triage and Utilization for Depression and Emergent Suicidality in Pediatric Primary Care” (ETUDES Center), P50MH115838, and the University of Pittsburgh Social Sciences Research Initiative and Pitt Seed Initiative funds.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Qualitative Review Committee at The University of Pittsburgh Medical Center (UPMC) (QI Project 2837 1 October 2020).

Informed Consent Statement

Informed adolescent assent and parent consent were obtained from all participants in this study.

Data Availability Statement

Data are contained within the article.


We acknowledge with gratitude David Brent’s Endowed Chair in Suicide Studies, as well as the Pennsylvania Legislature for their support of the STAR-Center. We are grateful to the adolescents who provided their perspectives. We thank the STAR-Center clinicians for their efforts in implementing SmartCAT into their work. Furthermore, the authors would like to thank the app development team, specifically Andi Saptono, Haomin Hu, Sichao Xue, Mengyuan Huang, Tianxun Hu, Gaoxiang Chen, and Yuhan Wang for their time and dedication in developing the latest version of the SmartCAT app. Finally, the authors would like to thank Margaret Lynch for her work in transcribing interview recordings.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

SmartCAT Engagement Interview Script
Introduction/Icebreaker: Thank you for agreeing to participate in this interview. Hi, my name is ______ and I work with Dr. Jennifer Silk at the University of Pittsburgh. The purpose of our conversation is to learn more about your experience using SmartCAT. We estimate that today’s interview will take about 30 minutes. We tried to make our questions as respectful as possible, but if at any time you feel uncomfortable answering any question, you can choose to skip the question.
As a reminder, I will be recording this interview on Microsoft Teams to make sure I capture all of your contributions. This recording will be kept on a secured portal and later transcribed by someone on our team. Any personal information that we talk about today will be taken out from the transcript. What we discuss today will not be shared with anyone else including your therapist, so we hope that you feel comfortable sharing your valuable perspectives and experiences.
Do you have any questions before we get started? Okay. I will go ahead and start recording.
First, I’d like to ask about your relationship with others in group therapy.
  • How did using SmartCAT affect your relationship with the group facilitators?
    Probe: Did using SmartCAT help/get in the way of your relationship (comfort sharing, working as a team) with the group facilitators? Did you have a good relationship with facilitators?
  • How did using SmartCAT affect your relationship with other group members?
    Probe: Did SmartCAT help/get in the way of your relationship with group members? Did you have a good relationship with group members?
  • What about SmartCAT can be changed to help people have better relationships with their group facilitators or other group members?
Now, I’d like to ask about your thoughts about group therapy and mobile mental health apps.
  • Before group therapy, what were your thoughts on using mobile apps as a part of group therapy?
    Probe: Did you expect mobile apps as a part of group therapy would be helpful for you/for people with social anxiety? What do you think of when you hear the words “mobile mental health apps”? What do you think of when you hear the words “group therapy”?
    Probe: What were your beliefs on how effective group therapy can be before using SmartCAT?
  • Now that group therapy is finished, has SmartCAT changed any of these thoughts or expectations?
    Probe: How has using SmartCAT changed your thoughts on how mobile apps can be used in group therapy/be used to reduce social anxiety?
    Probe: How has SmartCAT affected your beliefs on how effective group therapy can be? How helpful was SmartCAT for you?
  • What about SmartCAT can be changed to help people have more positive expectations about group therapy or mobile mental health apps?
Shifting gears again,
  • How did using SmartCAT affect your attendance to group therapy sessions?
    Probe: Did SmartCAT make it more/less convenient to attend sessions, easier/harder to remember to attend group therapy?
  • How did using SmartCAT affect your motivation to attend group therapy sessions?
    Probe: Did SmartCAT make you more/less motivated to attend therapy sessions?
  • What about SmartCAT can be changed to help people attend group therapy sessions?
Now, I’d like to ask about how SmartCAT impacted your understanding of therapy.
  • How did using SmartCAT affect your understanding of group therapy?
    Probe: Did using SmartCAT make your understanding of your or others role in treatment/what happens in and out of sessions/your goals for therapy more/less clear?
  • How did using SmartCAT affect your understanding of what social anxiety is?
    Probe: Did using SmartCAT increase your understanding of social anxiety symptoms/treatment?
  • What about SmartCAT can be changed to help people have a better understanding of group therapy or social anxiety?
Now, I’d like to ask about how using SmartCAT impacted your participation in and out of session.
  • How did using SmartCAT affect your participation during group therapy sessions?
    Probe: Did SmartCAT make it easier/harder to participate, help you feel prepared to discuss or practice skills in group sessions?
  • How did SmartCAT affect how you learned and applied skills outside of group therapy sessions?
    Probe: Did using SmartCAT make you more/less likely to use skills outside of group therapy sessions?
  • What about SmartCAT can be changed to help people participate during group therapy sessions or practice skills outside of session?
Any other questions or comments?
I’m going to stop the recording now.


  1. Albano, Anne Marie, and Patricia Marten DiBartolo. 2007. Cognitive-Behavioral Therapy for Social Phobia in Adolescents: Stand Up, Speak Out: Therapist Guide. Oxford: Oxford University Press. [Google Scholar]
  2. Axelson, David A., and Boris Birmaher. 2001. Relation between Anxiety and Depressive Disorders in Childhood and Adolescence. Depression and Anxiety 14: 67–78. [Google Scholar] [CrossRef] [PubMed]
  3. Becker, Kimberly D., Maya Boustani, Resham Gellatly, and Bruce F. Chorpita. 2018. Forty Years of Engagement Research in Children’s Mental Health Services: Multidimensional Measurement and Practice Elements. Journal of Clinical Child & Adolescent Psychology 47: 1–23. [Google Scholar] [CrossRef]
  4. Bolinski, Felix, Annet Kleiboer, Eirini Karyotaki, Judith E. Bosmans, Anna-Carlotta Zarski, Kiona K. Weisel, David D. Ebert, Corinna Jacobi, Pim Cuijpers, and Heleen Riper. 2018. Effectiveness of A Transdiagnostic Individually Tailored Internet-Based and Mobile-Supported Intervention for the Indicated Prevention of Depression and Anxiety (iCARE Prevent) in Dutch College Students: Study Protocol for a Randomised Controlled Trial. Trials 19: 1–13. [Google Scholar] [CrossRef]
  5. Branson, Christopher E., Philip Clemmey, and Preetika Mukherjee. 2013. Text Message Reminders to Improve Outpatient Therapy Attendance among Adolescents: A Pilot Study. Psychological Services 10: 298–303. [Google Scholar] [CrossRef] [PubMed]
  6. Broglia, Emma, Abigail Millings, and Michael Barkham. 2019. Counseling with Guided Use of a Mobile Well-Being App for Students Experiencing Anxiety or Depression: Clinical Outcomes of a Feasibility Trial Embedded in a Student Counseling Service. JMIR MHealth and UHealth 7: e14318. [Google Scholar] [CrossRef]
  7. Bunnell, Brian E., Lynne S. Nemeth, Leslie A. Lenert, Nikolaos Kazantzis, Esther Deblinger, Kristen A. Higgins, and Kenneth J. Ruggiero. 2020. Barriers Associated with the Implementation of Homework in Youth Mental Health Treatment and Potential Mobile Health Solutions. Cognitive Therapy and Research 45: 272–86. [Google Scholar] [CrossRef]
  8. Clarke, Angela T., Stephen A. Marshall, Jennifer A. Mautone, Stephen L. Soffer, Heather A. Jones, Tracy E. Costigan, Anwar Patterson, Abbas F. Jawad, and Thomas J. Power. 2015. Parent Attendance and Homework Adherence Predict Response to a Family–School Intervention for Children with ADHD. Journal of Clinical Child & Adolescent Psychology 44: 58–67. [Google Scholar] [CrossRef] [Green Version]
  9. Danko, Christina M., Tasha Brown, Lauren Van Schoick, and Karen S. Budd. 2016. Predictors and Correlates of Homework Completion and Treatment Outcomes in Parent–Child Interaction Therapy. Child & Youth Care Forum 45: 467–85. [Google Scholar] [CrossRef]
  10. Elo, Satu, and Helvi Kyngäs. 2008. The Qualitative Content Analysis Process. Journal of Advanced Nursing 62: 107–15. [Google Scholar] [CrossRef]
  11. Farabee, David, Stacy Calhoun, and Robert Veliz. 2016. An Experimental Comparison of Telepsychiatry and Conventional Psychiatry for Parolees. Psychiatric Services 67: 562–65. [Google Scholar] [CrossRef]
  12. Gonzalez, Araceli, V. Robin Weersing, Erin M. Warnick, Lawrence D. Scahill, and Joseph L. Woolston. 2011. Predictors of Treatment Attrition among an Outpatient Clinic Sample of Youths with Clinically Significant Anxiety. Administration and Policy in Mental Health and Mental Health Services Research 38: 356–67. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  13. Hill, Clara E., Sarah Knox, Barbara J. Thompson, Elizabeth Nutt Williams, Shirley A. Hess, and Nicholas Ladany. 2005. Consensual Qualitative Research: An Update. Journal of Counseling Psychology 52: 196–205. [Google Scholar] [CrossRef] [Green Version]
  14. Ip, Patrick, David Chim, Ko Ling Chan, Tim M. H. Li, Frederick Ka Ho, Benjamin W. Van Voorhees, Agnes Tiwari, Charlie Wai Leung Chan, and Matthew Ho. 2016. Effectiveness of a Culturally Attuned Internet-Based Depression Prevention Program for Chinese Adolescents: A Randomized Controlled Trial. Depression and Anxiety 33: 1123–31. [Google Scholar] [CrossRef]
  15. Landis, J. Richard, and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33: 159. [Google Scholar] [CrossRef] [Green Version]
  16. Lopez, Amy, Sarah Schwenk, Christopher D. Schneck, Rachel J. Griffin, and Matthew C. Mishkind. 2019. Technology-Based Mental Health Treatment and the Impact on the Therapeutic Alliance. Current Psychiatry Reports 21: 76. [Google Scholar] [CrossRef]
  17. Neal-Barnett, Angela, Robert Stadulis, Delilah Ellzey, Elizabeth Jean, Tiffany Rowell, Keaton Somerville, Kallie Petitti, Benjamin Siglow, Arden Ruttan, and Mary Hogue. 2019. Evaluation of the Effectiveness of a Musical Cognitive Restructuring App for Black Inner-City Girls: Survey, Usage, and Focus Group Evaluation. JMIR MHealth and UHealth 7: e11310. [Google Scholar] [CrossRef]
  18. Newton, Amanda, Alexa Bagnell, Rhonda Rosychuk, Janelle Duguay, Lori Wozney, Anna Huguet, Joanna Henderson, and Janet Curran. 2020. A Mobile Phone-Based App for Use during Cognitive Behavioral Therapy for Adolescents with Anxiety (MINDCLIMB): User-Centered Design and Usability Study. JMIR MHealth and UHealth 8: e18439. [Google Scholar] [CrossRef]
  19. Oruche, U. M., Sarah M. Downs, Ethan D. Holloway, Claire Draucker, and Matthew Aalsma. 2014. Barriers and Facilitators to Treatment Participation by Adolescents in a Community Mental Health Clinic. Journal of Psychiatric and Mental Health Nursing 21: 241–48. [Google Scholar] [CrossRef] [Green Version]
  20. Paquette, Catherine E. 2020. Treatment Engagement in Smartphone-Enhanced vs. Standard Behavioral Activation for Substance Use. Unpublished Doctoral Dissertation. Chapel Hill: University of North Carolina. [Google Scholar] [CrossRef]
  21. Patterson Silver Wolf, David A., Alex T. Ramsey, Joel Epstein, Sara Beeler-Stinn, and Autumn Asher Black Deer. 2020. Bridges to Sobriety: Testing the Feasibility and Acceptability of a Mobile App Designed to Supplement an Adolescent Substance Use Disorder Treatment Program. Clinical Social Work Journal. Advance online publication. [Google Scholar] [CrossRef]
  22. Philippi, Paula, Harald Baumeister, Jennifer Apolinário-Hagen, David Daniel Ebert, Severin Hennemann, Leonie Kott, Jiaxi Lin, Eva-Maria Messner, and Yannik Terhorst. 2021. Acceptance towards Digital Health Interventions—Model Validation and Further Development of the Unified Theory of Acceptance and Use of Technology. Internet Interventions 26: 100459. [Google Scholar] [CrossRef] [PubMed]
  23. Pramana, Gede, Bambang Parmanto, James Lomas, Oliver Lindhiem, Philip C Kendall, and Jennifer Silk. 2018. Using Mobile Health Gamification to Facilitate Cognitive Behavioral Therapy Skills Practice in Child Anxiety Treatment: Open Clinical Trial. JMIR Serious Games 6: e8902. [Google Scholar] [CrossRef] [PubMed]
  24. Radovic, Ana, Pamela L. Vona, Antonella M. Santostefano, Samantha Ciaravino, Elizabeth Miller, and Bradley D. Stein. 2016. Smartphone Applications for Mental Health. Cyberpsychology, Behavior, and Social Networking 19: 465–70. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  25. Rogers, Julia, Tracy Gladstone, Benjamin Van Voorhees, and Eduardo L. Bunge. 2021. The Role of Human Support on Engagement in an Online Depression Prevention Program for Youth. Social Sciences 10: 285. [Google Scholar] [CrossRef]
  26. Saldaña, Johnny. 2013. The Coding Manual for Qualitative. London: Sage Publications. [Google Scholar]
  27. Silk, Jennifer S., Gede Pramana, Stefanie L. Sequeira, Oliver Lindhiem, Philip C. Kendall, Dana Rosen, and Bambang Parmanto. 2020. Using a Smartphone App and Clinician Portal to Enhance Brief Cognitive Behavioral Therapy for Childhood Anxiety Disorders. Behavior Therapy 51: 69–84. [Google Scholar] [CrossRef] [PubMed]
  28. Strauss, A. L., and J. Corbin. 1997. Grounded Theory in Practice. Thousand Oaks: Sage Publications. [Google Scholar]
  29. Thatcher, Jason Bennett, Misty L. Loughry, Jaejoo Lim, and D. Harrison McKnight. 2007. Internet Anxiety: An Empirical Study of the Effects of Personality, Beliefs, and Social Support. Information & Management 44: 353–63. [Google Scholar] [CrossRef]
  30. Torous, John, Michael E. Levin, David K. Ahern, and Megan L. Oser. 2017. Cognitive Behavioral Mobile Applications: Clinical Studies, Marketplace Overview, and Research Agenda. Cognitive and Behavioral Practice 24: 215–25. [Google Scholar] [CrossRef]
  31. Whiteside, Stephen P. H., Bridget K. Biggs, Michael S. Tiede, Julie E. Dammann, Julie C. Hathaway, Marc E. Blasi, Deanna Hofschulte, and Kristin Vickers. 2019. An Online- and Mobile-Based Application to Facilitate Exposure for Childhood Anxiety Disorders. Cognitive and Behavioral Practice 26: 478–91. [Google Scholar] [CrossRef]
  32. Wu, Ashley, Matthew A. Scult, Emily D. Barnes, Jessica A. Betancourt, Avital Falk, and Faith M. Gunning. 2021. Smartphone Apps for Depression and Anxiety: A Systematic Review and Meta-Analysis of Techniques to Increase Engagement. Npj Digital Medicine 4: 20. [Google Scholar] [CrossRef]
Table 1. Engagement codes by category.
Table 1. Engagement codes by category.
Category Code Frequency Kappa
Relationship Comfort with sharing 4 0.85
Interactions and communications 2 1.00
Shared experiences 1 1.00
No impact on relationship 6 1.00
Suggestion about relationship 5 1.00
Expectancy Low expectations 7 0.93
Positive expectations 4 1.00
Positive shift in beliefs 8 1.00
Previous experiences 5 0.88
No impact on expectancy 3 0.85
Suggestion about expectancy 4 1.00
Attendance Behavioral attendance 1 1.00
Motivation to attend 3 1.00
No impact on attendance 4 1.00
Suggestion about attendance 3 1.00
Clarity Psychoeducation about social anxiety 5 0.65
Role and purpose of therapy 5 0.79
No impact on clarity 3 1.00
Suggestion about clarity 2 1.00
Homework App reminders 4 1.00
In-session participation 11 1.00
Reinforce practice 6 0.90
Usability 4 0.85
No impact on homework 2 0.66
Suggestion about homework 4 1.00
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Lu, C.; Chu, W.; Madden, S.; Parmanto, B.; Silk, J.S. Adolescent Perspectives on How an Adjunctive Mobile App for Social Anxiety Treatment Impacts Treatment Engagement in Telehealth Group Therapy. Soc. Sci. 2021, 10, 397.

AMA Style

Lu C, Chu W, Madden S, Parmanto B, Silk JS. Adolescent Perspectives on How an Adjunctive Mobile App for Social Anxiety Treatment Impacts Treatment Engagement in Telehealth Group Therapy. Social Sciences. 2021; 10(10):397.

Chicago/Turabian Style

Lu, Celine, Wendy Chu, Shannon Madden, Bambang Parmanto, and Jennifer Susan Silk. 2021. "Adolescent Perspectives on How an Adjunctive Mobile App for Social Anxiety Treatment Impacts Treatment Engagement in Telehealth Group Therapy" Social Sciences 10, no. 10: 397.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop