Students Tele-All: Self-Efficacy and Self-Reflection as Measures of Student Success in Telepractice Training

Abstract

With an increasing demand for telepractice services, the need for telepractice education is more important than ever. In addition to learning how to deliver these services, certain clinical and technological skills learned through telepractice apply more broadly to in-person care. Evaluating students’ abilities to master these skills is necessary to ensure clinical skill competence. We utilized self-efficacy ratings and self-reflections to determine student growth after hands-on telepractice training among nursing, occupational therapy, physical therapy, and speech-language pathology students. Students across disciplines demonstrated growth in all measured skill areas and reported overall increased confidence. Students with less prior telepractice experience reported greater increased confidence in seven measured competency areas than students with more prior telepractice experience. The number of completed sessions at the end of student placements was correlated with increased confidence for one measured skill area. Regardless of whether or not students had prior experience when starting their placement, as the number of telepractice sessions completed increased, thus did students’ confidence levels. The results of this study support telepractice as a viable clinical education tool for student growth and the use of self-efficacy and self-reflection as valuable tools for monitoring the effectiveness of telepractice clinical learning activities.

1. Introduction

1.1. Background

Health profession graduate students are required to have a working knowledge of current clinical practices before entering both clinical rotations and professional working roles. Traditional academic competencies and classroom didactic materials are beneficial and safe ways to practice both patient-interaction skills and documentation practices while allowing for reflective discussion. Common instructional methodologies in health profession education include the use of simulation, role play, in-person client interactions, and most recently, virtual client encounters, to ready students for entry into their fields. Preparing students specifically for the virtual patient-care environment must be valued to keep up with the pace of integrative technologies that allow healthcare systems to offer accessible, secure, and effective care, raising expectations that students arrive with foundational knowledge in telepractice [1,2,3]. In this study, we examine the value of utilizing telepractice as a teaching methodology in graduate education to ensure students’ growth in necessary clinical competencies across the advanced practice nursing (APRN), occupational therapy (OT), physical therapy (PT), and speech-language pathology (SLP) professions.

1.2. The Importance of Telepractice Education and Outcome Measurements

Prior to the pandemic, despite increases over time in the use of virtual client encounters in clinical practice, telepractice education had remained limited across health professions [15,16]. Given the pre-existing rise in telepractice service delivery and the now massive increase in its use due to the pandemic, the necessity for telepractice education within health profession programs has become pertinent for students to meet graduation requirements and market needs as they enter the workforce. Traditionally, health profession educators design and arrange learning experiences for graduate students and measure outcomes from both clinical learning and classroom experiences to capture skill competency and the understanding of instructional content. During in-person patient simulation and live-patient encounters, student skill competencies can typically be observed and quantified based on rubrics or checklists that mirror national board requirements and accreditation standards in APRN, OT, PT and SLP in areas such as patient handling, evaluation and intervention methodologies, professional reasoning, and documentation. Measuring student outcomes from learning experiences informs graduate programs that students are acquiring the necessary skills for clinical competence and that learning activities are effective in developing competence.

With the arrival of the novel coronavirus disease (COVID-19) as a global pandemic in the spring of 2020, clinical practices and health profession educators were thrust into the utilization of virtual platforms as a service-delivery medium [4]. For some time, telepractice has been accepted as a viable clinical service delivery modality by APRN, OT, PT and SLP professions when appropriate for client needs [5,6,7,8], yet relatively few clinicians utilized virtual visits prior to the pandemic. In OT, Furniss reported that 34% of therapists who responded to an American Occupational Therapy Association (AOTA) survey (n = 599) reported having completed at least one telepractice service prior to April 2020 [9]. Forty-six percent of the AOTA survey respondents reported having completed at least one session as of January 2021. In PT, an American Physical Therapy Association (APTA) survey (n = 5400 PTs, n = 1100 PTAs) completed between April and May 2020 reported that 2% of respondents provided live video consults prior to the COVID-19 pandemic, whereas, during those early months of the state of emergency, 50% of practitioners provided telepractice services [10]. The American Association of Nurse Practitioners (AANP) conducted two surveys in May 2020 (n = 4800) and August 2020 (n = 4000) to examine the impact of COVID-19 on APRN practice [11]. While pre-pandemic telepractice utilization was not reported, 51% of APRNs reported a transition to telepractice in May 2020, which increased to 63% by August 2020 [12]. An October 2020 survey (n = 13,539) completed by the American Speech-Language Hearing Association (ASHA) reported that prior to the pandemic, 4.8% of SLPs provided telepractice services [13], although a 2002 ASHA survey (n = 1667) found that as many as 11% of respondents utilized telepractice [14]. ASHA’s October 2020 survey found that at that time, 71.5% of respondents reported having utilized telepractice [13]. As a result of this increase in telepractice provision across professions, it is possible that consumers and providers will desire continued access to remote services, making the need for education in this delivery method more important than ever.

In health profession education, the foci of competency are typically centered around students’ successes in demonstrating knowledge, comprehension, and synthesis of concepts and skills foundational to client-centered care and practice. Ericsson and Lehman suggested that specific or deliberate practice focused on direct, required skills may support better outcomes for students, as they have the supports for repetition, engagement at appropriate levels of difficulty, time to correct errors, and informative feedback [17]. To enhance these learning encounters, educators are called to understand that how students apply learned concepts to deliverable skills is largely influenced by their self-efficacy, an essential component of Bandura’s social cognitive theory [18]. Relatedly, Dewey’s experiential learning theory [19] often guides the creation of learning experiences in health and medical profession education, where concrete experiences and active experimentation and practice are paired with self-reflection and meaning making. Assessment and intervention service delivery models (both in person and through telepractice), when paired with a self-reflection component, can support desired learning outcomes by allowing more opportunities for student self-assessment of their own learning, precipitate meaning making, concretize confidence, and support self-efficacy in acquiring and retaining skills [20]. Pasupathy and Bogshutz [21] and Lee and Schmaman [22] found that self-efficacy skills were positively correlated with SLP students’ clinical performance. Opportunities for self-assessment and self-reflection may enhance students’ understanding of where their skills lie, where there may be room for growth, and perceptions of self-confidence.

Although some health professions’ student outcome measurement tools may translate to some degree to the virtual service-delivery context, such as the Student Confidence Questionnaire [23] developed for use with occupational therapy students and the SLP Clinical Self-Efficacy Inventory [21] developed for use with speech-language pathology students, there are still limited resources for specifically assessing student competence through telepractice experiences. Some graduate programs, such as the University of Maine’s Speech Therapy Telepractice and Technology Program, have reported utilizing detailed practicum evaluation forms in which supervisors rate student performance on competency development at various timepoints in the training process [24]. Likewise, a University of Kentucky speech-language pathology program reported designing their pediatric, school-based telepractice curriculum around general knowledge and skills competencies established by ASHA’s Council on Academic Accreditation in Audiology Speech Language Pathology (CAA) [16], though details of how student outcomes were gathered were not reported.

Given the lack of widespread tools for tracking student outcomes in telepractice education, research on the use of outcome measures in this area is severely limited. Recently, McGill and Dennard reported on the use of a survey distributed to several speech-language pathology graduate programs that captured students’ perceptions of their knowledge and clinical skills related to telepractice [25]. In their study, only 20% of respondents reported having had experience providing telepractice services. They found that students with more experience in delivering telepractice services, as well as those who were further along in their graduate programs, demonstrated higher confidence in their knowledge and skills than those with less experience. McGill and Dennard’s study focused on capturing students’ perceptions of their knowledge and skills related to telepractice broadly at a single point in time, rather than before or after deliberate training, though their use of self-efficacy ratings and the content of the items used in their survey have potential for use as pre- and post-training measures for tracking student outcomes. In this paper, we will present a tool originally designed to measure speech-language pathology student outcomes during telepractice training. While our tool was created prior to the publication of McGill and Dennard’s survey, it shares similarities in the types of competencies measured as well as in self-reflection methodology.

1.3. Telepractice Competencies in Clinical Education

While many training requirements inherently differ across APRN, OT, PT and SLP professions given the individual scopes of practice of each discipline, there is some degree of overlap in competencies that students must develop to practice as competent clinicians. Examples of these shared skills include building rapport with clients, effective communication with clients, caregivers, and related professionals, and in reliably and efficiently collecting data, including case histories. Many skills that practitioners must acquire across and within professions can be conceivably addressed through telepractice training (e.g., client interviews across professions, handwriting analysis in OT), while other skills may present greater challenges (e.g., vocal fold examination in SLP, balance assessment in PT) given logistical, safety, and technological barriers.

In nursing education, APRN students are required by the American Nursing Credentialing Center (ANCC) to complete a minimum of 500 direct clinical hours for certification [26]. Telepractice in a variety of synchronous and asynchronous modalities is considered direct care and therefore, meets the certification requirements. The National Organization of Nurse Practitioner Faculties (NONF) and AANP support the integration of telepractice into the APRN curriculum and have developed competencies associated with telepractice etiquette, assessment and technology skills, policy, legal implications, and documentation [5,27,28,29].

In OT, one telepractice-specific 2018 Accreditation Council for Occupational Therapy Education (ACOTE^®) standard exists (std b.4.15) that addresses students’ need to understand the use of information and communication technologies in practice [30]. This accreditation requirement includes not only understanding telepractice technology but also knowing and using electronic medical record systems and virtual environments. AOTA’s position paper on telepractice outlines guidelines for use in practice, underscoring the importance of clinical reasoning and adhering to the AOTA Code of Ethics, following state regulatory boards’ rules and regulations and making decisions on a case-by-case basis [6]. As such, all these areas are important to develop through clinical education. Additionally, AOTA’s position statement emphasizes the value of telepractice use in OT practice, student education, and supervision [6].

In PT, there are no specific requirements in the 2020 Commission on Accreditation in Physical Therapy Education (CAPTE) standards related to telepractice [31]. Prior to the pandemic, students were not able to receive direct clinical hours that counted towards graduation requirements. Considering the pandemic, CAPTE provided allowances to graduate programs through a guidance document stating that clinical hours performed through telepractice could be counted as long as the learning experiences were equivalent to previous learning plans in ensuring entry-level competence upon graduation [32].

In SLP, the 2020 CAA standards [33] do not contain any requirements specific to the development of telepractice skills. However, ASHA’s Telepractice Practice Portal does outline guidelines for telepractitioners in recognition of the specialized skills necessary, such as the ability to understand appropriate models of technology used to deliver services, sensitivity to cultural and linguistic variables that relate to telepractice, and the need to remain current and knowledgeable in terms of national and state regulations [8]. Given the established guidelines, it can be presumed that practitioners should learn these required skills within the context of their certification process rather than on the job. While telepractice training has been viewed by the Council for Clinical Certification in Audiology and Speech-Language Pathology (CFCC) as valuable learning experiences for some time, prior to the COVID-19 pandemic, the CFCC did not allow telepractice hours to count towards graduation requirements. In response to the pandemic, the CFCC made accommodations to these clinical education requirements to allow telepractice sessions to count towards graduation requirements, currently through 31 December 2022 [34].

To support the continuation of and increased need for telepractice education in the health professions, we set out to evaluate students’ abilities to develop clinical skills across a variety of competency areas using self-efficacy and self-reflection measurements by asking the following questions:

• Do students demonstrate increased confidence in clinical skill development related to competency standards across health professions in the telepractice modality?
• Is telepractice a viable medium for clinical skill instruction for both telepractice-specific service delivery as well as the development of clinical skills that translate to in-person practice?

Regarding these questions, we hypothesized that:

• Students would demonstrate increased confidence across competency areas from pre- to post-training.
• Students who participated in more telepractice sessions would demonstrate greater increases in confidence than students who participated in a limited number of sessions.
• Students with little prior experience in telepractice would demonstrate greater increases in confidence than those with prior experience.
• Students with more telepractice experience would demonstrate greater confidence than those with less telepractice experience.

2. Materials and Methods

2.1. Participants

Participants in this study were 229 students enrolled in four health profession graduate programs at the MGH Institute of Health Professions in Boston, MA, USA. Of the 229 students who participated, 37 were Master of Science in Nursing students, 25 were Doctor of Occupational Therapy students, 6 were Doctor of Physical Therapy students, and 161 were Master of Science in Speech-Language Pathology students. Participation was encouraged but not required. Students were at varying points in their curricula; APRN students were in their third year, OT students were in their second or third year, PT students were in their second or third year, and CSD students were in their first or second year of their programs. However, the majority of students had no prior experience with telepractice. Students were enrolled in clinical experiences that were pre-existing in scope and populations served, though all placements were adapted from previous in-person models to telepractice. Participant consent was waived due to the nature of this research as a retrospective analysis of student outcome data collected for quality improvement.

2.2. Student Self-Efficacy Tool for Telepractice

The Student Self-Efficacy Tool for Telepractice (SSET-TP) was originally developed by faculty in the SLP master’s program as a part of a telepractice pilot program to capture outcome measures from a student-learning experience prior to the COVID-19 pandemic. At the onset of the pandemic, APRN, OT, PT, and SLP faculty were required to pivot to telepractice to continue clinical experiences for students that resulted in clinical hours towards meeting graduation requirements. With an identified need to track student outcome measures to ensure students would sufficiently acquire the necessary clinical skills to meet educational standards when learning through the telepractice medium, faculty across these disciplines came together to adopt the SSET-TP as a way to examine student-learning outcomes.

The SSET-TP was designed to measure student self-efficacy in performing a series of thirteen clinical skills using a 4-point rating scale (1—not yet confident, 2—minimally confident, 3—moderately confident, 4—very confident. Not applicable was also available) and to capture self-reflections on self-formulated S.M.A.R.T. goals related to learning following the goal-writing method of specific, measurable, achievable, realistic, and time-oriented goals [35]. Confidence-scale categories were influenced by the SLP departmental focus on encouraging students to embrace a growth mindset [36] in perceiving their skills as not having been acquired “yet” versus a fixed mindset of “not at all confident” or “lacking skill”. Items on the scale were both telepractice specific (e.g., setting up telepractice equipment) and clinical skill specific (e.g., rapport building, data collection) and were reflective of skills that students were required to develop to successfully complete their placements. Items on the SSET-TP were closely related to key clinical items used within the SLP department to track students’ progress in attaining the clinical competencies required by CFCC standards throughout the graduate program. Items contained within the SSET-TP self-efficacy scale can be found in

Table 1. Differences in scores before and after telepractice training. n = sample size, SD = standard deviation, Z = Z score, p-value = significance level of Wilcoxon signed ranks test.

Item	Pre-Training			Post-Training			Difference		Wilcoxon Signed Ranks
I Can Effectively:	n	Mean	SD	n	Mean	SD	Mean	SD	Z	p-Value
Set up telepractice equipment	104	2.62	1.03	104	3.68	0.56	−1.07	0.97	−7.572	<0.001
Plan for telepractice sessions	104	2.38	0.93	104	3.73	0.45	−1.36	1.03	−7.974	<0.001
Adapt/create new materials to be used in telepractice sessions	102	2.23	0.87	102	3.57	0.52	−1.34	0.95	−8.089	<0.001
Collect data during telepractice sessions	105	2.51	0.89	105	3.46	0.64	−0.94	1.06	−6.778	<0.001
Execute telepractice sessions in a timely manner	105	2.46	0.99	105	3.64	0.56	−1.18	1.08	−7.421	<0.001
Manage challenging behavior during telepractice sessions	101	1.85	0.86	101	3.00	0.58	−1.15	1.01	−7.330	<0.001
Troubleshoot technical issues during telepractice sessions	104	2.09	0.90	104	3.20	0.60	−1.12	0.91	−7.721	<0.001
Execute telepractice sessions with various client populations (ages, disorders)	103	2.04	0.88	103	3.06	0.78	−1.02	1.11	−6.795	<0.001
Build rapport with clients during telepractice sessions	104	2.66	0.91	104	3.72	0.47	−1.06	0.94	−7.611	<0.001
Communicate with caregivers during telepractice sessions	97	2.69	0.89	97	3.49	0.58	−0.80	1.00	−6.166	<0.001
Scaffold during telepractice sessions	96	2.19	0.90	96	3.46	0.66	−1.27	1.05	−7.347	<0.001
Execute therapy sessions in consideration of the needs and values of clients/caregivers in their homes	104	2.48	0.90	104	3.58	0.52	−1.10	0.96	−7.608	<0.001
Maintain privacy and confidentiality of client information gathered during telepractice sessions	103	3.32	0.88	103	3.90	0.30	−0.58	0.91	−5.525	<0.001

,

Table 2. Correlation between number of telepractice sessions at the end of training and change in scores. n = sample size, SD = standard deviation, tau = tau-b statistic, p-value = significance level of Kendall’s tau-b test.

Item	Kendall’s Tau-b
I Can Effectively:	n	Mean (Post-Pre)	SD (Post-Pre)	Tau (Post-Pre)	p-Value (Post-Pre)
Set up telepractice equipment	106	1.10	1.00	0.132	0.122
Plan for telepractice sessions	106	1.34	1.14	0.032	0.709
Adapt/create new materials to be used in telepractice sessions	106	1.37	1.02	0.043	0.612
Collect data during telepractice sessions	106	0.97	1.10	0.070	0.408
Execute telepractice sessions in a timely manner	106	1.21	1.11	0.087	0.300
Manage challenging behavior during telepractice sessions	106	1.16	1.16	0.069	0.413
Troubleshoot technical issues during telepractice sessions	106	1.11	0.99	0.096	0.263
Execute telepractice sessions with various client populations (ages, disorders)	106	1.04	1.19	0.016	0.847
Build rapport with clients during telepractice sessions	106	1.09	0.98	0.117	0.170
Communicate with caregivers during telepractice sessions	106	0.92	1.08	0.083	0.330
Scaffold during telepractice sessions	106	1.15	1.07	0.171	0.045
Execute therapy sessions in consideration of the needs and values of clients/caregivers in their homes	106	1.15	1.03	0.026	0.763
Maintain privacy and confidentiality of client information gathered during telepractice sessions	106	0.66	1.02	0.105	0.224

,

Table 3. Correlation between number of telepractice sessions prior to training and change in scores. n = sample size, SD = standard deviation, tau = tau-b statistic, p-value = significance level of Kendall’s tau-b test.

Item	Kendall’s Tau-b
I Can Effectively:	n	Mean (Post-Pre)	SD (Post-Pre)	Tau (Post-Pre)	p-Value (Post-Pre)
Set up telepractice equipment	106	1.10	1.00	−0.353	<0.001
Plan for telepractice sessions	106	1.34	1.14	−0.316	<0.001
Adapt/create new materials to be used in telepractice sessions	106	1.37	1.02	−0.268	0.001
Collect data during telepractice sessions	106	0.97	1.10	−0.151	0.072
Execute telepractice sessions in a timely manner	106	1.21	1.11	−0.256	0.002
Manage challenging behavior during telepractice sessions	106	1.16	1.16	−0.098	0.239
Troubleshoot technical issues during telepractice sessions	106	1.11	0.99	−0.214	0.011
Execute telepractice sessions with various client populations (ages, disorders)	106	1.04	1.19	−0.109	0.191
Build rapport with clients during telepractice sessions	106	1.09	0.98	−0.274	0.001
Communicate with caregivers during telepractice sessions	106	0.92	1.08	−0.136	0.106
Scaffold during telepractice sessions	106	1.15	1.07	−0.221	0.009
Execute therapy sessions in consideration of the needs and values of clients/caregivers in their homes	106	1.15	1.03	−0.125	0.138
Maintain privacy and confidentiality of client information gathered during telepractice sessions	106	0.66	1.02	−0.131	0.129

and

Table 4. Correlation between number of telepractice sessions and scores. n = sample size, SD = standard deviation, tau = tau-b statistic, p-value = significance level of Kendall’s tau-b test.

Item	Kendall’s Tau-b
I Can Effectively:	n	Tau	p-Value
Set up telepractice equipment	212	0.558	<0.001
Plan for telepractice sessions	211	0.653	<0.001
Adapt/create new materials to be used in telepractice sessions	207	0.597	<0.001
Collect data during telepractice sessions	212	0.443	<0.001
Execute telepractice sessions in a timely manner	212	0.540	<0.001
Manage challenging behavior during telepractice sessions	212	0.431	<0.001
Troubleshoot technical issues during telepractice sessions	212	0.508	<0.001
Execute telepractice sessions with various client populations (ages, disorders)	212	0.422	<0.001
Build rapport with clients during telepractice sessions	212	0.528	<0.001
Communicate with caregivers during telepractice sessions	212	0.379	<0.001
Scaffold during telepractice sessions	192	0.567	<0.001
Execute therapy sessions in consideration of the needs and values of clients/caregivers in their homes	211	0.498	<0.001
Maintain privacy and confidentiality of client information gathered during telepractice sessions	211	0.366	<0.001

.

Background (e.g., prior experience with telepractice) and demographic (e.g., time completed within the student’s graduate program) information were also collected. Background and demographic questions varied across programs based on program-relevance and program-specific factors (e.g., tracks within programs, clinical placement centers). Confidence-rating items were the same across all professions, except for one item regarding patient scaffolding that was determined to be irrelevant to APRN students (10 students) and was, therefore, removed for those students.

Students’ clinical placements for spring 2020 through summer of 2021 were included in this study. The SSET-TP was administered at two timepoints. At timepoint 1 (pre-training), background/demographics and the SSET-TP were administered at the start of the students’ telepractice placements prior to their first telepractice visit. At this time, students were asked to rate their confidence on the various clinical skills listed within the tool using the rating scale, as well as to formulate a S.M.A.R.T. goal for the practicum and to provide a written reflection on their perceived current skill level related to their goal. At the end of their placement, timepoint 2 (post-training), students were given the SSET-TP again, containing the same rating scale items, and were asked to provide a reflection on their progress with the S.M.A.R.T. goal they had established at the start of the learning experience. The clinical placements varied in length depending on accreditation standards for the various professions; students were measured before the start of their clinical experience and within one week of their last clinical visit.

2.3. Data Analysis

Quantitative and qualitative data were collected using REDCap [37], an electronic data collection tool, at the start of students’ telepractice clinical placements prior to their first telepractice visit (pre-training) and at the conclusion of placements after completing all telepractice visits (post-training). Questionnaires were completed asynchronously at the students’ convenience and without a time limit. Quantitative data for 114 students were analyzed. Students who did not complete both SSET-TP administrations (pre- and post-training) were excluded from the data set (8). Qualitative data for 223 students were analyzed. Students who did not complete more than three telepractice sessions were excluded (16). Students who completed only pre-training S.M.A.R.T. goal formulation or post-training reflections were included in the analyses for which data were available. PT students were not included in the qualitative analysis due to administrative error.

IBM SPSS version 24 was used for statistical analysis. Means and standard deviations were provided. Because data were not normally distributed, nonparametric trainings were used. To compare pre- and post-training ratings, a Wilcoxon signed ranks test was performed for each of the 13 questions on the survey. To determine correlations between the number of telepractice sessions and the ratings to each question, as well as between students’ experience with telepractice and ratings to each question, Kendall’s tau-b tests were performed due to the ordinal variable and categorical nature of the items. Spearman’s rank correlation analyses were not used because Kendall’s tau-b allows for continuous data to be analyzed and is more robust than Spearman’s rank, and there is no real difference between the two analytical methods when variables do not follow a normal distribution [38]. Statistical significance was concluded if p ≤ 0.05.

Content analyses were performed to analyze students’ chosen S.M.A.R.T. goals and their progress thereafter. Of note, many students wrote goals that did not meet all the parameters of S.M.A.R.T. goals, but they were analyzed for content nonetheless. Qualitative data were coded and analyzed using the six-step process described by Braun and Clarke [39]. Three faculty members coded the pre-training S.M.A.R.T. goals and post-training reflections independently, then met to achieve intercoder reliability and collapse codes into themes. At pre-training, S.M.A.R.T. goals were analyzed for themes on which students chose to focus on their clinical skill development. At post-training, student self-reflections related to their previously established S.M.A.R.T. goals were analyzed for thematic outcomes.

3. Results

The average time between pre- and post-training was 67.12 ± 27.69 days. The average number of sessions were as follows: pre-training = 2.45 ± 3.23, post-training = 10.17 ± 3.62, overall = 6.43 ± 5.19. Results of the Wilcoxon signed ranks test (Table 1) showed that there was a statistically significant difference between pre- and post-training ratings for all 13 questions (p < 0.001), with an average increase of 1.09 ± 1.09 points. Results of the first Kendall’s tau-b test (Table 2) showed that the number of telepractice sessions after training was correlated with the change in ratings in only one of the thirteen questions (p = 0.045). Results of a second Kendall’s tau-b test (Table 3) showed the number of telepractice sessions before training was negatively correlated with the change in ratings of seven questions (p ranged from <0.001 to 0.011), suggesting that those with fewer telepractice sessions before training had the greater change in ratings in approximately half of the questions. Results of a third Kendall’s tau-b test (Table 4) showed that the number of telepractice sessions students completed, including those prior to the evaluated placement, was positively correlated with the ratings of all 13 questions (p < 0.001), suggesting that as students’ amount of overall telepractice experience grew, thus did their confidence.

Qualitative data from pre-training S.M.A.R.T. goals and post-training reflections revealed two primary themes, building confidence and growth at the two timepoints, respectively. Sub-themes pre- and post-training were identified as organizational skills, interpersonal skills, technical skills, and diagnostic and intervention skills, reflecting the focus on the students’ areas in which they were initially underconfident and by the end of the telepractice experience demonstrated growth.

At pre-training, the majority of students formulated S.M.A.R.T. goals related to building confidence across four sub-themes related to organizational skills (e.g., documentation, planning, and time management), interpersonal skills (e.g., behavior management, rapport-building, communication, bias, and inclusion), technical skills (e.g., telepractice tools, telepractice execution, learning a new service-delivery modality), and diagnostic/intervention skills (e.g., service-specific skills, such as working with clients with aphasia, diagnosing rashes, executing physical exams, literacy intervention, and clinical reasoning). An example of a student-formulated S.M.A.R.T. goal that focused on session-planning was, “I will create functional telepractice lesson materials that can be used in an online platform through a shared screen for each of my clients without technical difficulties in 9/10 opportunities.” An example of a student-formulated S.M.A.R.T. goal that focused on interpersonal and behavior management skills was, “At the end of the practicum placement, I will have developed three strategies to manage challenging behavior in the setting of telepractice.”

At post-training, most students reported that they demonstrated growth in the areas on which they chose to focus with their S.M.A.R.T. goals. Many students also reported growth in other domains that were not originally included in their goals. The same sub-themes that were extracted from pre-training S.M.A.R.T. goals were found as areas of growth at post-training. Initially, some students felt that they would not be able to develop clinical skills in the telepractice environment and worried about being able to manage learning new clinical practice skills at the same time. However, analysis of reflections demonstrated growth in these areas and an appreciation for telepractice. As an example of a related reflection, one student reported, “I came into this practicum a little worried about telepractice and wondering whether it could be at all similar to in-person therapy, and I have found that I am able to [be] just as successful with [telepractice] as I was with in-person therapy. I believe I have truly grown as a clinician, and this experience has provided me with the foundation to provide effective intervention through this mode in the future.” In addition to examination skills and technical skills, students also reported improvements in clinical reasoning, which is an important component of clinical education. One student said, “I feel my experience with [telepractice] at the [Center] furthered all my goals listed [previously]. I am definitely more comfortable with what tools are available to conduct physical exams virtually, better able to convey warmth to clients, and the experience furthered my diagnostic reasoning as well. I am grateful for the experience, thank you!” Students also appreciated development in problem solving and innovating the use of items at home to substitute for equipment that would often be available in practice settings. One student reported, “I believe that I met [my] goal through creating interventions for use during [telepractice] sessions based on interventions I learned or researched. I adapted using a slant board for handwriting by having the client use a binder [they] had at home, found an online memory card game, mindfulness videos, and used items clients had in their own home as therapeutic tools (such as beads and art supplies).”.

4. Discussion

4.1. Interpretation of Results

Results of the quantitative and content analyses confirmed that students were able to develop greater confidence in their clinical abilities across all items measured by the SSET-TP, including those skills that were specific to telepractice service delivery as well as those that are more broadly applicable to in-person clinical practice. Similar to McGill and Dennard [25], we found a positive correlation between the amount of experience students had in delivering telepractice services and confidence level, adding to a necessary and growing body of support for intensive training with online service delivery to prepare students for competent and current clinical practice. While students in our sample who completed more telepractice sessions within the examined placement only demonstrated greater increases in confidence from pre- to post-training for the item regarding scaffolding, students with less prior experience at the start of the program demonstrated greater increases in confidence in many skill areas during the placement time period. Given these positive results in using live, online client interactions for skill development, our data promote the continuation of telepractice as an important teaching tool that could effectively be utilized by graduate programs, regardless of whether students plan to later practice in telepractice or primarily in in-person service delivery.

In addition to data supporting telepractice education and telepractice as a tool for student skill development, data gathered from the SSET-TP also provided valuable information to our graduate programs in determining the strengths and areas of growth in terms of program development. In a review of differences in pre- and post-training confidence as well as in reading student self-reflections on their goals, programs could identify needs and areas in which placements exceled in the training of various competencies.

In terms of feasibility of utilizing the SSET-TP as an outcome measurement, the tool was easy to implement within our clinical curricula with little time and effort required by clinical educators to distribute the pre- and post-surveys. Utilization of an electronic data-collection tool afforded an efficient means to collect, organize, and analyze the data. Overall, the use of the SSET-TP was found to be valuable across our programs in evaluating a predominantly new training methodology and will continue to be as clinical education through telepractice evolves.

The results of our analyses overall support the use of telepractice as a learning modality to attain clinical competencies. Telepractice provides increased accessibility to clients, supporting the missions of graduate student clinics to provide services to those in need. Because of this, clinical education programs are afforded the ability to broadly recruit a diverse array of clients that supports students in learning to provide services to populations that may have less presence in the area in which they are training. Additionally, telepractice eases graduate programs’ concerns related to insufficient numbers of placements to support the attainment of clinical hours necessary for graduation when programs are challenged by placement shortages related to factors such as geographical location (e.g., rural areas with few facilities available nearby) and competition (e.g., areas where several graduate programs of the same professions are located in the same area).

4.2. Limitations

Several limitations should be considered in relation to our methodology and conclusions. First, this study was a retrospective review of data collected by graduate programs as a method of evaluating student outcomes from clinical placements, and therefore, was not designed with a rigid research protocol in mind. Likewise, the SSET-TP is an unvalidated tool and requires psychometric analysis to ensure content validity and reliability.

In addition, the items of the SSET-TP are not inclusive of all discipline needs. In preparing for distribution of this tool, we did remove the item regarding scaffolding to the survey completed by APRN students given its irrelevance to the profession, but other modifications were not made to account for items that may have been desirable to specific programs. In general, items on the SSET-TP do not measure skills that relate to targeted, methodological clinical assessment and intervention tasks, such as determining speech-sound production accuracy or supporting patients in completing activities of daily living. Likewise, given the inherent limitations of serving clients at a distance, not all competencies that students must develop are measured by the SSET-TP, nor are we suggesting that all clinical skills can be developed through telepractice. Student competencies related to hands-on or manipulation skills, such as patient transfers or palpating a patient’s neck during a swallow, cannot be addressed in a telepractice session and thus, cannot be assessed with this measure.

The variety of training goals and professional training experience for students in this study must also be considered when analyzing the outcome data. For example, some OT students were in their fourth semester of graduate training and others in their fifth when delivering telepractice services, allowing for possible differences in targeted skill-development. While data were gathered on group differences, such as the programs in which students were enrolled and the number of semesters they had completed, we were unable to account for total program length and the timeline by which clinical skills would be expected to develop within the various curricula.

Limitations also exist regarding goals on which students reflected. First, not all goals were written in the correct S.M.A.R.T. goal format, resulting in reflections that were subjective in describing progress. Some students also reported being unable to recall their original goal and reported on other goals they had set for themselves throughout their placements. Likewise, many students provided reflections at post-training that addressed their experience more broadly, accounting for their overall progress within their placement. Finally, given that goals were self-formulated, the goals on which students reflected may not have been goals that would have otherwise been identified by their clinical educators as areas on which to focus. Regardless, the reflections students completed created a valuable opportunity for students to reflect on their growth and continued needs as well as for clinical educators to investigate their students’ perceptions of their clinical skill-development.

Inherent to self-reporting, subjectivity should also be considered as a factor that may have impacted self-efficacy ratings, creating limitations within the results, wherein the absence of rating criteria (e.g., rubrics), students’ perceptions of strengths and areas for growth are likely to vary. For example, some students may be overly confident while others may underestimate their abilities. Additionally, while the data collected in this project were anonymous, it is possible students may have inflated self-efficacy ratings if concerned that the results would be seen by their faculty.

4.3. Future Directions

Performing psychometric analyses to establish the validity and reliability of the SSET-TP is a necessary next step in developing this measure. Comparing the results of the SSET-TP with clinical educator assessments or with client feedback are future methods to consider for validation. To achieve reliability for each discipline, the SSET-TP may require refinement to measure the unique needs for each discipline’s specific skill development. This could involve examining each discipline’s SSET-TP data separately and examining students at the same level within each program (e.g., analyzing data from second year PT students only). Differences between levels of students within each program should then be analyzed to determine growth in telepractice skills across the program’s teaching methods and experiences. Establishing reliability and validity as well as implementing the SSET-TP regularly could provide justification for student hours engaged in telepractice and provide support for meeting each discipline’s required accreditation competencies and standards in this modality.

Given our interprofessional collaboration, we identified that there exists a significant amount of overlap in clinical competencies to be developed by students across APRN, OT, PT and SLP professions. Because of this, the SSET-TP was able to capture student progress regardless of profession. In utilizing this tool in the future as a single, consistent tool across programs, clinical educators could effectively develop interprofessional-learning opportunities that target overlapping needs efficiently. For example, in comparing data across programs, if students demonstrate less confidence in rapport building during telepractice visits than in other areas, clinical educators could develop an interprofessional simulation to foster development of this clinical competency, engaging many students within the same learning experience.

While evolving technologies often have great potential for breaking down barriers, they are often met with resistance. In the case of telepractice, despite its rapidly increased usage and the evidence-based recognition of its effectiveness by various health professions’ professional organizations, challenges remain in convincing these organizations on the importance of required, guided student education in this area. Relatedly, issues of reimbursement by Medicare, Medicaid, and other commercial insurers may discourage graduate programs from prioritizing telepractice training, creating a myriad of missed opportunities for students in developing skills for innovative clinical practice. With the results of this study, we hope to provide support for telepractice as a valuable and necessary focus of graduate education in the health professions and urge both graduate programs and accrediting bodies to examine and prioritize training in this area.