Next Article in Journal
Landslide Displacement Prediction Based on Variational Mode Decomposition and GA–Elman Model
Next Article in Special Issue
Two or Four Weeks Acute: Chronic Workload Ratio Is More Useful to Prevent Injuries in Soccer?
Previous Article in Journal
Resource Allocation for Network Slicing in RAN Using Case-Based Reasoning
Previous Article in Special Issue
Effect of a 10-Week Sensomotor Exercise Program on Balance and Agility in Adolescent Football Players: A Randomised Control Trial
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 13
Issue 1
10.3390/app13010446
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

The Effects of Different Training Interventions on Soccer Players’ Sprints and Changes of Direction: A Network Meta-Analysis of Randomized Controlled Trials

by
Kuan Dong
1,†,
Guyeol Jeong
2,† and
Buongo Chun
1,*
1
Graduate School of Physical Education, Myongji University, Yongin 17058, Republic of Korea
2
Department of Human Healthcare, College Affiliated with the Main Administration II, Gyeongsang National University, Jinju 52725, Republic of Korea
*
Author to whom correspondence should be addressed.
†
These authors contributed equally to this work.
Appl. Sci. 2023, 13(1), 446; https://doi.org/10.3390/app13010446
Submission received: 11 December 2022 / Revised: 22 December 2022 / Accepted: 27 December 2022 / Published: 29 December 2022
(This article belongs to the Special Issue Effects of Physical Training on Exercise Performance)
Browse Figures
Versions Notes

Abstract

:
Objective: To compare and rank the effects of different training interventions on the sprint and change of direction (COD) abilities of soccer players using a network meta-analysis. Methods: The PubMed, Cochrane, and Web of Science databases were searched for papers published up to June 2022. Inclusion criteria: (1) No distinction was made between nationality, region, or gender. No distinction was made between physical activity times or habits. Healthy and disease-free soccer players (age ≥ 18 years old) were eligible. (2) Different training methods and their combinations were used. (3) Groups of either no training or single training, or combined training were included. (4) Randomized controlled trials (RCTs) were included. (5) The outcome indicators included at least one of the following: sprint, agility, and change of direction (COD). Exclusion criteria: (1) studies in non-English were not included. (2) Individual studies, general public studies, literature review studies, qualitative studies, case studies, and studies with unclear data such as means and standard deviations were not included. (3) Studies using the same data were not included. (4) Interventions that could not be statistically analyzed because of insufficient numbers of studies were not included. RCTs that satisfied the inclusion criteria were included. Paired analyses and network meta-analyses were performed using random-effects models. The included studies were assessed using the Cochrane risk-of-bias assessment tool. The surface under the cumulative ranking curve (SUCRA) metric was used to rank the effectiveness of each treatment and identify the best treatment. Results: The network meta-analysis (NMA) included eleven RCTs with a total of two-hundred and seventy-seven participants and six interventions: plyometrics combined with sprint, agility, and resistance training (P+T+S+A), plyometrics combined with sprint and agility training (P+S+A), resisted sprinting combined with agility training (RS+A), plyometric training (P), resistance training (T), soccer skills, and a strategy training control group (C). P+T+S+A ranked highest in terms of improving soccer players’ 10 m sprint performance (SUCRA = 70.2%) and COD (SUCRA = 75.0%). P+S+A ranked highest in terms of improving soccer players’ 20 m sprint performance (SUCRA = 69.8%). Conclusions: Based on the network meta-analysis, for combined training, P+T+S+A was more effective at improving the sprinting and COD ability of soccer players. In the single training mode, plyometric training was the most effective. To improve sprint and COD ability, P+T+S+A should be chosen. Such improvements may be expected after P+T+S+A interventions for a duration of six or more weeks. However, gender, competitive level, and other factors will affect the assessment results. Given the limitations of the above analysis, these results should be interpreted with caution.

1. Introduction

Soccer is the world’s most popular sport and is unique, with more than 270 million participants [1,2]. Soccer players need to perform a lot of repetitive straight sprinting, direction changing, jumping, turning, and other actions whilst playing the game [3,4]. Among them, straight sprinting is the most common action. From analyzing the movements of scoring players in soccer games, we know that straight sprinting and agility account for 51% of the total number of movements [5,6,7]. Compared to previous seasons, high-intensity running distance and actions have increased by 30% and 50%, respectively [8]. Sprinting and agility can be used to distinguish the level of an athlete, as has been shown in the studies in [4,9].
Soccer players’ abilities such as sprinting and COD can be assessed through a variety of physical fitness tests (i.e., the 505 test and the linear speed 10 m test). These tests are inexpensive, easy to perform, and popular among coaches [10]. It should be highlighted here that agility is used to describe a “rapid whole-body movement with a change of velocity or direction in response to a stimulus”, whereas COD is used to describe a pre-planned testing environment, whereby athletes change their movement direction and velocity without the inclusion of perceptual–cognitive processes [11].
To improve soccer players’ performances in areas such as sprinting and agility, RCTs of different training interventions have been conducted. Good results have been obtained in strength training, plyometric training, speed–agility–quickness (SAQ) training, sled sprint training, and compound training [12,13,14,15,16]. In a previous systematic review and meta-analysis, the effect of strength training on the sprinting and jumping performance of professional soccer players was assessed [17]. Other studies have considered potential gender differences and different responses between men and women [8,18] and the effects of plyometric training on physical performance such as sprinting in male and female soccer players [19,20].
To find out which training method is the best, the effects of strength training and plyometric training on the sprinting, change of direction, and jumping performances of female soccer players were compared, and the effect of plyometric training was better than that of strength training [21]. When comparing the effects of sled sprint and vest-resistance sprint training on the sprint performance of young soccer players, sled sprint training was better [22]. In a comparison of the effects of weightlifting resistance training and plyometric training on players’ speed abilities, the magnitude of the improvement was similar [23]. Moreover, for short sprint performance, Plisk et al. [24] divided training modes into three levels based on task specificity and primary (free sprint), secondary (resistance sprint, downhill assist sprint, etc.), and tertiary (strength or plyometric training, etc.) training. However, there was no consistent conclusion as to how to improve soccer players’ sprinting and agility abilities or which training method was the best. As far as we know, there have been no network meta-analyses comparing more than two training interventions.
NMA is a relatively novel analysis method, which is different from a traditional meta-analysis. It combines the evidence of direct and indirect comparisons into a single analysis to compare the effects of two or more interventions [25]. It can be used to rank different interventions to inform the selection of an optimal training intervention [25].
Therefore, the purpose of this study was to perform a network meta-analysis to compare the effects of various training interventions using the results of previous RCTs and to derive the best interventions that relevant practitioners can refer to in selecting the most effective training. The research hypothesis was as follows: (1) combined training is more effective than single training, and (2) including more types of exercise makes training more effective.

2. Materials and Methods

Our research program was registered on Prospero, the international register of expectations for system evaluation, with the registration number CRD42022349079.

2.1. Search Strategy

The electronic databases searched included PubMed, Cochrane, and Web of Science from their inception to June 2022. The search strategy was constructed around the PICOS tool. (P) Subjects: healthy soccer players without disease. (I) Intervention: training. (C) Experimental group: no training or single training, or combined training. (O) Outcome measures: sprint, agility, and COD. (S) Experimental design: randomized controlled trial. The detailed search strategy is shown in Table 1. The search process is shown in Figure 1.

2.2. Inclusion Criteria

No distinction was made between nationality, region, or gender. No distinction was made between physical activity times or habits. Healthy and disease-free soccer players (age ≥ 18 years old) were eligible. (2) Different training methods and their combinations, including plyometric training, rapid stretching, compound training, etc., and in combination with each other, were used. (3) No training or single training, or combined training were used. (4) Randomized controlled trials were used. (5) The outcome indicators included at least one of the following: sprint, agility, and COD.

2.3. Exclusion Criteria

Studies in non-English were excluded. (2) Individual studies, general public studies, literature review studies, qualitative studies, case studies, and studies with unclear data such as means and standard deviations were excluded. (3) Studies using the same data were excluded. (4) Interventions that could not be statistically analyzed because of insufficient numbers of studies were excluded.

2.4. Study Selection

Studies were screened and excluded using the management software Endnote20. Two researchers (Dong and Chun) first used the Endnote software to deduplicate the articles. Secondly, by reading the titles and abstracts of the articles, the included and excluded articles were determined. Finally, we read the full texts and made further decisions about inclusion. If there was an objection, it was discussed and resolved by a third researcher (Jeong).

2.5. Data Extraction

We extracted data from original articles based on research, including (1) author; (2) sample size; (3) age; (4) publication year; (5) testing metrics; (6) training period, training frequency, and volume; (7) intervention measures; and (8) competitive level, i.e., professionals (the first division of their countries or above, Greek Super league, Champions League, Norwegian Premier League, Brazilian First Division, Moroccan First Division, Swedish First Division, Irish First Division etc.), and sub-level, i.e., semi-professionals [17].

2.6. Risk of Bias of Individual Studies

The risk of bias (ROB) was assessed using the Cochrane Handbook version 5.1.0 tool for assessing the ROB in RCTs. Seven aspects were considered: (1) random sequence generation; (2) allocation concealment; (3) the blinding of participants and personnel; (4) the blinding of outcome assessments; (5) incomplete outcome data; (6) selective reporting; and (7) other biases. Based on the number of components in the experiment that may have had a high ROB, there were three levels of ROB: high risk (five or more), intermediate risk (three or four), and low risk (two or fewer) [26].

2.7. Data Analysis

In studies where exercise was the intervention, all variables were continuous and were analyzed using standard mean difference (SMD) with a 95% confidence interval (CI) [27]. There were potential differences between the studies, so we chose a random-effects model for the analysis instead of a fixed-effects model [28].
We use the Stata14.1 software and, according to the PRISMA NMA instruction manual [29], a network meta-analysis was performed based on a frequentist framework. A nodal approach was used to quantify and demonstrate the agreement between indirect and direct comparisons; if p > 0.05, the agreement was verified [30]. In the resulting network graph, each node represented a different exercise intervention and a different control condition, and the lines connecting the nodes represented direct comparisons between interventions. The size of each node and the width of the connecting lines were proportional to the number of studies [31]. The exercise interventions were ranked using the surface under the cumulative ranking curve (SUCRA) value and the mean ranking (Mean Bank); the higher the SUCRA value, the better the ranking. To check for bias due to small studies, funnel plots were generated and checked visually using symmetry criteria [32].

3. Results

3.1. Study Identification and Selection

Our literature search yielded 1668 studies, with 460 duplicates being removed automatically or manually. Reviews, theoretical research, and survey-based research were initially screened by reading the title and abstract of the literature, and 1148 items were deleted. After reading the full texts, for a variety of reasons, 11 studies were eventually included (Figure 1).

3.2. Characteristics of the Included Studies

All the studies were published from the databases’ inception to June 2022, with a total sample size of 277 people. Most of the participants were men, and the average age of the participants ranged from 18 to 24. The experimental group was given single or combined training, and the control group was given technical and tactical soccer training or strength training. The number of training weeks ranged from five to twelve. The characteristics of the included studies are shown in Table 2.

3.3. Quality Assessment of the Included Studies

The methodological quality assessments are shown in Figure 2 and Figure 3. Eleven studies mentioned randomization. Five studies explained the specific methods related to allocation concealment. Nine studies were considered to have a high risk of performance bias. The risk of attrition bias and reporting bias was low in all eleven RCTs. Therefore, seven studies were at moderate risk of bias and four were considered to be at low risk of bias.

3.4. Network Meta-Analysis

A total of 11 studies were included to analyze the sprinting ability and agility of soccer players. Of these, seven studies examined the effect on 10 m sprints, four studies examined the effect on 20 m sprints, and seven studies examined the effect on COD. The full NMA figures are shown in Figure 4, Figure 5, Figure 6 and Figure 7.

3.4.1. Results for 10 m Sprint

The results of the NMA showed that, compared with the control group, group 4 (SMD = −0.51, 95% CI: −1.10–0.08), group 3 (SMD = −0.49, 95% CI: −1.49–0.50), and group 2 (SMD = −0.32, 95% CI: −1.06–0.43) were better than the control group at improving the 10 m sprint ability, as shown in Figure 4C. Group 4 ranked first for improving the 10 m sprinting ability with different training methods (SUCRA = 70.2%, as shown in Table 3).

3.4.2. Results for 20 m Sprint

The results of the NMA showed that, compared with the control group, group 5 (SMD = −1.02, 95% CI: −2.02–0.02), group 4 (SMD = −0.92, 95% CI: −1.86–0.01), 3 (SMD = −0.76, 95% CI: −2.29–0.77), and group 2 (SMD = −0.70, 95% CI: −1.95–0.56) were better than the control group in terms of improving the 20 m sprinting ability, as shown in Figure 5C. Group 5 ranked first for improving the 20 m sprinting ability with different training methods (SUCRA = 69.8%, as shown in Table 3).

3.4.3. COD

The results of the NMA showed that, compared with the control group, group 4 (SMD = –0.77, 95% CI: –1.71–0.16), group 3 (SMD = –0.61, 95% CI: –1.64–0.41), and group 2 (SMD = –0.29, 95% CI: –1.84–1.25) were better than the control group at improving the COD ability, as shown in Figure 6C. Group 4 ranked first at improving the COD ability (SUCRA = 75.0%, as shown in Table 3).

3.5. Publication Bias Test

We constructed separate funnel plots for all the outcome measures to test for possible publication bias. The visual inspection of the funnel plots was largely symmetrical, and there may have been some degree of publication bias [43]. The Egger test results showed that p > 0.05 (10 m, p = 0.590; 20 m, p = 0.117; COD, p = 0.085) with no publication bias. The details are shown in Figure 7.

4. Discussion

In this network meta-analysis (NMA), we considered 11 RCTs and compared the relative effects of six kinds of interventions for soccer players’ sprinting and COD abilities. The results of the study showed that, in terms of combined training, P+T+S+A had the best effect on soccer players’ 10 m sprint and COD abilities. P+S+A had the best effect on soccer players’ 20 m sprint abilities. For the single training mode, plyometrics had the best effect.
First, for the single training mode, our findings supported the previous findings of Pardos et al., in that plyometric training had better effects than strength training [21]. Plyometrics take advantage of the stretch–shortening cycle (SSC), the muscle–tendon eccentric stretch, concentric contraction, and the ability to transfer this energy to the concentric push-out phase [44,45]. Simultaneously, it improves neuromuscular function (intermuscular coordination, size structure, and mechanical changes) and does not involve external resistance, providing excellent performance conversion [46,47]. Squat jumping exercises can effectively enhance muscle power output, while reverse jumping improves muscle coordination and increases the ground reaction force [48]. Second, SSC exercises enhance single-fiber contractility through the force and contraction velocity of type I, IIa and IIa/IIx fibers, increased fiber strength, contraction velocity, increased Ca sensitivity, and fiber size [49]. Finally, Jiménez et al. verified the sprint and force–velocity (F–V) correlation; some jumping exercises based on the F–V curve can provide positive adaptation, and imitating the action pattern can explain its enhanced sprinting ability [50,51,52,53]. In terms of COD, the enhancement of neuromuscular function and eccentric strength can allow an athlete to better decelerate during a change of direction [54,55]. The insufficiency of simple training intensity, the generation of residual fatigue effects, and concurrent training reduce strength training adaptability and thus affect athletes’ sprinting performance [56].
Second, in the 20 m sprint, the combined training mode (plyometric training combined with resistance or sprint training) had better benefits than the single training mode. This is a training method recommended by many coaches and researchers [57,58]. Also supporting the previous findings of Rumpf et al. [59] and Nicholson et al. [60], combining level 3 training (plyometric or strength training) with specific sprint training (free sprint, assisted sprint, etc.) is more effective for short sprints than simple training. Combined training provides sufficient stimulation to simultaneously develop the physical fitness and mechanical efficiency of the lower extremities, reducing cadence and increasing stride length [61].
It is worth noting that P+S+A in the combined training mode was better than P+T+S+A in terms of 20 m sprinting. This kind of concurrent training will reduce the adaptability of training and interfere with the training effect [56]. Thus, P+T+S+A is not as effective. At the same time, the adjustment variable will also have an impact on the training effect [62]. The results of studies on the effects of strength training on soccer players of different ages show that younger players have greater training benefits [63]. As shown in our included studies with athletes aged 18 in the study of P+S+A and athletes aged 18–24 in P+T+S+A, this is one of the reasons why P+S+A works better.
Third, the experimental and control groups that we analyzed in the study conducted tactical soccer training and a small-side game. Some studies have reported that small-side games positively affect agility and 10 m and 20 m sprints [64,65,66]. Given these points, it is considered that these aspects require future research.
Finally, the size of many effect estimates depends on indirect comparisons based on reliability, and more research on multi-arm designs should be conducted in the future.

Limitations of the Study

(1) Heterogeneity between the original studies was inevitable, and some studies were unobtainable, which affected the validity of the NMA to a certain extent. (2) Since only English articles were included, the number of previous RCT studies was insufficient. Our NMA also did not compare assisted or sled sprints, iso-inertial eccentric-overload training methods, and the SAQ training method in COD. (3) The effects of moderator variables on the assessment results were also not statistically analyzed, e.g., age and sex. (4) As the capacity to detect bias will be limited when meta-analyses are based on a limited number of small trials, the results from the analyses should be treated with considerable caution [67]. (5) Only two studies were studied with professional players, and the others were semi-professional players.

5. Conclusions

In conclusion, our study has implications for relevant practitioners and researchers such as soccer coaches who could design conditioning protocols based on our results. In our analysis, the top-ranked intervention for improving sprint and COD performance in soccer players was plyometric combined with resistance, sprint, and agility training, while the highest-performing one of the single modes was plyometric training. Therefore, if a variety of training is allowed for a coach, he/she should choose P+T+S+A to improve sprint and COD performance. However, if time does not permit this or if there is only one training choice, choosing plyometrics would be the most effective way to improve an athlete’s sprinting and COD performance. In addition, positive effects can be expected if the training period is longer than six weeks. However, given the limitations of the above analysis, these results should be interpreted with caution. In the future, more multi-armed RCTs should be performed to provide more direct evidence for the relative effectiveness of various exercise interventions.

Author Contributions

Conceptualization, K.D. and B.C.; methodology, K.D. and B.C.; software, K.D. and B.C.; validation, G.J. and K.D.; formal analysis, K.D.; resources, K.D. and G.J.; data curation, K.D.; writing—original draft preparation, K.D.; writing—review and editing, K.D., G.J. and B.C.; visualization, K.D.; supervision, G.J. and B.C.; project administration, K.D. and B.C. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable. This study was a secondary analysis of publicly available data, therefore ethical review and consent were not required.

Informed Consent Statement

Not applicable. This study was a secondary analysis of publicly available data, therefore Informed consent statement were not required.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author, [B.C.], upon reasonable request.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Spiotta, A.M.; Bartsch, A.J.; Benzel, E.C. Heading in soccer: Dangerous play? Neurosurgery 2012, 70, 1–11. [Google Scholar] [CrossRef] [PubMed]
  2. Roth, T.S.; Osbahr, D.C. Knee Injuries in Elite Level Soccer Players. Am. J. Orthop. 2018, 47, 1–16. [Google Scholar] [CrossRef] [PubMed]
  3. Arnason, A.; Sigurdsson, S.B.; Gudmundsson, A.; Holme, I.; Engebretsen, L.; Bahr, R. Physical fitness, injuries, and team performance in soccer. Med. Sci. Sports Exerc. 2004, 36, 278–285. [Google Scholar] [CrossRef]
  4. Stølen, T.; Chamari, K.; Castagna, C.; Wisløff, U. Physiology of soccer: An update. Sports Med. 2005, 35, 501–536. [Google Scholar] [CrossRef]
  5. Faude, O.; Koch, T.; Meyer, T. Straight sprinting is the most frequent action in goal situations in professional football. J. Sports Sci. 2012, 30, 625–631. [Google Scholar] [CrossRef]
  6. Haugen, T.; Tønnessen, E.; Hisdal, J.; Seiler, S. The role and development of sprinting speed in soccer. Int. J. Sports Physiol. Perform. 2014, 9, 432–441. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  7. Castagna, C.; D’Ottavio, S.; Abt, G. Activity profile of young soccer players during actual match play. J. Strength Cond. Res. 2003, 17, 775–780. [Google Scholar] [CrossRef]
  8. Barnes, C.; Archer, D.T.; Hogg, B.; Bush, M.; Bradley, P.S. The evolution of physical and technical performance parameters in the English Premier League. Int. J. Sports Med. 2014, 35, 1095–1100. [Google Scholar] [CrossRef]
  9. Wright, M.D.; Atkinson, G. Changes in Sprint-Related Outcomes During a Period of Systematic Training in a Girls’ Soccer Academy. J. Strength Cond. Res. 2019, 33, 793–800. [Google Scholar] [CrossRef] [Green Version]
  10. Čović, N.; Jelešković, E.; Alić, H.; Rađo, I.; Kafedžić, E.; Sporiš, G.; McMaster, D.T.; Milanović, Z. Reliability, Validity and Usefulness of 30-15 Intermittent Fitness Test in Female Soccer Players. Front. Physiol. 2016, 7, 510. [Google Scholar] [CrossRef] [PubMed]
  11. Sheppard, J.M.; Young, W.B. Agility literature review: Classifications, training and testing. J. Sports Sci. 2006, 24, 919–932. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  12. Milanović, Z.; Sporiš, G.; Trajković, N.; Sekulić, D.; James, N.; Vučković, G. Does SAQ training improve the speed and flexibility of young soccer players? A randomized controlled trial. Hum. Mov. Sci. 2014, 38, 197–208. [Google Scholar] [CrossRef] [PubMed]
  13. Bianchi, M.; Coratella, G.; Dello Iacono, A.; Beato, M. Comparative effects of single vs. double weekly plyometric training sessions on jump, sprint and change of directions abilities of elite youth football players. J. Sports Med. Phys. Fit. 2019, 59, 910–915. [Google Scholar] [CrossRef] [PubMed]
  14. Torres Martín, L.; Raya-González, J.; Ramirez-Campillo, R.; Chaabene, H.; Sánchez-Sánchez, J. Effects of body mass-based resistance training on measures of physical fitness and musculotendinous injury incidence and burden in U16 male soccer players. Res. Sports Med. 2021. [Google Scholar] [CrossRef] [PubMed]
  15. Sinclair, J.; Edmundson, C.J.; Metcalfe, J.; Bottoms, L.; Atkins, S.; Bentley, I. The Effects of Sprint vs. Resisted Sled-Based Training; an 8-Week in-Season Randomized Control Intervention in Elite Rugby League Players. Int. J. Environ. Res. Public Health 2021, 18, 9241. [Google Scholar] [CrossRef]
  16. Ronnestad, B.R.; Kvamme, N.H.; Sunde, A.; Raastad, T. Short-term effects of strength and plyometric training on sprint and jump performance in professional soccer players. J. Strength Cond. Res. 2008, 22, 773–780. [Google Scholar] [CrossRef]
  17. Nuñez, J.; Suarez-Arrones, L.; de Hoyo, M.; Loturco, I. Strength Training in Professional Soccer: Effects on Short-sprint and Jump Performance. Int. J. Sports Med. 2022, 43, 485–495. [Google Scholar] [CrossRef]
  18. van de Hoef, P.A.; Brauers, J.J.; van Smeden, M.; Backx, F.J.G.; Brink, M.S. The Effects of Lower-Extremity Plyometric Training on Soccer-Specific Outcomes in Adult Male Soccer Players: A Systematic Review and Meta-Analysis. Int. J. Sports Physiol. Perform. 2019. [Google Scholar] [CrossRef]
  19. Sáez de Villarreal, E.; Requena, B.; Cronin, J.B. The effects of plyometric training on sprint performance: A meta-analysis. J. Strength Cond. Res. 2012, 26, 575–584. [Google Scholar] [CrossRef] [Green Version]
  20. Ramirez-Campillo, R.; Castillo, D.; Raya-González, J.; Moran, J.; de Villarreal, E.S.; Lloyd, R.S. Effects of Plyometric Jump Training on Jump and Sprint Performance in Young Male Soccer Players: A Systematic Review and Meta-analysis. Sports Med. 2020, 50, 2125–2143. [Google Scholar] [CrossRef]
  21. Pardos-Mainer, E.; Lozano, D.; Torrontegui-Duarte, M.; Cartón-Llorente, A.; Roso-Moliner, A. Effects of Strength vs. Plyometric Training Programs on Vertical Jumping, Linear Sprint and Change of Direction Speed Performance in Female Soccer Players: A Systematic Review and Meta-Analysis. Int. J. Environ. Res. Public Health 2021, 18, 401–420. [Google Scholar] [CrossRef] [PubMed]
  22. Fernández-Galván, L.M.; Casado, A.; García-Ramos, A.; Haff, G.G. Effects of Vest and Sled Resisted Sprint Training on Sprint Performance in Young Soccer Players: A Systematic Review and Meta-analysis. J. Strength Cond. Res. 2022, 36, 2023–2034. [Google Scholar] [CrossRef] [PubMed]
  23. Morris, S.J.; Oliver, J.L.; Pedley, J.S.; Haff, G.G.; Lloyd, R.S. Comparison of Weightlifting, Traditional Resistance Training and Plyometrics on Strength, Power and Speed: A Systematic Review with Meta-Analysis. Sports Med. 2022, 52, 1533–1554. [Google Scholar] [CrossRef] [PubMed]
  24. Plisk, S. Speed, agility, and speed-endurance development. Essent. Strength Train. Cond. 2000, 4, 471–491. [Google Scholar]
  25. Rouse, B.; Chaimani, A.; Li, T. Network meta-analysis: An introduction for clinicians. Intern. Emerg. Med. 2017, 12, 103–111. [Google Scholar] [CrossRef] [Green Version]
  26. Higgins, J.P.; Altman, D.G.; Gøtzsche, P.C.; Jüni, P.; Moher, D.; Oxman, A.D.; Savovic, J.; Schulz, K.F.; Weeks, L.; Sterne, J.A. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011, 343, d5928. [Google Scholar] [CrossRef] [Green Version]
  27. Li, D.; Chen, P. Effects of Aquatic Exercise and Land-Based Exercise on Cardiorespiratory Fitness, Motor Function, Balance, and Functional Independence in Stroke Patients-A Meta-Analysis of Randomized Controlled Trials. Brain Sci. 2021, 11, 1097. [Google Scholar] [CrossRef]
  28. Jackson, D.; Riley, R.; White, I.R. Multivariate meta-analysis: Potential and promise. Stat. Med. 2011, 30, 2481–2498. [Google Scholar] [CrossRef] [Green Version]
  29. Moher, D.; Shamseer, L.; Clarke, M.; Ghersi, D.; Liberati, A.; Petticrew, M.; Shekelle, P.; Stewart, L.A. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst. Rev. 2015, 4, 1. [Google Scholar] [CrossRef] [Green Version]
  30. Salanti, G.; Ades, A.E.; Ioannidis, J.P. Graphical methods and numerical summaries for presenting results from multiple-treatment meta-analysis: An overview and tutorial. J. Clin. Epidemiol. 2011, 64, 163–171. [Google Scholar] [CrossRef]
  31. Shim, S.; Yoon, B.H.; Shin, I.S.; Bae, J.M. Network meta-analysis: Application and practice using Stata. Epidemiol. Health 2017, 39, e2017047. [Google Scholar] [CrossRef] [PubMed]
  32. Khera, R.; Murad, M.H.; Chandar, A.K.; Dulai, P.S.; Wang, Z.; Prokop, L.J.; Loomba, R.; Camilleri, M.; Singh, S. Association of Pharmacological Treatments for Obesity With Weight Loss and Adverse Events: A Systematic Review and Meta-analysis. Jama 2016, 315, 2424–2434. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  33. Faude, O.; Roth, R.; Di Giovine, D.; Zahner, L.; Donath, L. Combined strength and power training in high-level amateur football during the competitive season: A randomised-controlled trial. J. Sports Sci. 2013, 31, 1460–1467. [Google Scholar] [CrossRef] [PubMed]
  34. Rodríguez-Rosell, D.; Torres-Torrelo, J.; Franco-Márquez, F.; González-Suárez, J.M.; González-Badillo, J.J. Effects of light-load maximal lifting velocity weight training vs. combined weight training and plyometrics on sprint, vertical jump and strength performance in adult soccer players. J. Sci. Med. Sport 2017, 20, 695–699. [Google Scholar] [CrossRef] [PubMed]
  35. Pedersen, S.; Heitmann, K.A.; Sagelv, E.H.; Johansen, D.; Pettersen, S.A. Improved maximal strength is not associated with improvements in sprint time or jump height in high-level female football players: A clusterrendomized controlled trial. BMC Sports Sci. Med. Rehabil. 2019, 11, 20. [Google Scholar] [CrossRef] [Green Version]
  36. Loturco, I.; Pereira, L.A.; Kobal, R.; Maldonado, T.; Piazzi, A.F.; Bottino, A.; Kitamura, K.; Cal Abad, C.C.; de Arruda, M.; Nakamura, F.Y. Improving Sprint Performance in Soccer: Effectiveness of Jump Squat and Olympic Push Press Exercises. PLoS ONE 2016, 11, e0153958. [Google Scholar] [CrossRef]
  37. Ribeiro, J.; Teixeira, L.; Lemos, R.; Teixeira, A.S.; Moreira, V.; Silva, P.; Nakamura, F.Y. Effects of Plyometric Versus Optimum Power Load Training on Components of Physical Fitness in Young Male Soccer Players. Int. J. Sports Physiol. Perform. 2020, 15, 222–230. [Google Scholar] [CrossRef]
  38. Spineti, J.; Figueiredo, T.; Willardson, J.; Bastos de Oliveira, V.; Assis, M.; Fernandes de Oliveira, L.; Miranda, H.; Machado de Ribeiro Reis, V.M.; Simão, R. Comparison between traditional strength training and complex contrast training on soccer players. J. Sports Med. Phys. Fit. 2019, 59, 42–49. [Google Scholar] [CrossRef]
  39. Ozbar, N.; Ates, S.; Agopyan, A. The effect of 8-week plyometric training on leg power, jump and sprint performance in female soccer players. J. Strength Cond. Res. 2014, 28, 2888–2894. [Google Scholar] [CrossRef]
  40. Shalfawi, S.A.; Haugen, T.; Jakobsen, T.A.; Enoksen, E.; Tønnessen, E. The effect of combined resisted agility and repeated sprint training vs. strength training on female elite soccer players. J. Strength Cond. Res. 2013, 27, 2966–2972. [Google Scholar] [CrossRef] [Green Version]
  41. Hwang, J.; Jooyoung, K. Effect of FIFA 11+ Training Program on Soccer-Specific Physical Performance and Functional Movement in Collegiate Male Soccer Players: A Randomized Controlled Trial. Exerc. Sci. 2019, 28, 141–149. [Google Scholar] [CrossRef]
  42. Ramírez-Campillo, R.; Vergara-Pedreros, M.; Henríquez-Olguín, C.; Martínez-Salazar, C.; Alvarez, C.; Nakamura, F.Y.; De La Fuente, C.I.; Caniuqueo, A.; Alonso-Martinez, A.M.; Izquierdo, M. Effects of plyometric training on maximal-intensity exercise and endurance in male and female soccer players. J. Sports Sci. 2016, 34, 687–693. [Google Scholar] [CrossRef] [PubMed]
  43. Wallace, B.C.; Schmid, C.H.; Lau, J.; Trikalinos, T.A. Meta-Analyst: Software for meta-analysis of binary, continuous and diagnostic data. BMC Med. Res. Methodol. 2009, 9, 80. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  44. Taube, W.; Leukel, C.; Gollhofer, A. How neurons make us jump: The neural control of stretch-shortening cycle movements. Exerc. Sport Sci. Rev. 2012, 40, 106–115. [Google Scholar] [CrossRef]
  45. Komi, P.V.; Gollhofer, A. Stretch reflexes can have an important role in force enhancement during SSC exercise. J. Appl. Biomech. 1997, 13, 451–460. [Google Scholar] [CrossRef] [Green Version]
  46. Markovic, G.; Mikulic, P. Neuro-musculoskeletal and performance adaptations to lower-extremity plyometric training. Sports Med. 2010, 40, 859–895. [Google Scholar] [CrossRef]
  47. Cormie, P.; McGuigan, M.R.; Newton, R.U. Developing maximal neuromuscular power: Part 2—Training considerations for improving maximal power production. Sports Med. 2011, 41, 125–146. [Google Scholar] [CrossRef]
  48. Bobbert, M.F. Drop jumping as a training method for jumping ability. Sports Med. 1990, 9, 7–22. [Google Scholar] [CrossRef]
  49. Malisoux, L.; Francaux, M.; Nielens, H.; Theisen, D. Stretch-shortening cycle exercises: An effective training paradigm to enhance power output of human single muscle fibers. J. Appl. Physiol. 2006, 100, 771–779. [Google Scholar] [CrossRef] [Green Version]
  50. Jiménez-Reyes, P.; Garcia-Ramos, A.; Párraga-Montilla, J.A.; Morcillo-Losa, J.A.; Cuadrado-Peñafiel, V.; Castaño-Zambudio, A.; Samozino, P.; Morin, J.B. Seasonal Changes in the Sprint Acceleration Force-Velocity Profile of Elite Male Soccer Players. J. Strength Cond. Res. 2022, 36, 70–74. [Google Scholar] [CrossRef] [PubMed]
  51. Jiménez-Reyes, P.; Samozino, P.; Brughelli, M.; Morin, J.B. Effectiveness of an Individualized Training Based on Force-Velocity Profiling during Jumping. Front. Physiol. 2016, 7, 677. [Google Scholar] [CrossRef] [PubMed]
  52. Jiménez-Reyes, P.; Samozino, P.; Morin, J.B. Optimized training for jumping performance using the force-velocity imbalance: Individual adaptation kinetics. PLoS ONE 2019, 14, e0216681. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  53. Mero, A.; Komi, P.V.; Gregor, R.J. Biomechanics of sprint running. A review. Sports Med. 1992, 13, 376–392. [Google Scholar] [CrossRef] [PubMed]
  54. Jones, P.A.; Thomas, C.; Dos’Santos, T.; McMahon, J.J.; Graham-Smith, P. The Role of Eccentric Strength in 180° Turns in Female Soccer Players. Sports 2017, 5, 42. [Google Scholar] [CrossRef] [Green Version]
  55. Falch, H.N.; Rædergård, H.G.; van den Tillaar, R. Association of strength and plyometric exercises with change of direction performances. PLoS ONE 2020, 15, e0238580. [Google Scholar] [CrossRef] [PubMed]
  56. Souza, E.O.D. Molecular adaptations to concurrent strength and endurance training. In Concurrent Aerobic and Strength Training; Springer: Berlin/Heidelberg, Germany, 2019; pp. 99–123. [Google Scholar]
  57. Brearley, S.; Bishop, C. Transfer of training: How specific should we be? Strength Cond. J. 2019, 41, 97–109. [Google Scholar] [CrossRef]
  58. Healy, R.; Kenny, I.C.; Harrison, A.J. Resistance Training Practices of Sprint Coaches. J. Strength Cond. Res. 2021, 35, 1939–1948. [Google Scholar] [CrossRef]
  59. Rumpf, M.C.; Lockie, R.G.; Cronin, J.B.; Jalilvand, F. Effect of Different Sprint Training Methods on Sprint Performance over Various Distances: A Brief Review. J. Strength Cond. Res. 2016, 30, 1767–1785. [Google Scholar] [CrossRef]
  60. Nicholson, B.; Dinsdale, A.; Jones, B.; Till, K. Authors’ Reply to Wewege et al.: Comment on: “The training of short distance sprint performance in football code athletes: A systematic review and meta-analysis”. Sports Med. 2021, 51, 1333–1334. [Google Scholar] [CrossRef]
  61. Lockie, R.G.; Murphy, A.J.; Schultz, A.B.; Knight, T.J.; Janse de Jonge, X.A. The effects of different speed training protocols on sprint acceleration kinematics and muscle strength and power in field sport athletes. J. Strength Cond. Res. 2012, 26, 1539–1550. [Google Scholar] [CrossRef] [Green Version]
  62. Gurevitch, J.; Koricheva, J.; Nakagawa, S.; Stewart, G. Meta-analysis and the science of research synthesis. Nature 2018, 555, 175–182. [Google Scholar] [CrossRef] [PubMed]
  63. Rodríguez-Rosell, D.; Franco-Márquez, F.; Mora-Custodio, R.; González-Badillo, J.J. Effect of High-Speed Strength Training on Physical Performance in Young Soccer Players of Different Ages. J. Strength Cond. Res. 2017, 31, 2498–2508. [Google Scholar] [CrossRef] [PubMed]
  64. Hammami, A.; Gabbett, T.J.; Slimani, M.; Bouhlel, E. Does small-sided games training improve physical fitness and team-sport-specific skills? A systematic review and meta-analysis. J. Sports Med. Phys. Fit. 2018, 58, 1446–1455. [Google Scholar] [CrossRef] [PubMed]
  65. Paul, D.J.; Gabbett, T.J.; Nassis, G.P. Agility in Team Sports: Testing, Training and Factors Affecting Performance. Sports Med. 2016, 46, 421–442. [Google Scholar] [CrossRef] [PubMed]
  66. Amani-Shalamzari, S.; Khoshghadam, E.; Donyaei, A.; Parnow, A.; Bayati, M.; Clemente, F.M. Generic vs. small-sided game training in futsal: Effects on aerobic capacity, anaerobic power and agility. Physiol. Behav. 2019, 204, 347–354. [Google Scholar] [CrossRef]
  67. Egger, M.; Davey Smith, G.; Schneider, M.; Minder, C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997, 315, 629–634. [Google Scholar] [CrossRef]
Applsci 13 00446 g001 550
Figure 1. PRISMA flowchart for inclusion and exclusion of studies.
Figure 1. PRISMA flowchart for inclusion and exclusion of studies.
Applsci 13 00446 g001
Applsci 13 00446 g002 550
Figure 2. Risk of bias graph.
Figure 2. Risk of bias graph.
Applsci 13 00446 g002
Applsci 13 00446 g003 550
Figure 3. Risk of bias summary. (Green: low risk; Yellow: unclear risk; Red: high risk) [33,34,35,36,37,38,39,40,41,42].
Figure 3. Risk of bias summary. (Green: low risk; Yellow: unclear risk; Red: high risk) [33,34,35,36,37,38,39,40,41,42].
Applsci 13 00446 g003
Applsci 13 00446 g004 550
Figure 4. (A) NMA figure for 10 m sprint, (B) SUCRA plot for 10 m sprint; (C) league table of 10 m sprints. 1 = control group; 2 = resistance training group; 3 = plyometric training group; 4 = plyometric + resistance + sprint + agility training group.
Figure 4. (A) NMA figure for 10 m sprint, (B) SUCRA plot for 10 m sprint; (C) league table of 10 m sprints. 1 = control group; 2 = resistance training group; 3 = plyometric training group; 4 = plyometric + resistance + sprint + agility training group.
Applsci 13 00446 g004
Applsci 13 00446 g005a 550Applsci 13 00446 g005b 550
Figure 5. (A) NMA figure for 20 m sprint; (B) SUCRA plot for 20 m sprint; (C) league table of 20 m sprint. 1 = control group; 2 = resistance training group; 3 = resisted sprint + agility training group; 4 = plyometric + resistance + sprint + agility training group; 5 = plyometric + sprint + agility training group.
Figure 5. (A) NMA figure for 20 m sprint; (B) SUCRA plot for 20 m sprint; (C) league table of 20 m sprint. 1 = control group; 2 = resistance training group; 3 = resisted sprint + agility training group; 4 = plyometric + resistance + sprint + agility training group; 5 = plyometric + sprint + agility training group.
Applsci 13 00446 g005aApplsci 13 00446 g005b
Applsci 13 00446 g006 550
Figure 6. (A) NMA figure for COD; (B) SUCRA plot for COD; (C) league table of COD. 1 = control group; 2 = resistance training group; 3 = plyometric training group; 4 = plyometric + resistance + sprint + agility training group.
Figure 6. (A) NMA figure for COD; (B) SUCRA plot for COD; (C) league table of COD. 1 = control group; 2 = resistance training group; 3 = plyometric training group; 4 = plyometric + resistance + sprint + agility training group.
Applsci 13 00446 g006
Applsci 13 00446 g007 550
Figure 7. Funnel plot of publication bias. (a) 10 m sprint; (b) 20 m sprint; (c) COD.
Figure 7. Funnel plot of publication bias. (a) 10 m sprint; (b) 20 m sprint; (c) COD.
Applsci 13 00446 g007
Table
Table 1. Search strategy on PubMed, Cochrane, and Web of Science.
Table 1. Search strategy on PubMed, Cochrane, and Web of Science.
PubMed#1 Athletes OR players OR player OR athlete; #2 training OR exercise; #3 soccer OR football; #4 speed OR sprint OR velocity OR agility OR performance OR change of direction OR COD; #5 randomized controlled trials (publication type)
#6: #1 AND #2 AND #3 AND #4 AND #5
Cochrane#1 Athletes OR players OR player OR athlete; #2 training OR exercise; #3 soccer OR football; #4 speed OR sprint OR velocity OR agility OR performance OR change of direction OR COD
#5: AND #1–#4
Web of Science#1 TS = (athletes OR players OR player OR athlete); #2 TS = (training OR exercise); #3 TS = (soccer OR football); #4 TS = (speed OR sprint OR velocity OR agility OR performance OR change of direction OR COD)
#5: #1 AND #2 AND #3 AND #4
Table
Table 2. Characteristics of study participants.
Table 2. Characteristics of study participants.
StudyCompetitive LevelTypeNo. of
Subjects		Age
(Years)		Exercise TypeTestDuration,
Frequency		Exercise Interventions
TotalMF
Faude et al., 2013
[33]		Semi-
professional		RCT88023.1 ± 2.7P+T+S+A10 m, COD7 weeks
2 sessions/week
30 min/session		Unilateral 4 × 90% 1RM + single-leg hurdle, jump 5 × 4 sets, rest 4 min, bilateral 50–60% 1RM, squat, sprint, etc., rest 1–2 min
88022.6 ± 2.4C
Rodríguez et al., 2017 [34]Semi-
professional		RCT10——24.5 ± 3.4P+T+S+A10 m, 20 m6 weeks
2 sessions/week		Squat 4–6 repetitions, I = 45–60% 1RM CMJ 3 sets × 5 reps, COD 3–5 × 10 s,
sprint 3–4 × 20 m, rest, 3 min
10——C
Pedersen et al., 2019 [35]Semi-
professional		RCT1901918 ± 3T10 m5 weeks
2 sessions/week		Free-barbell squats
1501519 ± 2C
Ronnestad et al., 2008 [16]ProfessionalRCT88023 ± 2P+T10 m7 weeks
2 sessions/week		Alternate-leg bound 2–4 × 8–10, double leg hurdle jump + single leg forward hop 2 × 5, etc.
66022 ± 2.5THalf squat, hip flexion, etc.
Loturco et al., 2016 [36]ProfessionalRCT99018.4 ± 1.2P10 m, 20 m, COD6 weeks
2 sessions/week		6 × 8–4, rest, 2 min, jump
880T6 × 8–4, rest, 2 min, olympic push
Ribeiro et al., 2020 [37]Semi-
professional		RCT88018.6 ± 0.52P10 m, COD7 weeks
2 sessions/1–5 week
12 sessions
35–45 min/session		Multi-direction hops 1–3 × 8, single-leg forward 2–3 × 8–12, box jump, drop jump 2–3 × 10–12, etc., rest, 2 min.
88018.4 ± 0.52TOlympic push, half squat.
Spineti et al., 2019 [38]Semi-
professional		RCT1010018.4 ± 0.4P+T+S+A10 m, 20 m, COD (93639)8 weeks
3 sessions/week
40 min/week		Plyometric hurdle hops, single-arm alternate-leg bound over barrier, diagonal jump over barrier, etc., 10 × 2, rest, 3 min.
12120TSmith half-squats, stiff-legged barbell deadlift, etc., 2–4 × 4–15 reps
Ozbar et al., 2014 [39]Semi-
professional		RCT90918.3 ± 2.6P+S+A20 m8 weeks
1 session/week
60 min/session		Standing long jump, single-leg horizontal jump, side-to-side slide and hops, etc., 3–5 × 5–12, sprint, COD, etc.
90918.0 ± 2.0C
Shalfawi et al., 2013 [40]Semi-
professional		RCT1001019.4 ± 4.4RS+A20 m10 weeks
1 session/week		6–3–9–9–6 agility, with resistance running band, repeated sprint, 4–5 × 40 m, exercise rest, 90 s, sets, rest, 10 min.
10010TLeg press, squat jump, leg extension, etc., 2–3 × 5–10 RM, rest, 2–3 min.
Hwang & Jooyoung, 2019 [41]Semi-
professional		RCT1010020.0 ± 0.6P+T+S+ACOD12 weeks
5 sessions/week
20–25 min/session		2 × 30 s squat, VJ, LJ, box jump, 2 × 30 s, agility running, etc.
1010020.1 ± 0.8C
Ramirez et al., 2016 [42]Semi-
professional		RCT1901922.4 ± 2.4P
(female)		COD
(Illinois)		6 weeks
2 sessions/week
40 min/session		Jump, reverse jump, etc., 5 sets
1901920.5 ± 2.5C
(female)
2121020.4 ± 2.8P (male)COD
(Illinois)		6 weeks
2 sessions/week
40 min/session		Jump, reverse jump, etc., 5 sets
2121020.8 ± 2.7C (male)
C = control group; A = agility training; T = resistance training; P+T = plyometric + resistance training; P+S = plyometric + sprint training; RS = resisted sprint; P+T+S+A = plyometric + resistance training + sprint + agility training. Ex. 1 = exercise 1 (908 isometric half squat exercise); s = seconds; reps = repetitions; Ex. 2 = exercise 2 (jump from the seated position); Ex. 3 = exercise 3 (single-leg jump, alternating right and left). 1/2 Squat = the degree of knee flexion was over 908; Skipping = maximal frequency on distance (10, 15, or 20 m); V. Jumps = vertical jumps; Stride J = stride jump; Sidelong = sidelong jumps; 28 triples = repeat the second movement of triple jump.
Table
Table 3. Rankings for 10 m sprint, 20 m sprint, and COD for different types of exercise.
Table 3. Rankings for 10 m sprint, 20 m sprint, and COD for different types of exercise.
Exercise10 m Sprint20 m SprintCOD
SUCRAP (%)Mean RankSUCRAP (%)Mean RankSUCRAP (%)Mean Rank
113.70.73.69.10.04.617.60.63.5
248.511.12.548.88.23.041.317.42.8
367.542.82.054.023.22.866.028.12.0
470.245.51.968.323.22.375.053.91.7
5 69.845.42.2
Note: higher SUCRA and lower mean ranks indicate better-performing treatments. P indicates the probability of it being the best treatment.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Dong, K.; Jeong, G.; Chun, B. The Effects of Different Training Interventions on Soccer Players’ Sprints and Changes of Direction: A Network Meta-Analysis of Randomized Controlled Trials. Appl. Sci. 2023, 13, 446. https://doi.org/10.3390/app13010446

AMA Style

Dong K, Jeong G, Chun B. The Effects of Different Training Interventions on Soccer Players’ Sprints and Changes of Direction: A Network Meta-Analysis of Randomized Controlled Trials. Applied Sciences. 2023; 13(1):446. https://doi.org/10.3390/app13010446

Chicago/Turabian Style

Dong, Kuan, Guyeol Jeong, and Buongo Chun. 2023. "The Effects of Different Training Interventions on Soccer Players’ Sprints and Changes of Direction: A Network Meta-Analysis of Randomized Controlled Trials" Applied Sciences 13, no. 1: 446. https://doi.org/10.3390/app13010446

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop