Next Article in Journal
Lightweight Software Architecture Evaluation for Industry: A Comprehensive Review
Next Article in Special Issue
A Multimodal Approach for Real Time Recognition of Engagement towards Adaptive Serious Games for Health
Previous Article in Journal
A High-Precision Algorithm for DOA Estimation Using a Long-Baseline Array Based on the Hearing Mechanism of the Ormia Ochracea
Previous Article in Special Issue
Can Immersive Virtual Reality Videogames Help Parkinson’s Disease Patients? A Case Study
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Augmented Reality, Serious Games and Picture Exchange Communication System for People with ASD: Systematic Literature Review and Future Directions

1
College of Computer Science and Engineering, Taibah University, Medina 42353, Saudi Arabia
2
ReDCAD Laboratory, University of Sfax, Sfax 3029, Tunisia
3
College of Community, Computer Sciences and Informatics, Taibah University, Medina 42353, Saudi Arabia
4
Department of Computer Science, Saba’a Region University, Mareb 14400, Yemen
5
Department of Information Technology, College of Computer, Qassim University, Buraydah 51452, Saudi Arabia
6
Department of Computer Science, College of Applied Sciences, Taiz University, Taiz 6803, Yemen
7
LaTICE Research Laboratory, University of Tunis, Tunis 1938, Tunisia
*
Author to whom correspondence should be addressed.
Sensors 2022, 22(3), 1250; https://doi.org/10.3390/s22031250
Submission received: 14 December 2021 / Revised: 1 February 2022 / Accepted: 3 February 2022 / Published: 7 February 2022
(This article belongs to the Special Issue Sensors in Serious Games for Health)

Abstract

:
For people with Autism Spectrum Disorder (ASD), using technological tools, such as augmented reality (AR) and serious games remain a new and unexplored option. To attract people with ASD who have communicative, social, emotional and attention deficit disorders to behavioral treatments, an attractive environment is needed that ensures continuity during treatment. The aim of the current work is to efficiently examine systematic reviews and relevant primary studies on ASD solutions from 2015 to 2020, particularly those using the traditional Picture Exchange Communication System (PECS), the application of augmented reality and those that propose serious games, thereby providing an overview of existing evidence and to identify strategies for future research. Five databases were searched for keywords that may be included within the broad Autism Spectrum Disorder ‘ASD’ umbrella term, alongside ‘augmented reality’, ‘serious games’ and ‘PECS’. We screened 1799 titles and abstracts, read, and retained 12 reviews and 43 studies. The studies scrutinized in our systematic review were examined to answer four primary and four sub-research questions, which we formulated to better understand general trends in the use of approaches for attracting people with ASD to behavioral therapies. Additionally, our systematic review also presents ongoing issues in this area of research and suggests promising future research directions. Our review is useful to researchers in this field as it facilitates the comparison of existing studies with work currently being conducted, based on the availability of a wide range of studies in three different areas (AR, SG and PECS).

1. Introduction

Autism Spectrum Disorder (ASD) is a neurodevelopment condition that affects many abilities including social, verbal and physical skills. It typically arises in early childhood, before the age of three years [1]. The World Health Organization stated that, based on studies conducted in the past 50 years, it is expected that there will be an increase in the prevalence of ASD globally, with estimation that one in every 160 children will have an ASD. Researchers believe that there are three possible means for investigation the condition’s prevalence, i.e., by providing diagnostic tools, improvement in diagnostic criteria and improved awareness for autism spectrum disorders. Through diagnosis, the severity of autism spectrum disorders levels (low, medium, high) based on individual verbal intelligence quotient and the degree of language delay [2], which can evaluate the extent to which daily life is negatively affected. In some instances involving those with ASD, individuals may appear ‘normal’ to a stranger; however, once they interact with the ASD individual, they may begin to observe differences [1]. Many who have ASD are shy in terms of communicating or panic when the need arises to have a simple conversation. As such from poor communication, it can be difficult for the person with ASD to be left alone because it will negatively affect society. Since those with ASD are part of society, they also need to communicate and express their feelings. If a person with ASD receives treatment intervention early on in childhood, they will be more likely to lead an independent life in adulthood.
Despite the limitations they experience regarding in-person communication and social skills when relating to others, people with ASD have shown a penchant for technology [3]. Given the repetitive and prolonged nature of behavioral therapies for people with autism, technological interventions allow for visual and auditory environments (stimuli) this ensures the patience of people with autism with its recurrence. Moreover, the use of technology in behavioral therapy for people with ASD has the advantage of being cost-effective, both in terms of treatment center costs and caregiver costs [4]. Studies have also shown that people with ASD are better at working with visual stimuli compared with other sensory stimuli [5]. Based on the findings of these studies, technology as digital behavioral therapy has been applied in a number of ways.
Augmented reality (AR) offers behavioral therapy in a way that enriches the experiences and skills of people with ASD, in addition to creating an integrated learning environment that provides for the possibility of three-dimensional (3D) visualization of learning content and the manipulation of physical objects [6]. Augmented reality enhances the imagination of ASD sufferers without negatively affecting it by creating ‘physical’ creatures to enhance selected targeted skills [7,8]. Moreover, using augmented reality, more interactive and attractive interfaces can be provided, removing the need for using traditional peripherals like the keyboard and the mouse [9].
One of the concepts integrated with digital behavior therapy technology is that of ‘serious games’. Serious games are used to measure the level of comprehension of an idea through a gaming environment to inspire motivation and commitment among people with ASD. Serious games provide many sensory stimuli, and studies have shown that it is possible to provide the attraction factor for participation through a framework or application that provides additional animation and images [10]. The technology provides a stimulating environment in which to effect traditional approaches, such as the Picture Exchange Communication System® (PECS) methodology. The traditional PECS method employs physical cards, which work as a Basis of Verbal and social skills acquisition. Typically, parents begin to practice PECS with their children at the age of 5 years; however, PECS-based technology allows for this practice to start at a younger age. In this regard, graphic interfaces provide a better solution compared with the traditional approach, as it is simple and easy to navigate.
In this survey, we review the above areas in detail and, as such, this paper contributes to addressing the gaps and restrictions that exist in each of these fields. The essence of this review is to provide information about the latest studies conducted from 2015 to 2020 in the following categories: (1) studies that use an AR environment for assisting people with ASD in all skills; particularly, we focus on communication and social skills and take advantage of solutions attached to other skills; (2) studies that employ serious games and present specific standards that are used to develop skills (3); studies that include PECS applications and knowledge of how to apply the methodology with it six stages.
The rest of the paper is organized as follows. Section 1 reviews existing surveys related to field of research and the contributions in this survey. Section 2 provides background for each of the fields included in an AR environment, as well as serious games and PECS applications. Section 3 details the research method used in this Sys-tematic Literature Review (SLR). Section 4 presents the results of the reviewed studies. Section 5 presents a discussion on how to use the reviewed studies via the primary and sub-research questions. Section 6 summarizes ongoing issues and future directions. Finally, the paper is concluded in Section 7.

1.1. Related Surveys

Caring for children with ASD has recently attracted broad interest in various fields, leading to researchers conducting numerous surveys on this topic. We identified selected areas in this context that were relevant to our survey as follows. Surveys related to AR combined with ASD; surveys that show the application of serious games with ASD; surveys that show applicatons used in PECS with ASD; recent surveys that bring all of these domains together with ASD. These surveys included the presentation of strategies and tools used in the specific fields in addition to additional research that focused on presenting the applications and systems used in treating ASD. Selected related surveys focused on the skills to be developed among children with ASD, followed by a comparison between these surveys with ours to identify the primary contributions of our work on this topic [11]. Revealed that most of the current AR applications that have been developed used four types of tools or strategies. Augmented reality applications rely on video modelling for visual cues, specific training and feedback, as well as final reinforcement components. The authors of [12] reviewed works targeting social interaction, emotion recognition, attention skills and functional learning using AR technology. The study found that AR techniques had a positive and effective impact on improving a range of areas, most notably social interaction, social communication skills, verbal/nonverbal communication, procedures for recognizing facial expression, attention skills and the career interests of children and adolescents with autism. The contributions of this survey appear to indicate the effectiveness of incorporating elements of AR in interventions aimed at improving the different skills of children and adolescents with ASD.
The survey conducted in [13] presents a review of recent studies related to the use of AR for children and adolescents with autism as a means for learning a variety of skills. In addition, a research classification was proposed to evaluate AR technology among children and adolescents with autism. This assessment focused on nine aspects, i.e., learning skills, characteristics, symptoms, technology, research design/study type, data collection methods, settings, evaluation parameters and evaluation. The presented survey discussed in [14] is presented in two parts. The first addressed the basic characteristics and design criteria required for AR systems that aid in the diagnosis and treatment of children with autism. The challenges associated with this type of AR system were also reviewed. In the second part, the evaluations in the first part were used to develop a new AR system, as the system included most of the standards required to design an effective AR system for children with ASD. This system aims to diagnose early autism in children by measuring their upper limb movements, and it consists of two main components, an augmented reality game and a program to record movements. Using augmented reality, children are stimulated to simulate a virtual object to move their hands, and a Microsoft Kinect sensor records all the children’s movements for analysis. The analysis process relies on the support vector machine (SVM) and the extreme learning machine (ELM), through which the diagnosis of autism is reached. In [15] authors provide a survey of evidence-based AR empirical solutions developed for ASD that focused on improving social communication skills in 14 related studies. During which the authors concluded that the effectiveness of augmented reality in teaching social communication skills to an individual with autism is not possible, due to the heterogeneity of the participants and the great diversity of skills targeted in the relevant studies despite the positive effect in the use of AR in education for people with autism. In [16], an overview of the use of AR technology is presented from a patient’s perspective. By identifying the strengths and weaknesses of the methods used in 16 current studies. This study focused on reviewing many diverse technologies such as smart glasses, mobile devices and video display systems, and focused on abilities such as social skills, the ability to communicate, attention and cognitive skills. In [17], about augmented reality in autism treatment there is two aspects were identified that were found to be common to almost all current studies on ASD: (1) the high communication and cognitive performance of people with ASD; (2) experimental designs aimed at improving social skills, language and relationships. Adnan et al. [18] focused on the satisfaction and confidence aspect of using AR applications among children with ASD.
The study found that using a visual approach in AR applications for training social interaction skills increased the enthusiasm and passion of ASD children. In [19], the purposes, limitations and effectiveness of AR and the quality and results of evaluation methods were analyzed among 10 recently published studies. These studies were divided into many categories based on content, social skills, emotion recognition and other factors. A central issue in the survey presented in [20] was determining which activities would be possible and most effective for applying the use of AR to develop techniques for ASD children and how to go about doing so. The authors note that few of studies recently published using different strategies (such as 3D cards) were directly related to approving treatments for ASD children.
On the other hand, for using the concept of serious games with ASD, three surveys were included in this area. In [21], the following meta-analyses revealed that limitations in applying the principles of serious game design may be computer-based socio-emotional interventions. Furthermore, a serious game assessment tool was developed to implement serious game principles when designing game-based interventions targeting ASD individuals. Studies such as [22] on the effectiveness of serious games for people with ASD deal with social skills, rather than targeting other adaptive behavior skills, since the field of serious game design has been successful in addressing many adaptive behavior skills and intellectual performance. Finally, through the survey presented in [23] measuring the effectiveness of information and communication technology and the use of serious games to treat people with ASD, authors Of the current study concluded that serious games incorporating ASD that are currently available have some limitations in terms of evidence of their clinical benefit.
In relation to PECS’s applications with ASD, as well as surveys that integrate the fields of AR, serious games and PECS’s technology with ASD, we found no surveys during the research process. Addressing these contexts represents one of the main contributions of the current survey. To identify all contributions, Table 1 provides a summary of previously published work and lists their limitations to identify the main contributions of the current survey.

1.2. Main Contribution of the Presented Work

Based on the previous review of the existing surveys, the current research presents the following primary contributions.
(1)
Conducted a systematic and accurate survey of studies applying the PECS approach. Our goal is to understand the way of using PECS with ASD children.
(2)
In contrast to existing surveys that focused on the number of targeted skills, as all previous surveys have demonstrated in related studies, and which primarily addressed social and communication skills, the current survey focused on improving and developing existing solutions. So we will try to review existing solutions using AR, SG and PECS with ASD children.
(3)
Aimed to find common factors in the field of AR environments, serious games and PECS’s applications, and to illustrate how combining these fields can contribute to the existing body of research.
(4)
Aimed to establish technical solutions for addressing different personality traits by defining specific criteria for people with ASD, as existing surveys did not consider solutions in terms of these differences and instead combined individuals with ASD with other groups (e.g., those with intellectual disabilities).
Table 1. Related surveys to the augmented reality environment and serious games with people with ASD.
Table 1. Related surveys to the augmented reality environment and serious games with people with ASD.
Augmented Reality with ASD
TitleAuthorJournalsYearCitationTimelineLiterature TypeNum Selected StudiesLimitation
Technology-Assisted Intervention for Children with Autism Spectrum Disorder using Augmented Reality [11]Suparjoh, Suriawati
Shahbodin, Faaizah
Ku, Che
Che, Nuraini
Mohd, Ku
International Journal of Recent Technology and Engineering202002012 to 2018systematic literature review13This SLR represents a presentation of previous studies without the opinions or criticism of researchers for these studies
Exploring the Impact of Augmented Reality in Children and Adolescents with Autism Spectrum Disorder: A Systematic Review [12]Berenguer, Carmen
Baixauli, Inmaculada
Gómez, Soledad
Andrés, María de El Puig
De Stasio, Simona
International Journal of Environmental Research and Public Health202002010 to 2020systematic literature
review
20
  • The low number of high-quality designs that have been implemented in studies as they are limited to communications and/or posters at international conferences and experimental studies.
  • The characteristics of the sample, as it contained samples from children and adolescents with high-performance autism and did not take into account the greater heterogeneity of the disorder in order to generalize the results.
  • A lack of longitudinal investigations to find out whether these techniques might really help children with autism improve social interactions. of augmented reality are considered.
Augmented reality for learning of children and adolescents with autism spectrum disorder (ASD): A systematic review [13]Khowaja, Kamran
Banire, Bilikis
Al-Thani, Dena
Sqalli, Mohammed Tahri Aqle, Aboubakr Shah, Asadullah Salim, Siti Salwah
IEEE Access2020
42005 to 2018systematic literature
review
30------------------------------
The use of augmented reality in the diagnosis and treatment of autistic children: a review and a new system [14]
Wedyan, Mohammad
AL-Jumaily, Adel
Dorgham, Osama
Multimedia Tools and Applications20201n.d.literature
review
24-------------------------------
Use of augmented reality for social communication skills in children and adolescents with autism spectrum disorder (ASD): A systematic review [15]
Khowaja, Kamran
Al-Thani, Dena
Banire, Bilikis
Salim, Siti Salwah
Shah, Asadullah
ICETAS 2019–2019 6th IEEE International Conference on Engineering, Technologies and Applied Sciences201912005 to 2018systematic literature
review
141-The need for a longitudinal study. Also, researchers need to determine the number and duration of sessions.
Using Augmented Reality in Patients with Autism: A Systematic Review [16]
Marto, Anabela
Almeida, Henrique A.
Gonçalves, Alexandrino
Lecture Notes in Computational Vision and Biomechanics20195n.d.systematic literature
review
161-The effect of individual differences in the study.2-Small sample sizes from 1 participant, up to 12.
New Technologies and Autism: Can Augmented Reality (Ar) Increase the Motivation in Children With Autism [17]Rega, Angelo
Mennitto, Andrea
Vita, Salvatore
Iovino, Luigi
INTED2018 Proceedings20189n.d.systematic literature
review
14------------------------------
Systematic review on augmented reality application for autism children [18]Adnan, Nur Hidayah
Ahmad, Ibrahim
Abdullasim, Nazreen
Journal of Advanced Research in Dynamical and Control Systems201802012 to 2018systematic literature
review
51-The effect of generalising potential for improvement remains to be examined using systematic intervention methods over a longer period.2-Engaging autism to explore the mechanism of augmented reality.
The Application of Augmented Reality for Intervention to People with Autism Spectrum Disorders [19]
Karamanoli, MSc PersefoniIOSR Journal of Mobile Computing & Application201712012 to 2016Literature Review10The methodology followed is imprecise and not enough clear
Augmented Reality and the Use of Alternative Communication for Children with Autism Spectrum Disorder: A Literature Review [24]Marcelo Marcio Soares(&) and Aline da Silva Oliveira Nevesspringer202002013 and 2018systematic literature
review
5few studies are studied and analysed
Serious Games with ASD
TitleAuthorJournalsYearcitationTimelineLiterature typeNum selected studiesLimitation
A systematic review and meta-analysis of social emotional computer based interventions for autistic individuals using the serious game framework [21]
Tang, Julia S.Y.
Chen, Nigel T.M.
Falkmer, Marita
Bölte, Sven
Girdler, Sonya
Research in Autism Spectrum Disorders201941990 to 2018systematic review and meta-analysis17----------------------------
Studying the effects of computer serious games on people with intellectual disabilities or autism spectrum disorder: A systematic literature review [22]
Tsikinas, Stavros
Xinogalos, Stelios
Journal of Computer Assisted Learning2019222005 to 2018systematic literature review58-------------------------------
Serious games to teach social interactions and emotions to individuals with autism spectrum disorders (ASD) [23] Grossard, Charline
Grynspan, Ouriel
Serret, Sylvie
Jouen, Anne Lise
Bailly, Kevin
Cohen, David
Computers and Education2017642001 to 2014systematic literature
review
31-------------------------------

2. Background

2.1. Autism Spectrum Disorder (ASD)

There are many definitions of ASD, all of which canter on the notion that ASD can be described in the simplest terms as impairment in social communication that develops early during childhood and is caused by a heterogeneous set of neurodevelopmental conditions, such as repetitive sensory and motor behaviours that are associated with a strong genetic component [25,26,27,28]. Autism can be defined in terms of the pathological anatomy of the cerebral cortex. It involves only a slight disturbance in the basic and diamond radial organization of neurons and glia in this particular area of the brain [29]. Studies have expressed it as a slight disorder that is extremely variable in dendritic orientation and in the decrease in size and spacing between the small radial columns of neurons in different cortical regions, including the frontal lobe [30,31]. As shown in Figure 1, small for some mandibular tracts (indicated by the red lines) and anterior temporal joints (indicated by the yellow lines) and are responsible for stabilizing human contact. However, the severity and size of the disorder may differ between people; accordingly, different types of autism present with different characteristics as shown in Table 2 [18].
This survey focuses on all autism types except Asperger’s syndrome and pervasive developmental disorder not otherwise because these represent severe disorders that require complex clinical procedures.
Although there is no drug treatment for autism spectrum disorder, early treatment can make a significant difference in the lives of children with autism [32]. Following the diagnostic process by a specialist, a treatment method will depend on determining the severity of the disorder and the child’s age. Typically, the earlier treatment can be started, the better the results will be. There are many therapeutic approaches that ASD children can practice at home, in school and in specialized clinics. Specific types of treatment methods can be presented as follows: behavior and communication treatments, applied behavior analysis, educational therapies, family therapies, speech therapy, occupational therapy, social skills classes, hippo therapy and using the PECS [33].
Table 2. Describe the types of autism [18].
Table 2. Describe the types of autism [18].
Type of AutismDescribe
Autistic disorderAbnormal or impaired development of social interaction and a limited repertoire of activities and interests, usually noticed during the first years of life
Asperger’s syndromeSevere and persistent impairment of social interaction and development of patterns of behavior, interests, and restricted and repetitive activities. No clinically significant delays in language acquisition
Rett’s disorderA very specific and distinct pattern of stunted growth after a period of normal functioning during the first five months after birth. Diagnosed only in females
Childhood disintegrativeDecline in multiple areas of function after a period of at least two years of apparent normal development
Pervasive developmental disorder not otherwiseSevere and pervasive impairment in the development of social reciprocal interaction, but criteria for a specific pervasive developmental disorder are not met.

2.2. The Picture Exchange Communication System

As previously noted, one of the methods of treating ASD is the PECS. The PECS system was developed by Andy Bundy and Laurie Frost in 1994 [34,35]. It is a communication system that mainly depends on exchanging images to express basic needs and communicate with others [36,37]. The PECS system targets children with a nonverbal autism spectrum disorder. It works using the exchange of well-known signals, images and symbols to enable effective communication as one of the Augmentative and Alternative Communication (AAC) strategies [38]. The PECS approach has the advantage of being a system supported with outdoor equipment at low technical costs because it relies only on painted drawings and recording the images in these drawings [39] as shown in Figure 2.
Furthermore, PECS provides more tangible reinforcement, as children receive tangible objects in its first stage, which teaches how to order first, which in turn enhances dealing with the components of the surrounding environment. Furthermore, PECS is distinguished by the fact that it does not require a child to have existing communication skills to learn about communication. Research by Bundy and Frost demonstrated that 59% of ASD children who were trained using PECS spontaneously developed independent speech [40].
The PECS approach comprises six stages. In the first stage, a picture is exchanged to discover how to deal and obtain the required item through pictures and this process develops until it reaches the sixth stage, in which simple questions and mutual comments are answered [41]. Table 3 briefly explains each of these stages. The PECS approach functions as per guidelines established by [34].
Table 3. Stages of PECS.
Table 3. Stages of PECS.
Stage NameDescriptionOutcome
  • How to Communicate
Children learn to exchange pictures of objects or activities they desire.Training to take the picture from the table and put the image in the hands of the communication partner.
2.
Distance and Persistence
Children learn to generalize this new skill if they continue to use individual images in different places and with different individuals. They are also taught to be more persistent and persistent in communication.He goes to the profile, grabs the photo, goes to the communication partner, gets attention, and leaves the picture in hand.
3.
Picture Discrimination
Children learn to choose between two or more pictures to order the things they want.To ask for the things he wants by moving to the communication book and choosing the appropriate image from among several images
4.
Sentence Structure
Children learn to attach a simple sentence to tape using the picture “I want” followed by a picture of the required thing.To request existing and non-existent things using multi-word phrases by navigating to the communication file and taking a picture (I want) + (picture of the desired thing) and placing it on the sentence bar and taking the sentence bar from the communication file and directing it to the communication partner and giving it to him
5.
Answering Questions
Children learn to use PECS to answer the question, “What do you want?”To automatically ask for many things and to answer the question (What do you want?)
6.
Commenting
Children are now learning how to comment in response to questions such as: “What do you see?”, “What do you hear?” And “What is this?” They also learn how to construct sentences starting with “I see”, “hear”, “feel”, “it”, and so on.To answer (What do you want?) (What do you see?) (What do you hear?) (What is this?) And automatically ask and comment on events he sees.

2.3. Serious Games

Other concepts that can be incorporated into ASD treatment is that of serious games. Although several different definitions can be applied, “serious games” involve many different types including training simulations, computer games and sports; serious games can be defined as digital games with several purposes, such as providing an experience of multimedia, as opposed to simply serving as entertainment [42]. They can also be defined as games that are designed with a primary purpose other than entertainment. Moreover, some [43] posit that the term “serious” in serious games refers to messages or input that is part of in-game content aimed at providing knowledge or training skills, i.e., providing the player with knowledge or experience management in a suitable environment [43]. To identify that serious games involve applications comprising experience, entertainment and multimedia, through the previous definition, it is clear that serious games are the result of the merging of three concepts, which are training simulation, computer games, and sports. As shown in Figure 3. Serious games can be combined with the latest technologies to contribute to increasing their benefits and purpose.
The rapid development of serious games is expected to increase in the future [44,45]. To ensure the success of serious games and to apply them to their full extent, several criteria that contribute to their benefits must be considered [43]. Five of these criteria include activity, modality, interaction style, and environment and application area. ‘Activity’ refers to determining the type of activity for which a game was designed. Many types of activity, including physical activity, target health and movement activities such as recreational games [46,47] and games aimed at obese children [48]. Another type of activity is physiological, which can be used in rehabilitation games [49] and to diagnose health conditions [50]. The activity can also be mental, i.e., one that is used to learn [51,52] and provide training in different fields. The ‘modality’ criterion refers to the senses that are used to transfer information from a device to the game participants. The most common senses in this regard are sight, hearing and touch, with recent attempts to include the olfactory sense [53]. To use serious games to their full extent, attempts have been made to integrate a wide range of senses in their mechanics, e.g., enabling participants to mix music or engage in tactile experiences [54]. The ‘interaction’ criterion refers to the specific input used within the game environment to enable participant interaction; these include traditional inputs, such as a keyboard, mouse and joystick, as well as more modern and smart inputs, such as enabling tactile experiences (touch) and tracking eye gaze and movement using sensors. Correctly selecting entries together with game requirements increases the success and effectiveness of serious games. One example that provides a good description of the input used for interaction was explored in [55] (Whitehead et al. 2010), i.e., the Sensor Network for Active Play system. This example involved a game aimed at effecting physical activity by connecting various sensors to the feet and arms of a game user.
The ‘environment’ criterion refers to the type of digital environment used in a game. Different types of serious game environments can be combined. The environment may be a two-dimensional (2D) or 3D game, or a combination of both. Alternatively, the game environment can employ an augmented, virtual or mixed reality. These types will be defined in more detail in the next section. The AR or virtual reality is widely used in serious games such as those defined in [54]. Mixed reality is also discussed in [48]. This criterion addresses whether the game requires permissions to locate the player. It also determines whether the game is portable. In addition, the criterion also determines whether play is affected online, in addition to the number of players in the game and whether it is individual or collective as discussed in [47]. The fifth and final criterion that must be determined in the design of serious games is the ‘application area’, which refers to serious games’ target areas for application. These include education and advertising, which dominate serious games by up to 57% according to [56], while the remaining percentage relates to less frequently addressed areas such as healthcare, rehabilitation, communication between people, luxury, tourism, antiquities and others. Figure 4 briefly illustrates the primary criteria for serious games.

2.4. Augmented Reality

One of the most important environments that can be used in serious games is known as AR. In the 1960s, AR prototypes were invented by Evan Sutherland and his students, in which code was used to display 3D graphics. Augmented reality was first recognized as an emerging technology in 2007 [57]. The technology involves 3D interactive recorded data that combines real and virtual objects [58]. Figure 5 (as presented by [59]) illustrates that AR is part of a connected chain that extends between real and virtual environments and includes AR and augmented virtual reality (AVR), where AR is closer to the real world and AVR is closer to the purely virtual environment. Two main types of AR can be defined as follows:
  • Location-based AR, which depends on devices or mobile phones that have a Global Positioning System (GPS) to display digital media as discussed in [60];
  • Visual-based AR, which depends on the user’s phone camera orientation to the object in the digital environment [61,62].
These two types of AR must, however, be able to preserve the shape of virtually created elements that merge with elements in reality, thereby rendering them compatible with the real environment. Furthermore, the performance of the virtual environment should be effective in real-time [63,64].
Augmented and AVR are often confused because while they are almost the same, there are subtle differences between them. In an AR, the user reference diagram is linked to reality but additional technical elements are added, making it appear as if these items are default aspects of reality. In an AVR environment, the user reference diagram is linked to a completely virtual world, and the player uses a virtual environment to engage in a particular experience [63,65,66]. Table 4 shows the most important differences between AR and AVR [14,67,68].
Augmented reality operates within a framework that includes three main phases. In the first stage, tracking, recording, display technologies and processing processes take place in real-time. This stage is important for creating the virtual object. The second stage includes interaction devices and procedures, in addition to presentation and configuration processes. The third stage is the application stage and includes the user interface. Figure 6 illustrates the different components of AR [69,70].

3. Review Methodology

This survey follows the methodology of a systematic review to examine current research work. A comprehensive view of recent proposals for rehabilitating children with ASD is provided and serves as a means for examining and evaluating current studies relevant to the topic to provide comprehensive information to the research community in several areas. In addition, this survey delivers a new proposition by developing the limitations of the current studies. This survey was conducted by following the recommended guidelines accordingly [71]. The survey process comprised three phases, i.e., planning, conducting and reporting. Figure 7 summarizes the steps required for a systematic review [72]. Details of these steps are explained in the following sections.

3.1. Planning the Review

3.1.1. The Need for a Systematic Review

Although many applications have been proposed in the field of rehabilitation for children with ASD, many technical interventions that contribute to increasing rehabilitation effectiveness are still required. This study reviewed several ways in which to develop rehabilitation for children with ASD by integrating approaches into three domains, i.e., augmented reality, serious games and the PECS approach. These areas were previously individually suggested in the reviewed literature; the current survey broadens the discussion and investigates integrating the suggested areas according to the following research questions [72]. Figure 8 shows the current relationship between these fields, considering the lack of research that combines them.

3.1.2. Defining the Research Questions

To achieve the main objectives of this study, the PICOC method [72] was used to define the primary and sub-research questions. The main research questions are as follows.
(1)
MRQ1: What are the effects of applying developed AR in the communication and social skills of people with ASD?
(2)
MRQ2: What is the effect of applying the standards of serious games developed for people with ASD?
(3)
MRQ3: How effective are the developed PECS applications for people with ASD?
(4)
MRQ4: Can the application of the PECS be developed using AR technology? Additionally, the following sub-research questions are proposed.
  • SRQ1: Can developing AR serious games attract people with ASD?
  • SRQ2: Can all the phases of PECS be implemented in a mobile application?
  • SRQ3: How long can people with ASD continue being attracted to the application of PECS?
  • SRQ4: How much can we benefit from previously reviewed papers that target people with ASD?

3.1.3. Review Protocol Development

The protocol development process contributes to the process of identifying the methods used in the systematic review components (review background, MRQ and SRQ development, search strategy, inclusion and exclusion criteria, data extraction and data synthesis).
The result of the review protocol development process aimed to reduce study bias and distinguish between SLR and traditional methods of reviewing the literature [73,74,75].

3.2. Conducting the Review

Search strategy: As shown in Figure 9, the search strategy aimed to extract data from the relevant research papers. Accordingly, a search strategy must be developed to ensure that scholars obtain the largest possible number of relevant studies. To ensure that additional academic publications are included, manual and automatic research methods were used, in addition to examining the content included in the review. An automatic search was used to find preliminary studies related to people undergoing ASD rehabilitation. The automatic search of library databases via the Internet was conducted based on search keywords. The following online databases (with their accompanying website link) were included in the search strategy:
The proposed steps in [13] were followed to derive the search terms as follows.
Seeking keywords using the main and sub-research questions, four main keywords/phrases were identified (autism spectrum disorder, augmented Reality, PECS, serious games).
(1)
Alternate spellings, synonyms and abbreviations for each keyword were defined.
(2)
The keywords used in the relevant searches were defined.
(3)
Similar keywords were defined and divided into four categories for each of the relevant proposed domains so that the common category for all domains was ‘autism spectrum disorder’.
(4)
Used the OR logical operator to combine keywords in each category.
(5)
Using the AND logic operator by combining keywords across categories.
(6)
Each of the search rules differed in terms of filtering searches; Table 5 shows the specifically applied filtering and the particular query formed from the keywords.
The search was restricted to the period 2015 to 2020 and found a good amount of literature within the three different fields. The total number of studies related to an AR environment was 569; 752 studies were found in the field of serious games and 478 were derived in the field of PECS applications. All of the documents identified were research articles. Next, all library databases were manually searched using the pre-defined keywords. Details of the overall search are shown based on the keywords identified in the libraries (identified in Figure 10).

3.3. Criteria for Including and Excluding Articles

From a search of five databases, 1799 related studies were found. Next, incorrect quotes screening reduced studies related by title, keywords, abstract and conclusion to 172 studies. Next, 31 duplicates were removed to reduce the related studies to 141. Using the forward and backward technique, the number of studies was further reduced to 84. First, the title of each study was considered. Next, the content of each primary study was reviewed starting with the abstract. The primary studies incapable of addressing one or more of the RQs related to this SLR were excluded from the list of related studies. Next, an analysis of the remaining studies was carried out based on the selection criteria. Then, inclusion and exclusion criteria (see Table 6) were applied as part of the selection criteria to reduce the number of included studies. The collective selection process reduced the related studies to 43. The references section of each remaining study was then manually checked to identify additional related studies but did not yield additional new and related studies. Finally, a qualitative assessment of the primary studies was performed; no additional new and related studies were found. The qualitative assessment of related studies is described below.
Table 6. Criteria for inclusion and exclusion of the articles.
Table 6. Criteria for inclusion and exclusion of the articles.
Inclusion CriteriaExclusion Criteria
Studies are written in English onlyStudies for which the full text is not available
Studies have been published from 2015 to 2020Duplicate studies
Studies type is Journal articles and conference papers.Studies that do not directly aim to help people with autism spectrum disorder only (without other disabilities).
All methodology except (Review, Survey)Studies involving VR and MR in studies related to augmented reality with ASD.
Studies that provide answers to the main research question or the sub-research questionsStudies that include all types of game-based learning and do not specialize in serious game.
Studies not related to the applications of PECS
Manual Search
The citations to related studies were traced using the forward and backward technique in [76,77], and the included databases were searched to find the studies cited in the selected studies. Among others, the Mendeley tool was used to sort and manage all studies and to remove duplicates. Using the manual search step, a systematic review of the research was confirmed as complete and relatively comprehensive.
Selection of Studies
As shown, using the steps for effecting automatic searches, 141 related studies were found. In the manual search, all articles were checked using the Mendeley tool. Duplicate articles were removed by checking the titles of each article. As a result, a total of 121 articles were obtained. The focus then moved to reviewing the summary of each article, which subsequently included 98 articles. In the final step, these articles were examined by focusing on their content, reducing the overall number to 43. Figure 11 shows the stages for selecting articles, based on checking the inclusion and exclusion criteria.
Quality Assessment of Primary Studies
Quality assessment (QA) was performed for each study to measure the quality of the content presented. A set of quality assessment questions was created to determine the suitability of the related studies.
(1)
QA1: Are the research goals/objectives clearly defined?
(2)
QA2: Is the solution based on AR/serious games/PECS clearly defined?
(3)
QA3: Is the research methodology specified in the article?
(4)
QA4: Are the search results reported?
The above four QA criteria were tested for the 43 included research papers to determine their reliability. The QA comprised three quality schema stages, i.e., high, medium and low, in which the quality of each paper relied on its loading score. For example, papers that satisfied the criteria were awarded a score of 2, papers that partially satisfied the criteria were awarded a score of 1 and papers that did not provide any information regarding the question and did not satisfy the criteria were awarded a score of 0. Consequently, based on the four QA defined criteria, studies with a score of 6 or above were considered to be of high quality, studies with a score of 5 or above were considered to be of medium quality, and studies with a score below 4 were considered to be of low quality. Table 7 presents the QA list for each study.
Table 7. Qualitative assessment of the related studies.
Table 7. Qualitative assessment of the related studies.
Augmented Reality
Paper #IDArticle_IDQA1QA2QA3QA4Score
[78]A_122228
[79]A_222228
[80]A_322228
[81]A_422105
[82]A_522228
[83]A_611114
[84]A_722228
[85]A_822228
[86]A_922228
[87]A_1022015
[88]A_1122026
[89]A_1221115
[90]A_1322217
[91]A_1421227
[92]A_1521126
[93]A_1621126
[94]A_1722228
[95]A_1822127
[96]A_1922105
[97]A_2021227
[98]A_2121227
[99]A_2222228
Serious games
Paper #IDS_IDQA1QA2QA3QA4Score
[100]S_121126
[101]S_222105
[102]S_322217
[103]S_421227
[104]S_521227
[105]S_621126
[106]S_720226
[107]S_822127
[108]S_922217
[109]S_1022228
[110]S_1121227
[111]S_1222222
PECS
Paper #IDP_IDQA1QA2QA3QA4Score
[112]P_121227
[113]P_221227
[114]P_321115
[115]P_422228
[116]P_521115
[117]P_621227
[118]P_722228
[119]P_822228
[120]P_920226
Data Extraction and Synthesis
A set of guidelines described in [121] and related to the data extraction process was followed to determine the relevant information from the related studies The data extraction method was designed to accurately record all information for each study in a summary. Features were determined considering all the research questions related to the systematic review.

4. Relevant Research

4.1. Combining an Augmented Reality Environment with Autism Spectrum Syndrome

A summary of related studies in the AR field, based on publication year, study type, country, participants, evaluation time, intervention, research strategy, methodology, device type, skills targeted and limitations are presented in Table 8.
Figure 12 shows the number of studies published between 2015 and 2020. As is evident, a number of studies were recently published due to the growing interest in AR technology. This interest was reflected in the announcements of large companies supporting AR technology in their products (e.g., Apple).
Among the related studies, 36% were published in journals, while the largest percentage was published as part of conference proceedings (64%). We note that the percentage of studies published as part of conferences was greater than the percentage of studies published in journals. We anticipated this to have been due to limitations that will be discussed in detail in the section on ongoing issues, which could be bypassed and published in conferences.
Through deeper analysis of the relevant studies, the following results emerged.
  • Figure 13 shows the number of included studies per country. The bulk of these works were from Taiwan (six published studies), followed by Malaysia, Brazil, Qatar, Portugal and the United States (one study per country); 11 studies did not clarify their country of publication. This indicates that researchers in Taiwan are interested in integrating AR in the rehabilitation of people with ASD and may also be attributed to the increased prevalence of autism. Based on a World Health Organization (https://www.who.int/news-room/fact-sheets/detail/autism-spectrum-disorders, accessed on 13 December 2021) report, the prevalence of autism worldwide is between 1 and 6. This prevalence is rated between 2.8% and 4.1% in Taiwan [122].
Table 8. Related work of using AR with ASD.
Table 8. Related work of using AR with ASD.
A_IDYearCountryEvaluation TimeInterventionResearch StrategyMethodologyDevice TypeSkills TargetedLimitations
A_1 [78]2016Taiwan4 weeksVideo-modeling storybookExperimentQualitativeTabletsFacial expressionsA bias in choosing participants in the experiment.
A_2 [79]2018n.d.n.d.n.d.ExperimentQualitativeMobile devicesSocial communication skills and cognitive skillsThey received all 4 applications and their interactions with AR applications were during one day.
There are no sound effects.
A_3 [80]2020n.d.8 weeksSocial story and sequence learning approachExperimentQualitativeTabletsSocial communication skillsTo use the system requires physical community storybooks and script cards
A_4 [81]2020Taiwan5 weeksVisual framework of the concept map (cm)ExperimentQualitativeComputersSocial communication skillsThere are several physical setup for the training system
A_5 [81]2020Qatarn.d.n.d.ExperimentQualitativeMobile devicescognitive skillsYou need a caregiver’s intervention in determining the level of a game based on the individual’s need and level.
A_6 [83]2020n.d.7 daysKinect sensorExperimentQualitativeComputersn.d.
A_7 [84]2015Portugaln.d.Alternative communication (pecs concept in some stages) and applied behaviour analysis strategies ExperimentQualitativeComputersSocial communication skillsThe system needs the correct cards to work
A_8 [85]2019n.d.n.d.Optical see-through (ost)Case study QualitativeHead-mounted devicesFacial expressionsOccluding real-world objects is not one of the strengths of ost ar displays.
The design works better in indoor settings without strong light coming through the headset.
Users with different pupil distances might experience differences in the depths of the rendering in 3d space.
A_9 [86]2015Taiwan1 month and 2 weeks A self-facial modeling learning system ExperimentQualitativeComputersFacial expressionsIt’s a complicated process while creating 3d characters.
The number of participants is small
A_10 [87]2019Taiwann.d.Key partial video with action
Auto organizational menu (aom)
ExperimentQualitativeSmartphonesSocial communication skillsThe number of participants is small.
The results are unclear
The complexity of scanning shapes to represent them with ar
A_11 [88]2018n.d.n.d.BooksCase studyQualitativen.d.cognitive skillsThe ambiguity of the research methodology
It did not clarify what is new in it
A_12 [89]2020n.d.n.d.Deep convolutional neural networksExperimentQualitativeSmartphonesFacial expressionsIt is impractical in the communication process as users will need to place their phone cameras and point them towards whomever they are talking to.
Participant details and duration of the experiment were not shown.
A_13 [90]2019n.d.n.d.Discrete trial teaching (dtt) approachCase studyQualitativeSmartglassesDaily living skillsThe system has not yet been evaluated to verify the extent to which autistic children acquire certain skills after using the system.
A_14 [91]2020Taiwan18-weekn.d.ExperimentQualitativeSmartphoneFacial expressionThey do not support independence and require a physical therapist for interaction and encouragement processes
A_15 [92]2018n.d.n.d.n.d.Case studyQualitativeMobile devicescognitive skillsThere are many technical glitches that make the use of the application restricted with some restrictions
A_16 [93]2018Usa3-weekFace2face moduleCase studyQualitativeSmartglassesSocial communication skillsThe face2face unit is not a precisely described unit that contains a series of levels and difficulty settings
A_17 [94]2018TaiwanFour months and two weeksConcept map (cm) strategyExperimentQualitativeTabletsSocial communication skillsGestures and facial expressions are not designed for avatar
A_18 [95]2018n.d.n.d.StorybookLeap motion controllerExperimentQualitativeComputersSocial communication skillsUsing computers that require many other devices to read the marker story
A_19 [96]2020Brazil3 yearsPecs conceptExperimentQualitativeMobile devicescognitive skills No numbers are showing the results of the trial during the intervention phase.
A_20 [97]2019Malaysian.d.Phonetic reading systemExperimentQualitativeMobile devicescognitive skillsDifficulty scanning the ar marker card due to the vibration of the mobile device or the light
A_21 [98]2020n.d.n.d.Kinect skeletal tracking (kst) systemExperimentQualitativeComputersSocial communication skillsAnimations are animated by coaches in 3d contextual backgrounds on the screen.
A_22 [99]2020n.d.n.d.n.d.ExperimentQualitativeMobile devicescognitive skillsThe system was built without the involvement of the autism specialist or caregivers
  • The devices used in AR systems for the rehabilitation of ASD patients were limited. Two studies used smart glasses; six studies used computers (the most common device used) in rehabilitation centres and 12 studies used mobile devices (smartphones, tablets and laptops). Three studies that used smartphones were obtained and another three used a tablet device. We found only one study that used a head-mounted device; one study did not mention the type of device used.
  • Figure 14 shows the various skills targeted in the related studies; these can be classified as follows. The largest number of studies (eight) targeted social communication skills, e.g., the effectiveness of asking for help and expressing abilities. Five studies involved facial expression skills and focused on six expressions (anger, fear, disgust, happiness, sadness and surprise). Six studies included cognitive skills, e.g., academic skills (arithmetic, geography, biology, numbers and letters), as well as attention skills, mental representations, focus skills and the recognition of objects. Daily living skills were represented in only one study. One study did not reveal specifically addressed skills.

4.2. Serious Games with Autism Spectrum Disorder

Table 9 presents a summary of studies related to the field of serious games. It is presented in terms of year of publication, study type, country, participants, evaluation time, intervention, research strategy, methodology, device type, targeted skills, activity, modality, interaction style, environment and limitations.
Figure 15 shows the number of studies published between 2015 and 2020. Only a small number was published during the past five years; the highest number was in 2016 (at a rate of three studies) and in 2019, this decreased to one study. We expect that this had been due to a lack of game designers interested in the field of ASD rehabilitation.
The percentage of conference publications (67%) was larger compared with journal articles (33%). Regarding the number of studies per country, six countries (Canada, the United Kingdom, Egypt, the Philippines and Serbia) each published one study. There were six studies for which the country of publication was not noted.
Table 9. The related studies of serious games with ASD.
Table 9. The related studies of serious games with ASD.
A_IDYearCountryEvaluation TimeResearch StrategyMethodologyDevice
Type
Targeted
Skills
ActivityModalityInteraction Style
EnvironmentLimitations
S_1 [100]2020n.d.n.d.case studyQuantityn.d.cognitive SkillsMentalVisual, AuditoryTangible interfaces
2DInterpreting and describing all possible reactions in a child ASD is difficult
S_2 [101]2017Canada2 daysExperimentQualitativecomputerSocial and communication skillsMentalVisual, Auditory, HapticTangible interfaces2DThe complexity of the proposed system building tools and needs news to operate.
The experiment was only performed on males
S_3 [102]2018UKn.d.ExperimentQualitativelaptopcognitive SkillsPhysical exertion,
Mental
Visual, AuditoryKinect and mouse2D,
multi-player games
Cessation and absence of some participants from the experiment.
S_4 [103]2015Germanyn.d.ExperimentQualitativeMobile devicesFacial expressions SkillsMentalVisual, AuditoryTangible interfaces2DThere are no motivational phrases when he finishes the stages to encourage the child
S_5 [105]2020Egyptn.d.ExperimentQualitativecomputercognitive SkillsMentalVisual, AuditoryKeyboarded2D/3DThe difficulty is configurable to make it more suitable for every child.
S_6 [123]2016n.d.n.d.ExperimentQualitativemobile deviceFacial expressions SkillsMentalVisual, AuditoryTangible interfaces2DIt depends on the presence of an official dealing with the program
S_7 [106]2016Philippinesn.d.ExperimentQualitativemobile devicecognitive SkillsMentalVisual, AuditoryTangible interfacesn.d.Lacks the side of interaction and attraction and does not benefit from the standards of serious games
S_8 [107]2019n.d.n.d.ExperimentQualitativeTabletsSocial and communication skillsMentalVisual, AuditoryTangible interfaces2DLimiting the time in the fourth, fifth, and sixth stages may cause the child to become distracted.
S_9 [108]2017n.d.n.d.ExperimentQualitativecomputercognitive SkillsMentalVisual, Auditorykeyboard and a mouse3DControl of the keyboard is difficult for children because there are many commands (such as space to jump, mouse to move the camera, etc.)
S_10 [109]2016Serbia2 weeksExperimentQualitativecomputerMotor and cognitive SkillsPhysical exertion,
Mental
Visual, AuditoryKinect2DThere are no specific periods in the game that the child loses focus
S_11 [110]2018n.d.n.d.ExperimentQualitativecomputercognitive SkillsMentalVisual, Auditoryn.d.2DThe serious game concept has not been used in an interestingly and attractively way to the target age group
S_12 [111]2015n.d.n.d.case studyQualitativemobile deviceEmotional SkillsMentalVisual, AuditoryTangible interfacesARThe number of attempts to reach the correct answer is open and may affect the results of the child’s evaluation
Devices used in serious games for ASD rehabilitation were small in number; five studies included computer use and six included mobile devices (smartphones, tablets and laptops). One study using a tablet device and one employing a laptop were obtained. Figure 16, shows the variety of skills targeted in related serious games studies. The largest number of studies targeted cognitive skills (six), which included skills such as understanding, academic skills, financial concept skills and computational thinking skills.
This was followed by two studies that involved facial expression skills; two additional studies included social and communication skills. Only one study included emotional skills, motor and cognitive skills.
We previously noted that the activity criterion included three types, i.e., mental, physical and physiological activity. However, physiological activity was not included in the current paper because it targets and treats autistic individuals with impaired movement, as well as visual and auditory sensitivity. All 12 related studies targeted mental activities, in contrast to physical activities, which were implicit in two studies with mental activities; no study focused only on physical activities.
Studies related to serious games included three styles: tangible interfaces, keyboard and/or mouse and the Microsoft Kinect system. Tangible interfaces were used at a rate of 58% and included physical or digital interfaces. Representing a traditional interaction style, a keyboard and/or mouse was used in up to 25% of studies. A relatively new style involved using the Kinect (17%). Figure 17 shows an unexpected consequence of the environments used in serious game studies. Despite the variety of environments in serious games, they were limited to 2D aspects in eight studies. Conversely, 3D, 2D/3D, AR and multi-player games were each represented by one study (excluding studies that included VR).

4.3. Applying the Picture Exchange Communication System with Autism Spectrum Disorder

Table 10 summarizes the results of the study on the application of PECS in terms of publication year, type of study, country, participants, and the number of phases applied, evaluation time, device type, intervention, search strategy, methodology and limitations. Only a few studies on the applications of PECS were published (nine in total); the highest value was found in 2018 (four) and in 2019 this decreased to two studies.
Unlike the fields of AR and serious games, 67% of related studies were published in journals, while conference papers represented 33%. We note that the percentage of studies published in journals was almost twice that of conferences papers.
The PECS studies included three countries, i.e., Taiwan (one study), Indonesia (two) and South America (one). Five studies did not denote a country of origin.
When considering studies related to the number of applied PECS phases, we found that they were limited to the first four phases (One study exception). Figure 18 shows that four studies applied four phases and three studies used three phases. Only one study used all six phases and another only applied the concept of PECS.
Table 10. The related studies of PECS applications with ASD.
Table 10. The related studies of PECS applications with ASD.
A_IDYearTypeCountryParticipantsNum_PhasesEvaluation TimeDevice TypeInterventionResearch StrategyMethodologyLimitations
P_1 [112]2015JournalTaiwan113four-weekTabletsChild-computer interactionExperimentQualitativeThe study was based on the concept of PECS, not the application of stages
P_2 [113]2016Journaln.d.n.d.3n.d.Mobile devicesAugmented realityquasi-experimentalQualitativeConducted on children between the ages of 6 and 11 years.
Is long-term use of mobile phones leads to further isolate the child?
The evaluation time is short.
P_3 [114]2018Journaln.d.46n.d.Mobile devicesAugmented realityCase studyQualitativeUsing virtual reality instead of augmented reality as the interfaces illustrate this.
P_4 [115]2018JournalIndonesia124n.d.smart phonesAugmented realityexperimentalqualitativeUse the traditional PIECE image.
The time for an experiment is not clear.
P_5 [116]2018JournalIndonesian.d.4n.d.n.d.n.d.quasi-experimentalQuantitativeLacks expressions of encouragement and attraction to the child
P_6 [117]2019ConferenceSouth America93n.d.Mobile devicesTEACCH
Methodology
And Autisdata software
experimentalqualitativeI lack attraction and encouragement in the app
P_7 [118]2018Journaln.d.n.d.4n.d.Mobile devicesn.d.experimentalQualitativeThe pull-out process can be complicated for a child
The images used are not interactive
P_8 [119]2019Conferencen.d.44n.d.Mobile devicesGoal-Directed Design (GDD) methodexperimentalQualitativeThe time is long in long sentences, which causes boredom for children
P_9 [120]2017Conferencen.d.n.d.n.d.n.d.Mobile devicesAided Learning (CAL), and Human-Computer Interaction (HCI),
Mobile Instant Messaging (MIM)
experimentalQualitativeOnly the PECS concept was used and not all stages were implemented
The reason for the most studies being limited to the first four phases was because phases five and six of PECS require people with ASD to answer a set of questions and talk about their feelings; this is not provided for in PECS’s applications.
We found PECS applications to be limited to mobile devices (100%). One study focused on the use of smartphones and one other on using a tablet; six studies all included mobile devices. One study made no mention of the device type.
Among the important results in the techniques and approaches for delivering intervention through the applications of PECS, we found that AR had been used in three studies. The remaining studies were more diversified and included many different technologies and approaches, such as child–computer interaction, the Treatment and Education of Autistic and Related Communication Handicapped Children methodology (TEACCH ®) and Autisdata software, the goal-directed design method, aided learning, human–computer interaction and mobile instant messaging.

5. Discussion

The main findings related to each RQ and SRQ of the current study is briefly described below.
  • MRQ1: What are the effects of applying developed AR to developing the communication and social skills of ASD children?
As previously noted, the studies that targeted social and communication skills included eight in all [A_2], [A_3], [A_4], [A_7], [A_10], [A_16], [A_18] and [A_21]. All results indicated that the use of AR technology had a positive effect on the progress of speech and language therapy in particular, as well as the social skills and communication in general for people with ASD. Moreover, the experience of AR technology was not limited to its impact on social skills and communication; other positive effects such as increased interest in the education process through participation, enjoyment and motivation were also indicated. The educational processes for people with ASD require long-term continuity; therefore, AR technology provides a very enjoyable and stimulating learning environment with a wide variety of sources for learning activities, which can help to avoid students becoming bored quickly. In addition, AR technology was able to deepen an understanding of the relationships between social situations and the different behaviours of individuals with ASD; in this context, it was observed that AR enhanced their social cues. Moreover, through exercises, role-playing games and interactive games using augmented reality, social skills were significantly affected.
Despite its positive effects, however, there are limitations to the use of AR in the rehabilitation of people with ASD. Scanning signs or objects using camera capturing are not designed in a manner that considers the capabilities of people with ASD. These signs and objects must be in a specific position, have good lighting and must be at a reasonable distance. For AR characters, these were represented by three types in the reviewed studies, i.e., cartoons, avatars and emojis. However, in most studies reviewed in this current research, the gestures and facial expressions of the characters were not considered. Only one study created an avatar; in this case, the faces of the participants were captured by taking several photos at different angles ([A_9]); however, this will be a complicated process if the number of participants is large.
  • MRQ2: What is the effect of applying the standards of the serious game developed for children of ASD?
Through their findings, studies relating to the efficient use of serious gaming standards in many learning practices enable their implicit understanding. These findings indicate a positive effect in terms of enhancing the motor skills and interactive functions of people with ASD, particularly children. The results also showed the effectiveness of serious games in terms of improving the language skills of children before and after using the game. Moreover, more interactive games motivated children to work towards common goals. Importantly, serious gaming standards were able to enhance the visual support factor, which must be taken into consideration when working on the rehabilitation of people with ASD. However, studies related to this context were limited to implementing traditional standards and these were not properly applied. Only one study among all those described, employed the AR environment ([S_12]), despite its positive impact on educational activities. Due to the low level of diverse environments in serious games studies, a lack of standard interaction patterns emerged because the environments were limited to the traditional mouse and keyboard interaction methods, as well as tangible interfaces.
  • MRQ3: How effective are the developed PECS applications for ASD children?
The effect of PECS applications was effective and positive in all respects. Studies showed that using the benefits of PECS applications effectively improved communication among ASD individuals with communication problems (verbal/social communication). The reasons for improvement included reducing the burden of carrying many traditional cards to make it easier to manage them; PECS applications apply a large range of symbols and images alongside the feature of adding sound effects to teach pronunciation and motivate people with ASD to learn. Moreover, PECS applications alleviated the difficulties faced by caregivers, such as reducing verbal repetition and accessing cards faster, thereby overcoming spatial and time constraints and enabling ASD progress monitoring without indirectly interfering at any time and any place after the completion of the session. Despite the many advantages we have mentioned, the related studies were small in number and several presented limitations. The applications lacked interaction and verbal communication (applied in phases five and six of PECS), providing one study ([P_3]), It is suggested to implement phases 5 and 6 in a cooperative manner between two players, through simple comments, asking for help and giving opinions while playing. Moreover, studies lacked participation and interaction with the caregivers, parents or friends of those with ASD; involving these groups could potentially increase the effectiveness of treatment using PECS’s applications, particularly in advanced stages of training. Most of the studies that addressed the need for the mouth movement feature which focus on clarifying how to pronounce the names of things and the mouth movement while the word is pronounced.
  • MRQ4: Can the application of PECS be developed with AR technology?
In the [P_2], [P_3] and [P_4] studies, it emerged that it is possible to design PECS applications using AR technology. The results also indicated an improvement in the ability of verbal communication, but more studies are needed over longer periods. Another major limitation is that the PECS approach has not been effectively integrated with AR technology, as AR was included only in some stages.
(1)
SRQ1: Can developing AR serious games attract ASD children?
As we previously noted, one study included in this review was related to serious games that used an AR environment ([S_12]). This study aimed to motivate children with ASD to acquire emotional recognition skills by attracting their attention and stimulating them with AR technology. What distinguishes this study is that its approach represented fun and interesting ways for people with ASD, where the GameBook was used to represent and interact the events of the story using AR technology. In contrast, also the study is a prototype that has not been evaluated for the reliability of serious games using AR technology.
(2)
SRQ2: Can all phases of PECS be applied in an AR application
Based on studies [P_3] and [S_8], it seems possible to create a PECS application that supports all six phases of this approach. Studies demonstrated positive results for developing communication and interaction skills through the application of PECS. However, These two studies is insufficient for demonstrating the effectiveness of supporting all phases of the PECS approach in AR application, as further testing involving a larger group of participants is required. Moreover, this study employed AR technology and, as such, are prone to the same limitations that we mentioned.
(3)
SRQ3: How long can ASD children continue to be attracted to PECS’s applications?
Maintaining the attention of individuals with ASD during treatment using the PECS approach is a primary obstacle, as treatment sessions for as long as five years may be needed to gain and retain minimum verbal skills. Therefore, attracting the attention of people with ASD for the longest possible period is required. None of the studies included herein could conduct PECS evaluation sessions for long periods; the longest time reached by related studies was four weeks ([P_1]). Accordingly, many more studies that evaluate the attractiveness of using PECS applications and that measure the maximum possible period for maintaining the attention of ASD individuals when using PECS applications are needed. The attraction factor mainly depends on the technologies that are incorporated in PECS applications.
(4)
SRQ4: How much can we benefit from previously reviewed papers that target ASD children?
Through an in-depth analysis of all related studies, many aspects emerged that present several areas of potential development for future studies. These aspects are described in the following points.
  • Participants: Based on the inclusion criteria for the current survey, all studies involving individuals with ASD that present with one specific symptom (mild, moderate, or severe) and no movement or cognitive impairments, special needs, or visual and auditory sensitivity were reviewed. Participants in the studies related to this survey included children and adolescents with ASD. This a group of ASD did not show symptoms of autism in strangers; but, the situation becomes embarrassing in dealing with them despite their high ability to learn, so they are not treated equally with other groups in which the symptoms of autism are a phenomenon. The different personality traits of people with ASD called for different techniques and interventions to be implemented in their rehabilitation, as most techniques that rely on sound or visual effects may negatively affect people with ASD who have auditory/visual sensitivity. Two included studies addressed the difference in personality traits of people with ASD during the development of the proposed learning systems ([P_3] and [S_4]). By allowing the systems to have flexible configurations to suit the needs of all those with ASD, such as choosing the gender, application color, and volume of sounds in the application etc. In terms of the number of participants, this ranged in most studies from one-to-four participants. These small samples may have been due to the difficulty of dealing with large numbers of individuals with ASD, based on the emotions they may experience during an intervention. Furthermore, each patient needs to be treated independently by a person they are familiar with.
  • Device type: Different opinions were observed in the studies regarding the type of device used in the rehabilitation of people with ASD. In 11 of the related studies ([A_4], [A_6], [A_7], [A_9], [A_18], [A_21], [S_2], [S_5], [S_9], [S_10] and [S_11]) the use of traditional devices (computers) was beneficial in terms of the variety of devices that could be linked to it, e.g., adding a motion sensor or large display screens; however, it was restricted in that they could only be used in rehabilitation centres. Additionally, many of these devices were expensive and it may also be confusing for people with ASD to interact with more than one device at the same time. Furthermore, advanced technology devices require an expert to operate and manage them. The use of smart glasses presented potential benefits in two studies ([A_13] and [A_16]). The use of smart glasses was found to be less distracting and less demanding in terms of cognitive workload. This was because smart glasses do not require the use of hands, which is beneficial in terms of not distracting the user and allowing them to only focus their visual attention during the learning process. However, the side- effects of using smart glasses for a long time for people with ASD remain unknown. Moreover, smart glasses are not able to address the visual problems of ASD individuals.
Significant support for the use of mobile devices (smartphones, tablets and laptops) was evident. Smartphones were included in 17 related studies ([A_2], [A_5], [A_15], [A_19], [A_20], [A_22], [S_3], [S_4], [S_6], [S_7], [S_12], [P_2], [P_3], [P_6], [P_7], [P_8] and [P_9]). Four related studies made use of laptops ([A_10], [A_12], [A_14] and [P_4]) and tablets were employed in five related studies ([A_1], [A_3], [A_17], [S_8] and [P_1]). These devices present multiple advantages, such as their portability, the ability to use them at anytime and anywhere, as well as their low cost compared with more traditional devices, which are limited in certain places and times. Moreover, people with ASD exhibited positive reactions when using smartphones and tablets. Despite promising results, however, some studies also expressed concerns about the use of smartphones and tablets, as prolonged and unregulated use can lead to isolation among ASD individuals and reduce their awareness of the surrounding (social and physical) environment; this is particularly the case for ASD individuals who primarily experience a lack of social communication skills. Selected studies showed that smartphones increased stress and pain due to remaining in the same position for an extended time.
Head-mounted devices were used in one study ([A_8]). These represent close-up displays that provide more control for those with ASD in terms of what they can observe, unlike the screens of mobile devices, which have a limited field of view. In addition, head-mounted devices enhance a sense of fun and can present a realistic AR experience. The disadvantage of head-mounted devices appears to be their need for operating in a low-light environment.
  • Culture: East Asian culture featured predominantly in the reviewed studies. Interestingly, no established studies were found in an Arabic culture setting. One study derived from Qatar ([A_5]) and another from Egypt ([S_5]); however, these studies depended on having knowledge of the English language or translating it only into Arabic.
  • Interventions: By combining the advantages of AR studies, serious games studies and PECS application studies, regardless of the characteristics of the participants, a diversified series of research can be collected that addresses current research problems in correlated studies.
We found one serious games study that employed an AR environment ([S_12]). The study aimed to create a game that would attract children’s attention to develop social and communication skills. This approach used the software Gamebook, which depends on a graphic interface, character interactions, voice and text narration and AR technology by using a children’s book. The AR in Gamebook provides an experience within a safe and attractive environment and can enhance a child’s attention and imagination without diminishing it. Moreover, through the game’s website, the system provides a feature that is able to monitor the results of children without direct intervention, thus supporting the child’s independence. All of these features were included in this work without financial costs or the use of traditional peripheral devices such as a keyboard and mouse.
Conversely, one serious games study also targeted social and communication skills but did so using the PECS method ([S_8]). The game in this study included 15 levels, which were divided as follows. (1) Levels that achieved differentiation and recognition skills through the use of the first, second and third stages of the PECS approach. (2) Levels that achieved the skills of building sentences and communicating with others through the use of the fourth stage of the PECS approach. (3) Levels (14 and 15), in which progress in communication skills was achieved by asking for help and making comments while playing with other players, and which followed a cooperative approach that achieved the values of participation and cooperation between players.
We also found three studies that combined AR with a PECS ([P_2], [P_3] and [P_4]) approach. In [P_2], the first three phases of the PECS approach were implemented alongside an AR technology intervention. Augmented reality technology displayed an animal class as 3D objects in a scene, thereby enhancing the attraction factor for ASD individuals. In the stage that was implemented, the goal was to achieve object recognition skills by clicking on the icon for each animal. The animal that was clicked display for viewing in a 3D image. In [P_3], the 3D objects were expanded because the application was designed with seven menus, each representing a stage of the PECS system; a final list was also presented to exit the game. The first stage comprised five categories, i.e., the classes, a dining room menu, a bathroom menu, a bedroom menu and a playground menu. In each category, all the components of the selected category were displayed. At the end of each stage, a short test was presented to assess ASD children. In [P_4], the first four phases of the PECS approach were used. Here, a conventional PECS card was scanned by smartphone camera and each card was represented by 3D objects. In the related studies, AR was used with PECS applications in a traditional manner, i.e., the representation of objects in 3D form only, without interaction or having a logical sequence of events. We believe that by leveraging the standards of serious gaming and the benefits of an AR environment to develop PECS applications, the problems that emerged in the reviewed studies could be avoided. None of the included studies adopted such an approach.
  • Independence: A relationship was observed between the type of devices used in studies and support for autonomy in working on the system without the presence of the device operator for the duration of the treatment session. In studies that relied on using a computer, an expert or care provider was required to operate them or to assist in using peripheral devices. Therefore, mobile devices support independence via their ease of use. Despite this, few studies were directly concerned with independence ([A_6], [A_9], [A_22], [S_1], [S_3], [S_4], [S_8], [S_11], [P_3] and [P_6]).
  • Monitoring progress: Enabling the independence of people with ASD and monitoring their progress presents some obstacles. People with ASD cannot be left unattended until the provided treatment has been used and recorded; concurrently, stimulating the independence of those with ASD should also be a focus. Monitoring progress should be an independent process that ensures that treatment is not directly interfered with. A group of studies suggested independent systems that help caregivers and parents monitor progress, either through websites ([S_7] and [S_10]) or by including them in the application ([A_22] and [P_5]). One study was entirely based on assessing the comprehension score of a child with ASD to provide accurate educational assistance ([S_1]). This approach will help caregivers to accurately identify concepts that the child did not fully understand. Conversely, all studies manually implemented progress monitoring, which can be considered inaccurate compared with automatic progress monitoring.
  • Cooperation: Selected studies ([A_15], [S_3] and [S_8]) indicated that games that specifically reinforce cooperative behaviours can help to develop the social behaviours of ASD individuals because they enhance an understanding of others, use simple interactive dialogue (such as asking for help) and make use of eye contact. However, these studies explained that some ASD individuals may more frequently engage in collaborative activities only to affect blame and to complain. In some instances, reactions made by ASD individuals may be unexpected in the event of mistakes on behalf of one of the parties involved.

6. Ongoing Issues and Future Research Directions

In this section, we provide a summary of ongoing issues and possible future directions for research based on the reviewed studies.

6.1. Ongoing Issues

Two types of ongoing issues (general and private) emerged from the included studies. The general issues refer to common issues in all of the reviewed studies, regardless of their particular field. Private issues are those that are specific to one area among those discussed herein.

6.1.1. General Ongoing Issues

  • Several participants
The number of children with ASD who participated in all of the reviewed studies was small, ranging in most (20) studies from one-to-four participants. Interestingly, there are few studies that recruited a large number of participants, as the number of study participants was [A_19] reach to 200 participants over a period time of up to 3 years in AR studies. Hence, the included studies do not represent a strong body of evidence as the findings have not been validated by more systematic evaluations.
  • Duration of the experiment
Behavioural therapies for people with ASD require long-term skills training. These issues are among the most prominent. In most of the reviewed studies that lasted up to four weeks, it is important to note that the number of sessions had not been counted because there was no clarification on the required sufficient period for acquiring skills. As already stated, one study ([A_19]) spanned a period of three years. We believe that the reason for the short duration of the experiment in this particular study can be ascribed to the fact that many of the studies had been conducted for specific conferences; accordingly, the duration of the experiment was adapted based on the timing of these conferences.
  • Completing the experiment
Completing an experiment can be a difficult task for people with ASD. We observed in most of the studies the withdrawal of one or two participants from the experiment for several reasons. One of these reasons related to experimental difficulties, that is, either because of the complexity of the system or because of the selection of participants whose characteristics did not match the experience. Other causes were related to the mood and physical state of people with ASD (aspects such as tension, fear and hunger must be considered). Furthermore, in some studies, people with ASD did not interact with researchers because they did not feel comfortable interacting with strangers.
  • Duration of use of mobile devices
Despite the reported positive results and stimulus interaction resulting from ASD individuals using mobile devices, there is a worrying trend related to the use of these devices. Evidence in studies indicates that the use of mobile devices can cause feelings of isolation for people with ASD, which may reduce their awareness of social and physical environments, particularly for ASD individuals with reduced communication and social skills. As such, prolonged use of these devices during the treatment period may contribute to a deterioration of these skills.
  • Culture
As previously noted, most of the reviewed studies were based in East Asia, indicating a lack of other additional cultural approaches in related studies. The bulk of studies were written in the English language or propose a studies based to the translation of the English language without carrying the original culture of the country of origin for the study. The images used in these studies represent selected cultures and may thus reduce the effectiveness of the social communication process. None of the included studies focused on studying the effect of culture and AR applications on the rehabilitation of people with ASD.

6.1.2. Private Ongoing Issues

  • Augmented Reality
The main problem in studies related to AR environments is that these environments involve only a display of objects with/without cards. As such, they lack a connection to any logical sequence of events. Moreover, although applications that rely on AR environments are being developed, few of them consider the needs of people with ASD in their communication with caregivers or with other ASD individuals. Moreover, a technical issue related to displaying AR objects currently appears unclear; the cause may be due to shaking of the camera device, particularly shaking caused by ASD individuals. Another reason may be attributed to lighting. In addition, in the studies included herein, these applications lacked response speed or a sense of real interaction with ASD sufferers.
  • Serious Games
One of the most prominent issues regarding serious games was the lack of diversity in the applied standards; furthermore, these standards were not properly employed. Due to the lack of diverse environments in the studies focused on serious games, an absence of standards for the interaction type was evident because this was limited to traditional methods. Furthermore, as was the case for the development of AR applications, which are limited to being created by software developers, most studies related to serious games often developed applications that lacked a sense of fun. A reason for this may be because the games are designed for sponsor providers or researchers who did not consider using creative game designers who are better equipped to create applications that ensure a focus on fun and engaging users.
  • Picture Exchange Communication System (PECS)
The lack of applications using PECS have makes it a poorly researched topic with several obstacles. Most PECS applications were applied in a traditional manner that simulated the use of a manual card. These were primarily 2D digital displays with added sounds; as such, they lacked visual and sensory interaction. Moreover, based on this design, the application made the process of moving between images difficult, either because of the size of the image, which was often small or, alternatively, because of the process of dragging images to form sentences. As we previously noted, most of the PECS applications were limited to the first four stages, creating issues linked to reduced social interaction, which the final two stages of PECS can provide.

6.2. Future Research Directions

The information presented in the ‘Discussion’ and ‘Open Issues’ sections can assist in predicting future trends that may contribute to additional studies linked to developing the field of rehabilitation for people with ASD.
Participants
By reviewing the number of participants and the personal characteristics that must be considered in studies related to the rehabilitation of people with ASD, future studies may opt to include their number of participants. Autism centres can be contacted to carry out a systematic evaluation of up to 12 participants, both male and female, with no other diagnoses except for ASD. The age group ranging from three to 12 years of age can also be included. This age group is able to rapidly acquire communication and social skills but is the least focused on by ASD research.
Device type
We support the use of mobile devices for several reasons including their attractiveness to people with ASD. Moreover, they can be used almost anywhere without significant financial costs and supports the pattern of repetition required by behavioural therapies for people with ASD. It is possible to avoid the isolation that accompanies frequent and prolonged use of mobile phones in future studies by imposing specific periods of use, e.g., during treatment sessions in clinical centres. This can be applied by adding a feature for caregivers through which to set the time for using the application, from one hour to three as a maximum.
Culture
Studies can be enriched in the future by integrating local culture in the rehabilitation of people with ASD. We aim to create an application that represents Arabic culture through the creation of characters depicted in local dress or engaging in social customs to help ASD individuals acquire and reinforce Arab social behaviours.
Interventions
The incorporation of PECS applications in serious AR environment-based games can provide solutions that ensure that the limitations present in each of the studied areas are improved in future research. An AR environment can provide 3D characters in a simulated real environment to help ASD individuals gain social skills in an attractive and comfortable environment for people with ASD. Moreover, serious games provide a logical sequence of events by following a storytelling approach, which can guarantee aspects such as fun and suspense and implicitly enhance social skills. By carefully diversifying the standards of serious games, educational environments can be created. In future studies, we aim to apply all of the standards shown in Figure 19. In this regard, for the PECS methodology, all six stages can be applied, including a treatment sequence that will provide an implicit aspect in serious games.
Independence
Using mobile devices can enable people with ASD to be autonomous because they can use the application without assistance. In future studies, this can provide independence through the creation of simple application interfaces, alongside tips that can help guide ASD individuals during the application experience.
Monitoring progress
Relevant studies on monitoring progress can be useful for future research. A progress monitoring feature can be included in the application to allow caregivers to accurately monitor the progress of ASD individuals. A breakdown of each stage completed and the time it took the participant to complete it can be included. Charts can also be added to facilitate the monitoring of participant progress.
Cooperation
Consideration should be given to the problem of how cooperation skills can be enhanced within the target skill framework, particularly when more than one player is involved and the game is conducted over the Internet. Accordingly, in future studies, an AR avatar will be added to represent a parent or friend of the ASD individual. This will be done by adding an image of the individual and having the application create an avatar using said image. People with ASD will then be able to engage in a range of simple conversations and collaborate with the avatar to finish the skill-set target frameworks alongside a familiar character.

7. Conclusions

Augmented reality environments have recently gained significant attention in a range of different contexts. This environment can be used efficiently if it is combined with the rehabilitation of people with ASD, e.g., using serious games and PECS applications. In this paper, a rigorous SLR was conducted about these approaches using 172 research papers in three domains published from 2015 to 2020. In the final stage, a total of 43 papers were examined. Augmented reality was represented in 22 studies, serious games studies represented 12 studies and a total of nine PECS application studies were included. We began by detailing the background and relationship between these three areas. We then reviewed existing surveys related to these three areas and found no other surveys related to PECS applications. This is one of the main contributions of this survey. We highlighted the studies related to each field in detail and noted their results. We also discussed the benefits of these studies for the research community and highlighted how future studies can be informed by making use of our review findings. Finally, we provided insights about the remaining issues and future directions for research to improve the application of PECS alongside AR-based serious games.

Author Contributions

Conceptualization, H.A., M.H., R.B. and S.K.; methodology, M.H., R.B. and S.K.; validation, R.B. and S.K., formal analysis, W.A.; investigation, M.H., R.B. and S.K.; resources, H.A., M.A.-S.; data curation, M.H., S.K. and M.A.-S.; writing—original draft, H.A., M.H., R.B. and W.A.; writing—review and editing, M.H.; visualization, M.H.; supervision, R.B. and S.K.; project administration, R.B.; funding acquisition, M.H. and M.A.-S. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

The researchers would like to thank the Deanship of Scientific Research, Qassim University for funding the publication of this project.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Gambrill, E. The Diagnostic and Statistical Manual of Mental Disorders as a Major Form of Dehumanization in the Modern World. Res. Soc. Work Pract. 2014, 24, 13–36. [Google Scholar] [CrossRef]
  2. Vahia, V. Diagnostic and statistical manual of mental disorders 5: A quick glance. Indian J. Psychiatry 2013, 55, 220. [Google Scholar] [CrossRef] [PubMed]
  3. Munson, J.; Pasqual, P. Using Technology in Autism Research: The Promise and the Perils. Computer 2012, 45, 89–91. [Google Scholar] [CrossRef]
  4. Knapp, M.; Romeo, R.; Beecham, J. The Economic Consequences of Autism in the UK; Foundation for People with Learning Disabilities: London, UK, 2007; pp. 1–29. [Google Scholar]
  5. Meadan, H.; Ostrosky, M.M.; Triplett, B.; Michna, A.; Fettig, A. Using Visual Supports with Young Children with Autism Spectrum Disorder. Teach. Except. Child. 2011, 43, 28–35. [Google Scholar] [CrossRef] [Green Version]
  6. Howorth, S.K.; Rooks-Ellis, D.; Flanagan, S.; Ok, M.W. Augmented Reality Supporting Reading Skills of Students with Autism Spectrum Disorder. Interv. Sch. Clin. 2019, 55, 71–77. [Google Scholar] [CrossRef]
  7. Martin, S.; Diaz, G.; Sancristobal, E.; Gil, R.; Castro, M.; Peire, J. New technology trends in education: Seven years of forecasts and convergence. Comput. Educ. 2011, 57, 1893–1906. [Google Scholar] [CrossRef]
  8. Dror, I.; Schmidt, P.; O’connor, L. A cognitive perspective on technology enhanced learning in medical training: Great opportunities, pitfalls and challenges. Med. Teach. 2011, 33, 291–296. [Google Scholar] [CrossRef] [PubMed]
  9. Richard, E.; Billaudeau, V.; Richard, P.; Gaudin, G. Augmented Reality for Rehabilitation of Cognitive Disabled Children: A Preliminary Study. In Proceedings of the 2007 Virtual Rehabilitation, Venice, Italy, 27–29 September 2007; IEEE: Piscataway, NJ, USA, 2007; pp. 102–108. [Google Scholar]
  10. Tissot, C.; Evans, R. Visual Teaching Strategies for Children with Autism. Early Child Dev. Care 2003, 173, 425–433. [Google Scholar] [CrossRef]
  11. Suparjoh, S.; Shahbodin, F.; Mohd, C.K.N.C.K. Technology-Assisted Intervention for Children with Autism Spectrum Disorder using Augmented Reality. Int. J. Recent Technol. Eng. 2020, 8, 2156–2162. [Google Scholar]
  12. Berenguer, C.; Baixauli, I.; Gómez, S.; Andrés, M.d.P.; de Stasio, S. Exploring the Impact of Augmented Reality in Children and Adolescents with Autism Spectrum Disorder: A Systematic Review. Int. J. Environ. Res. Public Health 2020, 17, 6143. [Google Scholar] [CrossRef] [PubMed]
  13. Khowaja, K.; Banire, B.; Al-Thani, D.; Sqalli, M.T.; Aqle, A.; Shah, A.; Salim, S.S. Augmented Reality for Learning of Children and Adolescents with Autism Spectrum Disorder (ASD): A Systematic Review. IEEE Access 2020, 8, 78779–78807. [Google Scholar] [CrossRef]
  14. Wedyan, M.; L-Jumaily, A.A.; Dorgham, O. The use of augmented reality in the diagnosis and treatment of autistic children: A review and a new system. Multimed. Tools Appl. 2020, 79, 18245–18291. [Google Scholar] [CrossRef]
  15. Khowaja, K.; Al-Thani, D.; Banire, B.; Salim, S.S.; Shah, A. Use of augmented reality for social communication skills in children and adolescents with autism spectrum disorder (ASD): A systematic review. In Proceedings of the 2019 IEEE 6th International Conference on Engineering Technologies and Applied Sciences (ICETAS), Kuala Lumpur, Malaysia, 20–21 December 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 1–7. [Google Scholar]
  16. Marto, A.; Almeida, H.A.; Gonçalves, A. Using Augmented Reality in Patients with Autism: A Systematic Review. In Proceedings of the ECCOMAS Thematic Conference on Computational Vision and Medical Image Processing, Porto, Portugal, 16–18 October 2019; Springer: New York, NY, USA, 2019; pp. 454–463. [Google Scholar]
  17. Rega, A.; Mennitto, A.; Vita, S.; Iovino, L. New Technologies and Autism: Can Augmented Reality (Ar) Increase the Motivation in Children with Autism? In Proceedings of the 12th International Technology, Education and Development Conference, Valencia, Spain, 5–7 March 2018; pp. 4904–4910. [Google Scholar]
  18. Adnan, N.H.; Ahmad, I.; Abdullasim, N. Systematic review on augmented reality application for autism children. J. Adv. Res. Dyn. Control Syst. 2018, 10, 26–32. [Google Scholar]
  19. Karamanoli, P. The Application of Augmented Reality for Intervention to People with Autism Spectrum Disorders. IOSR J. Mob. Comput. Appl. 2017, 4, 42–51. [Google Scholar] [CrossRef]
  20. di Bucchianico, G.; Kercher, P.F. (Eds.) Advances in Design for Inclusion 587; Springer International Publishing: Cham, Switzerland, 2018. [Google Scholar]
  21. Tang, J.S.Y.; Chen, N.T.M.; Falkmer, M.; Bölte, S.; Girdler, S. A systematic review and meta-analysis of social emotional computer based interventions for autistic individuals using the serious game framework. Res. Autism Spectr. Disord. 2019, 66, 101412. [Google Scholar] [CrossRef]
  22. Tsikinas, S.; Xinogalos, S. Studying the effects of computer serious games on people with intellectual disabilities or autism spectrum disorder: A systematic literature review. J. Comput. Assist. Learn. 2019, 35, 61–73. [Google Scholar] [CrossRef] [Green Version]
  23. Grossard, C.; Grynspan, O.; Serret, S.; Jouen, A.-L.; Bailly, K.; Cohen, D. Serious games to teach social interactions and emotions to individuals with autism spectrum disorders (ASD). Comput. Educ. 2017, 113, 195–211. [Google Scholar] [CrossRef] [Green Version]
  24. Soares, M.M.; Neves, A.d.O. Augmented Reality and the Use of Alternative Communication for Children with Autism Spectrum Disorder: A Literature Review. In Proceedings of the International Conference on Applied Human Factors and Ergonomics, New York, NY, USA, 24–28 July 2020; Springer: Cham, Switzerland, 2020; pp. 324–330. [Google Scholar]
  25. Lord, C.; Elsabbagh, M.; Baird, G.; Veenstra-Vanderweele, J. Autism spectrum disorder. Lancet 2018, 392, 508–520. [Google Scholar] [CrossRef]
  26. Quesnel-Vallières, M.; Weatheritt, R.J.; Cordes, S.P.; Blencowe, B.J. Autism spectrum disorder: Insights into convergent mechanisms from transcriptomics. Nat. Rev. Genet. 2019, 20, 51–63. [Google Scholar] [CrossRef]
  27. Hernawan, A.D.; Diningrum, A.; Jati, S.N.; Nasip, M. Risk Factors of Autism Spectrum Disorder (ASD). Unnes J. Public Health 2018, 7, 104–112. [Google Scholar] [CrossRef]
  28. Geschwind, D.H.; Levitt, P. Autism spectrum disorders: Developmental disconnection syndromes. Curr. Opin. Neurobiol. 2007, 17, 103–111. [Google Scholar] [CrossRef] [PubMed]
  29. Hutsler, J.J.; Love, T.; Zhang, H. Histological and Magnetic Resonance Imaging Assessment of Cortical Layering and Thickness in Autism Spectrum Disorders. Biol. Psychiatry 2007, 61, 449–457. [Google Scholar] [CrossRef] [PubMed]
  30. Buxhoeveden, D.P.; Semendeferi, K.; Buckwalter, J.; Schenker, N.; Switzer, R.; Courchesne, E. Reduced minicolumns in the frontal cortex of patients with autism. Neuropathol. Appl. Neurobiol. 2006, 32, 483–491. [Google Scholar] [CrossRef] [PubMed]
  31. Casanova, M.F.; van Kooten, I.A.J.; Switala, A.E.; van Engeland, H.; Heinsen, H.; Steinbusch, H.W.M.; Hof, P.R.; Trippe, J.; Stone, J.; Schmitz, C. Minicolumnar abnormalities in autism. Acta Neuropathol. 2006, 112, 287–303. [Google Scholar] [CrossRef] [PubMed]
  32. Matson, J. Behavioral treatment of autistic persons: A review of research from 1980 to the present. Res. Dev. Disabil. 1996, 17, 433–465. [Google Scholar] [CrossRef]
  33. Wyver, S. Book Review: Autism Spectrum Disorders: Interventions and Treatments for Children and Youth. Autism 2006, 10, 118–119. [Google Scholar] [CrossRef]
  34. Bondy, A.S.; Frost, L.A. Mands across the Water: A Report on the Application of the Picture-Exchange Communication System in Peru. Behav. Anal. 1993, 16, 123–128. [Google Scholar] [CrossRef] [Green Version]
  35. Cummings, A.R.; Carr, J.E.; LeBlanc, L.A. Experimental evaluation of the training structure of the Picture Exchange Communication System (PECS). Res. Autism Spectr. Disord. 2012, 6, 32–45. [Google Scholar] [CrossRef]
  36. Lerna, A.; Esposito, D.; Conson, M.; Russo, L.; Massagli, A. Social-communicative effects of the Picture Exchange Communication System (PECS) in Autism Spectrum Disorders. Int. J. Lang. Commun. Disord. 2012, 47, 609–617. [Google Scholar] [CrossRef]
  37. Charlop-Christy, M.H.; Carpenter, M.; Le, L.; LeBlanc, L.A.; Kellet, K. Using the Picture Exchange Communication System (Pecs) with Children with Autism: Assessment of Pecs Acquisition, Speech, Social-Communicative Behavior, and Problem Behavior. J. Appl. Behav. Anal. 2002, 35, 213–231. [Google Scholar] [CrossRef] [Green Version]
  38. Alsayedhassan, B.; Banda, D.R.; Griffin-Shirley, N. A Review of Picture Exchange Communication Interventions Implemented by Parents and Practitioners. Child Fam. Behav. Ther. 2016, 38, 191–208. [Google Scholar] [CrossRef]
  39. Ganz, J.B.; Parker, R.; Benson, J. Impact of the Picture Exchange Communication System: Effects on Communication and Collateral Effects on Maladaptive Behaviors. Augment. Altern. Commun. 2009, 25, 250–261. [Google Scholar] [CrossRef] [PubMed]
  40. Rosenwasser, B.; Axelrod, S. The Contributions of Applied Behavior Analysis to the Education of People with Autism. Behav. Modif. 2001, 25, 671–677. [Google Scholar] [CrossRef]
  41. Battaglia, D.; McDonald, M.E. Effects of the Picture Exchange Communication System (PECS) on Maladaptive Behavior in Children with Autism Spectrum Disorders (ASD): A Review of the Literature. J. Am. Acad. Spec. Educ. Prof. 2015, 8, 20. [Google Scholar]
  42. Kamath, H.S.; Rao, V.; Santhosh; Kamath, R. Simulation of long term evolution (LTE) based communication system with different protocols. In Proceedings of the 2017 IEEE 8th Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON), New York, NY, USA, 19–21 October 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 488–491. [Google Scholar]
  43. Laamarti, F.; Eid, M.; el Saddik, A. An Overview of Serious Games. Int. J. Comput. Games Technol. 2014, 2014, 358152. [Google Scholar] [CrossRef]
  44. De Freitas, S.; Liarokapis, F. Serious Games: A New Paradigm for Education? In Serious Games and Edutainment Applications; Springer: London, UK, 2011; pp. 9–23. [Google Scholar]
  45. Mouaheb, H.; Fahli, A.; Moussetad, M.; Eljamali, S. The Serious Game: What Educational Benefits? Proc. Soc. Behav. Sci. 2012, 46, 5502–5508. [Google Scholar] [CrossRef] [Green Version]
  46. Buttussi, F.; Chittaro, L. Smarter Phones for Healthier Lifestyles: An Adaptive Fitness Game. IEEE Pervasive Comput. 2010, 9, 51–57. [Google Scholar] [CrossRef]
  47. Silva, J.M.; el Saddik, A. Exertion interfaces for computer videogames using smartphones as input controllers. Multimed. Syst. 2013, 19, 289–302. [Google Scholar] [CrossRef]
  48. Scarle, S.; Dunwell, I.; Bashford-Rogers, T.; Selmanovic, E.; Debattista, K.; Chalmers, A.; Powell, J.; Robertson, W. Complete Motion Control of a Serious Game against Obesity in Children. In Proceedings of the 2011 Third International Conference on Games and Virtual Worlds for Serious Applications, Atheens, Greece, 4–6 May 2011; IEEE: Piscataway, NJ, USA, 2011; pp. 178–179. [Google Scholar]
  49. Cameirão, M.S.; Badia, S.B.; Oller, E.D.; Verschure, P.F. Neurorehabilitation using the virtual reality based Rehabilitation Gaming System: Methodology, design, psychometrics, usability and validation. J. Neuroeng. Rehabil. 2010, 7, 48. [Google Scholar] [CrossRef] [Green Version]
  50. McKanna, J.A.; Jimison, H.; Pavel, M. Divided attention in computer game play: Analysis utilizing unobtrusive health monitoring. In Proceedings of the 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MN, USA, 3–6 September 2009; IEEE: Piscataway, NJ, USA, 2009; pp. 6247–6250. [Google Scholar]
  51. Consolvo, S.; Klasnja, P.; McDonald, W.D.; Avrahami, D.; Froehlich, J.; LeGrand, L.; Libby, R.; Mosher, K.; Landay, J.A. Flowers or a robot army? Encouraging awareness & activity with personal, mobile display. In Proceedings of the 10th International Conference on Ubiquitous Computing—UbiComp’08, Seoul, Korea, 21–24 September 2008; Association for Computing Machinery: New York, NY, USA, 2008; p. 54. [Google Scholar]
  52. Shin, N.; Sutherland, L.M.; Norris, C.A.; Soloway, E. Effects of game technology on elementary student learning in mathematics. Br. J. Educ. Technol. 2012, 43, 540–560. [Google Scholar] [CrossRef] [Green Version]
  53. Chen, Y. Olfactory Display: Development and Application in Virtual Reality Therapy. In Proceedings of the 16th International Conference on Artificial Reality and Telexistence—Workshops (ICAT’06), Hangzhou, China, 29 November–1 December 2006; IEEE: Piscataway, NJ, USA, 2006; pp. 580–584. [Google Scholar]
  54. Arnab, S.; Petridis, P.; Dunwell, I.; de Freitas, S. Enhancing Learning in Distributed Virtual Worlds through Touch: A Browser-based Architecture for Haptic Interaction. In Serious Games and Edutainment Applications; Springer: London, UK, 2011; pp. 149–167. [Google Scholar]
  55. Whitehead, A.; Johnston, H.; Nixon, N.; Welch, J. Exergame Effectiveness: What the Numbers Can Tell Us. In Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games—Sandbox’10, Los Angeles, CA, USA, 28–29 July 2010; Association for Computing Machinery: New York, NY, USA, 2010; pp. 55–62. [Google Scholar]
  56. Djaouti, D.; Alvarez, J.; Jessel, J.-P.; Rampnoux, O. Origins of Serious Games. In Serious Games and Edutainment Applications; Springer: London, UK, 2011; pp. 25–43. [Google Scholar]
  57. Van Krevelen, D.W.F.; Poelman, R. A Survey of Augmented Reality Technologies, Applications and Limitations. Int. J. Virtual Real. 2010, 9, 1–20. [Google Scholar] [CrossRef] [Green Version]
  58. Yuen, S.C.-Y.; Yaoyuneyong, G.; Johnson, E. Augmented Reality: An Overview and Five Directions for AR in Education. J. Educ. Technol. Dev. Exch. 2011, 4, 11. [Google Scholar] [CrossRef]
  59. Milgram, P.; Fumio, K. A Taxonomy of Mixed Reality Visual Displays. IEICE Trans. Inf. Syst. 1994, 77, 1321–1329. [Google Scholar]
  60. McMahon, D.; Cihak, D.F.; Wright, R. Augmented Reality as a Navigation Tool to Employment Opportunities for Postsecondary Education Students with Intellectual Disabilities and Autism. J. Res. Technol. Educ. 2015, 47, 157–172. [Google Scholar] [CrossRef]
  61. Dunleavy, M.; Dede, C. Augmented Reality Teaching and Learning. In Handbook of Research on Educational Communications and Technology; Springer: New York, NY, USA, 2014; pp. 735–745. [Google Scholar]
  62. Herbst, I.; Braun, A.-K.; McCall, R.; Broll, W. TimeWarp. In Proceedings of the 10th International Conference on Human Computer Interaction with Mobile Devices and Services—MobileHCI’08, Amsterdam, The Netherlands, 2–5 September 2008; Association for Computing Machinery: New York, NY, USA, 2008; p. 235. [Google Scholar]
  63. Azuma, B.M.R.; Baillot, Y.; Behringer, R.; Feiner, S.; Julier, S. Recent Advances in Augmented Reality. IEEE Comput. Graph. Appl. 2001, 21, 34–47. [Google Scholar] [CrossRef] [Green Version]
  64. Phan, V.T.; Choo, S.Y. Interior Design in Augmented Reality Environment. Int. J. Comput. Appl. 2010, 5, 16–21. [Google Scholar] [CrossRef]
  65. Bhatt, S.K.; de Leon, N.I.; Al-Jumaily, A. Augmented Reality Game Therapy for Children with Autism Spectrum Disorder. Int. J. Smart Sens. Intell. Syst. 2014, 7, 519–536. [Google Scholar] [CrossRef] [Green Version]
  66. Raajan, N.R.; Suganya, S.; Priya, M.V.; Ramanan, S.V.; Janani, S.; Nandini, N.S.S.; Hemanand, R.; Gayathri, S. Augmented Reality Based Virtual Reality. Procedia Eng. 2012, 38, 1559–1565. [Google Scholar] [CrossRef] [Green Version]
  67. Gan, L.; Jiang, J.; Zhang, W.; Su, Y.; Shi, Y.; Huang, C.; Pan, J.; Lü, M.; Wu, Y. Synthesis of Pyrrolidine Ring-Fused Fullerene Multicarboxylates by Photoreaction. J. Org. Chem. 1998, 63, 4240–4247. [Google Scholar] [CrossRef]
  68. Orrell, A.; McKee, K.; Dahlberg, L.; Gilhooly, M.; Parker, S. Improving continence services for older people from the service-providers’ perspective: A qualitative interview study. BMJ Open 2013, 3, e002926. [Google Scholar] [CrossRef]
  69. Mekni, M.; Lemieux, A. Augmented Reality: Applications, Challenges and Future Trends. Appl. Comput. Sci. 2014, 20, 205–214. [Google Scholar]
  70. Bimber, O.; Raskar, R. Spatial Augmented Reality Merging Real and Virtual Worlds; A.K. Peters/CRC Press: Boca Raton, FL, USA, 2005. [Google Scholar]
  71. Snyder, H. Literature review as a research methodology: An overview and guidelines. J. Bus. Res. 2019, 104, 333–339. [Google Scholar] [CrossRef]
  72. Klumpner, C.; Wijekoon, T.; Wheeler, P. New Methods for the Active Compensation of Unbalanced Supply Voltages for Two-Stage Direct Power Converters. IEEJ Trans. Ind. Appl. 2006, 126, 589–598. [Google Scholar] [CrossRef] [Green Version]
  73. O’Connor, A.M.; Anderson, K.M.; Goodell, C.K.; Sargeant, J.M. Conducting Systematic Reviews of Intervention Questions I: Writing the Review Protocol, Formulating the Question and Searching the Literature. Zoonoses Public Health 2014, 61, 28–38. [Google Scholar] [CrossRef] [PubMed]
  74. Idris, R. How to Develop a Competitive Research Protocol; Geneva Foundation for Medical Education and Research: Geneva, Switzerland, 2018. [Google Scholar]
  75. Model Systems Knowledge Translation Center. A Guide for Developing a Protocol for Conducting Literature Reviews. Available online: https://msktc.org/guide-developing-protocol-conducting-literature-reviews (accessed on 13 December 2021).
  76. Webster, J.; Watson, R.T. Analyzing the Past to Prepare for the Future: Writing a Literature Review. MIS Q. 2002, 26, 7–18. [Google Scholar]
  77. Mullins, M.M.; DeLuca, J.B.; Crepaz, N.; Lyles, C.M. Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000–2010): Are the searches clearly explained, systematic and reproducible? Res. Synth. Methods 2014, 5, 116–130. [Google Scholar] [CrossRef] [Green Version]
  78. Chen, C.H.; Lee, I.J.; Lin, L.Y. Augmented reality-based video-modeling storybook of nonverbal facial cues for children with autism spectrum disorder to improve their perceptions and judgments of facial expressions and emotions. Comput. Hum. Behav. 2016, 55, 477–485. [Google Scholar] [CrossRef]
  79. Iftene, A.; Trandabǎt, D. Enhancing the Attractiveness of Learning through Augmented Reality. Procedia Comput. Sci. 2018, 126, 166–175. [Google Scholar] [CrossRef]
  80. Hsu, H.T.; Lee, I.J. Using augmented reality technology with serial learning framework to develop a serial social story situation board game system for children with autism to improve social situation understanding and social reciprocity skills. In Proceedings of the International Conference on Human-Computer Interaction, Copenhagen, Denmark, 19–24 July 2020; Springer International Publishing: Cham, Switzerland, 2020. [Google Scholar]
  81. Huang, Y.C.; Lee, I.J. Using augmented reality and concept mapping to improve ability to master social relationships and social reciprocity for children with autism spectrum disorder. 12189 LNCS. In Proceedings of the International Conference on Human-Computer Interaction, Copenhagen, Denmark, 19–24 July 2020; Springer International Publishing: Cham, Switzerland, 2020. [Google Scholar]
  82. Khowaja, K.; Al-Thani, D.; Hassan, A.O.; Shah, A.; Salim, S.S. Mobile augmented reality app for children with autism spectrum disorder (asd) to learn vocabulary (marvoc): From the requirement gathering to its initial evaluation. 12211 LNCS. In Proceedings of the International Conference on Human-Computer Interaction, Copenhagen, Denmark, 19–24 July 2020; Springer International Publishing: Cham, Switzerland, 2020. [Google Scholar]
  83. Hossein, N.A.S.M.; Mirinezhad, S.Y.; Ilanloo, A. Utilizing SURF Features and KLT Tracking Algorithm in Augmented Reality (AR), Using Kinect V. 2 with the Aim of Autism Therapy. In Proceedings of the 3rd National Conference on Computer, Information Technology and Applications of Artificial Intelligence, Ahvaz, Iran, 5 February 2020. [Google Scholar]
  84. Da Silva, C.A.; Fernandes, A.R.; Grohmann, A.P. Star: Speech therapy with augmented reality for children with autism spectrum disorders. Lect. Notes Bus. Inf. Process. 2015, 227, 379–396. [Google Scholar]
  85. Sun, R.; Haraldsson, H.; Zhao, Y.; Belongie, S. Anon-Emoji: An optical see-Through augmented reality system for children with autism spectrum disorders to promote understanding of facial expressions and emotions. In Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Beijing, China, 10–18 October 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 448–450. [Google Scholar]
  86. Chen, C.H.; Lee, I.J.; Lin, L.Y. Augmented reality-based self-facial modeling to promote the emotional expression and social skills of adolescents with autism spectrum disorders. Res. Dev. Disabil. 2015, 36, 396–403. [Google Scholar] [CrossRef]
  87. Wang, C.-P.; Tsai, C.-H. Requesting Help Module Interface Design on Speech-Generating Device and Augmented Reality for Children with Autism Spectrum Disorder. Educ. Innov. Appl. 2019, 395–398. [Google Scholar]
  88. Nazaruddin, M.A.; Efendi, M. The Book of Pop Up Augmented Reality to Increase Focus and Object Recognition Capabilities for Children with Autism. J. ICSAR 2018, 2, 9–14. [Google Scholar] [CrossRef]
  89. Lee, J.R.H.; Wong, A. AEGIS: A real-time multimodal augmented reality computer vision based system to assist facial expression recognition for individuals with autism spectrum disorder. arXiv 2020, arXiv:2010.11884. [Google Scholar]
  90. Machado, E.; Carrillo, I.; Saldana, D.; Chen, F.; Chen, L. An assistive augmented reality-based smartglasses solution for individuals with autism spectrum disorder. In Proceedings of the IEEE International Conference on Dependable, Autonomic and Secure Computing, Fukuoka, Japan, 5–8 August 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 245–249. [Google Scholar]
  91. Lee, L.C.; Kao, K.C. Applying facial expression of augmented reality for teaching students with autism spectrum disorders. In Proceedings of the International Conference on Human-Computer Interaction, Copenhagen, Denmark, 19–24 July 2020; Springer International Publishing: Cham, Switzerland, 2020. [Google Scholar]
  92. Kotzageorgiou, M.; Kellidou, P.M.; Voulgari, I.; Nteropoulou-Nterou, E. Augmented reality and the symbolic play of pre-school children with autism. In Proceedings of the 17th European Conference on e-Learning, Athens, Greece, 1–2 November 2018; Curran Associates, Inc.: Red Hook, NY, USA, 2018; pp. 273–280. [Google Scholar]
  93. Sahin, N.T.; Abdus-Sabur, R.; Keshav, N.U.; Liu, R.; Salisbury, J.P.; Vahabzadeh, A. Case Study of a Digital Augmented Reality Intervention for Autism in School Classrooms: Associated with Improved Social Communication, Cognition, Motivation via Educator and Parent Assessment. Front. Educ. 2018, 3, 13–31. [Google Scholar] [CrossRef]
  94. Lee, I.J.; Chen, C.H.; Wang, C.P.; Chung, C.H. Augmented Reality Plus Concept Map Technique to Teach Children with ASD to Use Social Cues When Meeting and Greeting. Asia-Pac. Educ. Res. 2018, 27, 227–243. [Google Scholar] [CrossRef]
  95. Syahputra, M.F.; Arisandi, D.; Lumbanbatu, A.F.; Kemit, L.F.; Nababan, E.B.; Sheta, O. Augmented reality social story for autism spectrum disorder. J. Phys. Conf. Ser. 2018, 978, 012040. [Google Scholar] [CrossRef]
  96. Cirino, C.B.A.A.; Fernandes, A.C.A.F.; Silva, J.D.P.E.; Serra, H.P. ERA—Augmented Reality Teaching—Assistive Technology Developed for the Literacy Process of Children with ASD. 12202 LNCS. In Proceedings of the International Conference on Human-Computer Interaction, Copenhagen, Denmark, 19–24 July 2020; Springer International Publishing: Cham, Switzerland, 2020. [Google Scholar]
  97. Mahayuddin, Z.R.; Mamat, N. Implementing augmented reality (AR) on phonics-based literacy among children with autism. Int. J. Adv. Sci. Eng. Inf. Technol. 2019, 9, 2176–2181. [Google Scholar] [CrossRef] [Green Version]
  98. Lee, I.J. Kinect-for-windows with augmented reality in an interactive roleplay system for children with an autism spectrum disorder. Interact. Learn. Environ. 2020, 29, 688–704. [Google Scholar] [CrossRef]
  99. Wang, K.; Zhang, B.; Cho, Y. Using Mobile Augmented Reality to Improve Attention in Adults with Autism Spectrum Disorder. In Proceedings of the CHI EA’20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 25–30 April 2020; Association for Computing Machinery: New York, NY, USA, 2020; pp. 1–9. [Google Scholar]
  100. Najoua, T.; Mohamed, E.A. A new evaluation technique through serious games for children with ASD. Int. J. Emerg. Technol. Learn. 2020, 15, 202–217. [Google Scholar]
  101. Barajas, A.O.; al Osman, H.; Shirmohammadi, S. A Serious Game for children with Autism Spectrum Disorder as a tool for play therapy. In Proceedings of the 2017 IEEE 5th International Conference on Serious Games and Applications for Health (SeGAH), Perth, WA, Australia, 2–4 April 2017; IEEE: Piscataway, NJ, USA, 2017. [Google Scholar]
  102. Tsikinas, S.; Xinogalos, S. Towards a serious games design framework for people with intellectual disability or autism spectrum disorder. Educ. Inf. Technol. 2020, 25, 3405–3423. [Google Scholar]
  103. Shoukry, L.; Caserman, P.; Göbel, S.; Steinmetz, R. Blitzmerker: Learning idioms with a mobile game. Lect. Notes Comput. Sci. 2015, 9090, 77–88. [Google Scholar]
  104. Vukićević, S.; Đorđević, M.; Glumbić, N.; Bogdanović, Z.; Jovičić, M.Đ. A Demonstration Project for the Utility of Kinect-Based Educational Games to Benefit Motor Skills of Children with ASD. Percept. Mot. Ski. 2019, 126, 1117–1144. [Google Scholar] [CrossRef] [PubMed]
  105. Elshahawy, M.; Aboelnaga, K.; Sharaf, N. CodaRoutine: A serious game for introducing sequential programming concepts to children with autism. In Proceedings of the 2020 IEEE Global Engineering Education Conference (EDUCON), Porto, Portugal, 27–30 April 2020; IEEE: Piscataway, NJ, USA, 2020; pp. 1862–1867. [Google Scholar]
  106. Eder, M.S.; Diaz, J.M.L.; Madela, J.R.S.; Mag-usara, M.U.; Sabellano, D.D.M. Fill me app: An interactive mobile game application for children with Autism. Int. J. Interact. Mob. Technol. 2016, 10, 59–63. [Google Scholar] [CrossRef]
  107. Nawahdah, M.; Ihmouda, W. Implementing a serious game to improve communication and social skills for children with autism. 11677 LNCS. In Proceedings of the International Conference on Collaboration Technologies and Social Computing, Kyoto, Japan, 4–6 September 2019; Springer International Publishing: Cham, Switzerland, 2019. [Google Scholar]
  108. Vallefuoco, E.; Bravaccio, C.; Pepino, A. Serious games in autism spectrum disorder: An example of personalised design. In Proceedings of the CSEDU 2017—9th International Conference on Computer Supported Education, Porto, Portugal, 21–23 April 2017; SciTePress: Setúbal, Portugal, 2017; pp. 567–572. [Google Scholar]
  109. Roglic, M.; Bobic, V.; Djuric-Jovicic, M.; Djordjevic, M.; Dragasevic, N.; Nikolic, B. Serious gaming based on Kinect technology for autistic children in Serbia. In Proceedings of the 2016 13th Symposium on Neural Networks and Applications (NEUREL), Belgrade, Serbia, 22–24 November 2016; IEEE: Piscataway, NJ, USA, 2016; Volume 360, pp. 24–27. [Google Scholar]
  110. Caria, S.; Paternò, F.; Santoro, C.; Semucci, V. The Design of Web Games for Helping Young High-Functioning Autistics in Learning How to Manage Money. Mob. Netw. Appl. 2018, 23, 1735–1748. [Google Scholar] [CrossRef]
  111. VCarvalho, H.; Brandão, J.; Cunha, P.; Vasconcelos, J.; Soares, F. Tobias in the Zoo—A Serious Game for Children with Autism Spectrum Disorders. Int. J. Adv. Corp. Learn. 2015, 8, 23. [Google Scholar] [CrossRef] [Green Version]
  112. Chien, M.E.; Jheng, C.M.; Lin, N.M.; Tang, H.H.; Taele, P.; Tseng, W.S.; Chen, M.Y. iCAN: A tablet-based pedagogical system for improving communication skills of children with autism. Int. J. Hum. Comput. Stud. 2015, 73, 79–90. [Google Scholar] [CrossRef]
  113. Hosseini, E.; Foutohi-Ghazvini, F. Play Therapy in Augmented Reality Children with Autism. Mod. Rehabil. 2016, 10, 110–115. [Google Scholar]
  114. Taryadi, T. Improved communication skills of children with Autism Spectrum Disorder using Augmented Reality based on PECS (Picture Exchange Communication System). Jaict 2018, 3, 1–8. [Google Scholar] [CrossRef]
  115. Kurniawan, I. The improvement of autism spectrum disorders on children communication ability with PECS method Multimedia Augmented Reality-Based. J. Phys. Conf. Ser. 2018, 947, 012009. [Google Scholar]
  116. Mirnawati, D.; Amka, D. Application of PECS (Picture Exchange Communication System) to Improve The Expressive Language Skills of Autism Children. In Proceedings of the 1st International Conference on Creativity, Innovation and Technology in Education (IC-CITE 2018), Banjarmasin, Indonesia, 23–24 November 2018; Volume 274, pp. 154–157. [Google Scholar]
  117. MOliveira, S.; Pereira, C.P.; de Santana, K.C.; Rossinholli, K.O.C. Autisdata: Software to help the development of people with ASD based on TEACCH and PECS methodologies. In Proceedings of the 11th International Conference on Computer Supported Education (CSEDU) 2019, Heraklion, Greece, 2–4 May 2019; pp. 331–338. [Google Scholar]
  118. Soomro, N.; Soomro, S. Autism children’s app using PECS. Ann. Emerg. Technol. Comput. 2018, 2, 7–16. [Google Scholar] [CrossRef]
  119. Susanti, F.; Junaedi, D.; Effendy, V. Communication learning user interface model for children with autism with the goal-directed design method. In Proceedings of the 2019 7th International Conference on Information and Communication Technology (ICoICT), Kuala Lumpur, Malaysia, 24–26 July 2019; IEEE: Piscataway, NJ, USA, 2019. [Google Scholar]
  120. Raja, P.; Saringat, M.Z.; Mustapha, A.; Zainal, A. Prospect: A Picture Exchange Communication System (PECS)-based Instant Messaging Application for Autism Spectrum Condition. IOP Conf. Ser. Mater. Sci. Eng. 2017, 226, 012088. [Google Scholar] [CrossRef] [Green Version]
  121. Elamin, M.B.; Flynn, D.N.; Bassler, D.; Briel, M.; Alonso-Coello, P.; Karanicolas, P.J.; Guyatt, G.H.; Malaga, G.; Furukawa, T.A.; Kunz, R.; et al. Choice of data extraction tools for systematic reviews depends on resources and review complexity. J. Clin. Epidemiol. 2009, 62, 506–510. [Google Scholar] [CrossRef] [PubMed]
  122. Li, H.-J.; Chen, C.Y.; Tsai, C.H.; Kuo, C.C.; Chen, K.H.; Chen, K.H.; Li, Y.C. Utilization and medical costs of outpatient rehabilitation among children with autism spectrum conditions in Taiwan. BMC Health Serv. Res. 2019, 19, 354. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  123. Sturm, D.; Peppe, E.; Ploog, B. EMot-iCan: Design of an assessment game for emotion recognition in players with Autism. In Proceedings of the 2016 IEEE International Conference on Serious Games and Applications for Health (SeGAH), Orlando, FL, USA, 11–13 May 2016; IEEE: Piscataway, NJ, USA, 2016. [Google Scholar]
Figure 1. Definition of ASD.
Figure 1. Definition of ASD.
Sensors 22 01250 g001
Figure 2. Materials for creating a PECS system.
Figure 2. Materials for creating a PECS system.
Sensors 22 01250 g002
Figure 3. Definition of serious games (as presented in [40]).
Figure 3. Definition of serious games (as presented in [40]).
Sensors 22 01250 g003
Figure 4. Serious Games Standards.
Figure 4. Serious Games Standards.
Sensors 22 01250 g004
Figure 5. Reality-virtuality continuum (as presented by [59]).
Figure 5. Reality-virtuality continuum (as presented by [59]).
Sensors 22 01250 g005
Figure 6. The component of AR.
Figure 6. The component of AR.
Sensors 22 01250 g006
Figure 7. Overview of the research methodology.
Figure 7. Overview of the research methodology.
Sensors 22 01250 g007
Figure 8. Overlapping domains of ASD patient rehabilitation, augmented reality environment, serious games, and PECS applications.
Figure 8. Overlapping domains of ASD patient rehabilitation, augmented reality environment, serious games, and PECS applications.
Sensors 22 01250 g008
Figure 9. Search Strategies.
Figure 9. Search Strategies.
Sensors 22 01250 g009
Figure 10. Search results from search databases.
Figure 10. Search results from search databases.
Sensors 22 01250 g010
Figure 11. Filtering with Checking Inclusion and Exclusion Criteria.
Figure 11. Filtering with Checking Inclusion and Exclusion Criteria.
Sensors 22 01250 g011
Figure 12. Publications per year in augmented reality environment.
Figure 12. Publications per year in augmented reality environment.
Sensors 22 01250 g012
Figure 13. Publications per country in augmented reality environment.
Figure 13. Publications per country in augmented reality environment.
Sensors 22 01250 g013
Figure 14. Skills Targeted in augmented reality environment.
Figure 14. Skills Targeted in augmented reality environment.
Sensors 22 01250 g014
Figure 15. Publications per year in serious games.
Figure 15. Publications per year in serious games.
Sensors 22 01250 g015
Figure 16. Targeted skills in serious games.
Figure 16. Targeted skills in serious games.
Sensors 22 01250 g016
Figure 17. Environment in serious games.
Figure 17. Environment in serious games.
Sensors 22 01250 g017
Figure 18. Num phases in PECS applications.
Figure 18. Num phases in PECS applications.
Sensors 22 01250 g018
Figure 19. Serious Games Criteria for Future Study.
Figure 19. Serious Games Criteria for Future Study.
Sensors 22 01250 g019
Table 4. The difference between augmented reality and virtual reality environment.
Table 4. The difference between augmented reality and virtual reality environment.
Augmented Reality EnvironmentVirtual Reality Environment
Provides digital components in a real environmentA completely digital environment
Most of the time, peripheral devices are not required.Most of the time, many peripheral devices are required (such as controllers, physical devices, etc....)
Requires technical tools to bring together the real and digital environmentsRequires technical tools to show digital environmental
It enhances feelings of experiencing in the real worldIt enhances feelings of experience in a whole new and safer environment
Table 5. Database query and filter actions.
Table 5. Database query and filter actions.
DatabaseQueryFilters Applied
Science Direct(“Autism” OR “Autism Spectrum Disorder” OR “ASD”) AND (“Augmented reality App” OR “AR”) AND (“Skills”) AND (“educate” OR “learn”)
Year: 2015–2020
Article type: Research articles
Subject areas: Computer Science and Engineering
Without “*”
Use fewer Boolean connectors (max 8 per field)
(“Autism” OR “Autism Spectrum Disorder” OR “ASD”) AND (“Serious games” OR “SG” OR “Serious games App”) AND (“Skills”) AND (“educate” OR “learn”)
(“Autism” OR “Autism Spectrum Disorder” OR “ASD”) AND (“PECS” OR “Picture Exchange Communication System” OR “PECS App”) AND (“educate” OR “learn”)
Springer Link(“Autism” OR “Autism Spectrum Disorder” OR “ASD”) AND (“Augmented reality” OR “Augmented Reality Therapy” OR “AR” OR “Rehabilitation with augmented reality” OR “ Augmented reality App*”) AND (“*Skills”) AND (“educat*” OR “learn*”)
Year: 2015–2020
Article type: Articles
Language: English
Discipline: Computer Science
(“Autism” OR “Autism Spectrum Disorder” OR “ASD”) AND (“Serious games” OR “Applied game” OR “SG” OR “Serious games App*”) AND (“*Skills”) AND (“educat*” OR “learn*”)
(“Autism” OR “Autism spectrum disorder” OR “ASD”)AND (“PECS” OR “Picture Exchange Communication System” OR “PECS APP*”) AND (“educat*” OR “learn*” OR “train*” OR “gam*”)
Semantic Scholar(“Autism*” OR “ASD”) AND (“Augmented reality” OR “Augmented reality App*”) AND (“*Skills”) AND (“educat*” OR “learn*” OR “gam*”)
Year: 2015–2020
Article type: Journal Article and Conference
Subject areas: Computer Science and Engineering
(“Autism*” OR “ASD) AND (“Serious games” OR “SG” OR “Serious games App*”) AND (“*Skills” ) AND (“educat*” OR “learn*” OR “train*”)
(“Autism*” OR “ASD”) AND (“PECS” OR “Picture Exchange Communication System”) AND (“App*”) AND (“educat*” OR “learn*” OR “gam*”)
IEEE Explorer(“Autism” OR “Autism spectrum disorder” OR “ASD” OR “Asperger syndrome” OR “Rett syndrome” OR “Childhood disintegrative disorder” OR “Pervasive Developmental Disorder–Not Otherwise Specified”) AND (“Augmented reality” OR “Augmented reality therapy” OR “AR” OR “Rehabilitation with augmented reality” OR “Augmented reality App*”) AND (“Social skills” OR “communication skills” OR “*Skills”)AND (“educat*” OR “learn*” OR “train*” OR “gam*” OR “simulat*”)
Year: 2015–2020
Article type: Journal and Conference
(“Autism” OR “Autism spectrum disorder” OR “ASD”OR “Asperger syndrome” OR “Rett syndrome” OR “Childhood disintegrative disorder” OR “Pervasive Developmental Disorder–Not Otherwise Specified”) AND (“Serious games” OR “Applied game” OR “SG” OR “serious games App*”) AND (“Social skills” OR “communication skills” OR “*Skills”)AND (“educat*” OR “learn*” OR “train*”OR “simulat*”)
(“Autism” OR “Autism spectrum disorder” OR “ASD” OR “Asperger syndrome” OR “Rett syndrome” OR “Childhood disintegrative disorder” OR “Pervasive Developmental Disorder–Not Otherwise Specified”) AND (“PECS” OR “Picture Exchange Communication System” OR “PECS App*”) AND (“educat*” OR “learn*” OR “train*” OR “gam*”)
Google Scholar(“Autism*” OR “ASD”) AND (“Augmented reality” OR “Augmented reality App*”) AND (“*Skills”) AND (“educat*” OR “learn*” OR “gam*”)
Year: 2015–2020
Language: English
(“Autism” OR “Autism spectrum disorder” OR “ASD” OR “Rett syndrome” OR “Childhood disintegrative disorder” OR “Pervasive Developmental Disorder–Not Otherwise Specified”) AND (“Serious games” OR “Applied game” OR “SG” OR “Mobile serious games”) AND (“Social skills” OR “communication skills” OR “*Skills”)AND (“educat*” OR “learn*” OR “train*”OR “simulat*”)
(“Autism*” OR “ASD”) AND (“PECS” OR “Picture Exchange Communication System”) AND (“App*”) AND (“educat*” OR “learn*”OR “gam*”)
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Almurashi, H.; Bouaziz, R.; Alharthi, W.; Al-Sarem, M.; Hadwan, M.; Kammoun, S. Augmented Reality, Serious Games and Picture Exchange Communication System for People with ASD: Systematic Literature Review and Future Directions. Sensors 2022, 22, 1250. https://doi.org/10.3390/s22031250

AMA Style

Almurashi H, Bouaziz R, Alharthi W, Al-Sarem M, Hadwan M, Kammoun S. Augmented Reality, Serious Games and Picture Exchange Communication System for People with ASD: Systematic Literature Review and Future Directions. Sensors. 2022; 22(3):1250. https://doi.org/10.3390/s22031250

Chicago/Turabian Style

Almurashi, Haneen, Rahma Bouaziz, Wallaa Alharthi, Mohammed Al-Sarem, Mohammed Hadwan, and Slim Kammoun. 2022. "Augmented Reality, Serious Games and Picture Exchange Communication System for People with ASD: Systematic Literature Review and Future Directions" Sensors 22, no. 3: 1250. https://doi.org/10.3390/s22031250

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop