A Systematic Mapping Study about IoT Platforms ^†

Abstract

Today, specialized software (called IoT platforms or, alternatively, IoT middleware) is available to build and manage IoT solutions from scratch, without having to write endless lines of code. Unfortunately, the selection of the IoT platform that best fits the requirements of the application to be developed is not trivial and may take a long time. The challenge comes from the huge number of candidates available today. The present work aims to relieve IT developers of the need to take charge of the IoT platform selection process. A systematic mapping study (SMS) was carried out. The final finding was that, at present, ThingsBoard is the most mature open-source IoT platform to be used for the prototyping of IoT applications without having to be an expert programmer.

1. Introduction

The target audience of the present paper are IT firms which are interested in expanding the number of their clients without being forced to allocate money to designing and developing sophisticated IoT systems before they obtain a formal assignment by the potential stakeholders. The study addresses all the IT players independently of their size. Small and Medium-sized Enterprises (SMEs) (the definition of SME adopted by the European Commission may be found at https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2003:124:0036:0041:en:PDF (accessed on 29 June 2023)) and Very Small Entities (VSEs) (the term VSE was defined by the ISO/IEC JTC1/SC7 Working Group 24 and subsequently adopted for use in the new ISO/IEC 29110) are the dominant force of the global software industry. A survey carried out in Belgium in 2015 (in connection with the development of complex software) [1] pointed out that time and cost overrun issues are more severe for SMEs and VSEs than for big firms. So, the only viable way for SMEs and VSEs to remain competitive or even to survive is to make extensive use of free-of-charge IT tools.

The present work aims to relieve IT developers of the need to take charge of the IoT platform selection process. The paper reports that ThingsBoard is, at present, the most mature open-source IoT platform to be used for the prototyping of IoT applications without having to be an expert programmer. An SMS is accomplished to achieve the goal.

The next sections are structured as follows. Section 2 is about the related work, while Section 3 first recalls the phases that comprise an SMS, then it describes our SMS. The results are discussed in Section 4.

2. Related Work

Today, specialized software (called IoT platforms or, alternatively, IoT middleware) is available. It is suitable to build and manage IoT solutions from scratch, without having to write endless lines of code. These tools belong to the category of low-code development platforms [2]. In the literature, IoT platforms are classified, and then compared, from many alternative points of view. Below, we mention just a few of those perspectives. IoT Analytics (https://iot-analytics.com/(accessed on 24 June 2023)) distinguishes five different types of platforms: application enablement/management, device management, data management, telco connectivity/management, and  IoT-based Infrastructure as a Service. Ref. [3], instead, mentions four categories of belonging: publicly traded, open-source, developer friendly and end-to-end connectivity. Additionally, ref. [4] defines IoT platforms in terms of functions and services, i.e., in terms of the characteristics of the IoT infrastructure and the IoT application to be developed; in [5], platforms are classified according to two complementary dimensions: Functional Blocks and Qualities. Finally, ref. [6] classifies the IoT platforms with respect to the application domain (e.g., healthcare, transportation, smart agriculture, smart building, etc.). It follows that choosing the IoT platform suitable for the prototyping of IoT applications is not trivial at all.

There is another issue that hampers the selection of the IoT platform that best fits the requirements of the application to be developed. The challenge comes from the huge number of candidates available today. Latest research by IoT Analytics found 613 IoT platform companies currently in operation (https://iot-analytics.com/iot-platform-companies-landscape/(accessed on 22 March 2023)). In 2018, Ismail et al. [7] reported that more than 300 IoT platforms were mentioned in the literature. Capabilities and features of IoT platforms are variable; as a consequence, selecting one of them is a time-consuming task. More recently, the same message was conveyed by Mijuskovic et al. [8] and Held et al. [9]. It is worth noting that, despite the fact that the approach in [8] is rooted in statistics, the authors conclude their paper saying that the results obtained have to be interpreted with caution because “The selection process about which platform to use is still difficult and resembles comparing apples and oranges”. Ref. [10] lists twenty-one key factors to choose an IoT platform, which indirectly confirms the complexity of such a task.

The present paper differs from the previously mentioned studies since its main goal is to collect and organize the available knowledge on IoT platforms suitable to carry out the prototyping of IoT applications.

3. Materials and Methods

Scholars have adopted several methods for carrying out literature reviews. SMSs, as they are called in the seminal paper by Kitchenham and Charters [11], are a category of the latter. An SMS is designed to provide a wide overview of a research area in order to establish whether research evidence exists on a topic and to provide, contextually, an assessment of the quantity of the evidence. The SMS was devoted to search the state-of-the-art of open-source projects with the final aim of maturing an overall picture of IoT platforms ready to be used for prototyping IoT applications.

According to the guidelines in [11], our SMS comprised the following three main phases: Planning, Conducting and Reporting the review results. The third phase is self-explanatory, while the second one consists in the implementation of the first phase. Therefore, below, we detail the Conducting phase of the SMS. It was articulated in terms of three activities: (a) definition of the study need; (b) definition of the research questions; and (c) definition of the mapping protocol (Figure 1).

Study need

The aim of this stage was to find out a mature, powerful, and hopefully easy-to-use open-source IoT platform to be adopted for the prototyping of IoT applications.

Research Question (RQ)

Are there published studies which compare IoT platforms?

Mapping protocol

This step comprised the following six activities (Figure 1).

• Search Process. It was implemented as a manual search of articles in the Scopus repository. Scopus, created by Elsevier in 2004, is the largest curated scientific database. It has around 200 million references (http://www.elsevier.com/solutions/scopus/content (accessed on 27 June 2023)). Major publishers (such as ACM, Elsevier, IEEE, Springer, Wiley, etc.) are indexed in Scopus. The search was carried out at the beginning of June 2023.
• Search string. The search string was the following: “Compar* of IoT Platforms” OR “Compar* IoT Platforms” OR “Compar* of IoT Middleware” OR “Compar* IoT Middleware” OR “Compar* of IoT Middleware Platforms” OR “Compar* IoT Middleware Platforms” OR “Comparative Analyses of IoT Middleware Plat-- forms” OR “Survey on IoT platform*” OR “Review on IoT platform*” OR “analys* of Internet-of-Things platforms” (“Compar* of […]” stands for Comparison of […], while “Compar* IoT […]” stands for Comparing […].)
• Inclusion criteria. Papers published (any time) in journals, surveys, conferences, and book chapters.
• Exclusion Criteria. The pool of candidates was restricted by excluding (a) conference reviews and (b) documents not written in English. As output, we obtained 10 items.
• Data Collection.

  Table 1. Metadata about the items from Scopus.

  Document Type	Number of Publications	Distribution over the Years	Number of Publications
  Article	3	2023	1
  Book Chapter	1	2022	2
  Conference Paper	5	2021	2
  Survey	1	2020	1
  		2018	1
  		2017	2
  		2016	1

  shows the distribution across publication type and years of the output items, while

  Table 2. The list of the retrieved items from Scopus. The documents are in the same order as they were returned by the Scopus’s engine.

  ID	Reference to the Publication	ID	Reference to the Publication
  i1	[12]	i6	[3]
  i2	[13]	i7	[14]
  i3	[9]	i8	[15]
  i4	[6]	i9	[16]
  i5	[8]	i10	[17]

  collects such items.
• Data Analysis. At this stage, the items in Table 2 were downloaded and carefully read. Section 4 provides an overview of each paper; hence, it summarizes the findings arising from the SMS.

4. Results

Table 3. Metadata about the ten documents in Table 2. NA stands for Not Applicable.

i	Year	Number of Analyzed IoT Platforms	Number of Open-Source IoT Platforms	Number of Comparison Criteria	Returns a Numerical Ranking
$i1$	2023	3	1	2	No
$i2$	2022	7	3	15	No
$i3$	2022	8	8	14	Yes
$i4$	2021	8	1	7	No
$i5$	2021	5	0	16	Yes
$i6$	2020	11	2	24	No
$i7$	2018	20	1	6	No
$i8$	2017	NA	NA	NA	NA
$i9$	2016	4	3	5	No
$i10$	2017	39	10	7	No

collects metadata about the documents listed in Table 2. The documents are reviewed starting from the oldest and moving toward the newest.

Ref. [16] introduces an abstract IoT reference architecture that is used to compare and evaluate three open-source alternatives (OpenMTC, FIWARE, and SiteWhere) and the Amazon proprietary IoT platform.

Ref. [15] reviews the role of MEMS sensors in IoT platforms. So, this study does not help us to reach the final goal of ours.

Ref. [17] compares 39 IoT platforms, both proprietary (29) and open-source (10). Such a study constitutes the largest and more accurate analysis conducted up to 2015. The 39 platforms are evaluated via a gap analysis devoted to highlight the deficiencies of those solutions with respect to the integration of heterogeneous devices, the data management mechanisms, and the support offered to application developers. The output of the analysis consists of a table where the basic features of the 39 IoT platforms are summarized. The table provides a quick visual information for those interested in selecting the most appropriate IoT platform to be deployed in a specific application domain. For each IoT platform, the table collects information about (a) the supported devices; (b) the type; (c) the architecture; (d) the nature, open-source or proprietary; (e) whether it includes REST APIs; (f) the data access control mechanism; and (g) the service discovery mechanism.

Ref. [14] takes into account 20 IoT platforms. The study suffers from two major shortcomings: (a) the comparison among the alternative solutions is qualitative; (b) the output of the analysis is a table that authors do not discuss in any way. From the perspective of our study, a further limit is the fact that this review takes into account just one open-source IoT platform.

Ref. [3] categorizes IoT platforms into the following four categories: publicly traded, open-source, developer friendly and end-to-end connectivity. The IoT platforms taken into account are compared with respect to (a) basic features (6), (b) sensing features (3), (c) communication features (7), and (d) application development features (8). The numbers in parenthesis denote the sub-features taken into account for comparison. By explicit choice of the authors, the study does not provide an exhaustive review of all platforms belonging to each of these categories. For example, the list of platforms in the open-source category is limited to Kaa IoT and ThingSpeak.

In [8], five proprietary IoT platforms (namely Azure IoT, AWS, SaS IoT, ThingWorx, and Kaa IoT) are compared by making recourse to statistical and visualization techniques to select the most suitable platform. The selection process takes into account 16 functional requirements clustered into three complementary perspectives: (a) Management Perspective; (b) Security and Privacy Perspective; and (c) Data Management and Analytics Perspective.

Ref. [6] is an in-depth survey about eight popular IoT platforms from different application domains (

Table 4. The IoT platforms analyzed in [6].

Platform Name	Application Domain	Distribution
AWS IoT	General purpose	Proprietary (Amazon)
FarmBeats	Agriculture	Proprietary (Microsoft)
HomeKit	Home automation	Proprietary (Apple)
openHAB	Home automation	Open source
SmartThings	Home automation	Proprietary (Samsung)
ThingWorx	Industry	Proprietary (Tutorials Point)
Watson IoT	Industry	Proprietary (IBM)
Windows IoT	General purpose	Proprietary (Microsoft)

). Only two platforms are general purpose, while the remaining are domain oriented; moreover, only openHAB is open source. The study, comparison, and evaluation of the different platforms is guided by a conceptual framework which takes into account seven different technical criteria in order to highlight their distinguishing attributes on communications, security, and privacy.

Ref. [13] proposes an IoT reference architecture (called SANAD) suitable to be used in different application domains. SANAD is compared against existing IoT reference architectures, as well as against proprietary and open-source IoT platforms (namely AWS IoT, Watson IoT, Azure IoT, Samsung SmarThings, Webinos, FIWARE, and OpenMTC).

In [9], authors compare the following seven IoT platforms: Eclipse Kapua, Mainflux, OpenRemote, SiteWhere, ThingsBoard, FIWARE, and Node-RED, from an initial set of 135 candidates available before June 2021. The initial pool is restricted by eliminating (a) not open-source platforms (i.e., not published on GitHub); (b) inactive projects (i.e., systems with no release after 2019); (c) excessively narrow projects (i.e., systems devoted to a niche application domain); (d) systems not ready for production; (e) platforms with low popularity. The seven candidates are rated with respect to 14 distinct features (

Table 5. The features taken into account in [9] and the score attributed to ThingsBoard.

#	Feature	Score
1	Abstraction and Model terminology	2
2	Connectivity	3
3	Deployment	3
4	Persistent communication and Message handling	3
5	Security	3
6	User management—Authorization	3
7	Data and Action processing	3
8	Resilience mechanisms	2
9	Scaling technologies	3
10	Cloud hosting support	3
11	Data visualization	3
12	Data analytics	1
13	License model	3
14	Scalability and Stability performance	3

) by applying a quantitative formal method. Each feature is weighted with a score in the range of [0…3]. It is worth mentioning that ThingsBoard obtained the maximum score (i.e., 3) for 11 out of 14 features. The worst score (i.e., 1) concerns Feature 12 (Data Analytics). Data Analytics is concerned with gaining further insights on the collected data by employing methods as, for instance, machine learning. As of January 2023, ThingsBoard has released Trendz Analytics, which allows turning data into insights and making better decisions easily. Such feature of ThingsBoard was not available at the time of publication of [9].

Table 6. Quantitative evaluation of the platforms compared in [9].

Platform	Score	Platform	Score
ThingsBoard	30.5	Kapua	23.0
Mainflux	24.75	SiteWhere	21.5
Node-Red	24.5	OpenRemote	20.5
Fiware	24.25

shows the total score of each IoT platform. As we see, ThingsBoard has the highest score.

In [12], the authors compare three IoT platforms (namely Ubidots, Thingspeak, and Antares) with respect to delay and data error rate parameters.

In light of the previous analysis, it follows that at the present, ThingsBoard is, among the available open-source IoT platforms, the most suitable middleware for carrying out the prototyping of IoT applications from scratch, without having to write endless lines of code.