Research on Logistics Service Assessment for Smart City: A Users’ Review Sentiment Analysis Approach

Abstract

The innovative development of logistics has become a powerful starting point and strong support for the construction of smart cities. An accurate evaluation of logistics service quality can promote intelligent transformation, and upgrading logistics enterprises can improve the urban public service infrastructure. In this study, we propose a logistics service quality evaluation model based on a combination of sentiment analysis technology and a traditional index evaluation system. With the help of sentiment analysis technology, the model focuses on extracting the sentiment characteristics of logistics service quality from user reviews and analyzing user attitudes from different aspects. We designed a new logistics service quality evaluation index system by improving the SERVQUAL model. The system uses sentiment analysis technology to explore evaluation content through feature extraction and builds relations between the evaluation content and indices. Additionally, we use sentiment orientation analysis with different indices to comprehensively evaluate service quality. The experimental analysis shows that the proposed model and algorithm have high accuracy.

1. Introduction

As the infrastructure at the lowest level of society, logistics not only connects production and consumption, but also supports the pillar industries and advantageous industries of the city. Recently, the rapid popularity of online shopping has significantly promoted the development of the logistics service industry. Logistics services, which are indispensable parts of online shopping, are directly related to customer satisfaction and influence purchase intentions [1,2]. To focus on improving their own service level and attracting more consumers, many e-commerce enterprises usually outsource their logistics services to third-party enterprises because they are not core competitive. However, consumers tend to regard the entire process of online shopping as a whole, and their attitude towards the quality of logistics services directly affects their attitudes towards the service and commodity quality, thereby affecting the sales of commodities [3]. Consumers’ attitudes towards logistics service quality have become an important factor affecting the development of e-commerce.

In the current field of logistics services, competition among logistics service enterprises has been very fierce, and the price standard of logistics services gradually tends to be consistent. Choosing an express company with high service quality is a primary concern for consumers [4,5]. To provide a reliable reference for consumers, most e-commerce platforms focus on users’ feedback on the quality of delivery services. Most studies allow users to score the quality of logistics services by constructing relevant evaluation index systems. The quality level of the relevant logistics service can be evaluated, and reasonable logistics services are recommended for consumers by analyzing users’ scores based on the index evaluation model [6,7,8,9]. However, the quality of logistics services involves many aspects, and different users may have different experiences. The scoring-mechanism-based evaluation system can only score and evaluate the quality of a logistics service. However, it is difficult to describe the advantages and disadvantages of users in terms of specific aspects.

Sentiment analysis, also known as opinion mining and propensity analysis, is an important branch of natural language processing. The main task of sentiment analysis is to help people obtain, sort out, and analyze relevant comment information using computer resources, as well as analyze, process, induce, and reason about subjective text information with emotional color [10]. Methods based on sentiment analysis have been applied to social networks [11], e-commerce [12], online public opinion analysis [13], and many other areas. In the field of logistics services, e-commerce platforms have accumulated various user evaluation information on the quality of logistics services. Most reviews describe various aspects of logistics services in text format, which can correctly reflect the quality of the logistics service. Manual processing of such large-scale information is difficult; thus, an automated machine analysis method must be adopted. With the help of sentiment analysis technology, sentiment characteristics of reviews on logistics service quality can be extracted to help the platform analyze users’ attitudes towards logistics services in different aspects and describe logistics service quality. This method is greatly significant in helping users choose high-quality logistics services and improving the service levels of enterprises.

This study proposes a sentiment-analysis-based logistics service quality evaluation model. An evaluation index system for logistics service quality based on the characteristics of logistics service review information is proposed. The mapping relationship between the sentiment characteristics of reviews and the index system is established to implement the multi-faceted sentiment classification of reviews and then implement the evaluation of various indices by extracting the sentiment characteristics of logistics service quality reviews. In addition, we developed detailed methods to solve the problem of sentiment intensity analysis and index weight calculation. The main contributions of this study are as follows:

(1)

This study proposes a sentiment-analysis-based logistics service quality evaluation model. The model uses sentiment analysis as the core method for evaluating quality of service. The mapping relations between sentiment features and the evaluation index system are established by extracting sentiment features from user reviews. The proposed model solves the problem of sentiment classification in various aspects of the reviews and the problem of sentiment analysis with different indices.

(2)

An improved logistics service quality evaluation index system is proposed. In the improved evaluation system, some indicators from the original evaluation system, such as service reliability, service responsiveness, and service empathy, are reserved. Meanwhile, the tangibility of the service is replaced by the standardization of the logistics service, and the assurance of the logistics service is replaced by the credibility of the logistics service. This improvement is more in line with the characteristics of user demand for logistics service quality, and the evaluation index can comprehensively cover the quality of logistics services.

(3)

This study proposes an algorithm for sentiment feature extraction from user reviews. The sentiment feature extraction algorithm for reviews is directly related to the accuracy of sentiment analysis. We collected publicly available datasets, and experimental studies show that both the recall and accuracy of the proposed algorithm are considerably high.

The remainder of this paper is organized as follows: Section 2 presents the related work on smart cities, service quality evaluation related to express, and logistics. In Section 3, we describe our logistics service quality evaluation model based on sentiment analysis. Section 4 presents the simulation results and analysis of the accuracy of the proposed model. Section 5 concludes the research and discusses the possible future work.

2. Related Works

Smart logistics is the nerve ending of smart cities; it can realize the reuse of idle resources and the comprehensive allocation of resources across functions and industrial chains in the ecology of cities, communities, and industrial clusters. Logistics, highways, and wharves are the infrastructure of the country [14,15]. Existing methods to evaluate logistics service quality or smart logistics focus on mathematical statistics, analytic hierarchy processes, and review text analysis.

2.1. Mathematical Statistics-Based Methods

Mathematical statistics-based methods are used to evaluate service quality. It generally adopts a questionnaire survey using mathematical statistical methods to perform statistics and analysis of the sample data. Therefore, factors that affect the quality of logistics services can be determined. Parmata et al. [16] developed a scale based on Parasuraman’s SERVQUAL scale for the measurement of distributor-perceived service quality at the distributor–manufacturer interface of the pharmaceutical supply chain. They designed a questionnaire based on SERVQUAL, a widely used service quality measurement scale. A personal survey was conducted among selected distributors across three major cities in the Indian pharmaceutical market. The study used an exploratory factor analysis approach to identify the critical factors of service quality, followed by confirmatory factor analysis. Stević et al. [17] presented an original integrated model for measuring RL service quality of reverse logistics. A modified SERVQUAL model was used to measure the service quality of logistics using an original questionnaire with a total of 21 statements arranged in five standard dimensions. Luyen et al. [18] combined fuzzy theory and the multi-criteria decision-making model for the evaluation and selection of potential third-party logistics providers. They developed a comprehensive approach to assess the quality of the logistics service industry. The combined method of the SERVQUAL and FAHP–TOPSIS models provides reasonable results and allows decision makers to visualize the impact of different criteria on the outcome. Wang et al. [19] used the Pythagorean fuzzy set as an efficient mathematical tool to capture indeterminacy. They explored various interactive Hamacher power aggregation operators for Pythagorean fuzzy numbers and presented Pythagorean fuzzy interactive Hamacher power aggregation operators.

The SERVQUAL model has been widely used in the service industry to understand the service needs and perceptions of target customers and to provide enterprises with a set of methods to manage and measure service quality. The latest research shows that the SERVQUAL models have high accuracy in service quality evaluation that involves users’ feedback. For example, the model is used to evaluate the service quality of the reverse logistics channels [20], medical services [21], and many other domains [22]. Thus, we establish the logistics service quality evaluation index system based on the improved SERVQUAL model.

2.2. Analytic Hierarchy-Based Methods

The analytic hierarchy process (AHP) completes the quantitative analysis of the qualitative index by constructing a set of multilevel evaluation index systems. In logistics service quality evaluation, there is a deep analysis of the factors affecting the quality of logistics services and their internal relations. AHP uses less quantitative information on logistics services to make the quality evaluation process more mathematical. This method can provide a simple approach for complex decision-making problems with multi-objective, multi-criteria, or no structural characteristics.

Lou et al. [23] constructed an index-based evaluation system for the service quality of the express delivery industry through market investigation and data analysis. This system applies AHP to survey expert options and obtain index weights using the fuzzy comprehensive evaluation method. Fu et al. [24] analyzed the key influencing factors of the service quality of crowdsourcing logistics and constructed a crowdsourcing logistics service quality evaluation index system with four first-level indices and fifteen s-level indices based on the AHP and entropy weight method. Bakır et al. [25] evaluated e-service quality in the airline industry from the consumers’ perspective. They presented an integrated fuzzy analytical hierarchy process (F-AHP), and the fuzzy measurement alternatives and ranking according to compromise solution (F-MARCOS) approach was proposed to handle the uncertain and imprecise nature of e-service evaluation. Alkharabsheh et al. [26] applied a more sophisticated measure of group agreement than the rank correlation. They used FAHP to determine the aggregated weights and then computed the Kendall correlation values to reveal stakeholder opinions.

2.3. Sentiment-Analysis-Based Methods

Sentiment analysis, also known as opinion mining, is an important research field in text mining. Its main task is to determine the sentiment tendencies that people express on entities, such as products, services, issues, events, topics, and their attributes [27]. The sentiment-analysis-based logistics service quality evaluation method usually builds a sentiment dictionary in the field of express reviews by analyzing relevant reviews of logistics services on the Internet. Based on the specified rules, positive and negative sentiment words are matched to judge the sentiment polarity of the review and analyze the sentiment tendency of consumers. With machine-learning methods, sentiment analysis can be transformed into text classification tasks.

The application of the sentiment analysis method in smart city governance has been widely explored. The smart city research community has already recognized that sentiment analysis can contribute to a better understanding of the public’s needs and concerns [28]. Content analysis illustrates the applicability of sentiment analysis for citizens’ engagement to solve social problems like traffic congestion, crime prediction, disaster management, etc. [29]. Research in multiple fields of the smart city has shown the feasibility and effectiveness of sentiment analysis technology. Li et al. [30] proposed a sentiment model for the task of sentiment analysis on Twitter and a methodological framework to map citizen sentiment from Twitter in helping governments to monitor their citizens’ moods. Al Nuaimi et al. [31] proposed a social media data analysis approach to explore smart city data collected from various social media sources. Abbasi-Moud [32] investigated the automatic extraction of users’ preferences in smart tourism and proposed a method to extract users’ preferences through semantic clustering of the comments and sentiment analysis. Researchers have studied the limitations of sentiment analysis for Arabic. Alotaibi [33] provided a resource on sentiment analysis in the Arabic and Saudi dialects. Alam et al. [34] initially proposed a domain-specific distributed word representation method for aconsiderably small corpus compiled from textual resources on social media for smart city applications. Also, descriptive statistics and sentiment analysis methods were used to identify car-sharing problems based on customer reviews in Poland [35]. Yue [36] used LDA and CNN-BiLSTM-based topic mining and sentiment analysis models for user comments to study the current state of public perception of smart city concepts. Also, there are some studies that combine location information with sentiment analysis to explore the variability of people’s attitudes towards health- and tourism-related services [37,38].

Bahamdain et al. [39] conducted sentiment analysis to assess customer satisfaction with logistics services in the private and public sectors. They used lexicon-based tweets collected and classified as positive, negative, or neutral. A support vector machine model was used for the classification. Chen et al. [40] presented a model for the evaluation of logistics service quality based on sentiment analysis of comment text. They constructed a triple graph neural network (TGNN) to perform sentiment analysis of the service texts. Kim et al. [41] presented a model based on text sentiment analysis which aimed to substitute the survey-based data-gathering process of Kano models with sentiment analysis. They used morpheme analysis to extract quality elements from a set of opinion texts, and the polarity text items were transformed into equivalent Kano survey questions. Tika et al. [42] conducted sentiment analysis on an online taxi transportation service. They used social media to communicate with customers. This study obtained reviews from Instagram, which is used to track public opinion of online Gojek transportation services. Reviews from the page are processed by text preprocessing and then classified using the naive Bayes classifier method to obtain the public value for online transportation services.

3. Methodology

3.1. Definitions and Symbols

This section defines and explains the related issues involved in the model to describe the logistics service quality evaluation system, review sentiment feature extraction, and express quality evaluation in detail.

Definition 1: 

Logistics service quality evaluation index. This refers to all kinds of data that have the meaning of logistics services and can reflect the characteristics of logistics services.

Definition 2: 

Sentiment characteristics of reviews. They refer to the descriptions of the logistics service, such as price, attitude, and speed, included in user reviews.

Definition 3: 

Index sentiment intensity. This refers to the sentiment tendency degree of the text evaluation content in each index of the evaluation index system. The sentiment intensity index describes the sentiment level of each index component.

Definition 4: 

Index weight. This refers to the weight of each index that affects quality evaluation with different levels of sentiment. The index weight describes the ratio between the number of features occupied by an index at a certain sentiment level and the number of features occupied by all sentiment levels.

Table 1. Notations in logistics service quality evaluation model and algorithm.

Variable	Description
$\mathrm{R}=\left\{{\mathrm{r}}_1,{ \mathrm{r}}_2, \dots ,{ \mathrm{r}}_{\mathrm{n}}\right\}$	the set of users’ review
${\mathrm{F}}_{\mathrm{e}}^{\prime }=\left\{{\mathrm{x}}_1,{ \mathrm{x}}_2, \dots ,{ \mathrm{x}}_{\mathrm{n}}\right\}$	the set of candidate sentiment feature words extracted from users’ review
${\mathrm{F}}_{\mathrm{e}}=\left\{{\mathrm{f}}_1,{ \mathrm{f}}_2, \dots ,{ \mathrm{f}}_{\mathrm{n}}\right\}$	the set of logistics service sentiment feature words
$\mathrm{N}\left({\mathrm{x}}_{\mathrm{i}}\right)$	the number of times a feature word appears in a review
${\mathrm{k}}_{\mathrm{i}}$	$\mathrm{the} \mathrm{weight} \mathrm{of} \mathrm{feature} \mathrm{word} {\mathrm{x}}_{\mathrm{i}}$
${\Delta }_{\mathrm{i}}$	The weight of a feature word check value
${\mathrm{L}}_{\mathrm{j}}$	the level of index sentiment
$\mathrm{Score}\left({\mathrm{L}}_{\mathrm{j}}\right)$	the score of index sentiment intensity grade
${\mathrm{Num}}_{\mathrm{i}}\left({\mathrm{L}}_{\mathrm{j}}\right)$	$\mathrm{the} \mathrm{number} \mathrm{of} \mathrm{feature} \mathrm{words} \mathrm{with} {\mathrm{L}}_{\mathrm{j}}$ sentiment level on the i-th index
${\mathrm{N}}_{\mathrm{i}}$	the sequence of the number of feature words with different sentiment levels on the i-th index
$\mathrm{E}\left(\mathrm{j}\right)$	the influence weight of each level of sentiment on the index
$\mathrm{Index}\left({\mathrm{N}}_{\mathrm{i}}\right)$	the score of the i-th index
$\overline{\mathrm{E}}\left({\mathrm{N}}_{\mathrm{i}}\right)$	the average value of the number of feature words accounting for the total sentiment words
${\mathrm{w}}_{\mathrm{i}}$	the influence weight of each index on the total score of logistics service quality
${\mathsf{\sigma }}_{\mathrm{i}}$	mean square deviation of an index
$\mathrm{Quality}\left(\mathrm{S}\right)$	comprehensive score of logistics service quality
$\mathrm{S}$	logistics service enterprise or a kind of logistics service to be evaluated
$\mathrm{sim}\left(\mathrm{X},\mathrm{Y}\right)$	the similarity value of feature segmentation and evaluation feature segmentation; X and Y are the multi-dimensional vectors of text feature segmentation and evaluation index feature segmentation, respectively

lists the notations used in this study to describe the QoS models and algorithms.

3.2. Sentiment-Analysis-Based Logistics Service Quality Evaluation Model

The sentiment-analysis-based logistics service quality evaluation model uses sentiment analysis as the core method to evaluate service quality. It builds mapping relations between sentiment features and evaluation indices by extracting sentiment features from user’s reviews and improving the evaluation index system. Figure 1 illustrates the framework of the proposed model.

The logistics service quality evaluation process can be divided into four steps as follows:

First, the evaluation index system of logistics service quality is reconstructed. Based on the existing SERVQUAL model, the characteristics of logistics services are analyzed, and the core issues that users care about are identified. The service quality index system can be improved by replacing the tangibility of the service with the standardization of the service and the assurance of the service with the credibility of the service. After retaining the reliability, responsiveness, and empathy of the original model, a logistics service quality index system is constructed. Simultaneously, the effectiveness of the new logistics service quality index system is evaluated.

Second, we extract the sentiment characteristics of the user reviews. The user review set is preprocessed to remove incomplete review text and then processed based on the stop word table. The nouns in the preprocessed text are used as candidate feature words to generate the candidate feature set, and the weight of the candidate feature words is determined according to the frequency of the candidate feature words in the review. If the candidate feature words satisfy the requirements, they are classified into a feature word set.

Third, we construct mapping relations between the sentiment characteristics and the evaluation index system. The sentiment feature segmentation and evaluation index feature segmentation of user reviews are represented by multi-dimensional vectors. According to the multi-dimensional vectors, the similarity between the feature segmentation of a review text and the index feature words can be calculated using the cosine method. If the calculated similarity value is greater than a preset threshold, the word belongs to the scope of the index feature.

Fourth, we calculate the sentiment intensity and index weight. We construct a sentiment intensity table based on the degree-level word list in the HowNet sentiment dictionary.

3.3. Logistics Service Quality Evaluation Index System Design

3.3.1. Characteristics of Logistics Service

Logistics delivers items to the recipient or a place designated by the sender within the promised timeframe. Timeliness, accuracy, security, and convenience are the four key characteristics [43,44,45]. Logistics service quality is the difference between consumers’ perceived and expected service levels. The characteristics of logistics service quality can be summarized as subjectivity, interactivity, and processes.

Subjectivity: There are differences in service quality perceived by consumers in different environments. This uncertainty indicates that the quality of service depends on consumers’ subjective feelings to a certain extent.

Interactivity: Logistics services provide consumers with interactive processes in both production and consumption stages. Consumers can propose improvements to service providers during the process of enjoying services, and service providers can provide new services to consumers in a timely manner through these suggestions.

Process: Process quality is an important part of service quality, and the experience in the service process may affect the overall perception of consumers.

3.3.2. Logistics Service Quality Evaluation Index System

The SERVQUAL model clearly shows the connotation of service quality with five divisions: tangibility, reliability, responsiveness, assurance, and empathy. According to the characteristics of logistics services, we establish a logistics service quality evaluation index system based on SERVQUAL. In the proposed logistics service quality evaluation index system, tangibility and assurance are replaced by the standardization and credibility of the logistics service, respectively. Simultaneously, service reliability, service responsiveness, and service empathy are preserved by appropriately adjusting specific secondary indicators. Customers can perceive the degree of reliability of the enterprise’s service commitment from both the enterprise and employees in logistics-service-related scenarios.

Table 2. Proposed logistics service quality evaluation index system.

First-Level Index (Dimension)	Second-Level Index
Standardization	1. Advanced nature of logistics service standards
	2. Cleanliness of staff attire and appearance
	3. Transparency of delivery costs
	4. Advanced uniformity of means of transport
	5. Ease use of site
Reliability	1. Satisfaction with the speed of delivery by express
	2. Accuracy of delivery by express
	3. Object loss phenomenon
	4. Object damage phenomenon
Responsiveness	1. Accessibility of the service hotline
	2. Timeliness of service delivery
	3. Timeliness of courier information tracking
Credibility	1. Credibility of employee services
	2. The professionalism of the staff
	3. The service quality of the staff
Empathy	1. Fluency in customer communication
	2. Effectiveness in handling customer complaints
	3. Focus on the effectiveness of customer questions

presents the profile of the proposed logistics service quality evaluation index system.

3.3.3. Effectiveness Examination of the Proposed Logistics Service Quality Evaluation Index System

We examined the reliability of the constructed logistics service quality evaluation index system using a questionnaire. The questionnaire was divided into three parts. The first part introduced the main content and use of the questionnaire, the second part described the personal information of the respondents, and the third section provided the evaluation indicators of logistics service quality. Questionnaires were sent through e-mail to people with an online shopping experience in the urban areas of Ningbo, China. The questionnaire survey lasted for approximately three months, with a total of 600 questionnaires distributed. Finally, 284 questionnaires were recovered, 262 of which were valid, with an effective rate of 92.3%.

In this study, SPSS22.0 was used to analyze the reliability of the collected questionnaires, and Cronbach’s measurement method [46] was used for reliability measurements.

Table 3. Reliability coefficient.

Project	Cronbach’s Reliability Coefficient
Standardization	0.780
Reliability	0.821
Responsiveness	0.854
Credibility	0.742
Empathy	0.726
Questionnaire overall	0.862

listed the Cronbach’s reliability coefficients of the proposed index system.

Based on

Table 4. Overall correlation coefficient between indices.

Index	Overall Correlation Coefficient	Sort
Advancement of logistics service standards	0.502	11
Cleanliness of staff attire and appearance	0.434	15
Transparency of delivery costs	0.461	12
Advancement of the means of transport	0.445	13
Ease of use of the website	0.630	6
Satisfaction with the speed of delivery	0.525	10
Accuracy of the delivery	0.633	5
Package loss	0.741	3
Package damage	0.582	7
Accessibility of the service hotline	0.810	2
Timeliness of service delivery	0.694	4
Timeliness of delivery information tracking	0.813	1
Credibility of employee services	0.552	8
Professional level of the staff	0.438	14
Service quality of the staff	0.417	18
Fluency in customer communication	0.531	9
Customer complaint handling efficiency	0.432	16
Attitude towards customer feedback	0.424	17

, the Cronbach’s reliability coefficients of the five evaluation indices are all greater than 0.7, indicating that the data have a certain reference significance, and the dimension of this index design is reasonable. Furthermore, the overall correlation coefficient between the indicators was used to complete the final selection, and 18 indicators were determined. Table 4 presents the overall correlations between the indicators. The overall correlation coefficient for each indicator was greater than 0.4, indicating that the selected indicators were reasonable.

The established logistics service quality evaluation index system is feasible based on Cronbach’s reliability coefficient of the index system and the correlation coefficient between the indicators.

3.4. Analysis of Sentiment Characteristics of Logistics Service Reviews

3.4.1. Sentiment Feature Extraction of Reviews

In the process of sentiment feature extraction of logistics service reviews, we started with a user review preprocess to screen out incomplete reviews. The preprocessed text is further processed with stop words that frequently appear in the commentary text and have no meaning for sentiment analysis, such as articles, prepositions, adverbs, and conjunctions. In this study, product features are treated as stop terms generated from a common stop word list and a manually annotated set.

The feature words studied in this study refer to the feature words price, attitude, and speed in user reviews. The noun in the preprocessed text is used as the candidate feature word, and the number of times and total number of feature words of each candidate in the users’ reviews are recorded to generate the candidate feature word set. The weight of a feature word can be calculated as

$${\mathrm{k}}_{\mathrm{i}}=\frac{\mathrm{N}\left({\mathrm{x}}_{\mathrm{i}}\right)}{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{N}\left({\mathrm{x}}_{\mathrm{i}}\right)},$$

(1)

where ${\mathrm{x}}_{\mathrm{i}}$ is a feature word, $\mathrm{N}\left({\mathrm{x}}_{\mathrm{i}}\right)$ is the frequency of ${\mathrm{x}}_{\mathrm{i}}$ appearing in the users’ review, ${\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{N}\left({\mathrm{x}}_{\mathrm{i}}\right)$ is the total number of times all candidate feature words appear in the user review, and ${\mathrm{k}}_{\mathrm{i}}$ is the weight of the feature word ${\mathrm{x}}_{\mathrm{i}}$.

We designed the logistics service review sentiment feature extraction algorithm with the following two steps: (1) nouns and noun phrases were selected from user reviews to construct a candidate feature word set, and (2) the candidate feature word set was traversed with $\mathrm{k}$ value to determine whether to add the candidate feature word to the feature word set. The process of $\mathrm{k}$ value check was implemented as

$${\Delta }_{\mathrm{i}}=\left\{\begin{array}{c}1, {\mathrm{if} \mathrm{k}}_{\mathrm{i}}>a\\ 0, \mathrm{else}\end{array},\right.$$

(2)

where ${\Delta }_{\mathrm{i}}$ is a candidate feature word and $\mathrm{a}(0<\mathrm{a}<1)$ is the preset constant value.

For the candidate feature word as ${\mathrm{x}}_{\mathrm{i}}$ and the feature extraction threshold as $\mathrm{a}(0<\mathrm{a}<1)$, the candidate feature can be determined based on Equation (2). When ${\mathrm{k}}_{\mathrm{i}}>\mathrm{a}$, ${\mathrm{x}}_{\mathrm{i}}$ equals 1, and the candidate features can be used as logistics service features. Otherwise, it was not a feature word. All the features with ${\mathrm{k}}_{\mathrm{i}}$ values greater than $\mathrm{a}$ in the candidate feature set were selected to construct the logistics service feature set. The sentiment feature extraction algorithm for logistics service review content is shown in Algorithm 1.

Algorithm 1. Logistics service review feature word extraction.

Input: User reviews $R=\left\{r_1, r_2, \dots , r_n\right\}$. Output: Logistics service $F_e=\left\{f_1, f_2, \dots , f_n\right\}$. Step1 extract a review $r_i$ from $R$ and preprocess it. Step2 save the preprocessed clause $c$ in Array $C\left. [m\right]$. Each review $r_i$ can produce m clauses at most. Step3 select a $c$ from $C\left. [m\right]$ and set it as $c_j\left(1\le j\le m\right)$, Step4 for $c_j$ with the following processes: (1) If $c_j$ is a noun or noun phrase, the label is as $x_i$ and it is saved into the candidate feature word set $F_e^{\prime }$. (2) If $j<m$, continue to extract $c$ from $C\left. [m\right]$ and repeat Step4 using $j++$. Step5 determine whether there are any unprocessed reviews $r_i$$$. If yes, repeat Steps 1 to 4. Step6 traverse $F_e^{\prime }$$\mathrm{and} \mathrm{calculate} {\Delta }_i$ value of each candidate feature word $x_i$. Step7 remove duplicates in $F_e^{\prime }$. Step8 traverse $F_e^{\prime }$$\mathrm{to} \mathrm{extract} \mathrm{the} {\Delta }_i$ with the following processes: (1) If ${\Delta }_i>a$ is true, the label is as $f_i$ and put into $F_e$ (2) Check whether $F_e^{\prime }$ is not empty. If not, return to Step 8. Step9 output $F_e$ and end the algorithm.

3.4.2. Mapping between Emotional Characteristics of Reviews and Evaluation Indices

The mapping relationship between them must be established to determine the review content that can reflect a specific index. The specific idea is to determine whether the weight value of the relationship between the sentiment segmentation of a review and index feature word is sufficiently large. If the weight is greater than a preset threshold, the article segmentation belongs to the scope of the index. Based on existing research results, we used the cosine similarity method to calculate the relations between text sentiment feature segmentation words and evaluation index feature words. The specific approach is shown as

$$\mathrm{sim}\left(\mathrm{X},\mathrm{Y}\right)=\frac{\mathrm{X}\times \mathrm{Y}}{\left|\mathrm{X}\right|\times \left|\mathrm{Y}\right|}=\frac{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{X}}_{\mathrm{i}}\times {\mathrm{Y}}_{\mathrm{i}}}{\sqrt{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{({\mathrm{X}}_{\mathrm{i}})}^2}\times \sqrt{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{({\mathrm{Y}}_{\mathrm{i}})}^2}},$$

(3)

where $\mathrm{X}$ and $\mathrm{Y}$ in the formula are the multi-dimensional vectors of text sentiment feature segmentation and evaluation index feature segmentation, respectively, ${\mathrm{X}}_{\mathrm{i}}$ and ${\mathrm{Y}}_{\mathrm{i}}$ represent the i-th component of $\mathrm{X}$ and $\mathrm{Y}$, respectively, and n is the number of dimensions of the multi-dimensional vector. In this study, the similarity threshold β was set to 0.9. Word segmentation can be included in the indicator review text set when cosine similarity is greater than or equal to 0.9. The index system includes 18 evaluation indices, and all the sentiment features of the review text can be divided into the corresponding evaluation index set after word segmentation and classification. The evaluation index set was used for sentiment analysis of the corresponding index. The sentiment tendency of the evaluation index content described in user reviews is the user’s sentiment towards the index.

3.4.3. Index Sentiment Intensity and Single Index Score Calculation

The index sentiment intensity is calculated according to the specific content of the evaluation index and the emotional tendency of the comment text under a certain evaluation index. In the index sentiment intensity calculation process, sentiment words, negative words, and adverbs of degree are combined to build a sentiment intensity table based on the degree level word list in the HowNet sentiment dictionary [47]. In this study, sentiment intensity was divided into six levels corresponding to different intensity values, as shown in

Table 5. Sentiment intensity of index.

Sentiment Intensity Levels	Combinations	Intensity Values
1	“S + P”, “D + E + N”	2
2	“M + P”	1.5
3	“P”, “E + P”, “D + S + N”, “D + N”, “D + D + P”	1
4	“N”, “E + N”, “D + S + P”, “D + P”	−1
5	“M + N”	−1.5
6	“S + N”, “D + E + P”	−2

.

In Table 5, “D” stands for negative word, “P” stands for positive sentiment word, “N” stands for negative sentiment word, “S” stands for advanced degree adverb, “M” stands for intermediate degree adverb, and “E” stands for primary degree adverb.

According to the sentiment intensity value, the degree of satisfaction of the user for a certain aspect of the service can be obtained. In this study, we established the mapping relationship between the emotional intensity value and the user’s score in a hundred-mark system. Specifically, sentiment intensity levels 1, 2, 3, 4, 5, and 6 correspond to scores (90–100], (80–90], (70–0], (60–70], (50–60], and [0–50], respectively. If the intermediate value of each interval is taken as the scoring value, the scoring function is given by Equation (4).

$$\mathrm{Score}\left(\mathrm{L}\right)=\left\{\begin{array}{c}\begin{array}{c}95, \mathrm{L}=1\\ 85, \mathrm{L}=2\end{array}\\ \begin{array}{c}75, \mathrm{L}=3\\ 55, \mathrm{L}=4\end{array}\\ \begin{array}{c}45, \mathrm{L}=5\\ 35, \mathrm{L}=6\end{array}\end{array},\right.$$

(4)

where L is the sentiment intensity level ($\mathrm{L}=\left\{1, 2, 3, 4, 5, 6\right\}$).

If a user’s review is related to $\mathrm{i}$ evaluation indices, the relationship between the review and the indices can be serialized according to Equation (5).

$$\begin{gathered} {\mathrm{N}}_1=\left\{{\mathrm{Num}}_1\left({\mathrm{L}}_1\right),{ \mathrm{Num}}_1\left({\mathrm{L}}_2\right), \dots ,{\mathrm{Num}}_1\left({\mathrm{L}}_{\mathrm{j}}\right),\dots ,{\mathrm{Num}}_1\left({\mathrm{L}}_{\mathrm{m}}\right)\right\} \\ {\mathrm{N}}_2=\left\{{\mathrm{Num}}_2\left({\mathrm{L}}_1\right),{ \mathrm{Num}}_2\left({\mathrm{L}}_2\right), \dots ,{\mathrm{Num}}_2\left({\mathrm{L}}_{\mathrm{j}}\right),\dots ,{\mathrm{Num}}_2\left({\mathrm{L}}_{\mathrm{m}}\right)\right\} \\ \dots \\ {\mathrm{N}}_{\mathrm{i}}=\left\{{\mathrm{Num}}_{\mathrm{i}}\left({\mathrm{L}}_1\right),{ \mathrm{Num}}_{\mathrm{i}}\left({\mathrm{L}}_2\right), \dots ,{\mathrm{Num}}_{\mathrm{i}}\left({\mathrm{L}}_{\mathrm{j}}\right),\dots ,{\mathrm{Num}}_{\mathrm{i}}\left({\mathrm{L}}_{\mathrm{m}}\right)\right\}, \end{gathered}$$

(5)

where i ($1\le \mathrm{i}\le 18$) is the number of second-level evaluation indices, $\mathrm{m} \left(\mathrm{m}=6\right)$ is the number of sentiment intensity levels, ${\mathrm{Num}}_{\mathrm{i}}\left({\mathrm{L}}_{\mathrm{j}}\right)$ is the number of feature words with ${\mathrm{L}}_{\mathrm{j}}$ sentiment intensity level on the i-th index, and ${\mathrm{N}}_{\mathrm{i}}$ is the sequence of the number of feature words with different sentiment levels.

A single index score contains a comprehensive score for the feature word at different sentiment levels, which is expressed as

$$\mathrm{Index}\left({\mathrm{N}}_{\mathrm{i}}\right)={{\displaystyle \sum }}_{\mathrm{j}=1}^{\mathrm{m}}\mathrm{E}\left(\mathrm{j}\right)\times \mathrm{Score}\left({\mathrm{L}}_{\mathrm{j}}\right),$$

(6)

where $\mathrm{E}\left(\mathrm{j}\right)$ is the influence weight of different sentiment intensity levels on the i-th index and can be calculated using Equation (7).

$$\mathrm{E}\left(\mathrm{j}\right)=\frac{{\mathrm{Num}}_{\mathrm{i}}\left({\mathrm{L}}_{\mathrm{j}}\right)}{{{\displaystyle \sum }}_{\mathrm{j}=1}^{\mathrm{m}}{\mathrm{Num}}_{\mathrm{i}}\left({\mathrm{L}}_{\mathrm{j}}\right)}$$

(7)

Therefore, the score of the i-th index can be expressed as

$$\mathrm{Index}\left({\mathrm{N}}_{\mathrm{i}}\right)={{\displaystyle \sum }}_{\mathrm{j}=1}^{\mathrm{m}}\frac{{\mathrm{Num}}_{\mathrm{i}}\left({\mathrm{L}}_{\mathrm{j}}\right)}{{{\displaystyle \sum }}_{\mathrm{j}=1}^{\mathrm{m}}{\mathrm{Num}}_{\mathrm{i}}\left({\mathrm{L}}_{\mathrm{j}}\right)}\times \mathrm{Score}\left({\mathrm{L}}_{\mathrm{j}}\right)$$

(8)

3.4.4. Index Weight and Total Logistics Service Quality Evaluation

The impact of different indices on total logistics service quality varies, and we need to set a set of weights to reflect the importance of different indicators. In this study, the standard deviation method [48] was used to determine the influence weight of the index. The specific calculation steps for the index weight are given in Equations (9)–(11).

(1)

Calculate the average value of index $i$ for all sentiment intensity levels.

$$\overline{E}\left(N_i\right)=\frac{{{\displaystyle \sum }}_{j=1}^{m}Nu{m}_i\left(j\right)}{m}$$

(9)

(2)

Calculate the mean square deviation of the index.

$${\mathsf{\sigma }}_i=\sqrt{\frac{{{\displaystyle \sum }}_{j=1}^m{(\overline{E}\left(N_i\right)-Nu{m}_i\left(j\right))}^2}{m}}$$

(10)

(3)

Calculate the weight of the index.

$$w_i=\frac{{\mathsf{\sigma }}_i}{{{\displaystyle \sum }}_{i=1}^n{\mathsf{\sigma }}_i}$$

(11)

In these formulas, $\mathrm{m}(\mathrm{m}=6$) is the number of sentiment intensity levels, $\mathrm{n}(\mathrm{n}=18$) is the number of evaluation indices, ${\mathsf{\sigma }}_{\mathrm{i}}$ is the mean square error, and ${\mathrm{w}}_{\mathrm{i}}$ is the weight of index i.

Considering all indices in the evaluation system, the score of the logistics service quality can be evaluated according to Equation (12).

$$\mathrm{Quality}\left(\mathrm{S}\right)={\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}\left({\mathrm{w}}_{\mathrm{i}}\times \mathrm{Index}\left({\mathrm{N}}_{\mathrm{i}}\right)\right),$$

(12)

where $\mathrm{S}$ is the logistics service enterprise or a certain logistics service that needs to be evaluated.

3.5. Evaluation Method

Recall and precision were selected to compare the number of extracted feature words with the actual number of feature words to evaluate the accuracy of the proposed logistics service review feature word extraction algorithm.

Table 6. Algorithm evaluation approach.

Extraction Results	Number of Correct Feature Words	Number of Incorrect Feature Words
Number of extracted feature words	A	B
Number of not extracted feature words	C

lists the contingency tables used in this study.

The calculation formulas for recall and precision are shown in Equations (13) and (14), respectively.

$$\mathrm{Recall}=\frac{\mathrm{A}}{\mathrm{A}+\mathrm{B}}$$

(13)

$$\mathrm{Precision}=\frac{\mathrm{A}}{\mathrm{A}+\mathrm{C}}$$

(14)

The mean squared error (MSE) method was used to evaluate the accuracy of the proposed logistics service quality evaluation index system. This method measures the mean value of the square of the difference between the evaluated and actual values. The MSE of the proposed logistics service quality evaluation index system can be calculated using Equation (15).

$$\mathrm{MSE}=\frac{1}{\mathrm{n}}{\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}{\left(\mathrm{Quality}\left({\mathrm{S}}_{\mathrm{i}}\right)-\mathrm{True}\left({\mathrm{S}}_{\mathrm{i}}\right)\right)}^2,$$

(15)

where $\mathrm{True}\left({\mathrm{S}}_{\mathrm{i}}\right)$ is the score manually obtained, which is treated as the actual value, and ${\mathrm{S}}_{\mathrm{i}}$ is the collection of reviews for a logistics service.

4. Experimental Analysis of the Evaluation Model

The effectiveness of the proposed logistics service quality evaluation index system depends on the accuracy of the logistics service feature extraction algorithm. First, the accuracy of the proposed logistics service feature extraction algorithm is qualitatively evaluated using a publicly available dataset, and then a comparison and validation analysis of the proposed logistics service quality evaluation index system is performed.

4.1. Dataset Collection

Logistics enterprises’ services have different quality effects on different products. The most direct reflection is that when transporting certain products, the service quality is good; however, the service quality is poor when transporting other products. Therefore, it is reasonable to evaluate the service quality of different types of logistics enterprise products. We used Amazon user review data [49] as the dataset for the evaluation of the proposed model. The selected review ranged from January 2017 to October 2018 and covered five product categories: shoes, food and drinks, books, sports, and electronics and accessories. The selected reviews contained 50,000 records, with each category having the same number of comments. All comments pertained to the logistics services of a certain product. We randomly selected 10,000 reviews that covered all product categories as the baseline dataset, and the remainder served as the test dataset. The baseline dataset was manually annotated and scored, and the test dataset was generally used to test the feasibility and effectiveness of the proposed algorithm.

4.2. Experimental Result Analysis

4.2.1. Recall of Feature Extraction Algorithm

The proposed logistics service reviews the feature word extraction algorithm implemented in MATLAB and executed with the test dataset to check its feasibility and effectiveness. The experimental results of the algorithm using the baseline dataset are shown in Figure 2. Based on the figure, the recall rate of the algorithm was small, and the gap with the actual result was large. With an increase in the number of baseline reviews, the recall rate of the algorithm increases significantly and tends stabilize, indicating that the algorithm-extracted feature words are close to the manually labeled ones. The recall rate becomes stable at a high level when the number of baseline reviews increases to a certain value. In addition, we found that the recall values of different product categories are slightly different with 10,000 reviews, but they are generally approximately 80%, indicating that the error rate of the algorithm is relatively low.

4.2.2. Precision of Feature Extraction Algorithm

Similarly, the precision evaluation of the proposed logistics service review feature word extraction algorithm is implied with baseline reviews that cover five product categories. The precision of the algorithm based on Equation (14) is shown in Figure 3.

As shown in Figure 3, the precision of the algorithm increases as the baseline review number increases. Precisions of all product categories almost reached their highest stable values when the baseline review number increased to 10,000. This shows that the logistics service review feature word extraction algorithm can extract most sentiment features with high accuracy.

4.2.3. MSE of the Proposed Logistics Service Quality Evaluation Index System

The logistics service quality evaluation index system proposed in this study was compared and analyzed with three benchmark models from Knop [50], Ma [9], and Obiedat [51]. Knop’s model combines unweighted SERVQUAL and weighted SERVQUAL, which jointly uses the SERVQUAL index system and AHP, and Obiedat’s model integrates sentiment analysis technology and the support vector machine (SVM) algorithm. The experiment is conducted using the baseline dataset, and the MSE values of the four methods are shown in Figure 4.

As shown in Figure 4, the MSE values of the four models significantly decrease with an increase in the baseline review number. However, the MSE reduction trend of the proposed logistics service quality evaluation index system was more pronounced than those of the other three models. Specifically, the MSE of Ma’s model is the highest, with 1000 reviews, but rapidly drops as the baseline view number increases. With 10,000 reviews, the MSE of Ma’s model is the second lowest among the four models. The MSE of Knop’s model is the smallest at first, but it booms to the second largest value when the baseline view number increased to 10,000. Similarly, the MSE of Obiedat’s model is the second smallest, with 1000 reviews, but it becomes the largest when the baseline view number increased to 10,000.

In Figure 4, when the number of comments is below 2000, the MSE value of Knop’s model is significantly higher than that of the model proposed in our paper. However, with the number of comments greater than 2000, the MSE value of Knop’s model improves slowly, while the MSE value of the proposed model improves rapidly. The main reason is that the number of indicators of Knop’s model is 22, and the proposed model has 18 indicators. The indicators of Knop’s model are more detailed in evaluating service quality. In the case of a low number of comments, its accuracy is higher. Sentiment analysis relies heavily on the number of comments, and the accuracy with a small number of comments is lower. When the number of comment samples reaches a certain level, it is difficult to distinguish users’ evaluation of service quality with more detailed indicators. Instead, many indicators can be clustered into the same category, which can better reflect the experience of most users. Therefore, the model proposed in this paper shows its superiority.

5. Conclusions and Future Work

With the continuous development of smart cities, smart logistics as a service and support has garnered considerable attention. The construction of urban smart logistics systems includes two important aspects: the operation network of logistics centers to meet the needs of urban production and life and the distribution system at the end of the distribution body. Worldwide, scientists have worked extensively to evaluate the level of logistics development in smart cities. However, assessing it directly based on the user’s perspective is difficult. Based on user reviews of e-commerce systems, this study proposed a semantic analysis approach to extract user satisfaction words towards logistics services to achieve the objective of logistics service evaluation. First, a logistics service that reviews feature word extraction was proposed for mining the sentiment features of user reviews based on the existing sentiment dictionary. Subsequently, the mapping relations between sentiment features and logistics-service-evaluation-related indices can be established. Finally, a logistics service quality evaluation index system was established based on the SERVQUAL model.

We validated the proposed logistics service review feature word extraction algorithm using a publicly available user review dataset. The recall and precision check results indicated that both the applicability and effectiveness of the algorithm were considerably high. A comparative analysis shows that the proposed logistics service quality evaluation index system has a lower MSE than the other three benchmark models. All these results show that the user-review-sentiment-analysis-based logistics service quality evaluation approach has high application potential.

However, there are still some methodological or technological constraints that might be addressed in future academic exercise. First, we made an effort with users’ review data from the e-commerce platform to extract the sentiment characteristics of logistics service quality. In the e-commerce environment, users’ feeling on logistics services can be easily affected by many factors, such as commodity quality, payment, customer service, etc. This may lead to deviation of users’ comments to logistics services. Second, it is difficult for existing sentiment analysis technology to identify intentionally malicious evaluations. There are studies that show that some organizations or individuals can use improper means to make a large number of malicious comments to discourage their competitors. However, the existing sentiment analysis technique applied in this paper cannot identify this issue. Finally, bias could exist in semantic analysis without location information. Logistics services are closely related to location, but the dataset used in the experiment does not contain location information. Therefore, the semantic analysis possibly biases the interpretation.

Due to data availability reasons, we only collected publicly available user reviews from a well-known e-commerce platform to conduct the study. In the future, we will focus on gathering smart logistics review datasets to conduct a comparative study of benchmark models, especially the logistics service evaluation methods based on SERVQUAL.