A Large-Scale Reviews-Driven Multi-Criteria Product Ranking Approach Based on User Credibility and Division Mechanism

Abstract

Massive online reviews provide consumers with the convenience of obtaining product information, but it is still worth exploring how to provide consumers with useful and reliable product rankings. The existing ranking methods do not fully mine user information, rating, and text comment information to obtain scientific and reasonable information aggregation methods. Therefore, this study constructs a user credibility model and proposes a large-scale user information aggregation method to obtain a new product ranking method. First, in order to obtain the aggregate weight of large-scale users, this paper proposes a consistency modeling method of text comments and star ratings by mining the associated information of user comments, including user interaction information and user personalized characteristics information, combined with sentiment analysis technology, and then constructs a user credibility model. Second, a double-layer group division mechanism considering user regions and comment time is designed to develop the large-scale group ratings aggregation approach. Third, based on the user credibility model and the large-scale ratings aggregation approach, a product ranking method is developed. Finally, the feasibility and effectiveness of the proposed method are verified through a case study for automobile ranking and a comparative analysis is furnished. The analysis results of the application case of automobile ranking show that there is a significant difference between the ranking results obtained by the ratings aggregation method based on the arithmetic mean and the ranking results obtained by this method. The method in this study comprehensively considers user credibility and group division, which can be reflected in user aggregation weights and the group aggregation process, and can also obtain more scientific and reasonable decision results.

1. Introduction

With the increasing popularity and development of the Internet, e-commerce and review platforms have emerged. These platforms allow consumers to post online reviews about their experiences and opinions on various product criteria, including quality and functionality. Online reviews offer valuable information to consumers who lack expert knowledge of the products they wish to purchase and help inform their purchase decisions. However, the abundance of online review data and the fake reviews posted by malicious users make it challenging for consumers to determine the authenticity of reviews and make informed purchase decisions. Thus, it is crucial to identify credible reviews from the vast amount of review data available. This study focuses on the credibility of online product reviews.

To date, numerous experts and scholars have conducted extensive research on the credibility of online product reviews, and they applied their findings to evaluate information credibility across different review platforms and social networks [1,2]. These studies involve developing credible models for online reviews to validate their reliability. Researchers examine the factors that influence the credibility of reviews and investigate the distribution patterns of ratings. For instance, Verma et al. proposed a credibility model for online reviews by analyzing the influencing factors of content, communicator, context, and consumer [3]. They also explored credibility variables associated with these factors and established a causal relationship between the variables and the credibility of online reviews by exploring 22 propositions. Banerjee et al. proposed a theoretical model of reviewer credibility based on dimensions such as positivity, engagement, experience, reputation, competence, and social connections [4]. They employed robust regression to determine the significance of these factors. Furthermore, Sun et al. developed a reputation rating method based on user rating bias and rating characteristics [5]. They discovered that the ratings provided by reliable users exhibit a peak distribution, whereas those provided by malicious users are substantially biased. On the other hand, scholars have presented research on the credibility of reviews from review text, ratings, user information, etc. Xiang et al. analyzes the reliability of review data from the aspects of review text semantic features, sentiment, and ratings [6]. Meel et al. put forward a holistic view of how the information is being weaponized to fulfil the malicious motives and forcefully make a biased user perception about a person, event, or firm [7]. In addition, review text and ratings are also key factors in the study of the credibility of reviews. For instance, Hazarika pointed out that there is an inconsistency between the review text and the rating in product reviews [8]. Almansour et al. proposed to build a system by fusing review text, ratings, and sources [9]. Lo et al. studied the credibility of reviews from the consistency of review text and ratings [10]. However, these studies have rarely introduced the latest text analysis tools to analyze the sentiment of texts. Additionally, traditional statistical and mathematical approaches are not sufficiently precise and efficient in reviewing texts. In this study, we employ the latest text analysis tool to develop a model based on sentiment analysis to assess the usefulness of comments.

Sentiment analysis (SA) technology is an important means of obtaining emotional tendencies in large-scale comments. SA is the process of gathering and analyzing people’s opinions, thoughts, and impressions regarding various topics, products, subjects, and services [11]. SA involves analyzing text or speech using computer techniques to determine the sentiment or emotional state within the text [12,13]. The latest text analysis model, which can be pretrained on large-scale text data, can be fine-tuned to address sentiment analysis tasks [14]. Yang et al. proposed a new SA model, SLCABG [15], and the related experimental results show that the model can effectively improve the performance of text SA. At the same time, SA is also used to analyze large-scale review sets. For instance, Haque et al. used a supervised learning method on a large-scale Amazon dataset to polarize it and achieve satisfactory accuracy [16]. Guo et al. identified the key dimensions of customer service voiced by hotel visitors using a data mining approach, latent Dirichlet analysis (LDA) [17], and the related set included 266,544 online reviews for 25,670 hotels located in 16 countries. Thus, this research employs the latest text analysis model, BERT, to conduct SA on comment text, and quantify the SA of the comment text to five levels corresponding to user ratings. This study constructs a sentiment score acquisition method for text comments based on manually annotated sentiment training libraries and combined with the BERT model.

Currently, there are numerous Internet-based review platforms exclusively for automobile brands, which offer rich and standardized data, providing information resources for useful research. Therefore, this research focuses on automobiles as the research subject and develops a model of review usefulness. Research on ranking decisions for automotive products is divided into two primary areas of investigation, namely rating-driven ranking decisions and text review-driven ranking decisions. In the research on rating-based ranking decisions, distributed linguistic term sets remain the core representation tool for transforming rating information. PROMETHEE-II and TODIM are extended to the linguistic term set environment to propose product ranking approaches [18,19]. In the research on ranking decisions based on text reviews, by combining the sentiment ratings and star ratings based on the output of the DUTIR sentiment dictionary, scholars developed a PageRank algorithm product ranking technique based on a directed graph model [20]. Additionally, scholars have addressed the accuracy problem of sentiment intensity recognition by developing a ranking decision approach based on ideal solutions and introducing two interval type fuzzy sets [21]. Scholars used sentiment analysis techniques to output five types of sentiment ranking by considering the advantages of probabilistic linguistic term sets in characterizing sentiment tendencies and their distribution forms. They combined these sentiment rankings with TODIM and evidence theory to construct a related product ranking decision approach [22]. The above studies primarily aggregate group wisdom knowledge in large-scale online reviews from a statistical viewpoint, using fuzzy sets, linguistic term sets, and other representational methods. However, they have not fully combined the current advanced text analysis techniques to analyze review texts, or considered word-of-mouth credibility and aggregation weights of heterogeneous individuals, and the inconsistency between text reviews and star ratings, as well as reviewer information disclosure, to conduct research. Thus, the current identification of false reviews is not precise enough. On the other hand, the current research does not consider the aggregation of large-scale ratings from group users, which is easily affected by large-scale fake reviews. To address this issue, this study proposes the construction of a user credibility model, and applies the text analysis model to analyze the user credibility weight of each review during the process of aggregating the wisdom knowledge of group online reviews. A group user score aggregation method is also built to calculate the comprehensive score of automobile brands. Based on this analysis, an automobile ranking decision method is developed.

In summary, this study examines how to weaken the influence of fake reviews and extract real and credible reviews for product ranking, and proposes a user credibility model based on the consistency of review sentiment orientations and ratings to solve the problem of difficult automobile ranking decisions. Compared with existing approaches, this approach examines the credibility of reviews in terms of online review text content, performs sentiment analysis on the review text, uses the high accuracy of the text analysis model, quantifies the sentiment intensity of each review text, and further analyzes the user disclosure information to compute user credibility weights. The contributions of this approach can be summarized as follows.

(1)

A user weight model based on user disclosure information is constructed, which includes authentication information, interaction information, and driving information. Then, the sentiment analysis techniques and expert knowledge are used to measure the degree of consistency between ratings and text comments, and a comprehensive user weight calculation model is developed.

(2)

The large-scale group ratings aggregation approach based on user region and comment time division is proposed, and a product ranking method is developed.

2. Problem Description and Data Description

Many consumers encounter difficulties in selecting the appropriate automobile for themselves because of their lack of professional experience and knowledge about automobiles. To address this issue, this research proposes a review credibility model based on user disclosure information and consistency to examine the aggregation of automobile reviews, and compute and rank the overall rating of each automobile brand. The notation defined below is employed to denote the aggregation and variables for this problem.

$X=\left\{x_1,x_2\dots x_n\right\}$: $n$ represents the number of alternative target automobiles, and $x_i$ represents the i-th target automobile, $i=1,2,\dots ,n$.

$A=\left\{a_1,a_2,\dots ,a_8\right\}$: The data analysis demonstrates that there are eight automobile criteria and $a_j$ is the j-th criteria of the automobile, $j\in \left[1,8\right]$, corresponding to {space, power, control, electricity/fuel consumption, comfort, exterior, interior, value for money}.

$U=\left\{u_i^1,u_i^2,\dots ,u_i^{K_i}\right\}$: $u_i^k$ represents the k-th user who commented on the target automobile $x_i$, and $K_i$ represents the number of users who commented on the target automobile $x_i$.

$T=\left\{t_{ij}^1,t_{ij}^2,\dots ,t_{ij}^{K_i}\right\}$: $t_{ij}^k$ represents the text of a comment made by user $e_i^k$ on a criterion $a_j$ of target automobile $x_i$. In this study, a user can only make one comment on an automobile in the dataset. Therefore, the number of comments is equal to the number of users making comments, $k=1,2,\dots ,K_i$.

$S=\left\{s_{ij}^1,s_{ij}^2,\dots ,s_{ij}^k\right\}$: $s_{ij}^k$ represents the star rating of user $e_i^k$ for a criterion $a_j$ of the target automobile $x_i$, $k=1,2,\dots ,K_i$.

Finally, the automobile’s overall rating is determined from the above dataset by computing the mapping function: $G=F\left(X,A,T,S\right)$, $G$ represents the composite rating and $F$ represents the mapping function.

3. A User Credibility Model Based on Consistency and User Disclosure Information

In this section, a user credibility model based on consistency and user disclosure information is developed to compute user credibility weights, as shown in Figure 1.

The target automobile $X=\left\{x_1,x_2\dots x_n\right\}$ is filtered according to constraints such as price, budget and model, and relevant comments are crawled on the Autohome.com forum, using Python crawlers, which comprise comment text, ratings, and user disclosure information. Python and SQL tools are e employed to eliminate illegal comment data, and the data are normalized and stored in the format shown in

Table 1. Comment text and rating scale.

Automotive Indicators	$\mathbf{Space} \left({\mathit{a}}_1\right)$	$\mathbf{Power} \left({\mathit{a}}_2\right)$	$\mathbf{Control} \left({\mathit{a}}_3\right)$	$\mathbf{Consumption} \left({\mathit{a}}_4\right)$	$\mathbf{Comfort} \left({\mathit{a}}_5\right)$	$\mathbf{Exterior} \left({\mathit{a}}_6\right)$	$\mathbf{Interior} \left({\mathit{a}}_7\right)$	$\mathbf{Vfm} \left({\mathit{a}}_8\right)$
Comment text	$t_{i1}^k$	$t_{i2}^k$	$t_{i3}^k$	$t_{i4}^k$	$t_{i5}^k$	$t_{i6}^k$	$t_{i7}^k$	$t_{i8}^k$

.

3.1. User Weights Based on the Consistency of Ratings and Text Reviews

• Step 1. Building automobile text review sentiment dataset

(1) Method construction

Step 1.1 Obtaining text review training set

The automobile reviews are crawled on the Autohome.com forum using Python crawlers, and prepossessing is performed to remove garbled characters, missing information, and other data that do not meet the specifications. At the same time, according to the types of automobiles currently on the market, they are divided into new energy automobiles and gasoline automobiles, and the review data are screened according to the score distribution to make the score distribution even. Finally, the data matrix $\overline{M}={\left(D_h\right)}_{H_0\times 1}$ is obtained.

Step 1.2 Obtaining text review training set with expert sentiment values

The obtained automobile review sample library is uploaded to the built automobile review labeling system to ensure that the data labeling conforms to the specification. At the same time, in order to obtain accurate data, L experts were hired to mark the comment text. Each review text is annotated once by L experts. The experts formulate the rating rules through discussion, and then mark the emotional strength of the comment text according to the rules, as illustrated in Figure 2. Finally, the data matrix $M={\left(D_h^l\right)}_{H_0\times L}$ is obtained.

Step 1.3 Aggregating of expert sentiment values for text review training set

The variance of the marked sentiment score of each comment is calculated and a threshold set to filter the data, so as to maintain the stability of the comment data. Then, the average value of the expert sentiment values $S{V}_h$ for text review $D_h$ is calculated as the emotional strength of the label.

$$S{V}_h=\frac{1}{L}{\displaystyle \sum _{l=1}^L{D}_h^l}$$

(1)

(2) Method execution

Step 1.1 Obtaining text review training set

Using crawlers to obtain more than 4000 new energy automobile reviews and more than 4000 gasoline automobile reviews in the forum, a total of more than 8000 automobile review data were obtained. Review data including missing information and garbled characters were removed through preprocessing, and finally 7361 automobile review data were obtained.

$${\overline{M}}_0={\left(D_h\right)}_{7361\times 1}$$

(2)

Step 1.2 Obtaining text review training set with expert sentiment values

Eight experts were hired to discuss and formulate the sentiment rating rules for automobile review texts; they logged in to the annotation system to annotate the above-mentioned obtained review texts. Finally the matrix $M_0$ of the text emotional annotations of the eight experts was obtained.

$$M_0={\left(D_h^l\right)}_{7361\times 8}$$

(3)

Step 1.3 Aggregating of expert sentiment values for text review training set

First, the variance of the sentiment labeling rating of eight experts for each comment was calculated, and the labeled data with a variance greater than 2 were removed. Then, the sentiment annotation matrix of the above-mentioned automobile review text was aggregate. Finally, 6563 automobile review text sentiment annotation data were obtained to form a sentiment analysis model training dataset. This included eight automobile attributes, where each attribute has a comment text. Finally 52,504 texts were obtained. The distribution of sentiment intensity is shown in

Table 2. Distribution of sentiment analysis model training dataset.

Sentiment Rating	Dataset
1	9888
2	9200
3	10,952
4	12,216
5	10,248

.

$$\begin{gathered} S{V}_h=\frac{1}{8}{\displaystyle \sum _{l=1}^8{D}_h^l} \\ SM={\left(S{V}_h\right)}_{6563\times 1} \end{gathered}$$

(4)

• Step 2. Training text review sentiment values based on BERT sentiment analysis model

Step 2.1 Building the automobile review sentiment analysis model based on BERT

In this study, the deep learning framework pytorch was used to build the automobile review sentiment analysis model. The main process is shown as the Figure 3.

The process was as follows: First, remove stop words, stemming, and other preprocessing of the automobile review text. Then extract the topics in the comment text, and then select an efficient deep learning model according to the short text processing effect. This study used the BERT model to build a sentiment analysis model.

The overall framework of the model is a stack of encoders with multiple layers of transformers, with a single-layer structure, as illustrated in Figure 4. $E_1,E_2,\dots ,E_N$ is the embedded word after the embedding process, $Trm$ represents the transformer encoding layer of the host, and $T_1,T_2,\dots ,T_N$ represents the word encoding after the multilayer process. The affective intensity prediction process includes embedding, multi-head self-attention, feedforward, and layer normalization.

At the same time, the model parameters are preliminarily set according to the length and data volume of the automobile review text data. The parameter settings of the model are shown in

Table 3. BERT model parameters.

Parameters	Name	Value
Max_length	Maximum text length	512
Epoch	Training batches	5
Batch_size	Number of batch gradients down	16

.

Step 2.2 Training text review sentiment values

In this study, the aforementioned prepared training dataset was employed to train the BERT model based on the pytorch framework to obtain a sentiment analysis model with eight automobile features. Figure 5 shows the process of sentiment analysis model training.

Table 4. Sentiment analysis model accuracy.

Automobile Properties	${\mathit{a}}_1$	${\mathit{a}}_2$	${\mathit{a}}_3$	${\mathit{a}}_4$	${\mathit{a}}_5$	${\mathit{a}}_6$	${\mathit{a}}_7$	${\mathit{a}}_8$
Model accuracy	89%	81%	80%	76%	76%	87%	77%	85%

shows the accuracy of the model.

• Step 3. Predicting text review sentiment values

To quantitatively predict the sentiment intensity of all comments, the trained sentiment analysis model was employed. The model predicts an emotional intensity based on the input utterances using the above module, as illustrated in Figure 6. The top part [CLS] is the identification of the beginning of the text, which contains the information of the entire sentence, but has no real meaning. The output of all other positions will be biased by placing more weight on the weight of their position, so the first place is output. Then, this is processed by the linear classifier module to predict a label.

The sentiment intensity S of the $a_j$ ($j\in \left[1,8\right]$) indicator review text of user $e_i^k$ for an automobile $x_i$ is obtained by examining the review text $t_{ij}^k$ with the sentiment analysis model. This yields the sentiment intensity prediction matrix $S_p=\left\{s_{pij}^1,s_{pij}^2,\dots ,s_{pij}^k\right\}$ for all reviews of the automobile $x_i$. The granularity $s_{pij}^k$ is the same as the user star rating $s_{ij}^k$, which takes on a range of values $\left\{1,2,3,4,5\right\}$.

• Step 4. User weights based on the consistency of ratings and text reviews

Generally, fake reviews are characterized by inconsistencies between the star rating $s_{ij}^k$ and the sentiment intensity $s_{pij}^k$ of their review texts. Thus, this section proposes approaches to compute the consistency weights of the two factors. The higher the weight, the higher the degree of consistency and the more credible the review. The computation process is as follows:

$$r_{ij}^k=\frac{1}{4}\left(4-\left|s_{ij}^k-s_{pij}^k\right|\right)$$

(5)

The consistency weight matrix is obtained for online reviews of automobiles, $R=\left[r_{ij}^k\right]$, $r_{ij}^k\ge 0$.

3.2. User Weights Based on Disclosure Information

User disclosures include whether they are authenticated, their interaction index (number of replies, number of likes, and number of views), as illustrated in

Table 5. User disclosure information form.

Automotive Series	User	$\mathbf{Certified} \left({\mathit{I}}_1\right)$	$\mathbf{Interaction} \mathbf{Index} \left({\mathit{I}}_2\right)$			$\mathbf{Usage} \mathbf{Rate} \left({\mathit{I}}_3\right)$
			Views	Points	Replies	Daily Travel Rate	Mileage
$x_i$	$e_i^k$	Yes/No	$p_1$	$p_2$	$p_3$	$q_1$	$q_2$

, and their daily travel rate, expressed as $I=\left\{I_1,I_2,I_3\right\}$. User disclosures provide a side view of the credibility of reviews.

• Step 1. Certification indicators

Users who post by word of mouth on the platform can be categorized as certified and non-certified owners. Certified owners are users who have purchased the automobile. The platform’s automobile owner certification requires uploading personal information such as certified automobile models and driving licenses. This information is audited by the platform. In contrast, uncertified owners may be users who have not purchased the automobile in question. Reviews from certified owners are more credible. The weights $w_{i{I}_1}^k$ are computed, as illustrated in the following formula:

$$w_{i{I}_1}^k=\left\{\begin{array}{l}1,Certified\\ 0.5,Non-certified\end{array}\right.$$

(6)

• Step 2. Travel rate indicator

The trip rate weighting is determined by combining the daily trip rate and mileage. Research suggests that many indicators of an automobile require sufficient mileage to test its performance. Thus, the higher the usage of the automobile, the deeper the user’s experience of the automobile’s performance and the more credible the reviews they publish. The usage rate of an automobile can be computed based on its daily driving rate and mileage driven. Based on statistics, the daily driving rate of the automobile is around $15\sim 78 \mathrm{km}/\mathrm{d}$, and this study divides the interval accordingly to compute the daily driving rate weights, as shown in

Table 6. Daily travel rate weights rule.

$\mathbf{Daily} \mathbf{Travel} \mathbf{Rate} \left(\mathbf{km}/\mathbf{d}\right)$	$[78,+\mathit{\infty })$	$[57,78)$	$[36,57)$	$[15,36)$	$[0,15)$
$q_1$	$1.0$	$0.8$	$0.6$	$0.4$	$0.2$

. Furthermore, statistics on mileage posted by word of mouth in automobile forums indicate that mileage is concentrated at $0\sim 1000 \mathrm{km}$. The higher the mileage, the lower the number of published word-of-mouth entries, according to which the following intervals are divided, as illustrated in

Table 7. Mileage weights rule.

$\mathbf{Mileage} \left(\mathbf{km}\right)$	$[8000,+\mathit{\infty })$	$[5000,8000)$	$[3000,5000)$	$[1000,3000)$	$[0,1000)$
$q_2$	$1.0$	$0.8$	$0.6$	$0.4$	$0.2$

.

The usage weights $w_{i{I}_2}^k$ can be computed by summing up the travel rate weights and the mileage weights as follows, where $\mu$ represents the automobile usage weight computation parameter:

$$w_{i{I}_2}^k=\mu q_1^{ik}+\left(1-\mu \right)q_2^{ik}$$

(7)

• Step 3. Interactive indicators

In this study, word-of-mouth entries posted on automobile forums are viewed, liked, and replied to by other users, and the ratio of the sum of the three to the length of posting is called the interaction index. A higher interaction index indicates that the review is more recognized and considered more credible. To assign weights to the interaction index, the following intervals were computed, as illustrated in the following formula:

$$w_{i{I}_3}^k=\left\{\begin{array}{l}1.0,I_3\in \left[15199,\infty \right]\\ 0.8,I_3\in \left[5424,15199\right]\\ 0.6,I_3\in \left[1769,5424\right]\\ 0.4,I_3\in \left[674,1769\right]\\ 0.2,I_3\in \left[0,674\right]\end{array}\right.$$

(8)

• Step 4. The weight of k-th user $e_i^k$ for automobile $x_i$ is given as follows based on the above three indicators:

  $$f_i^k=\frac{1}{3}\left(w_{i{I}_1}^k+w_{i{I}_2}^k+w_{i{I}_3}^k\right)$$

  (9)

3.3. User Comprehensive Weight Calculation

Fusing the consistency weight of online reviews with the user disclosure weight yields a user credibility weight for reviews $c_{ij}^k$. The formula is as follows, where $\mu$ is the parameter for computing the credibility weight of online reviews:

$$c_{ij}^k=\mu r_{ij}^k+\left(1-\mu \right)f_{ij}^k$$

(10)

where $f_{ij}^k=f_k^i\left(j=1,2,\cdots ,8\right)$.

4. Large-Scale Ratings Aggregation Based on Group User Division

This section proposes a multi-criteria ratings aggregation method for group users to address the issue of weakening the role of user reputation weights in large groups of users, thus weakening the impact of false reviews on the overall rating. The approach first divides users into multiple sets based on their purchase location. Then, all user sets are divided into sets based on their purchase time. User ratings are then computed using user reputation weights. Finally, the aggregation of all user sets is distributed to compute the overall rating.

• Step 1. Group division method

Step 1.1 Group division method based on user geography

As the users of the platform are automobile owners from different regions of the country, their experience and needs of the automobile may vary. Therefore, the users of each automobile $x_i$ purchase $u_i^k$ are divided into eight collections based on geography. The geographical divisions of China are set up as

$$D=\left\{\left.D_d\right|d=0,1,\dots ,7\right\}=\left\{\mathrm{No} \mathrm{region}, \mathrm{Northeast}, \mathrm{North}, \mathrm{Central}, \mathrm{East}, \mathrm{South}, \mathrm{Southwest}, \mathrm{Northwest}\right\},$$

The seven sets of users by geography are represented as

$${\cup }_{d=0}^7\left\{u_i^{D_d}\right\},u_i^{D_d}=\left\{\left.u_i^{dk}\right|k=1,2,\dots ,\left|D_d\right|\right\}, d=0,1,\dots ,7.$$

Step 1.2 Group division method based on the time of user comments

Time is a crucial factor that should not be overlooked, and reviews are even more time sensitive, with different references at different times. Additionally, the automobiles themselves are being updated and the purchase of services and prices are constantly changing. Thus, the study of reviews must also be approached according to different time periods. This study divides the collection based on geographical divisions using years as the research step, and the difference between the earliest and latest reviews, $n$, as the research quotient. The time collection is $T=\left\{\left.T_t\right|t=1,2,\dots ,n\right\}$. The set of users divided by geography and time can be expressed as ${\cup }_{d=1}^7{\cup }_{t=1}^n\left\{u_i^{D_d{T}_t}\right\}$, $u_i^{D_d{T}_t}=\left\{\left.u_i^{dtk}\right|k=1,2,\dots ,\left|D_d{T}_t\right|\right\}$, $d=0,1,\dots ,7$,$t=1,2,\dots ,n$, where $\left|D_d{T}_t\right|$ represents the number of users who commented during time period $t$ in region $d$. The group division structure is shown in Figure 7.

Each user is assigned a weight based on the set division of users, and the final credibility weight for each comment is $u_{ij}^{dtk}$,$k=1,2,\dots ,\left|D_d{T}_t\right|$, which denotes the credibility weight of the $a_j$ ($j\in \left[1,8\right]$) indicator for the automobile $x_i$ by the k-th user in year $T_t$ under a geographical region $D_d$. This is then normalized to give $c_{ij}^{dtk}$.

$$c_{ij}^{dtk}=\frac{u_{ij}^{dtk}}{{\displaystyle \sum _{k=1}^{\left|D_d{T}_t\right|}{u}_{ij}^{dtk}}}$$

(11)

• Step 2. Calculating the overall user rating

To further mitigate the impact of false reviews, the original star rating $s_{ij}^k$ of each online review was arithmetically averaged with the predicted sentiment intensity $s_{pij}^k$ of its text $t_{ij}^k$ to obtain a new rating ${\overline{S}}_{ij}^k$ for each review on each criteria of the automobile.

$${\overline{S}}_{ij}^k=\lambda s_{ij}^k+\left(1-\lambda \right)s_{pij}^k$$

(12)

where ${\overline{S}}_{ij}^k$ denotes user $u_i^k$ rating of indicator $a_j$ for automobile $x_i$.

• Step 3. Aggregating group user ratings

After the aforementioned group segmentation, the rating corresponding to user $u_i^{dtk}$ is ${\overline{S}}_{ij}^{dtk}$,$\left\{{\overline{S}}_{ij}^{dtk}\left|k=1,2,\dots ,\left|D_d{T}_t\right|\right.\right\}$. The explanation of the related parameters is shown in

Table 8. Explanation of formulas ${\overline{S}}_{ij}^{dtk}$.

Parameter	Meaning
$d$	The d-th purchase region
$t$	The t-th time period
$k$	The k-th user in the user set of the t-th time period in the d-th region
$i$	The i-th alternative automobile
$j$	The j-th feature of the automobile
$\overline{S}$	Composite score of text sentiment strength and raw rating

.

Finally, the rating is multiplied by the credibility weight and summed to obtain the final rating ${\overline{S}}_{ij}$.

$${\overline{S}}_{ij}=\frac{1}{8}{\displaystyle \sum _{d=0}^7\left(\frac{1}{n}{\displaystyle \sum _{t=1}^n\left({\displaystyle \sum _{k=1}^{\left|D_d{T}_t\right|}{c}_{ij}^{dtk}\times {\overline{S}}_{ij}^{dtk}}\right)}\right)}$$

(13)

where $k=1,2,\dots ,\left|D_d{T}_t\right|$, and ${\overline{S}}_{ij}$ represents the combined rating of all users of the $a_j$ indicator for automobile $x_i$.

• Step 4. Aggregating multi-criteria ratings

Finally, the eight indicators were aggregated to find the computed composite rating ${\overline{S}}_i$ for automobile $x_i$. $w_j$ is the weight of criteria $a_j$.

$${\overline{S}}_i={\displaystyle \sum _{j=1}^8{w}_j{\overline{S}}_{ij}}$$

(14)

5. Product Ranking Methods

• Step 1. Collect the data and structure it to obtain the comment dataset of the automobile.
• Step 2. Obtain the user weights.
  • Step 2.1 Calculate the user weights based on information disclosure.
  • Step 2.2 Calculate the user weights based on the consistency of ratings and text reviews.
  • Step 2.3 Obtain the user comprehensive weight.
• Step 3. Aggregate large-scale ratings.
  • Step 3.1 Group division based on user geography and comment time.
  • Step 3.2 Calculate the overall user rating based on ratings and emotional analysis value of text comments.
  • Step 3.3 Aggregate group user ratings.
  • Step 3.4 Aggregate multi-criteria ratings.
• Step 4. The ranking results for alternative target automobiles $x_i\left(i=1,2,\dots ,n\right)$ are obtained based on the final overall ratings ${\overline{S}}_i\left(i=1,2,\dots ,n\right)$

  $$x_{\sigma \left(i\right)}\succ x_{\sigma \left(i+1\right)}\left(i=1,2,\dots ,n\right)$$

  where ${\overline{S}}_{\sigma \left(i\right)}\succ {\overline{S}}_{\sigma \left(i+1\right)}\left(i=1,2,\dots ,n\right)$.

6. Application of the Method

The popularity of e-commerce has resulted in the development of numerous review platforms. As a large commodity, the market for automobiles is huge, and this has resulted in the emergence of specialized automobile review platforms, such as AutoZone. These platforms offer reviews of almost all automobiles and provide comprehensive information, making them one of the most crucial sources of information for consumers. However, many consumers are plagued by false reviews because of their lack of automobile-related knowledge, making it challenging for them to make an informed choice. Thus, this section is based on user disclosure information and a consistency user-credibility model analysis approach to assist consumers in making informed purchasing decisions.

• Step 1. Determining product sets, criteria sets, and data acquisition

As shown in

Table 9. Alternative target automobile brands.

Automobile Model	Number of Reviews	Overall Rating	Review Time	Price (in RMB)
GAC Toyota-Hanlanda ($x_1$)	3762	4.51	2006–2022	26.88–34.88
Dongfeng Nissan-Xuan Yi ($x_2$)	2689	4.53	2016–2022	9.98–17.49
SAIC Volkswagen-ToucanL ($x_3$)	3329	4.52	2018–2022	19.90–28.38
Dongfeng Honda-XR-V ($x_4$)	2392	4.86	2015–2022	13.29–15.29
SAIC-Volkswagen-Polo ($x_5$)	3647	4.71	2014–2022	9.09–12.49
FAW-Volkswagen-Golf ($x_6$)	3756	4.86	2014–2022	12.98–22.98

, six alternative target automobiles were selected based on consumers’ budgets and models. All criteria of each automobile brand were analyzed, with the computation process detailed below.

A Python crawler was written to crawl the review data of the corresponding target automobile in AutoZone as of 30 December 2022, as each automobile brand was released at a different time, and thus its review count was different. The crawled data were structured using the Python program. Table 9 shows the review data obtained after removing illegal review data, such as garbled codes and null values.

• Step 2. Use a user credibility model based on consistency and user disclosures to obtain user credibility weights.
  • Step 2.1 User weights based on the consistency of ratings and text reviews
    • Step 2.1.1 Sentiment analysis based on text comments

The trained sentiment analysis model was employed to predict the sentiment intensity of the preprocessed online review text for each automobile. A new rating, denoted $s_{pij}^k$, was obtained for each of the eight automobile features of each online review, as illustrated in

Table 10. Predicted emotional intensity data table for Dongfeng Nissan-Xuan Yi ($x_2$).

No.	${\mathit{a}}_1$	${\mathit{a}}_2$	${\mathit{a}}_3$	${\mathit{a}}_4$	${\mathit{a}}_5$	${\mathit{a}}_6$	${\mathit{a}}_7$	${\mathit{a}}_8$
0	4	4	3	4	4	5	3	4
1	5	3	3	5	2	4	2	5
2	4	4	5	5	5	5	5	4
3	4	4	4	3	5	5	4	4
4	5	4	3	3	5	5	5	5

.

• • Step 2.1.2 Consistency weights based on ratings and text

After obtaining the predicted sentiment intensity $s_{pij}^k$ of all the review texts, a consistency analysis was conducted with their corresponding original star ratings to determine a consistency weight $r_{ij}^k$.

Table 11. Consistency weights of ratings and texts for Dongfeng Nissan-Xuan Yi ($x_2$).

No.	${\mathit{a}}_1$	${\mathit{a}}_2$	${\mathit{a}}_3$	${\mathit{a}}_4$	${\mathit{a}}_5$	${\mathit{a}}_6$	${\mathit{a}}_7$	${\mathit{a}}_8$
0	4	3	3	4	4	3	4	4
1	3	3	4	3	4	4	4	3
2	3	3	3	3	3	3	4	1
3	4	2	4	4	4	4	3	3
4	3	4	4	1	4	4	4	4

illustrates the data for the consistency weighting $r_{ij}^k$ component of the Dongfeng Nissan-Xuan Yi ($x_2$).

Step 2.2 Calculation of weights based on user information disclosure

The characteristic information weights of all review publishers for each automobile were computed. The authentication weight ($I_1$), usage rate weight ($I_2$), and interaction index weight ($I_3$) were computed using the computation approach proposed in the previous section. The parameters $\mu =0.3$ for computing automobile usage rate weight were set, and aggregation was used to obtain the characteristic information weight $f_i^k$ for each review publisher, as illustrated in

Table 12. Information on indicators for calculating the weighting of the Nissan-Xuan Yi user feature.

No.	Certified	Purchase Time	Comment Time	Mileage	${\mathit{p}}_1$	${\mathit{p}}_2$	${\mathit{p}}_3$	${\mathit{I}}_1$	${\mathit{I}}_2$	${\mathit{I}}_3$	${\mathit{f}}_{\mathit{i}}^{\mathit{k}}$
0	NO	2017-08	2017-08	1500	2035	0	0	0.5	0.6	0.3	1.4
1	YES	2016-10	2017-08	8788	1919	0	0	1.0	0.6	0.6	2.2
2	NO	2017-08	2017-08	998	673	0	1	0.5	0.4	0.2	1.1
3	NO	2017-08	2017-09	760	2529	0	0	0.5	0.6	0.2	1.3
4	NO	2018-01	2018-01	33	134,664	166	95	0.5	1.0	0.2	1.7

for the partial data of Dongfeng Nissan-Henyi.

Step 2.3 Combined user weighting calculation

Combining user feature information weights and review consistency weights, and setting $\mu =0.7$, yields a credibility weight $c_{ij}^k$ for each review about all automobile criteria for each automobile brand, as illustrated in

Table 13. Combined weighting of users of Dongfeng Nissan- Xuan Yi ($x_2$).

No.	${\mathit{a}}_1$	${\mathit{a}}_2$	${\mathit{a}}_3$	${\mathit{a}}_4$	${\mathit{a}}_5$	${\mathit{a}}_6$	${\mathit{a}}_7$	${\mathit{a}}_8$
0	3.19	2.49	2.49	3.19	3.19	2.49	3.19	3.19
1	2.79	2.79	3.49	2.79	3.49	3.49	3.49	2.79
2	2.34	2.34	2.34	2.34	2.34	2.34	3.04	0.94
3	3.07	1.67	3.07	3.07	3.07	3.07	2.37	2.37
4	2.46	3.16	3.16	1.06	3.16	3.16	3.16	3.16

.

• Step 3. Aggregation of group user ratings
  • Step 3.1 Group division
    • Step 3.1.1 Grouping based on user geography

According to the purchase area of the reviews divided into eight collections, the Python location function was used to achieve regional collection division, and the number of reviews in each collection was determined, as illustrated in

Table 14. Dongfeng Nissan—Xuan Yi ($x_2$) regional breakdown set.

Area	Number of Reviews	Area	Number of Reviews
No region	91	East	836
Northeast	210	South	408
North	240	Southwest	240
Central	438	Northwest	210

.

  Step 3.1.2 Group segmentation based on user purchase time periods

In addition to the regional set division, each regional set was again divided into sets by observing the distribution of users’ time to purchase an automobile, as illustrated in

Table 15. Dongfeng Nissan—Xuan Yi ($x_2$) Regional—Time Group Segmentation Set.

Area	2016–2018	2019	2020–2022
No region	1	39	51
Northeast	25	44	141
North	32	138	70
Central	43	172	223
East	96	386	354
South	70	231	107
Southwest	16	106	118
Northwest	13	48	165

.

  Step 3.1.3 Combined user weighting normalization

Table 16. Set internal confidence weights normalized.

No.	${\mathit{a}}_1$	${\mathit{a}}_2$	${\mathit{a}}_3$	${\mathit{a}}_4$	${\mathit{a}}_5$	${\mathit{a}}_6$	${\mathit{a}}_7$	${\mathit{a}}_8$
1	0.082	0.097	0.055	0.082	0.101	0.075	0.071	0.078
2	0.080	0.095	0.075	0.056	0.099	0.073	0.089	0.099
3	0.086	0.100	0.058	0.110	0.082	0.079	0.094	0.105
4	0.094	0.108	0.088	0.094	0.113	0.108	0.101	0.113
5	0.062	0.078	0.102	0.085	0.081	0.100	0.073	0.059
6	0.112	0.081	0.105	0.088	0.084	0.102	0.096	0.107
7	0.093	0.085	0.087	0.070	0.089	0.085	0.100	0.089
8	0.112	0.081	0.105	0.112	0.107	0.102	0.076	0.107
9	0.097	0.111	0.113	0.074	0.116	0.089	0.104	0.093
10	0.089	0.103	0.105	0.113	0.085	0.103	0.097	0.085
11	0.091	0.062	0.107	0.115	0.042	0.083	0.099	0.065

shows the results of normalizing the combined weights of users after dividing each set.

  Step 3.1.4 Calculation of the overall user rating

(1)

Rating and text emotional intensity combined

The average of the raw ratings of each comment and the sentiment intensity obtained from the text sentiment quantification was computed to obtain the rating ${\overline{S}}_{pij}^k$ for each comment.

Table 17. Mean ratings of Dongfeng Nissan- Xuan Yi ($x_2$) ratings and textual sentiment intensity.

No.	${\mathit{a}}_1$	${\mathit{a}}_2$	${\mathit{a}}_3$	${\mathit{a}}_4$	${\mathit{a}}_5$	${\mathit{a}}_6$	${\mathit{a}}_7$	${\mathit{a}}_8$
0	5.0	2.5	4.5	5.0	4.0	4.5	3.0	4.0
1	3.5	3.5	3.0	3.5	4.0	4.0	4.0	4.5
2	3.5	3.5	4.5	4.5	4.5	3.5	4.0	3.5
3	5.0	3.0	4.0	3.0	5.0	5.0	2.5	4.5
4	3.5	4.0	3.0	3.5	5.0	3.0	3.0	4.0

presents the average ratings for some of the comments.

(2)

Overall user rating calculation

Based on each review’s rating and its user reputation weighting to compute its composite rating,

Table 18. Overall rating of Dongfeng Nissan- Xuan Yi ($x_2$).

No.	${\mathit{a}}_1$	${\mathit{a}}_2$	${\mathit{a}}_3$	${\mathit{a}}_4$	${\mathit{a}}_5$	${\mathit{a}}_6$	${\mathit{a}}_7$	${\mathit{a}}_8$
0	0.068	0.102	0.098	0.047	0.095	0.041	0.104	0.096
1	0.097	0.039	0.052	0.109	0.036	0.041	0.055	0.051
2	0.077	0.080	0.077	0.045	0.093	0.095	0.041	0.065
3	0.075	0.050	0.058	0.054	0.102	0.048	0.080	0.057
4	0.099	0.083	0.069	0.111	0.096	0.098	0.105	0.097

illustrates the composite ratings computed for a collection of group users.

  Step 3.1.5 Aggregation of group user ratings

(1)

Group aggregation by user purchase time

Aggregation is based on a collection of time-of-purchase users to compute an overall rating, as shown in

Table 19. Rating matrix for aggregation of Dongfeng Nissan- Xuan Yi ($x_2$) by the time interval in the Northeast.

Time Interval	${\mathit{a}}_1$	${\mathit{a}}_2$	${\mathit{a}}_3$	${\mathit{a}}_4$	${\mathit{a}}_5$	${\mathit{a}}_6$	${\mathit{a}}_7$	${\mathit{a}}_8$
2016–2018	4.49	3.46	3.65	3.91	4.36	4.06	3.93	4.46
2019	4.45	3.89	3.96	4.16	4.50	4.62	4.26	4.34
2020–2022	4.68	4.09	4.49	4.47	4.66	4.74	4.57	4.65

.

(2)

Group aggregation by user geography

A composite rating is computed based on the aggregation of the set of users in the area of purchase, as illustrated in

Table 20. Dongfeng Nissan- Xuan Yi ($x_2$) geographical aggregation rating matrix.

No.	${\mathit{a}}_1$	${\mathit{a}}_2$	${\mathit{a}}_3$	${\mathit{a}}_4$	${\mathit{a}}_5$	${\mathit{a}}_6$	${\mathit{a}}_7$	${\mathit{a}}_8$
0	4.38	4.09	4.14	4.38	4.68	4.63	4.03	4.49
1	4.45	3.99	4.12	4.22	4.53	4.57	4.22	4.49
2	4.54	4.02	4.12	4.26	4.51	4.61	4.34	4.51
3	4.50	3.98	4.15	4.27	4.52	4.54	4.30	4.48
4	4.47	3.92	4.16	4.24	4.52	4.53	4.25	4.40
5	4.49	3.76	4.07	4.18	4.48	4.48	4.22	4.40
6	4.45	3.78	4.17	4.28	4.44	4.51	4.16	4.40
7	4.54	3.81	4.03	4.18	4.50	4.47	4.25	4.48

. Each serial number represents a geographical group.

(3)

Aggregation of multi-geographical ratings

The multi-criteria composite ratings were obtained by aggregating the aggregated ratings of all regional user groups.

Table 21. Multi-criteria automobile composite rating table.

Automobile Criteria	${\mathit{x}}_1$	${\mathit{x}}_2$	${\mathit{x}}_3$	${\mathit{x}}_4$	${\mathit{x}}_5$	${\mathit{x}}_6$
$a_1$	4.68	4.48	4.65	4.43	3.86	3.65
$a_2$	4.26	3.92	4.42	4.22	4.08	4.30
$a_3$	4.25	4.12	4.42	4.32	4.54	4.40
$a_4$	4.00	4.25	3.89	4.14	4.16	3.92
$a_5$	4.54	4.52	4.09	3.51	4.08	3.87
$a_6$	4.68	4.54	4.66	4.67	4.72	4.61
$a_7$	4.03	4.22	3.99	3.73	3.92	4.21
$a_8$	4.37	4.46	4.28	4.12	4.42	3.96

illustrates the multi-criteria composite ratings for the six automobile brands.

Step 3.2 Aggregation of multi-criteria ratings

A multi-criteria rating aggregation approach was implemented, and consumers set all weights $w_j=0.125$, $j=1,2,\dots ,8$ to obtain the overall ratings of the automobile brands.

Table 22. Overall ratings for alternatives with different ratios ($\lambda$) of ratings to text sentiment scores.

Automobile Brand	$\mathbf{Original} \left({\mathit{l}}_1\right)$	Ranking	$\mathit{\lambda }=0$$\left({\mathit{l}}_2\right)$	Ranking	$\mathit{\lambda }=0.5$$\left({\mathit{l}}_3\right)$	Ranking	$\mathit{\lambda }=0.2$$\left({\mathit{l}}_4\right)$	Ranking	$\mathit{\lambda }=0.8$$\left({\mathit{l}}_5\right)$	Ranking
$x_1$	4.51	6	4.15	1	4.35	1	4.23	1	4.48	1
$x_2$	4.53	4	4.14	2	4.31	2	4.21	2	4.42	2
$x_3$	4.52	5	4.13	3	4.30	3	4.20	3	4.41	3
$x_4$	4.86	1	3.93	5	4.14	5	4.02	5	4.27	5
$x_5$	4.71	3	4.02	4	4.22	4	4.11	4	4.34	4
$x_6$	4.86	1	3.91	6	4.11	6	3.99	6	4.25	6

illustrates the calculation results of the comprehensive score when $\lambda$ in Formula (12) takes different values, and their corresponding rankings for the six automobile brands. And Figure 8 shows the changing trend. Figure 6 demonstrates the pattern of the composite ratings obtained from the three different methods.

• Step 4. Obtain the ranking results

The composite ratings of each automobile brand were obtained and ranked based on the aforementioned composite rating computation, and the findings are presented in

Table 23. Overall scoring ranking table.

Case1	$x_4=x_6>x_5>x_2>x_3>x_1$
Case2	$x_1>x_2>x_3>x_5>x_4>x_6$
Case3	$x_1>x_2>x_3>x_5>x_4>x_6$
Case4	$x_1>x_2>x_3>x_5>x_4>x_6$
Case5	$x_1>x_2>x_3>x_5>x_4>x_6$

.

The ranking order of the automobiles was changed by performing a consistency analysis of the ratings with the review text and the fusion of the text feature information to compute the overall rating of the automobiles, which differed from the original rating of the automobiles. Six automobiles were ranked with the original overall rating of $x_4=x_6>x_5>x_2>x_3>x_1$. The ranking results of the overall rating and sentiment analysis overall rating computed using the above approach were $x_1>x_2>x_3>x_5>x_4>x_6$. The change in rating and brand ranking demonstrates that fake reviews affect the overall rating and ranking of an automobile brand. Users can employ this analysis to choose an automobile brand that suits their needs for each automobile criterion by simply adjusting the weight W of each automobile criteria. For instance, if they prefer an automobile with comfortable space, Toucan L is the recommended choice; if they prioritize low fuel consumption, Dongfeng Nissan- Xuan Yi is a suitable choice.

This study employs a text sentiment analysis approach as the foundation to examine the consistency between ratings and review text while fusing review text features, which can well verify the authenticity of each review. This study’s experimental findings demonstrate that the proposed analysis approach effectively mitigates the impact of false reviews, allowing consumers to obtain comprehensive ratings of automobile brands devoid of the influence of false reviews, as well as criteria-specific ratings for each brand. Therefore, consumers can personalize the selection of automobile brands based on their needs.

7. Conclusions

Considering that user feature information and content feature information can also reflect the credibility of reviews, this study also calculates the weight of user feature information and content feature information of each review. Considering the inconsistency between online review texts and the corresponding star ratings, this study uses a deep learning model to analyze the sentiment of online review texts, predict the sentiment intensity rating of each text, and compare it with the corresponding star ratings given by users to obtain the consistency weight. By combining objective and subjective factors, the feasible weight of each review can be more accurately calculated. In recalculating the composite ratings, this study split the reviews of an automobile from multiple sets by the location and time of purchase and calculated the composite ratings for each set within the set, taking full account of the impact of the location and time of purchase on the credibility of the reviews. Compared with similar existing studies, the research process, methods, and results of this paper are more interpretable and enlightening. In particular, in terms of user credibility, the proposed approach closely explores the personality characteristics disclosed by users. However, there are still limitations in this study; for example, the product quality complaint data are not considered, and the information is not comprehensive enough. Missing values for future user reviews are also not considered. This research is aimed at automobiles, and further research is needed on how to deal with other fields. In the future, further research and attempts will be made regarding the consideration of user personalized preferences in the ratings aggregation process, and the integration of product quality complaint data provided by users for product rankings.