Theory and Data-Driven Competence Evaluation with Multimodal Machine Learning—A Chinese Competence Evaluation Multimodal Dataset

Abstract

In social interactions, people who are perceived as competent win more chances, tend to have more opportunities, and perform better in both personal and professional aspects of their lives. However, the process of evaluating competence is still poorly understood. To fill this gap, we developed a two-step empirical study to propose a competence evaluation framework and a predictor of individual competence based on multimodal data using machine learning and computer vision methods. In study 1, from a knowledge-driven perspective, we first proposed a competence evaluation framework composed of 4 inner traits (skill, expression efficiency, intelligence, and capability) and 6 outer traits (age, eye gaze variation, glasses, length-to-width ratio, vocal energy, and vocal variation). Then, eXtreme Gradient Boosting (XGBoost) and Shapley Additive exPlanations (SHAP) were utilized to predict and interpret individual competence, respectively. The results indicate that 8 (4 inner and 4 outer) traits (in descending order: vocal energy, age, length-to-width ratio, glasses, expression efficiency, capability, intelligence, and skill) contribute positively to competence evaluation, while 2 outer traits (vocal variation and eye gaze variation) contribute negatively. In study 2, from a data-driven perspective, we accurately predicted competence with a cutting-edge multimodal machine learning algorithm, low-rank multimodal fusion (LMF), which exploits the intra- and intermodal interactions among all the visual, vocal, and textual features of an individual’s competence behavior. The results indicate that vocal and visual features contribute most to competence evaluation. In addition, we provided a Chinese Competence Evaluation Multimodal Dataset (CH-CMD) for individual competence analysis. This paper provides a systemic competence framework with empirical consolidation and an effective multimodal machine learning method for competence evaluation, offering novel insights into the study of individual affective traits, quality, personality, etc.

1. Introduction

Competence, a quality that distinguishes an individual from ordinary people, can be explained by a combination of traits, such as being skillful, efficient, knowledgeable, and intelligent [1,2,3]. Some researchers have defined competence as an underlying characteristic of an individual that is causally related to effective or superior job performance [4,5,6]. Others have related competence to a series of personal traits (describing the qualities of competent people, such as confidence, intelligence, and efficiency) and behaviors (describing the actions of competent people, such as analytical thinking, interpersonal understanding, and information seeking) [7,8,9]. Meanwhile, the social role of individual competence has been widely emphasized since it represents the advanced knowledge and skills required for communication and collaboration, information management, learning and problem solving, and meaningful participation, as well as attitudes related to the strategic use of these skills in intercultural, critical, creative, responsible, and autonomous manners [10]. For instance, in politics, highly competent candidates receive more support [11]. In marketing, competent salespersons earn more customer trust, thus leading to more extensive sales [12]. In an organization, a CEO who even “looks competent” is more likely to be selected and receive more salary compensation [13].

Given the importance of competence, researchers have conducted much research on competence frameworks [1,14,15,16]. Bogdan Wojciszke (1993) proposed that competence is related to three factors: “intelligence, willpower, and courage” (1993, p. 329) [14]. S. T. Fiske et al. (2007) built a classical competence framework including “perceived ability, intelligence, skill, creativity, and efficacy” (2007, p. 77) [15]. In subsequent studies, new dimensions for competence evaluation, such as independence, education, and effectiveness, have continued to be added [16,17,18]. Researchers have offered valuable insights into the concept of competence, and among the scales they developed, there are four main dimensions used as measures of competence: skill, efficiency, intelligence, and capability [1,14,19]. However, previous research has mainly focused on measuring competence based on individuals’ self-perception or others’ judgment using surveys, and the hard evidence of competence-related individual traits is quite limited. Moreover, existing studies have been disproportionately focused on a series of inner traits (e.g., intelligence and capability) related to competence. However, individual competence can actually be evaluated based on both inner and outer traits. For example, among inner traits, skill, intelligence, capability, and efficiency highlight individuals’ previous studies and are thus correlated to competence evaluation [14,16,19,20,21,22,23,24]. Among outer traits, facial and vocal cues play an important role in competence assessment [25,26,27,28].

Given the above, not only inner competence traits (i.e., skill, intelligence, capability, and efficiency) but also outer traits convey rich social information about competence evaluation. Therefore, there is a chance to evaluate perceived competence directly through easily observed individual traits. To fill these gaps, we developed a two-step empirical study that leverages machine learning and computer vision methods and proposed a framework and a predictor of competence evaluation based on multimodal data. This study aims to address the following questions:

(1)

What are the multimodal variables (i.e., outer and inner traits) that affect individual competence?

(2)

How do large-granular multimodal competence variables and small-granular multimodal competence features contribute to competence prediction?

(3)

How can cutting-edge deep learning models be utilized to make highly accurate predictions of competence by utilizing visual, vocal, and textual information?

To answer these questions, we conducted two studies from knowledge-driven and data-driven perspectives. In the knowledge-driven study (study 1), we first extracted 10 competence-related traits (both outer and inner traits) from the previous literature. Second, we obtained 4473 GB of 2573 video files that include the competence behavior. Third, two groups of experts annotated the perceived competence of individuals in the videos. Fourth, we employed machine learning methods, eXtreme Gradient Boosting (XGBoost) and SHapley Additive exPlanations (SHAP), to predict and understand individual competence. Fifth, to comprehensively understand competence and validate the results of machine learning, ordinary least squares [29] regression was used to provide economic evidence of the impact of 10 traits on competence evaluation. In the data-driven study (study 2), we accurately predicted competence with a cutting-edge multimodal machine learning algorithm, low-rank multimodal fusion (LMF), which exploits the intra- and intermodal interactions among all the visual, vocal, and textual features composing an individual’s competence behavior. In total, 43 dimensions of visual features were extracted by OpenFace [30], 24 dimensions of vocal features were extracted by Librosa [31], and 768 dimensions of textual features were extracted by BERT [32].

In study 1, the results of XGBoost prediction indicate that 10 traits contribute to competence evaluation, with a mean absolute error (MAE) value of 7.335 × 10⁻⁶, which is highly accurate, and the SHAP results indicate that 8 traits (vocal energy, age, length-to-width ratio, glasses, expression efficiency, capability, intelligence, and skill) contribute positively to competence evaluation, while vocal variation and eye gaze variation contribute negatively. In study 2, the LMF model with 3 modalities (visual, vocal, and textual) has high accuracy, with a value of 87.20%, highlighting the importance of information on individuals’ visual, vocal, and textual competence behaviors. In addition, the results indicate that the visual modality contributes the most to individual perceived competence, followed by the vocal modality and then the textual modality. It seems that “how an individual looks” and “how he speaks” are more important than “what he says.”

This paper offers the following four theoretical contributions: (1) This research extends prior research by integrating a more comprehensive competence evaluation framework with 10 traits, including not only 4 inner traits but also 6 outer traits that can be easily observed, thus helping to clarify the conceptual ambiguity surrounding competence [33]. (2) This study is among the first to directly and empirically predict competence with both 10 knowledge-driven traits and 835 data-driven features extracted from videos, bringing new insights into the evaluation of competence and generating more convincing results. While previous studies used small samples and evaluated competence based on human perception [1,16,18], the use of data-driven empirical evidence in this study provides a deeper understanding of an individual’s competence evaluation. (3) Methodologically, an advanced deep learning algorithm, LMF, is employed for the analysis of psychology-based issues. Both intra- and intermodal interactions among an individual’s visual, vocal, and textual information are mapped, thus leading to a more precise prediction of individual competence, which offers insights into machine learning analysis in psychological research [34,35,36]. (4) We provide the Chinese Competence Evaluation Multimodal Dataset (CH-CMD) for individual competence evaluation to the public. This dataset, comprising rich information on visual, vocal, and textual information, along with annotated competence scores, could encourage further studies of the relationship between inner and outer traits and individual competence. The framework of this paper is shown in Figure 1.

2. Theoretical Background and Hypothesis Development

2.1. Competence Framework

The word “competence”, an important concept in the management strategy literature since the 1990s, is “shrouded in theoretical confusion” [37]. There are three main ways to establish a coherent definition of competence: relating competence with a series of behaviors (e.g., learning style, work feedback, collaboration) [38,39,40], treating competence as a key dimensional stereotype according to social judgment theory [9,15], and associating competence with superior work performance [8].

Although the concept of competence has been constantly confused [33], academic endeavors have focused on building competence frameworks and applying them to a wide range of fields, such as personnel selection and performance evaluation. For instance, Bogdan Wojciszke (1993) proposed that competence is related to three factors: “intelligence, willpower, and courage” (1993, p. 329) [14]. Fiske (2007) built a classical competence framework including “perceived ability, intelligence, skill, creativity, and efficacy” (2007, p. 77) [15]. In subsequent studies, adjustments have continued to be made on this basis, and a series of new factors, such as independence, education, and effectiveness, have been added [16,17,18].

Table 1. Competence evaluation dimensions.

Source	Measurement			Our Measurement
Awale (2019) [18]	Incompetent	vs.	Competent
	Unconfident	vs.	Confident
	Incapable	vs.	Capable
	Inefficient	vs.	Efficient
	Unintelligent	vs.	Intelligent
	Unskillful	vs.	Skillful
Bogdan Wojciszke (1993) [14]	Unintelligent	vs.	Intelligent
	Timid	vs.	Courageous
	Lack of Will-power	vs.	Will-power
Aaker (2010) [1]	Incompetent	vs.	Competent
	Ineffective	vs.	Effective
	Inefficient	vs.	Efficient
Lebowitz (2015) [16]	Not Confident	vs.	Confident		1. Unskillful vs. Skillful
	Incompetent	vs.	Competent		2. Inefficient vs. Efficient
	Unintelligent	vs.	Intelligent		3. Unintelligent vs. Intelligent
	Incapable	vs.	Capable		4. Incapable vs. Capable
	Not Independent	vs.	Independent
	Not Competitive	vs.	Competitive
	Unskilled	vs.	Skilled
	Uneducated	vs.	Educated
van de Ven (2017) [17]	Incompetent	vs.	Competent
	Insecure	vs.	Self-assured
	Incapable	vs.	Capable
	Clumsy	vs.	Skilled
Fiske (2002) [19]	Incompetent	vs.	Competent
	Unconfident	vs.	Confident
	Not Independent	vs.	Independent
	Uncompetitive	vs.	Competitive
	Unintelligent	vs.	Intelligent

is a summary of the research on competence evaluation. In following previous work, we selected the four most commonly used dimensions to understand the nature of competence: skill, expression efficiency, intelligence, and capability (Table 1).

2.2. Competence-Related Traits

We integrated a more comprehensive competence evaluation framework that includes four inner traits and six outer traits of competence. In this subsection, the previous literature is reviewed to provide a theoretical background for the framework we propose.

2.2.1. Inner Traits

The inner traits used in this study include four traits extracted from prior studies: skill, expression efficiency, intelligence, and capability.

(a) Skill. Skills refer to particular ways to perform tasks well. Skill is a comprehensive concept that includes professional skills, learning skills, communication skills, and comprehension skills [16,19,20]. People tend to associate skill with competence since it is an indicator of expertise and high performance [41]. For instance, salespersons with analytical skills earn higher sales [42], adaptive selling skills enable salespersons to better position a firm’s products over those of its competitors [43], and interpersonal mentalizing skills help salespersons reduce burnout, thus improving their performance [44]. Thus, skill is anticipated to be positively correlated with competence.

(b) Expression efficiency. Efficiency refers to carrying out work processes efficiently [19,20]. Here, efficiency is mainly manifested by expression efficiency, such as organizing ideas into points clearly and using rhetoric (e.g., metaphors and exemplification) to convey information vividly [45]. Researchers have proven that the use of metaphors enhances the idea quality, as determined by listeners [46]; the use of exemplification helps listeners understand complex topics since it increases vividness and authenticity [47,48]. Expression efficiency is a critical component of high service quality [49,50] and client trust [51]. Thus, expression efficiency is anticipated to be positively correlated with competence.

(c) Intelligence. Intelligence is a measure of the extent to which an individual masters knowledge about a product. As individual-level intellectual capital, intelligence increases salespersons’ information-based behavior and provides a basis for identifying competence cues [12,52]. Higher performance in terms of selling activities is obtained by salespersons who are able to display product knowledge intelligently [53,54]. Therefore, intelligence is anticipated to be positively correlated with competence.

(d) Capability. Capability means that individuals possess the ability to perform a certain task [55], for instance, being capable of satisfying the needs of customers or having decision-making control over selling issues such as offering discounts and free shipping. Prior research has indicated that, in addition to skills, staff members also need empowerment to complete work tasks [56]. Salespersons who are capable of determining their sales tasks and courses of action have higher self-efficiency and sales performance [57]. Researchers have also found that by introducing capability into the competence framework, the robustness of the framework improves [24]. Given the above, capability is anticipated to be positively correlated with competence.

2.2.2. Outer Traits

The outer traits included six easily observed physical traits: age, eye gaze variation, glasses, length-to-width ratio, vocal energy, and vocal variation (Table 2).

(e) Age. Age represents the estimated age of an individual based on his appearance (rather than his exact age). People hold a series of positive stereotypes about older staff [58], since age is often associated with working experience [59]. For this reason, age has been recognized as a dimension in which individual skills and capabilities may systematically grow over time [60]. For instance, older staff often master higher physical [61] and organizational social skills [59], and they are perceived to be more professional than younger staff [62]. Therefore, age is anticipated to be positively correlated with competence.

(f) Glasses. Glasses refer to whether an individual is wearing glasses. In general, people believe that glasses are worn by those who are more intelligent, knowledgeable, and professional; thus, wearing glasses is an indicator of competence [63,64,65]. For instance, wearing glasses increases the perceived competence of candidates in political elections, thus leading to electoral success, and this effect is especially stronger in situations where intelligence is desirable [66]. For these reasons, wearing glasses is anticipated to be positively correlated with competence.

(g) Eye-gaze variation. Eye-gaze variation is a measure of the extent to which an individual shifts his gaze during a speech. Previous studies have validated the negative impact of gaze variation on individuals’ social perceptions, such as attractiveness [67], dominance [68], and confidence [69]. In video-mediated communication, gaze is defined as directly looking at a camera; thus, eye gaze variation means that an individual fails to maintain constant eye contact with the camera. However, even in online communication, eye gaze is very important. For instance, in clinical situations, physicians with a constant gaze in videoconferencing teletherapy are perceived to be more reliable [70,71]. In online education situations, instructors with limited gaze variation are deemed to be more confident and have higher educational performance [72]. Therefore, gaze variation is anticipated to be negatively correlated with competence.

(h) Length-to-width ratio. The length-to-width ratio relates to the shape of one’s face, and it is calculated by dividing the face length (measured from the top of the eyelids to the upper lip) by the width. Previous studies have examined whether the length-to-width ratio is positively related to an individual’s persuasiveness [73,74]. For women, a higher length-to-width ratio is associated with social capability, emotional capability, and attractiveness [75]. For men, a higher length-to-width ratio indicates less aggressiveness and is more appealing [76,77]. Therefore, the length-to-width ratio is anticipated to be positively correlated with competence.

(i) Vocal energy. Vocal energy is the energy of a human voice and is usually quantified by voice amplitude. A wealth of research has proven that voice energy is positively related to self-confidence [78,79,80] and competence [81]. It also provides information associated with an individual’s perceived dominance (Burgoon, Johnson, and Koch, 1998 [82]) and increases persuasiveness (Van Zant and Berger, 2020 [83]). Therefore, vocal energy is anticipated to be positively correlated with competence.

(j) Vocal variation. Vocal variation refers to shifts in one’s voice amplitude during speech. Vocal variation also carries rich, socially relevant information [84]. For instance, speakers who lack confidence and background knowledge about a topic tend to shift their voice during speech [85]. Previous studies have also suggested that larger vocal variations have a negative effect on speakers’ perceptual performance [86]. Therefore, while vocal energy is considered a positive indicator of competence, vocal variation appears to be considered the opposite.

Table 2. Competence-related variables.

Inner Traits	Definition	References
Skill	A comprehensive concept including the ability to adapt, learn, communicate, comprehend, and question.	[41,43,44]
Expression efficiency	The extent to which an individual exploits oral expression skills when they introduce products (i.e., organizing ideas clearly and using metaphors, examples, and other rhetoric).	[45,46,47,48,49,50,51]
Intelligence	The extent to which a salesperson is knowledgeable about the technical features and capabilities of products.	[12,52,53,54]
Capability	The extent to which an individual has decision-making control over selling issues.	[87,88,89,90,91]
Outer Traits	Definition	References
Age	The estimated age of an individual is based on his appearance.	[58,59,60,61,62]
Glasses	Whether an individual wears glasses	[63,64,65,66]
Eye gaze variation	Whether an individual makes direct eye contact with a camera	[69,70,71,72,92]
Length-to-width ratio	A measure of an individual’s face length (measured from the top of the eyelid to the upper lip) divided by the width.	[73,74,75,76,77]
Vocal energy	A term that expresses the perceived power of a voice and is quantified by amplitude.	[78,79,80,82,83]
Vocal variation	Shifts in a speaker’s voice amplitude during a speech.	[84,85,86,93]

Table 2. Competence-related variables.

Inner Traits	Definition	References
Skill	A comprehensive concept including the ability to adapt, learn, communicate, comprehend, and question.	[41,43,44]
Expression efficiency	The extent to which an individual exploits oral expression skills when they introduce products (i.e., organizing ideas clearly and using metaphors, examples, and other rhetoric).	[45,46,47,48,49,50,51]
Intelligence	The extent to which a salesperson is knowledgeable about the technical features and capabilities of products.	[12,52,53,54]
Capability	The extent to which an individual has decision-making control over selling issues.	[87,88,89,90,91]
Outer Traits	Definition	References
Age	The estimated age of an individual is based on his appearance.	[58,59,60,61,62]
Glasses	Whether an individual wears glasses	[63,64,65,66]
Eye gaze variation	Whether an individual makes direct eye contact with a camera	[69,70,71,72,92]
Length-to-width ratio	A measure of an individual’s face length (measured from the top of the eyelid to the upper lip) divided by the width.	[73,74,75,76,77]
Vocal energy	A term that expresses the perceived power of a voice and is quantified by amplitude.	[78,79,80,82,83]
Vocal variation	Shifts in a speaker’s voice amplitude during a speech.	[84,85,86,93]

3. Study 1. Knowledge-Driven Competence

3.1. Data

Online livestream videos provide abundant materials to comprehensively understand individual competence, including influencers’ inner and outer traits, which can be obtained through visual, vocal, and textual cues detected from their facial traits, voice, and speech transcripts. In this section, videos used for competence evaluation were randomly selected from an online livestream platform to extract both inner and outer competence-related traits (see Table 2). Four steps were taken to eliminate the heterogeneity of the videos. (1) The number of videos in each product category was roughly balanced to avoid product-type interference. (2) Videos of the top or bottom influencers were filtered out since these videos may contain too large or too small impacts on the prediction outcome, respectively. (3) The data were cleaned by filtering out videos (a) with no influencer or more than one influencer, (b) with no sound, and (c) with a moving camera. (4) All the transcripts transferred from audio files were proofread to guarantee the validity of our empirical results.

After these steps were completed, 4473 GB of video files were obtained. The videos were evenly distributed in terms of product categories, including beauty products, baby products, sporting goods products, and furnishings.

3.2. Variables Extraction

3.2.1. Inner Traits Extraction

The inner traits (skill, expression efficiency, intelligence, and capability) were obtained by computing the text similarity between the influencer’s transcript and an example transcript collection. First, we invited 10 experts in sociology and psychology to create an example transcript collection, which includes 100 sentences displaying the inner traits of skill, expression efficiency, intelligence, and capability. Next, the transcripts were cleaned and text segmentation conducted with the Jieba module in Python. Then, bidirectional encoder representations from transformers (BERT)-based word embeddings were used to transfer the transcripts into 768-dimensional sequence embeddings [32]. Finally, the cosine text similarity between the influencers’ transcripts and the example collection was calculated to measure to what extent an influencer exhibits these inner traits (see Appendix A).

3.2.2. Outer Traits Extraction

Among the outer traits, age, glasses, and length-to-width ratio were obtained using Face++ https://www.faceplusplus.com.cn/ (accessed on 17 June 2022), a platform that provides a face detection application programming interface [94]. This tool applies 106 facial landmarks on the human face in the video and marks the outline of the face, eye, eyebrow, lip, and nose contour, enabling the precise detection of facial traits. For instance, facial length can be measured by calculating the distance between the landmark on the top of the eyelid and that on the upper lip. Facial width is measured in similar a way, so the length-to-width ratio could be measured. Glasses can be measured by detecting whether the landmarks around the eyes are obstructed. Age can be evaluated through the skin detection API of Face++.

Gaze information was obtained using OpenFace 2.0 https://openface.io/ (accessed on 17 June 2022), an interactive application for facial behavior analysis. OpenFace 2.0 applies the CLNF framework, a general deformable shape registration approach, to detect the location of the eyeball and the pupil. CLNF gives the pupil location in 3D camera coordinates and computes the pupil’s intersection with the eyeball sphere. In this way, the gaze vector can be obtained by calculating the vector from the 3D eyeball center to the pupil location. Then, eye-gaze variation was measured by the coefficient of variation between an individual’s gaze from left to right and up to down.

Vocal energy and vocal variation were extracted from audio files using the Librosa speech toolkit at 22,050 Hz Here, all the audio files were segmented from the livestream videos by calling the moviepy module in Python. Vocal energy measures the amplitude of the influencer, and vocal variation measures the standard deviation of amplitude. The variable summary of our research is shown in

Table 3. Variable summary.

Variable	Description	Mean	SD	Min	Max	API Algorithm
Dependent Variable
Competence	Numeric, the value from the perceived competence evaluation	40.664	9.751	10	80	/
Independent Variables
Glasses	Binary, whether an individual wears glasses	0.011	0.104	0	1	Face++
Eye gaze variation	Numeric, calculation of the coefficient of variation between an individual’s gaze from left to right and up to down	0.056	0.055	0	0.402	OpenFace 2.0
Length-to-width ratio	Numeric, calculation of the length of an individual’s face (measured from the top of the eyelid to the upper lip) divided by the width	0.705	0.03	0.596	0.897	Face++
Vocal energy	Numeric, the amplitude of an individual’s voice	0.022	0.023	0.002	0.219	Librosa
Vocal variation	Numeric, the amplitude variation in an individual’s voice	0.011	0.009	0.001	0.102	Librosa
Expression efficiency	Numeric, a measure of the extent to which an individual exploits his oral expression skills	0.042	0.2	0	1	Text similarity
Skill	Numeric, a measure of the extent to which an individual demonstrates his sales skills	0.053	0.705	0.03	0.596	Text similarity
Intelligence	Numeric, a measure of the extent to which an individual demonstrates knowledge about a product’s technical features and capabilities	0.121	0.326	0	1	Text similarity
Capability	Numeric, a measure of the extent to which an individual uses a powerful style of speech	0.075	0.263	0	1	Text similarity
Age	An individual’s perceived age	25.454	4.354	13	47	Face++
Control Variable
Gender	Female influencers are labeled as 0, male influencers are labeled as 1.	0.096	0.276	0	1	Face++

Note: OpenFace 2.0 is a framework to implement a modern facial behavior analysis algorithm. Librosa is a Python package for audio analysis. Face++ is a platform offering computer vision technologies such as face detection, face search, face landmarks, and face masks.

.

3.3. Competence Factors with a Machine Learning Method

3.3.1. Model

In this section, our empirical research consists of three steps. First, to consolidate the framework of 10 competence-related traits and generate an accurate competence evaluation, a machine learning algorithm, XGBoost, was used to build the prediction model. Then, to make the results interpretable, SHAP was applied to rank the importance of the 10 traits. Finally, to further validate the results of the machine learning model, OLS regression was performed on the 10 traits and individual competence.

(1)

XGBoost

In this section, we employed the XGBoost model to study whether our framework can be used to predict individual competence. XGBoost, a scalable machine learning system for tree boosting, has been widely proven to yield state-of-the-art results in many prediction tasks [95]. As essentially a gradient tree boosting algorithm, in XGBoost, a scalable end-to-end tree learning system is applied. The core idea of XGBoost is to continuously conduct feature splitting to train new regression tree functions $f\left(x\right)$ and use them to fit the residual of the prediction in the previous round, as shown in Equations (1)–(4).

$${\widehat{y}}_i{}^{\left(0\right)}=0$$

(1)

$${\widehat{y}}_i{}^{\left(1\right)}=f_1\left(x_i\right)={\widehat{y}}_i{}^{\left(0\right)}+f_1\left(x_i\right)$$

(2)

$$\begin{gathered} {\widehat{y}}_i{}^{\left(2\right)}=f_1\left(x_i\right)+f_2\left(x_i\right)+={\widehat{y}}_i{}^{\left(1\right)}+f_2\left(x_i\right) \\ \dots \end{gathered}$$

(3)

$${\widehat{y}}_i{}^{\left(t\right)}={{\displaystyle \sum }}_{k=1}^t{f}_k\left(x_i\right)={\widehat{y}}_i{}^{\left(t-1\right)}+f_t\left(x_i\right)$$

(4)

Here, ${\widehat{y}}_i{}^{\left(t\right)}$ is the prediction at training round $t$.$f_t\left(x_i\right)$ is the new tree generated in round t. In every round, the new function $f_t\left(x_i\right)$ should be the one that minimizes the objective in Equation (5).

$$Ob{j}^{\left(t\right)}={{\displaystyle \sum }}_{i=1}^{n}l\left(y_i , {\widehat{y}}_i{}^{\left(t-1\right)}+f_t\left(x_i\right)\right)+\Omega \left(f_t\right)+constant$$

(5)

l denotes the loss function, and $\mathsf{\Omega }\left(f_t\right)$ is the regularizer that penalizes the complexity of the model. Finally, the prediction ${\widehat{y}}_i$ is the summation of the predicted values of all trees, as shown in Equation (6).

$${\widehat{y}}_i=\varnothing \left(x_i\right)={\displaystyle \sum }_{k=1}^K{f}_k\left(x_i\right)$$

(6)

By explicitly adding a regularizer to the model, XGBoost excels at avoiding overfitting and thus improving the generalizability of the model. In addition, a tree learning algorithm and justified weighted quantile sketch procedure are applied; thus, this model also excels at handling sparse data and achieving efficient calculation.

In our study, we trained and tested the model with video clips from the CH-CMD, which contains 2374 samples in total. We randomly selected 80% of the video clips as training data and the remaining video clips as test data. We set the learning rate to 0.01 and the number of trees to 100. Finally, in the task of predicting perceived individual competence, a mean square error (MSE) of 7.335 × 10⁻⁶ and a root mean square error (RMSE) of 0.002708 were obtained using this model. These results indicate that our competence evaluation framework contains valid information about perceived competence.

(2)

SHAP

In this step, SHAP, a machine learning interpretability approach, was applied to rank the importance of 10 inner and outer individual traits to competence evaluation. Based on cooperative game theory, SHAP uses Shapley values to measure the importance of each variable. It is assumed that when given the input of a set of variable values (the $j^{th}$ feature of the $i^{th}$ sample is $x_{ij}$) and the baseline of model $Y_{base}$, the machine learning model can output a particular prediction of $Y_i$. Since different feature values have different impacts, $f\left(x_{ij}\right)$ denotes a specific contribution of $x_{ij}$ to the prediction $Y_i$. Here, $f\left(x_{ij}\right)$ is the SHAP value of $x_{ij}$ and follows Equation (7).

$$Y_i=y_{base}+f\left(x_{i1}\right)+f\left(x_{i2}\right)+\dots +f\left(x_{ij}\right)$$

(7)

Here, $x_{ij}$ represents the 10 traits in the framework, $f$ is the XGBoost model built in the last step, and $Y_i$ (the output of the model) reflects the evaluation of the competence of each sample. SHAP exhibits two apparent advantages to meet our research requirements: (1) It presents not only the impact extent but also the impact direction of each variable: when the SHAP value is positive ($f\left(x_{ij}\right)>0)$, this specific variable of the $i^{th}$ sample contributes positively to the prediction of $Y_i$, and the reverse also holds true. (2) SHAP can exhibit individual-level importance and better visualize the contribution of each trait.

3.3.2. Results

Figure 2 displays the mean SHAP value of the 10 traits, which are sorted from high to low according to the mean SHAP value. The mean SHAP values reflect the significance of the traits in the competence evaluation.

In addition, Figure 3 shows the density scatter plots of the aggregate SHAP values across all input video samples to visualize individual-level trait importance. In this panel, traits are sorted in the same order as in Figure 2 and are denoted by lines consisting of scattered dots. Every pink or blue dot denotes the SHAP value of an input video sample. Pink dots indicate high trait values, while blue dots indicate low trait values.

From Figure 2 and Figure 3, vocal energy and vocal variation are among the two most impactful traits. While vocal energy has a positive impact on competence evaluation, vocal variation has a negative impact. These results are consistent with the theoretical analysis in Section 2.2. Eye-gaze variation is also important in the competence evaluation framework. However, its impact is contextual; that is, while it contributes to the competence evaluation of some individuals, it may hinder that of others (see Section 3.5). Facial traits, such as age, glasses, and the length-to-width ratio, also play important roles in the model. A higher estimated age could either increase or decrease the predicted competence. The length-to-width ratio contributes positively to competence evaluation. Wearing glasses positively impacts competence, but not wearing glasses negatively impacts competence.

Finally, expression efficiency, capability, intelligence, and skill have smaller impacts on competence evaluation, and their impact is not monotonic. They exhibit similar patterns in the scatter plot, indicating that a larger value has either a positive or negative impact on the predicted outcome.

3.4. Competence Factors with an Economical Method

To validate the results, we also used OLS regression to investigate the impact of individual traits on perceived competence (

Table 4. OLS model regression results.

Variable	(1)	(2)	(3)
Skill	0.0795		0.0987
	(−0.0757)		(−0.0741)
Expression efficiency	0.207 **		0.229 ***
	(−0.0851)		(−0.0834)
Intelligence	0.0644		0.0386
	(−0.052)		(−0.0511)
Capability	0.139 **		0.110 *
	(−0.0645)		(−0.0633)
Age		0.0195 ***	0.0191 ***
		(−0.0039)	(−0.0039)
Glasses		1.153 ***	1.158 ***
		(−0.162)	(−0.162)
Eye gaze variation		−0.316	−0.353
		(−0.305)	(−0.304)
Length-to-width ratio		1.489 ***	1.447 **
		(−0.565)	(−0.565)
Vocal energy		7.095 ***	7.001 ***
		(−1.18)	(−1.181)
Vocal variation		−6.427 **	−5.997 *
		(−3.113)	(−3.116)
Constant	4.035 ***	2.436 ***	2.447 ***
	(−0.0196)	(−0.414)	(−0.414)
Observations	3300	3300	3300
R-squared	0.004	0.044	0.048

Standard errors in parentheses *** p < 0.01, ** p < 0.05, * p < 0.1.

). Table 4 shows the regression results of the OLS model with the CH-CMD. In general, the results are consistent with those of the machine learning approach. Therefore, the credibility of the XGBoost model we built is validated, and our proposed competence evaluation framework is adequate for the prediction task.

3.5. Heterogeneity Analysis

In the previous subsections, we presented a ranking of the contribution of individual trait variables to the prediction of competence. Moreover, individual heterogeneity exists in the SHAP results (Figure 4). Figure 4 shows the SHAP values of two examples to demonstrate this heterogeneity.

For instance, age (with a value of 0.35), length-to-width ratio (with a value of 0.16), vocal energy (with a value of 0.12), and vocal variation (with a value of 0.08) contribute positively to individual 1′s competence but contribute negatively to individual 2′s competence. In addition, even though eye gaze variation contributes negatively to the competence of both individuals 1 and 2, the degree is different. Eye gaze variation negatively impacts individual 2′s competence by 0.24, while it negatively impacts individual 1′s competence by only 0.09. Therefore, the influence of these traits is contextual.

3.6. Discussion

In Section 3, we conducted study 1, that is, an experiment to explain and predict perceived individual competence from a knowledge-driven perspective. By considering the previous competence framework, we developed an enhanced competence evaluation framework (including 4 inner traits and 6 outer traits) to explain individual competence. To validate this framework, we conducted empirical research following two pathways: (1) First, we utilized a machine learning model. We applied the XGBoost model to predict individual perceived competence with the abovementioned variables and the SHAP method to realize the impact direction and feature importance of these variables. (2) Second, we used a classical econometric model. We conducted an OLS regression analysis to explore the general relationships between competence and these variables. Finally, an MSE of 7.335 × 10⁻⁶ and an RMSE of 0.002708 were obtained using the XGBoost model, and the results of these two pathways are consistent, indicating that the proposed framework is adequate for the competence prediction task and consolidates the prior competence theory.

First, the appearance traits of individuals, such as age, glasses, and length-to-width ratio, are positively related to perceived competence. From the SHAP results, it is noted that these variables contribute greatly to the prediction results. Second, eye gaze variation has a negative effect on the evaluation of individual competence. However, this impact is contextual, according to the SHAP results. Third, voice traits have the greatest contribution to the evaluation of individual competence. Between them, vocal energy has a positive effect, while vocal variation has a negative effect.

Meanwhile, expression efficiency, capability, intelligence, and skill correlate positively to individual competence in general. However, these traits contribute the least to the prediction of competence. This may indicate that inner traits are not critical to the embodiment of individual competence in a fast-paced livestream context.

4. Study 2. Knowledge-Driven and Data-Driven Competence

4.1. Dataset Construction

Extracting individual competence-related features from massive unstructured data in videos is a continuous challenge. In this section, we introduce the CH-CMD to capture and structure individual competence with deep learning at a smaller granularity. In the following subsections, we explain the details of the data acquisition and crawl system, dataset split, final statistics, preprocessing steps, and annotation.

4.1.1. Data Acquisition

All the videos used to extract features in the three modalities in study 2 were randomly crawled from a livestream platform. We built a new crawl system based on existing crawlers that allows us to filter out videos with more than one influencer on the screen (to ensure monologue videos) and with a moving camera (to ensure that all influencers’ intentions are focused on the camera). Then, the original videos (often with hours of content) were segmented into pieces of video clips that lasted for only a few seconds and contained only one sentence. A total of 1973 videos were crawled using this process.

4.1.2. Dataset Split

The Natural Split CH-CMD crawled without any guidance and comprises a representative sample of the natural distribution of competence polarity on the selected platform. However, it is not suitable for deep learning. In this subsection, we first discuss the statistics of the Natural Split CH-CMD and then discuss the process of guided crawling to acquire videos with more polarized competence to balance the dataset.

(1)

Natural split dataset

In this paper, the 1973 video samples were crawled from the selected livestream platform to capture the distribution of competence (Figure 5). Figure 5 demonstrates that the majority of the individuals in the video clips are neutrally competent (962), followed by weakly competent (356), weakly incompetent (319), incompetent (228), and competent (108). Without any guidance, the Natural Split dataset contains fewer polarized videos and reflects the original distribution of the livestream platform. However, for deep learning, the distribution of this unprocessed data is not ideal since there are too many videos with neutrally competent individuals.

(2)

Guided crawl

To compensate for the lack of polarized competence and balance the distribution of samples, we implemented stratified sampling in the crawl system: (1) First, we crawled videos of medium length to observe a greater fluctuation of competence; (2) Second, we analyzed the videos while crawling with data from three modalities (instead of analyzing them after crawling); and (3) Third, we filtered out videos of the top and bottom influencers since they generate too much or too little influence, respectively. After three rounds of filtering, controllable crawling by stratified sampling can be realized. In addition, we added manually annotated polarized videos from the Persuasive Opinion Multimedia (POM) dataset [96].

A total of 2374 videos were obtained for the Guided Crawl CH-CMD, which has a more balanced distribution compared to that of the Natural Split CH-CMD (see Figure 5) and facilitates the prediction of individual competence with deep learning approaches.

4.1.3. Preprocessing

Some video restrictions are set to filter out inappropriate samples:

(1) Using face detection technology, we chose only videos with one influencer on the screen to ensure that the videos contained only monologues. (2) Videos with moving cameras were filtered to ensure that influencers were concentrated on the camera.

With the high-quality transcripts we obtained by automatic speech recognition [34], we tokenized the final pool of videos into sentences using punctuation markers and randomly sampled a sentence from the transcript of each livestream video. Thus, we obtained video clips containing only one sentence that lasted for approximately 8 s.

4.1.4. Final Statistics

The final pool consists of 2374 livestream video clips in total. These videos were crawled from different commodity types, including beauty products, baby products, sporting goods products, and furnishings. We limited the number of IDs for each commodity type. These steps ensured the diversity of the videos.

The diversity of videos leads to the following generalized advantages: (1) The diversity of commodity types enables models to achieve generalizability across various commodity types and marginalize the definition of the dataset domain; (2) the diversity of commodity types leads to the diversity of individuals, allowing the training model to be implemented across different individuals and enabling generalization; (3) the diversity of commodity types also leads to the diversity of record settings, allowing the training model to be generalized through microphones and cameras with different intrinsic parameters.

4.1.5. Annotation

The videos chosen for the dataset were then annotated for competence by expert annotators on a Chinese crowdsourcing platform, and measures were taken to ensure the reliability of the annotation results. In this subsection, we first introduce the selection of annotators, review the definition of competence, and describe the annotation interface.

(1)

Annotator selection

The annotation of the CH-CMD was conducted by expert annotators from three sources: academia, industry, and crowdsourcing. Since we aimed to learn about individuals’ intuitively perceived competence, experts in academia were exclusively from business schools in universities (instead of psychology or medical schools). Experts in the industry included livestream and platform salespeople.

In addition, measures were taken to guarantee annotation quality. Expert annotators were trained before they conducted the annotation task. For instance, to avoid implicit prejudice and subjectivity, they received the following notification: “Please watch the video and annotate the competence of the speaker. Please note that you will only evaluate the individual competence of the speaker, not whether you agree with them”. Only the annotators with an acceptance rate greater than 98% were allowed to perform this task.

(2)

Competence annotation

Previous studies have provided a variety of definitions of competence and used different terms as a measure of competence. Table 1 presents a brief summary of the measurement of competence in previous studies. Among them, we chose 4 pairs of opposite items, “unintelligent/intelligent”, “unskillful/skillful”, “incapable/capable”, and “inefficient/efficient”, and averaged them to measure individual competence. Since these items have been widely used in published work, the reliability and validity of our annotation can be assured (see Section 2.2). The annotators were also required to utilize a 7-point competence scale, ranging from 1 to 7 (e.g., “unintelligent/intelligent”; 1 = “unintelligent” and 7 = “intelligent”).

(3)

Annotation user interface

The interface viewed by annotators in the process of annotation is shown in Figure 6. Before annotating, the annotators viewed a five-minute training video that provided instructions on how to use the annotation system. Then, a short livestream video was played. The annotators were allowed to rewatch the video at any time but were not allowed to perform annotations before the video had completely played at least once. In addition, at any moment during this process, the annotators were allowed to review the training video.

4.1.6. Extracted Features

The extracted features in each modality are as follows (for all experiments in Section 4.2, we extracted the same features):

(1)

Visual

Visual features are traits that can be captured visually. To extract visual features, we cut frames from the video clips at 30 Hz. First, with the multitask cascaded convolutional neural network (MTCNN) face detection algorithm [97], we obtained aligned faces. In addition, we applied the facial action coding system (FACS) to influencers’ faces to detect facial action units and better understand their facial expressions. Finally, we obtained 35 facial action units and 8-dimensional gazes using the OpenFace2.0 toolkit. Forty-three-dimensional visual features were extracted, as shown in Figure 7.

(2)

Vocal

Vocal features are the acoustic features of an influencer’s voice. Vocal features were obtained from the MP3 audio files that were separated from the video clips by calling the moviepy module in Python. Then, we used the Librosa speech toolkit to extract voice features from these audio files at 22,050 Hz. Finally, 3-dimensional frame-level voice features, including 20-dimensional mel-frequency cepstral coefficients (MFCCSs), RMS, zero crossing rate, spectral rolloff, and spectral centroid, were extracted. Twenty-four-dimensional vocal features were extracted, as shown in Figure 8.

(3)

Textual

Textual features are the characteristics of individuals’ language content when they introduce products. All text was obtained by transferring individuals’ voices into text with the ASR technique of the Alibaba Cloud. Since all the videos were obtained from a livestream platform, the transcripts were in Chinese. We used two tokens to label the beginning and end of each text. Then, we generated word vectors by using pretrained Chinese BERT-based word embeddings. Finally, each word was represented as a 768-dimensional word vector, as shown in Figure 9 and

Table 5. Extracted features.

Modality	Toolkit	Dims	Features
Visual	OpenFace	43	Eye features (8 dims)
			Facial action unit (35 dims)
Vocal	Librosa	24	MFCCSs (20 dims)
			RMS
			Zero crossing rate
			Spectral rolloff
			Spectral centroid
Textual	BERT	768	BERT-base word embeddings

.

4.2. Model Specifications

In prior works, we utilized visual, vocal, and textual features to predict competence scores. However, one problem remained unsolved (see Figure 10). If prediction outcomes generated by models with a single modality are inconsistent with each other, how can we integrate all the information to obtain a comprehensive competence score? In other words, how can the model map not only the intramodality information but also the cross-modality interaction information to obtain a prediction output?

To address this challenge, we applied a cutting-edge deep learning algorithm, LMF, for individual competence evaluation at the feature level [98]. This algorithm outperforms other deep learning algorithms in three ways: (a) LMF is characterized by using tensor representation, enriching the intermodal information in the model and leading to a more convincing prediction. (b) LMF is an efficient light computation method characterized by using low-rank tensors instead of explicitly creating a high-dimensional tensor. Therefore, LMF requires relatively low computing power, making it suitable for application and generalization in various scenarios. (c) LMF has also been proven to achieve high performance in prior studies, with relatively high accuracy and low MAE [98].

Given the above advantages of LMF, we chose it for competence prediction. We divided the 2374 video samples into an 80% training set, a 10% validation set, and a 10% test set for the LMF model. The operating principle of this method is the transformation of the input representation into a high-dimensional tensor $Z$ and then mapping it back to a lower-dimensional output vector space. Here, ${\left\{Z_m\right\}}_{m=1}^M$ denotes the unimodal vectors of M different modalities, and $h$ denotes the multimodal output representation. However, in the following steps, LMF does not explicitly create a tensor $Z$. Instead, three steps are taken to simplify the calculation:

First, the weight factor $W$ is decomposed. Generally, to map tensor $Z$ back to a lower-dimensional output vector space, it needs to be multiplied by a weight factor $W$, which is naturally a tensor of order −($M+1$) and denotes the weight of a linear layer. It can be partitioned into ${\tilde{W}}_k$(${\tilde{W}}_k\in R^{d_1\times d_2\times \dots \times d_M}, k=1, 2, \dots ,d_h)$. Moreover, every ${\tilde{W}}_k$ can be decomposed by Equation (8).

$${\tilde{W}}_k={{\displaystyle \sum }}_{i=1}^R\underset{m=1}{\overset{M}{{\displaystyle \otimes }}}{w}_{m,k}^{\left(i\right)}, w_{m,k}^{\left(i\right)}\in R_m^d$$

(8)

Here, the minimal R that makes the decomposition valid is referred to the rank of the tensor. Thus, the weight factor $W$ is expressed in Equation (9).

$$W={\displaystyle \sum }_{i=1}^r\underset{m=1}{\overset{M}{{\displaystyle \otimes }}}{w}_m^{\left(i\right)}$$

(9)

Second, tensor $Z$ must be decomposed. Currently, $h$ can be calculated, as shown in Equation (10).

$$h=\left({\displaystyle \sum }_{i=1}^r\underset{m=1}{\overset{M}{{\displaystyle \otimes }}}{w}_m^{\left(i\right)}\right)·Z$$

(10)

We continue the optimization by decomposing $Z$ in Equation (11).

$$\begin{gathered} h=\left({\displaystyle \sum }_{i=1}^r\underset{m=1}{\overset{M}{{\displaystyle \otimes }}}{w}_m^{\left(i\right)}\right)·Z \\ ={\displaystyle \sum }_{i=1}^r(\underset{m=1}{\overset{M}{{\displaystyle \otimes }}}{w}_m^{\left(i\right)}·Z) \\ ={\displaystyle \sum }_{i=1}^r(\underset{m=1}{\overset{M}{{\displaystyle \otimes }}}{w}_m^{\left(i\right)}·\underset{m=1}{\overset{M}{{\displaystyle \otimes }}}{Z}_m) \end{gathered}$$

(11)

Third, $h$ is reconstructed. In practice, a slightly different form is generally used to reconstruct $h$ in Equation (12).

$$\begin{gathered} h={\wedge }_{m=1}^M({\displaystyle \sum }_{i=1}^r{w}_m^{\left(i\right)}·Z_m) \\ ={\displaystyle \sum }_{i=1}^r{\{\underset{m=1}{\overset{M}{\wedge }}\left[w_m^{\left(1\right)}, w_m^{\left(2\right)}, \dots , w_m^{\left(r\right)}\right]·\widetilde{z_m}\}}_i \end{gathered}$$

(12)

Here, ${ \mathsf{\Lambda }}_{m=1}^M$ denotes the elementwise product of a sequence of tensors. Thus, h can be computed without explicitly creating the tensor Z from the input representations. $Z_m$, allowing the LMF model to be applied to any number of modalities and improving computational efficiency.

Using the LMF algorithm, our prediction task with inputs of individuals’ features in three modalities can be explained, as shown in Figure 11. First, we passed the unimodal inputs. $x_a$, $x_v$, and $x_l$ into three subembedding networks $f_a$, $f_v$, and $f_l$, respectively, so that we could obtain the unimodal representations $z_1$, $z_2$, and $z_3$, respectively. Here, every $z_i$ was appended to 1. Second, with the method of low-rank multimodal fusion with modality-specific factors, we obtained the multimodal output representation h. This method makes multimodal fusion efficient without compromising the model’s performance. Finally, the multimodal representation, h, was used to generate an evaluation of individual competence.

4.3. Results

An ablation study was conducted by comparing the performance of the 6 models (including 3 models with unimodal data, 2 models with biomodal data, and 1 full fusion model with trimodal data) that combine data in different modalities (textual, vocal, and visual). The results are shown in

Table 6. Modality ablation results.

Method		Deep Learning Model	Data Modalities Used by the Model			(a) Accuracy	(b) F1-Score	(c) MAE	(d) Corr
			Textual	Vocal	Visual
(1)	No fusion + unimodal data	LMF	√			0.8055	0.8057	0.7353	0.644
(2)				√		0.8207	0.8211	0.7015	0.6951
(3)					√	0.8328	0.832	0.6701	0.6893
(4)	Partial fusion + bimodal data	LMF	√	√		0.8419	0.8418	0.724	0.6471
(5)			√		√	0.845	0.8453	0.6581	0.6983
(6)				√	√	0.8419	0.7293	0.724	0.6471
(7)	Full fusion + trimodal data	EF_LSTM	√	√	√	0.8297	0.8299	0.7415	0.6304
(8)		LMF	√	√	√	0.8723	0.8717	0.6347	0.7188

. For the no-fusion model with unimodal information, relatively optimal performance, with a prediction accuracy of 0.8328, was obtained for the visual modality (see line 3 in Table 6). For the partial fusion model with bimodal information, the model with textual and visual modalities performed better than the other two models (see lines 4 to 6 in Table 6). For the full fusion model, the accuracy reached 0.8723 and was higher than the accuracies obtained using the no fusion and partial fusion models (see line 8 in Table 6). These results indicate that the LMF algorithm is adequate for the prediction task, validating the necessity of multimodal analysis. Figure 12 is the visualization of the model evaluation in terms of accuracy (a), Corr (b), F1-score (c), and MAE (d) values. According to Figure 12, the LMF model with visual, vocal, and textual modalities performed better than the other bimodal and unimodal models, highlighting the necessity of multimodal analysis.

In addition, we conducted a benchmark experiment with the same dataset using the early fusion long short-term memory recurrent neural network (LSTM-RNN), another classical deep learning model [99]. The results showed that the LMF model displayed better performance than the LSTM model in terms of multimodal analysis, with higher accuracy, F1-score, and Corr values and lower MAE values (see lines 7 and 8 in Table 6).

4.4. Discussion

In study 2, we explored individual competence from livestream videos at a smaller granularity from a data-driven perspective. With the crawled livestream videos from the livestream platform, we first constructed the CH-CMD and sent the livestream videos to a crowdsourcing platform for competence annotation. Then, we extracted features based on three modalities: visual, vocal, and textual. Finally, we utilized an efficient deep learning algorithm, LMF, to build our competence prediction model with the input data from the three modalities.

Our research confirmed the necessity of multimodal analysis in the livestream context, with the trimodal model (accuracy reaching 87.20%) outperforming the unimodal (accuracy by over 7%) and bimodal models (accuracy by approximately 3%). Most prior studies examined individual competence with intramodality information, but in the real world, modalities interact with each other, and they may influence individuals’ perceptions. For instance, visual information can influence the impact of vocal behaviors [100], and an individual’s verbal statements may affect his visual perception [101]. Therefore, it is essential to make predictions with full cross-modal information and employ multimodal analysis in our research.

In addition, the results of study 2 provided valuable insights into the impact of individual features in the visual, vocal, and textual modalities on perceived competence. First, the results of unimodal models indicate that models with visual features perform better than models with vocal or textual features (see lines 1 to 3 in Table 6). Second, the bimodal model with the visual modality performed better (see lines 4 to 6 in Table 6). These results indicate that the visual modality contributes the most to perceived individual competence, followed by the vocal modality, and then the textual modality. It seems that “how an individual looks” and “how he speaks” are more important than “what he says”. This is because the transcripts used by influencers in livestreams generally have similar patterns and wording [102], so competence is mainly demonstrated through visual and vocal features rather than textual features. This result is consistent with study 1: according to the SHAP results, influencers’ facial traits (such as eye gaze variation, length-to-width ratio, and age) and vocal traits (such as vocal energy and vocal variation) rank among the top traits, while traits extracted from text (such as intelligence, capability, expression efficiency, and skill) rank among the bottom traits. All of these results can be explained by “visual and demeanor biases”, which means that audiences often rely more on visual and vocal cues than textual cues to make judgments about an individual [100,103].

5. Conclusions

5.1. Discussion

In this article, we introduce two studies on individual competence from knowledge-driven and data-driven perspectives. In the knowledge-driven study (study 1), based on the XGBoost and SHAP results, both outer and inner traits contribute to competence evaluation. Moreover, outer traits (age, eye gaze variation, glasses, length-to-width ratio, vocal energy, and vocal variation) demonstrate even more significance than inner traits (skill, expression efficiency, intelligence, and capability) in competence prediction. This result was verified in the data-driven study (study 2), which indicates that the visual and vocal modalities are more important than the textual modality. In the no-fusion LMF, better performance is achieved by the model with visual and vocal features than by the model with textual features, and in the partial-fusion LMF, models with the visual modality outperform the model without this modality. Both studies convey a message: “what you say” is less important than “how you look” and “how you speak” [100,103].

5.2. Theoretical Contributions

This research makes five contributions to the theory.

First, this study extends the literature by integrating a more comprehensive competence evaluation framework. By including easily observed outer and inner traits, this framework provides a more convenient way to understand competence evaluation and helps to address the ambiguity in previous competence studies.

Next, this research empirically validates the extant literature and successfully predicts individual competence. While previous studies mainly mapped competence using surveys [1,14,19], this research measures competence with 10 knowledge-driven traits and 835 data-driven features extracted from videos, realizing more convincing prediction outcomes and reducing subjectivity [37].

This research also employs a cutting-edge deep learning algorithm, LMF, in competence evaluation. The full-fusion LMF model outperforms other models, demonstrating the necessity and legitimacy of multimodal analysis. The results also provide information about which modality is more important for competence evaluation.

Finally, this research provides academia with the CH-CMD, which contains rich multimodal information for individual competence evaluation. This dataset is among the first to be used to explore individual competence through livestream videos, encouraging further research on individual affective traits, quality, personality, etc.

5.3. Managerial Contribution

The competence evaluation framework in the research can be adopted by both individuals and organizations. First, at the individual level, influencers in livestreams, online teachers, and even candidates in remote interviews can apply this framework to reexamine and improve their competence in front of a camera. Simple adjustments can be made to one’s behaviors according to the framework (such as wearing glasses and adjusting one’s voice energy).

Second, at the organizational level, this framework is useful. For instance, online stores could select influencers with high competence scores before formally signing contracts with them instead of choosing inadequate influencers who may have poor performance on a livestream. Online education platforms could apply this framework to send lectures and courses given by the most competent teachers to the home page or recommend them to users. Personnel departments could apply this framework to self-introduction videos from job seekers and decide on the list of candidates who can enter the next round of interviews.

Third, technically, this research adopts cutting-edge machine learning algorithms to analyze individual traits. With the light computation LMF method, the model can conduct competence evaluations efficiently without consuming excessive computing power and time [98]. Therefore, this technique can be easily applied and generalized by businesses.

Finally, the CH-CMD generated in this research is publicly available, encouraging the industry to analyze individual competence and explore its business application. Moreover, this dataset also fills the gap that few datasets contain information on Chinese influencers’ competence traits.

6. Limitations

Finally, three limitations of this study call for further research.

First, our research mainly explores the correlation between individual competence and the 10 traits (4 inner traits and 6 outer traits). However, it did not explain the causality between them. Whether these traits indeed cause an increase or decrease in competence evaluation and “how” these traits influence competence (a mechanistic study) require further research.

Second, we trained and tested our competence evaluation model with livestream platforms, since livestream videos facilitate our research on individual competence in three modalities. However, regretfully, due to difficulties in data acquisition, we did not validate our model with other video platforms or other business settings this time, which calls for further research to study the application of this model in different contexts, such as online education, online interviews, and remote diagnosis.

Third, this research focused on providing a Chinese competence evaluation multimodal dataset and testing and training the models with Chinese livestream videos. Whether these models can be generalized to other ethnic groups and cultural backgrounds remains an open issue. This limit calls for more diverse samples from different ethnic groups and cultural backgrounds to be involved in the dataset in future research and for more advanced algorithms supporting cross-language analysis to be applied in this area.