Using LinkedIn Endorsements to Reinforce an Ontology and Machine Learning-Based Recommender System to Improve Professional Skills

Abstract

Nowadays, social networks have become highly relevant in the professional field, in terms of the possibility of sharing profiles, skills and jobs. LinkedIn has become the social network par excellence, owing to its content in professional and training information and where there are also endorsements, which are validations of the skills of users that can be taken into account in the recruitment process, as well as in the recommender system. In order to determine how endorsements influence Lifelong Learning course recommendations for professional skills development and enhancement, a new version of our Lifelong Learning course recommendation system is proposed. The recommender system is based on ontology, which allows modelling the data of knowledge areas and job performance sectors to represent professional skills of users obtained from social networks. Machine learning techniques are applied to group entities in the ontology and make predictions of new data. The recommender system has a semantic core, content-based filtering, and heuristics to perform the formative suggestion. In order to validate the data model and test the recommender system, information was obtained from web-based lifelong learning courses and information was collected from LinkedIn professional profiles, incorporating the skills endorsements into the user profile. All possible settings of the system were tested. The best result was obtained in the setting based on the spatial clustering algorithm based on the density of noisy applications. An accuracy of 94% and 80% recall was obtained.

1. Introduction and Related Studies

Employability requirements have changed to address the new realities and challenges faced by organizations. According to [1] cited by [2], in several fields, some discrepancies emerge between employee competences and the labor market. In view of this, professionals must improve and/or develop their competences and skills according to trends in the labor market in order to be competitive.

Nowadays, organizations can use different Social Networks Sites (SNSs) to research into candidates [3], as well as to gather information on labor trends so as to adjust academic plans, analyze student profiles and extract their skills, and identify the most desired skills for the purpose of adjusting curricula [4].

In the professional field, LinkedIn has played a very important role in terms of the dissemination of jobs, where, according to the work presented by one of its missions is to match jobs with suitable professionals [5].

To support this process, the Recommender Systems (RSs) have emerged, which collect the information linking users to items and use it to make relevant and meaningful suggestions. By [6] the RSs base the prediction of user interests, according to their explicit or implicit preferences. RSs are not only used to find the ideal candidate for a job position, but also to offer educational services to professionals to develop professional skills.

RSs use different filtering algorithms, in which the predominant approaches are collaborative filtering, content-based filtering and hybrid filtering. This combines the two approaches so as to overcome any problems that may arise through individual use of the different techniques, such as the problem of data sparsity, or owing to the lack of information provided by users about the recommended items, which is known as cold start [7], and thus improve recommendation performance [8].

On the other hand, the quality of recommendations is highly dependent on information retrieval and representation. According to [2], RSs handle a large amount of data that, depending on their nature, can derive from different sources such as SNSs, which are characterized by their heterogeneity. In this regard, semantic RSs have emerged, which make use of the semantic web to represent reality in a specific domain, in a machine-readable way [9].

The work presented by [2] involves a review of articles that propose the use of ontology in order to create a model to describe the relationship between a domain area and the users that belong to it.

In this review, the importance of the representation and management of information is highlighted. Studies [10,11] pointed out that ontologies allow information of interest to users to be inferred, which can be used to enrich recommendations. Likewise, [12] indicated that the amount of information handled is a key factor in the success of recommendations. Studies [11,13] pointed out that the use of ontology improved cold start and data sparsity problems, which limit RS performance. [14] also suggested that the use of ontologies facilitates the analysis, reasoning, sharing and reuse of knowledge.

In recent years, artificial intelligence techniques have been introduced in RSs, where Machine Learning (ML) techniques are included [15]. According to [16], the use of these techniques has proved to be a promising solution when designing RSs in the era of Big Data.

According to [17] ML algorithms have become dominant in big data analysis, visualization and modeling. For [18] ML applies mathematical models and algorithms to gradually improve the performance of a task and can be categorized into two main groups: supervised and unsupervised learning. ML algorithms are performing tasks like classification, regression, clustering, dimensionality reduction, and ranking.

In the systematic review presented by [19], attention should be drawn to the work shown in

Table 1. Use of ML in recommender systems.

Ref.	Year	Objective	Filtering Technique	ML Technique	Metrics
[20]	2015	Recommend LLL courses according to common professional interests	Collaborative + ML	ML supervised	Accuracy/Recall
[21]	2018	Recommend courses according to academic profile and jobs on offer	Collaborative + Content + ML	ML supervised	Accuracy/Recovery
[22]	2019	Recommend exercises based on information regarding academic performance	ML	ML non-supervised	NDCG **, MAE * and F1 ***
[23]	2019	Recommend courses to maximize learning based on past performance	ML	ML by way of support	Accuracy/F1
[24]	2019	Recommend consultancy according to academic-industrial research interests	ML	ML supervised	NDCG
[25]	2020	Recommend learning resources according to context, taking colleagues’ learning into account	ML	ML non-supervised	Accuracy/Recall/F1

MAE *: Mean absolute error; NDCG **: Normalized Discounted Cumulative Gain; F1 ***: Harmonic mean between precision and recall.

in this area.

In some cases, ML is used in RSs to make inferences about how past actions correspond to future outcomes. [23] applies this technique to predict future performance based on students’ academic record in order to re-schedule courses for target course preparation and exam preparation. [24] proposes an RS for different users in academia, using ML to extract information about research areas of interest to professors from publications and curricula, and then combining them with their background to assign courses and research work for supervisory purposes.

Additionally, semantic web techniques have been combined with ML techniques, allowing knowledge to be exploited from available information, updated and new relationships between data inferred [14]. The combination of an ontology-based recommender system with ML techniques has been used as an approach to improve the accuracy of recommendations, in addition to addressing information overload, with a view to solving cold start problems [26].

Based on the above [2], they propose a hybrid RS to suggest lifelong learning (LLL) courses to improve professional competences for users whose profile is built from their LinkedIn profile data. The RS core involves semantic filtering that uses ontology to model employment sectors and areas of knowledge to represent professional competences. The ontology is updated using events based on professional record profile data extracted from LinkedIn, using ML to cluster entities in order to make predictions about the new data. As a line of future work, they recommend considering the use of LinkedIn endorsements to enrich the user profile and thus improve recommendations.

Recently, there have been several research work oriented analyses of data provided by LinkedIn, which in addition to data related to academic and professional training, also highlights endorsements, which is one of the key features of LinkedIn, asking the viewer to endorse a skill for the candidate as proof of their skill level [27].

In a review of relevant literature, we found the research shown in

Table 2. Research related to the use of LinkedIn endorsements.

Ref.	Year	Objective	Use of Endorsements	RS
[4]	2015	Determine a set of competences required by the market, in which more focus should be placed on the curriculum.	To determine the weight of a skill (beginner, intermediate, advanced).	YES
[28]	2016	Identify the motivation behind behavior in the use of LinkedIn.	To analyze the reliability of skills endorsements.	NO
[29]	2017	Determine new skills endorsements.	To deduce new endorsements.	NO
[30]	2018	Propose an evaluation framework for evaluation methods of competences via regression.	To analyze endorsements, to determine which ones are false.	NO
[27]	2019	Estimate the experience gained by LinkedIn members and objectively identify skills.	To measure the level of skills.	NO
[31]	2020	Measure reliability of a candidate for LinkedIn profile startups.	To determine a user’s skills.	NO
[32]	2021	Job recommendation.	To determine false endorsements.	YES

, related to the use of LinkedIn endorsements.

This is highlighted in [4], in which an RS is proposed that classifies profiles according to skills and peer endorsements, identifying the most desired skills that should be covered by curricula and areas of learning, and suggesting possible corrections in learning programs. [32] indicates that in addition to this information, endorsements of the skills of LinkedIn users can be analyzed to complement the profile.

The system proposed by [32] recommends relevant job opportunities for users seeking employment in the area of information technology, based on the curriculum to be entered into the system. In addition, it makes it easy for recruiters to search for the best talent based on job requirements by analyzing not only the resume, but also LinkedIn endorsements, which could improve the user profile.

Based on the above, an improvement in RS of [2] is proposed, incorporating LinkedIn endorsements as an additional system input. These are taken into account in order to validate the skills determined in the RS profiles to improve recommendations.

In literature on the subject, we found some RSs aimed at developing professional skills through a wide range of training. In this respect, the objective of the RSs proposed by [33,34,35] is to identify the professional competencies that users need to develop in their current job, so that they can be taken into account in their training plan.

For [4], along with RSs, the analysis of SNSs data has become a common practice for collection of information about users, since [36], SNSs such as Facebook and LinkedIn have become integrated into everyday life.

Ref. [37] indicates that social media provide innovative pedagogical frameworks for teaching and learning that allow students to develop digital skills deemed useful for a successful professional career. He also points out that semantic social networks offer a series of advantages related to their strategic use, such as the creation of a network of contacts, which can have an impact on their professional development by connecting with professionals and following labor market trends.

As the use of SNSs has increased decade by decade, so has interest in using SNSs in recruitment and hiring processes [3]. In this sense, for [28], LinkedIn is the most influential web tool in terms of professional use, unlike other SNSs such as Facebook or Twitter, which focus on social relationships. In LinkedIn, users have greater visibility in professional terms, as they can share their training, skills and work experience [4].

For [37], LinkedIn focuses on professional networking and career development, designed to help people make connections, share experiences and resumes, and find jobs. It is also a tool that can be used to find relevant, quality content. According to [3], allowing users to provide written recommendations and endorsements of skills that appear on a user’s profile, LinkedIn has also introduced endorsements, in which users are asked to endorse the skills of other users [27].

The data contained in SNSs are very diverse [3], and RSs also obtain information from multiple sources; to handle heterogeneity, they have adopted semantic knowledge representation as one of the theories to help deal with this [26].

Based on this review and lending continuity to the RS presented by [2], it is proposed that LinkedIn skills endorsements be incorporated in order to validate the skills obtained in the user profiling stage to optimize performance and prediction of the continuing education course recommendation system. The system uses LinkedIn user records in the area of software development.

This article is structured as follows: Section 2 details the RS proposal; Section 3 describes the evaluation of the proposal; and Section 4 provides conclusions and recommendations for future research.

2. Materials and Methods

2.1. Materials

In the field of learning and in personalized recommendation services, different user dimensions, such as personal data, interests, knowledge levels, and context, among others, must be taken into account in order to respond appropriately to their needs [38]. Based on this, LinkedIn records used by [2] were employed to create user profiles, which contained personal, academic and professional data of users, as they in turn contain relevant information to help determine the area of knowledge, skills and labor sector that are taken into account when recommending courses. Additionally, LinkedIn endorsements were incorporated. For the purposes of evaluating the proposal, this was confined to the specific domain of software development.

For user and course profiles, professional skills and job sectors were coded using the taxonomies defined in [2].

The ontology used was stored in Neo4j, and selection of the tool in [2] was based on the fact that it is a NoSQL graph database manager, whereby the relationship is the most important database element representing the interconnection between nodes. This makes it ideal to represent knowledge graphs, allowing nodes and relationships and efficient operations to be managed on them.

As for the different programs that make up the system, the ones developed by [2] were used as a basis and programmed using Python 3. Cypher, the Neo4j query language, was used to query the graph.

2.2. Methods

In order to recommend continuing education courses to improve and develop professional skills, a hybrid RS based on ontology and ML is planned to determine the skills to be updated and/or developed according to labor market trends. User profiles are created from their LinkedIn records, taking into account endorsements, in order to validate and determine the level of skills. The recommendation process has three filtering stages at whose core is a semantic filtering, which is combined with filtering by content for the initial prediction of courses, and a heuristic one to obtain the final range of courses to be recommended.

The following is established as a basis for designing the RS:

The system input data is obtained from LinkedIn, to create user profiles, and from the web for courses, without requiring any additional information upload by the users.

Courses will be recommended using the skills they develop as a criterion. Skills in the user and course profiles need to be defined under the same terms in the recommendation process.

Modular design allows new algorithms to be incorporated into the system and multiple configuration options offered, as well as including parameterizations for the different algorithms, by enabling or disabling filtering stages in order to validate and evaluate the proposals.

The RS architecture proposed is shown in Figure 1.

The system consists of two main phases: the off-line phase (I), in which the user profiles (A) and (B) are built, definitions are loaded and data models are built from the ontology (C) and ML (D) update processes; an on-line phase (II), in which the course recommendation process is performed, for which purpose the user profile is to be given the recommendation (1) must be built, and which is performed in three filtering stages (2, 3 and 4).

2.2.1. Off-Line Phase

Profile Creation

A semantic profiling based on ontology is performed in order to generate the different profiles used, making use of taxonomies. This process consists of the user profiling process (A) and the course profiling process (B).

After analyzing the records of LinkedIn users and LLL courses extracted from the Web, two taxonomies were then defined to hierarchically code the areas of knowledge and job performance sectors needed for the recommendation process. The levels defined for the areas of knowledge are: area; sub-area; specialty; sub-specialization; and knowledge. For job performance sectors, the hierarchy levels are: sector; field; type; domain; and position.

The following are coded with a taxonomy of areas of knowledge: user skills with their level of specialization (LS) and their degree of updating (DU), the skills to be developed on LLL courses with an LS, and also the skills required to qualify for a course. The job performance sector taxonomy is used to hierarchically code the different positions held by users.

The user profile information is given by:

-

Demographic information.

-

Job performance sectors. This is a set of hierarchical codes defined by a taxonomy for job performance sectors, their description and dates.

-

Skill sets, where each skill is coded with three arguments: a hierarchical code that is defined by a taxonomy of areas of knowledge, a level of specialization and degree of update.

Course profile information is provided by:

-

Demographic information.

-

Skills to be developed. Each skill is coded with two arguments: a hierarchical code defined by a taxonomy of areas of knowledge and a level of specialization.

-

Required skills, where each is coded with a hierarchical code that is defined by a taxonomy of areas of knowledge.

-

Related skills. These are given by a set of codes of areas of knowledge, where the skills developed on the course may be of interest or complement them.

Ontology and ML

A key element for smooth performance of RSs is to ensure that there is a wealth of semantic data mining, and to prevent the loss of information obtained from data retrieval. Another process associated with this phase aims at building data models to represent the domain of skills and job performance sectors, according to areas of knowledge.

The ontology, in its broadest sense, seeks to represent positions, skills and existing relationships between them. Additionally, it should offer the capacity to represent different groupings of positions under the criterion of similarity according to the skills they use. Likewise, the different relationships that can be found between skills must be represented. The classes defined in the ontology are shown in

Table 3. Classes of ontology.

Class	Description	Type
Area	General category, into which an area of knowledge is divided, e.g.,: Computing	Entity
Sub-area	Different general categories, which contain a sub-area of knowledge, e.g.,: Languages	Substrate
Specialization	Different branches of specific knowledge, which contain a sub-area of knowledge, e.g., Programming Languages	Substrate
Sub-specialization	Different categories, which contain a specialization. Groups together common knowledge of different specific areas of knowledge, e.g., Object-oriented Programming Languages	Substrate
Knowledge	Specific knowledge belonging to a sub-specialization. Corresponds to the most specific level (leaves) in the area of knowledge hierarchy. May denote the domain of technologies, tools, software, etc., e.g., Java Programming Language	Substrate
Sector	General category, into which labor domain is divided, e.g., Information and Communication Technology	Substrate
Field	Different general categories, which contain a labor sector, e.g., Software Development	Substrate
Type	Nature of the different positions contained in field, e.g., Developer	Substrate
Domain	Specific areas contained in a type of position, e.g., Development of Mobile Apps	Substrate
Position	Specific position in a position domain: Corresponds to the most specific level (leaves) in the job performance sector hierarchy, e.g., Development of iOS Native Mobile Apps	Substrate
Requirement	Specific requirements of a position. There may be different types (desirable, mandatory). Related to age, experience, and academic background, among others	Substrate
Skill	Level of experience and degree of update associated with an area of knowledge	Updated via event
Cluster	Group of similar positions	ML

.

Likewise, class hierarchy is given by the different relationships between classes of ontology as described in

Table 4. Ontology relationships.

Relationship	Description	Cardinality	Classes
Is a/an	Belongs to a category	1:1	Between underlying classes belonging to a hierarchy
Ia the same	Descriptions are synonymous	1:1	Between the same classes belonging to a hierarchy
Is of interest	Knowledge is of interest to other areas of knowledge	1:N	Between areas of knowledge
Involves	A skill involves or refers to a specific area of knowledge	1:1	Skill ⟶ Knowledge
Possesses	A position makes use of a set of skills A cluster possesses a set of skills such as Centroide	1:N	Position ⟶ Skill Cluster ⟶ Skill
Includes	A cluster includes a set of positions	1:N	Cluster ⟶ Position
Has	A position has a set of requirements	1:N	Position ⟶ Requirements

.

The ontology is updated in two stages from user profiles: through events (C), where the model is updated with the relationships between job performance sectors and areas of knowledge according to user profile competences; through machine learning (D), the positions of the entities in the ontology are grouped together using the density based spatial clustering of applications with noise algorithm (DBSCAN) and K-means algorithms.

For [39] K-means algorithm mainly uses the Euclidean distance function to measure the similarity between data objects, and the sum of squared errors is used to found spherical clusters with more uniform distribution of data.

The work presented by [40] defines DBSCAN as a clustering algorithm based on data point distribution density. The algorithm can identify the degree of data density and classify the data points in the distribution. At the same time, sporadic data points can be identified as noise, rather than being classified within a class.

In the event driven update, the ontology stores the relationship between the characters and areas of knowledge, which is given by skills with attributes, as well as the number of times that skill is present, its LS average and the promise of DU, which is calculated taking into account the number of users that possess the skill. The relationships are determined, and their attributes calculated, from a training dataset of user profiles via events in the ontology.

For the ontology substrate, new entities, relationships and attributes are defined in the taxonomies and inferences that are allowed to be made, which will be used in semantic filtering. Among the new relationships, synonyms and uses of terms in other languages are contemplated—for the purposes of testing this proposal, the use of terms in Castilian Spanish and English were both considered, due to their common use in the domain used. In the case of areas of knowledge, relationships of the type “is of interest” are used in their entities to indicate that users who possess a skill related to those entities may be interested in developing other skills with other knowledge, in order to obtain a more comprehensive profile.

Additionally, in work performance sectors, the position entities can have “opt for” relationships, where from one position one can opt for similar or higher positions in an orderly relationship defined by a ladder.

Job performance clusters are created in the machine learning update. An alternative for calculation of related skills involves using ML. It is proposed that unsupervised clustering algorithms be used in instances of position entities, in order to group similar positions, with these groupings being based on the similarity of skills associated with the relevant positions. Job performance charge clustering, based on the similarity of the skill set, can be used to determine the set of related positions for a particular position, or a particular user’s skill set. From the job grouping determined, related skills can be established as those skills that appear most frequently in the positions in the grouping, and which the user does not possess.

2.2.2. On-Line Stage

The course recommendation process is carried out at this stage. As shown in Figure 1, this is made up of the objective user profile construction process (1) and recommendation process, which is carried out in three filtering stages (2), (3) and (4).

• (1) Process involved in creating the target user profile

This process is similar to the profile construction of the off-line stage. It differs in the semantic transduction stage in that it incorporates coding of the endorsements in the user profile. In the information obtained from the LinkedIn record of the user to whom the recommendation is to be made, are the endorsements and use of skill coding algorithms, and the hierarchical code of the area of knowledge is determined. To determine the LS and DU, heuristics are used that take into account the relative frequency of the number of users who endorse the skill. The endorsements are part of the profile of the user to whom the recommendation is to be made and are taken into account in the recommendation process as skills that the user has acquired.

• Recommendation process

The recommendation process followed is the one proposed by our previous work [2], where the first stage corresponds to a modification of the semantic filtering algorithm (2) in order to determine the user’s own, related and ontological skills, as given by the incorporation of LinkedIn endorsements. Then a second content-based filtering (3) is applied for initial course prediction, and finally, filtering and sorting heuristics (4) are applied for the final recommendation of LLL courses to improve and/or develop professional skills based on a user’s record.

The process diagram represented in Figure 2 is followed in order to prepare course recommendations for each user.

• (2) Semantic filtering to determine similar skills

In order to determine related skills for a user, we first determine his or her related work performance areas using the ontology, based on his or her skills. For this process, the set of work performance sectors where the user’s own skills appear is determined and filtered as follows: those related work performance sectors whose skills are covered by a given percentage of the user’s skills are selected, and/or those related work performance sectors whose associated skills cover a percentage of the user’s own skills are selected. In any case, the user’s job performance sectors are excluded from the set of related job performance sectors.

Alternatively, from the user’s job performance sectors it can be determined to which groupings (clusters) they belong, constructed according to ML algorithms. In the case of new data, i.e., a job performance sector that is not part of any of the clusters obtained during training, it is possible to predict the cluster, to which the user belongs, by making use of the user’s skills. Given the groupings, the set of job performance sectors that belong to each of them will form the user’s related performance sectors.

In the case of DBSCAN, a core node with a distance less than an epsilon value is located and associated with the cluster, to which it belongs, to predict clustering for new data; in the case of k-means, the centroid with the smallest distance is located and associated with that cluster. In both cases, the difference with one of the similarity functions is used as distance, according to Equation (1).

$$d=1-Sim\left(S{K}_u,S{K}_c\right),$$

(1)

Finally, which related work performance sectors are deemed prior to the user’s work performance sectors are determined in an orderly relationship imposed by a ladder, and these are removed from related performance areas. Subsequently, with sectors of work performance, the user’s own and related skills are determined, which are the most frequent skills associated with related sectors of work performance that add value, in other words, that the skill does not exist among the set of skills or, if it does, that it has a higher level of specialization or degree of update than the user’s skills; finally, those skills that add value and are included under the relationship “is of interest” in the knowledge network are added to related skills.

Given the inputs:

$\mathrm{UJS}$, Set of user job performance sectors.

$\mathrm{USK}$, Set of user skills.

$\mathrm{UESK}$, Set of validated user skills.

The following Algorithm 1 is used to semantically determine related skills (RSK), the result of modifying the algorithm described in [2], after taking endorsements into consideration:

Algorithm 1: Semantic determinining of similar skills.
Input$: UJS, USK, UESK$
1:	$RJS\leftarrow \varnothing$ // Related job sector
2:	$RS{K}^{’}\leftarrow \varnothing$
3:	To$sk\in UKS$ obtain
4:	$\hspace{1em}RJS\leftarrow RJS\cup Onto\_GetJobs\left(js\right)$
5:	$RJS\leftarrow RJS-UJS$
6:	$RJS\leftarrow Onto\_CutOff\_Jobs\left(RJS,USK,ujkP,jskP\right)$
7:	$RJS\leftarrow RJS-Onto\_Previous\left(RJS,UJS\right)$
8:	To js ∈ RJS obtain
9:	$\hspace{1em}RS{K}^{’}\leftarrow RS{K}^{’} \cup \left\{sk, SV\left(USK\cup \mathrm{UESK},sk\right)\wedge sk\in Onto\_Skills\left(js\right)\right\}$
10:	To$sk\in \left(USK\cup MS2S\left(RSK,\phi \right)\right)$ obtain
11:	$\hspace{1em}RS{K}^{’}\leftarrow RS{K}^{’}\cup \left\{s{k}^{’}, SV\left(USK\cup \mathrm{UESK},s{k}^{’}\right)\wedge s{k}^{’}\in Onto\_IntSkills\left(sk\right)\right\}$
12:	$RSK\leftarrow MS2S\left(RS{K}^{’},\lambda \right)$
Output$: RSK$

The following Algorithm 2 is used to alternately determine similar skills using ML clustering:

Algorithm 2: Calculation of similar skills.
Input$: UJS, USK, UESK$
1:	$RJS\leftarrow \varnothing$ // Related job sector
2:	$RS{K}^{’}\leftarrow \varnothing$
3:	To$js\in UJS$ obtain
4:	$\hspace{1em}RJS\leftarrow RJS\cup \left(Onto\_Cluster\_Pred\left(js\right)-\left\{js\right\}\right)$
5:	$RJS\leftarrow RJS-Onto\_Previous\left(RJS,UJS\right)$
6:	To$js\in RJS$ obtain
7:	$\hspace{1em}RS{K}^{’}\leftarrow RS{K}^{’} \cup \left\{sk, SV\left(USK\cup UESK,sk\right)\wedge sk\in \mathrm{O}nto\_Skills\left(js\right)\right\}$
8:	To$sk\in \left(USK\cup MS2S\left(RSK,\phi \right)\right)$ obtain
9:	$\hspace{1em}RS{K}^{’}\leftarrow RS{K}^{’} \cup \left\{s{k}^{’}, SV\left(USK \cup UESK,s{k}^{’}\right)\wedge s{k}^{’}\in Onto\_IntSkills\left(sk\right)\right\}$
10:	$RSK\leftarrow MS2S\left(RS{K}^{’},\lambda \right)$
Output$: RSK$

Where:

Onto_Get_Jobs($sk$) performs job performance sectors whose users have the sk skills registered via events in the ontology and can be configured to use registered skills belonging to the same subspecialty sub-specialization of sk when no job performance sectors are obtained for the same.

Onto_CutOff_Jobs(jobs,sks,uskp,jskp) performs filtering of job performance sectors, js,js ∈jobs, according to a percentage jskp coverage of sks skill set over the skills associated with job performance sector js,js ∈jobs or a percentage skp coverage of sector skills js,js ∈jobs over the sks skill set.

Onto_Cluster_Pred($js$) predicts the cluster for a js job sector according to clustering by Machine Learning in the ontology.

Onto_Previous($RJS,UJS$), the previous sectors in $RJS$ to $UJS$ according to the orderly relationship given by the ladder in the ontology.

Onto_Skills($JSi$) are the skills of $JSi$ represented in the ontology, a product of update via events involving user profiles.

Onto_IntSkills($sk$) are the skills of interest associated with the sk skills represented in the ontology.

SV($SK,sk$) refers to the function that verifies that the sk skill is of value for SK skills.

MS2S($MS,c$) constructs a set with the most frequent c elements in the MS multiset.

RSK’, Multiset of related skills.

RSK refers to the set of similar skills.

• (3) Content-based filtering to predict initial course recommendation

In this stage, the algorithm proposed in [2] is applied, where filtering by content is used to filter the courses from the course catalog that raise the level of specialization and/or degree of update of the user’s own skills, or help develop the user’s related skills. From this filtering, an initial recommendation is obtained, which is refined in the next stage.

• (4) Heuristics of filtering and ordering courses for final prediction

The initial course prediction from the previous phase is complemented and filtered using heuristics proposed by [2], while the following is verified for the user profile and course catalog:

1.

That the user has the necessary skills to approach the courses, and otherwise, courses that develop them are selected by applying demographic restrictions;

2.

User demographic restrictions are applied to initial course prediction and to the result of the previous stage;

3.

In the course prediction, the courses whose skills to be developed are the same or a subset of another are eliminated, retaining those with the highest score, to thus determine course recommendation for the user.

3. Evaluation and Results

According to [7], RS evaluation can be conducted through on-line and off-line experimentation. The work presented by [2] performed off-line RS evaluation. This type of evaluation allows different algorithms and approaches to be assessed, which is used in experimental environments, since the utmost consistency is desired in order to compare the performance of different proposals under the same conditions. Likewise, the off-line evaluation involves metrics to reflect the effectiveness of the system from the user perspective and provide a widely accepted evaluation, due to the robustness of the metrics used.

To evaluate our proposal, and to compare the results with our previous work [2], we performed an off-line evaluation and calculated the metrics used in it (

Table 5. Evaluation metrics for recommendation systems.

Metrics	Description	Calculation for the System
Coverage	$\mathrm{Users} \mathrm{to} \mathrm{whom} \mathrm{the} \mathrm{system} \mathrm{has} \mathrm{made} \mathrm{a} \mathrm{recommendation}, CRi$$\mathrm{set} \mathrm{of} \mathrm{recommendations} \mathrm{from} \mathrm{user} u_i\in$ U	$C\left(S\right)=\frac{\mid \left\{u_i\in U,C{R}_i\not\equiv \varnothing \right\}\mid }{\mid U\mid }$
Precision	The fraction of the recommendation that is relevant to the user.	$P\left(S\right)=\frac{1}{\mid U\mid }{\displaystyle \sum _{u_i\in U}}P\left(u_i\right)$
MAE	$\mathrm{Mean} \mathrm{absolute} \mathrm{error} \mathrm{of} \mathrm{the} \mathrm{given} \mathrm{recommendation} \mathrm{vs}. \mathrm{the} \mathrm{recommendation} \mathrm{expected} \mathrm{by} \mathrm{the} \mathrm{user}, CEi$	$MAE=\frac{{{\displaystyle \sum }}_{i=1}^n|(C{E}_i\cup C{R}_i)-\left(C{E}_i\cap C{R}_i\right)|}{{{\displaystyle \sum }}_{i=1}^n|C{E}_i\cup C{R}_i|}$
RMSE	Root mean square error	$RMSE=\sqrt{\frac{{{\displaystyle \sum }}_{i=1}^n|(C{E}_i\cup C{R}_i)-(C{E}_i\cap C{R}_i){|}^2}{{{\displaystyle \sum }}_{i=1}^n|C{E}_i\cup C{R}_i|}}$
Recall	Calculates the ratio between recommendation and user preference	$R\left(S\right)=\frac{1}{\mid U\mid }{\displaystyle \sum _{u_i\in U}}R\left(u_i\right)$
Novelty	The portion of recommendations made to the user that the user is not familiar with or has not seen before	$N\left(S\right)=\frac{1}{\mid U\mid }{\displaystyle \sum _{u_i\in U}}N\left(u_i\right)$
Serendipity	The fraction of the recommendation that is unexpected and valuable to the user.	$S\left(S\right)=\frac{1}{\mid U\mid }{\displaystyle \sum _{u_i\in U}}S\left(u_i\right)$
F1	Harmonic mean between the precision and recall	${\mathrm{F}}_1=\frac{2\ast Precisi\mathrm{o}n\ast Recall}{Precisi\mathrm{o}n+Recall}$

).

RSs are evaluated in batches, with a set of data containing the profiles of both users and courses, as well as user preferences. These were taken from a survey conducted using a Google form, where it was requested that a list of courses be classified in the categories of desirable, preferred, novel and serendipitous, in order to ascertain the preferences of users regarding choice of LLL courses.

The off-line phase of the system was implemented, whereby data were loaded into the Neo4j database that had previously been populated by the ontology substrate definitions. This was updated via events using the training dataset, and clusters of job performance sectors were determined using ML, k-Means and DBSCAN algorithms.

In order to evaluate RS performance and compare it to our previous work, metrics were calculated for each of the following configurations [2]:

1: Content filtering using only the user’s own skills; no related skills are determined, although skills of interest are considered;

2: Collaborative filtering using only the user’s own work performance sectors; semantically related skills are determined;

3: Semantic filtering using rules to determine related job performance sectors and related skills;

4: Semantic filtering using 75% coverage of user skills to determine related job performance sectors. In other words, those positions that cover 75% of the user’s skills were selected as related job performance sectors;

5: Semantic filtering using 50% coverage of skills from the job performance sector to determine related job performance sectors. In other words, those job performance sectors in which the user’s skills cover 50% of the skills associated with the position;

6: Semantic filtering using DBSCAN clustering with ε = 0.3, to determine related job performance sectors and semantic rules to determine related skills;

7: Semantic filtering using k-Means clustering to determine related job performance sectors and semantic rules to determine related skills;

8: Semantic filtering using DBSCAN and k-Means clustering to determine related job performance sectors and semantic rules to determine related skills.

Each of the configurations described above were run through the different data sets, training, testing and total, obtaining the following results.

The test results are next shown along with the results obtained in the previous work [2], to compare the results and evaluate improvement in the RS.

The results, using the training data (70% of the data set) to validate the model performance, are shown in

Table 6. Metrics for the different RS configurations used to train data.

Setting	Ref.	MAE	RMSE	Coverage	Precision	Recall	Novelty	Serendipity
1	[2]	0.58	9.13	0.64	1.00	0.36	0.14	0.09
	Author’s Proposal	0.58	9.13	0.64	1.00	0.36	0.14	0.09
2	[2]	0.51	8.22	0.73	0.97	0.41	0.21	0.07
	Author’s Proposal	0.51	8.17	0.73	0.97	0.41	0.21	0.07
3	[2]	0.25	4.44	0.91	0.83	0.82	0.46	0.07
	Author’s Proposal	0.24	4.14	0.91	0.85	0.82	0.46	0.07
4	[2]	0.22	3.65	0.82	0.93	0.71	0.45	0.07
	Author’s Proposal	0.21	3.38	0.82	0.95	0.71	0.45	0.07
5	[2]	0.38	6.62	0.91	0.80	0.74	0.39	0.07
	Author’s Proposal	0.36	6.35	0.91	0.83	0.74	0.39	0.07
6	[2]	0.18	2.92	0.91	0.91	0.80	0.52	0.07
	Author’s Proposal	0.16	2.58	0.91	0.94	0.80	0.52	0.07
7	[2]	0.49	9.25	0.91	0.67	0.63	0.31	0.04
	Author’s Proposal	0.48	9.05	0.91	0.69	0.63	0.31	0.04
8	[2]	0.30	5.24	0.95	0.80	0.80	0.48	0.04
	Author’s Proposal	0.28	4.95	0.95	0.82	0.80	0.48	0.04

.

The results obtained from use of the different configurations with the total number of samples from the dataset are shown in

Table 7. Metrics for the different RS for the whole sample.

Setting	Ref.	MAE	RMSE	Coverage	Precision	Recall	Novelty	Serendipity
1	[2]	0.59	11.05	0.61	0.99	0.37	0.11	0.06
	Author’s Proposal	0.59	11.05	0.61	0.99	0.37	0.11	0.06
2	[2]	0.50	9.49	0.71	0.97	0.45	0.17	0.05
	Author’s Proposal	0.50	9.45	0.71	0.97	0.45	0.17	0.05
3	[2]	0.25	5.16	0.94	0.82	0.87	0.39	0.06
	Author’s Proposal	0.23	4.82	0.94	0.84	0.87	0.39	0.06
4	[2]	0.23	4.49	0.81	0.94	0.73	0.39	0.05
	Author’s Proposal	0.22	4.26	0.81	0.96	0.73	0.39	0.05
5	[2]	0.34	7.03	0.87	0.84	0.75	0.34	0.06
	Author’s Proposal	0.33	6.78	0.87	0.86	0.75	0.34	0.06
6	[2]	0.27	5.63	0.94	0.84	0.86	0.45	0.05
	Author’s Proposal	0.25	5.05	0.94	0.86	0.86	0.45	0.05
7	[2]	0.51	11.72	0.94	0.62	0.65	0.25	0.04
	Author’s Proposal	0.50	11.36	0.94	0.65	0.65	0.25	0.04
8	[2]	0.35	7.64	0.97	0.75	0.86	0.42	0.03
	Author’s Proposal	0.33	7.26	0.97	0.77	0.86	0.42	0.03

.

Finally, the system was run under the different configurations, using the test portion (30%) of data, with the results shown in

Table 8. Metrics for the different configurations for the set of test data.

Setting	Ref.	MAE	RMSE	Coverage	Precision	Recall	Novelty	Serendipity
1	[2]	0.62	6.24	0.56	0.96	0.41	0.04	0.00
	Author’s Proposal	0.62	6.24	0.56	0.96	0.41	0.04	0.00
2	[2]	0.47	4.75	0.67	0.96	0.56	0.07	0.00
	Author’s Proposal	0.47	4.75	0.67	0.96	0.56	0.07	0.00
3	[2]	0.23	2.64	1.00	0.80	1.00	0.24	0.04
	Author’s Proposal	0.22	2.48	1.00	0.80	1.00	0.24	0.04
4	[2]	0.26	2.63	0.78	0.96	0.78	0.26	0.00
	Author’s Proposal	0.26	2.63	0.78	0.96	0.78	0.26	0.00
5	[2]	0.25	2.60	0.78	0.94	0.78	0.21	0.02
	Author’s Proposal	0.25	2.60	0.78	0.94	0.78	0.21	0.02
6	[2]	0.43	5.42	1.00	0.67	1.00	0.28	0.00
	Author’s Proposal	0.40	4.92	1.00	0.68	1.00	0.28	0.00
7	[2]	0.56	7.24	1.00	0.49	0.69	0.12	0.06
	Author’s Proposal	0.55	6.90	1.00	0.54	0.69	0.12	0.06
8	[2]	0.45	5.79	1.00	0.62	1.00	0.28	0.00
	Author’s Proposal	0.43	5.54	1.00	0.63	1.00	0.28	0.00

.

From the results obtained, it can be observed that both this proposal and the system proposed in [2] evidence similar behavior. As such, the configurations showing the best performance in the case of all data sets were semantic filtering using rules to determine related job performance sectors and related skills (3) and semantic filtering using DBSCAN clustering with ε = 0.3 to determine related job performance sectors, and semantic rules to determine related skills (6). However, there was an improvement in MAE and RMSE scores, and a slight increase in precision.

A measure that summarizes both precision and recall is the harmonic mean between them (F1).

Table 9. Harmonic mean (F₁) for configurations with the best RS performance.

Configuration	Ref.	F1 Training Data	F1 Total Data	F1 Test Data
3	[2]	0.82	0.84	0.89
	Author’s Proposal	0.83	0.84	0.89
6	[2]	0.85	0.85	0.80
	Author’s Proposal	0.86	0.85	0.81

shows the harmonic means for the best performing configurations.

From the harmonic mean comparison table, it can be seen in Figure 3, that there is a slight improvement in configuration 6, which makes use of the DBSCAN algorithm, while configuration 3, which corresponds to semantic filtering, remains unchanged.

From the different tests, the clustering performed by k-Means evidences an inferior performance, which could be explained by the nature of the domain and distribution of the positions in space. In general, in the graph analysis we observe better clustering (related positions) performed by DBSCAN than that performed by k-Means. Improvements in the recall and serendipity metrics can be associated with use in the ontology and the estimation of associated skills or skills of interest and filtering of positions prior to the current one in an orderly relationship imposed by a ladder.

In our review of different works, we found some similar RS proposals, and so to evaluate the performance of ours it is advisable to compare results. The work by [41] is geared to university students and company coordinators, recommends jobs according to user skills, while that by [21] is designed to recommend courses to students, using multiple data sources. On the other hand, the work by [42] recommends career paths and skills required for different jobs to users, based on their skills and interests. Finally, [35] recommends online courses to professionals according to their professional competences and professional development preferences.

Table 10. Comparison of evaluation metrics in similar work.

RSs Related	Precision	Recall	F1
[41]	0.83	0.82	0.82
[21]	0.92	0.7	0.80
[42]	0.64	-	-
[35]	-	0.70	-
Author’s Proposal	0.94	0.80	0.85

provides a summary of the results obtained from each of these papers, which use the precision and recall metrics for the evaluation of their RSs, and the best performing configuration from our proposal.

When comparing these works with the best results obtained from our proposal, an improvement in metrics of all similar RS proposals can be seen, as can an increase in accuracy, recall and harmonic mean; the latter measures being the combined performance between accuracy and recall.

4. Discussion and Conclusions

In this paper we have presented a new version of the RS proposed by [2], taking the incorporation of endorsements of user LinkedIn skills to evaluate RS performance into consideration. In the review of literature on LinkedIn endorsements, we noted that very few papers analyze this element. These are more oriented to analyze the validity of endorsements, as an element to be taken into account in the data extracted from SNSs.

The strategy proposed for recommendation of LLL courses was based on establishing a relationship between users according to work performance sectors and professional skills in order to identify those skills that should be improved or developed for their current job, or to access another, higher level job and, based on these identified skills, to determine an initial prediction of courses that may develop them. This strategy made it possible to establish a mechanism for relating the data, which generally speaking, did not initially have relationships on which to base recommendations.

By incorporating the endorsements, it was possible to obtain more information on the user profile, which in turn made it possible to incorporate skills that were not evident in the data related to current employment, and which were useful when refining course recommendation. When evaluating the incorporation of endorsements in creating user profiles, an improvement in RS performance was observed. The configuration that makes use of the DBSCAN algorithm improves the precision value by 3%, and results in a decrease in the root mean square error (RMSE) and mean absolute error (MAE).

When we wanted to compare our proposal with similar works, we did not find RSs that made use of endorsements, therefore, we selected some similar proposals in terms of RSs objectives. One of the main differences focuses on the nature of the data as it was observed that most work was with structured data. The work by [42] obtained data directly from a university database, while [41] obtained them from a data repository used for ML experimentation, and [35] extracted them from employee profiles and the training plan directly from a company. The work by [21], on the other hand, proposed using information from LinkedIn to complement the data entered directly from users through the application or obtained from the university database. From these data a taxonomy was proposed to represent course, job and student information.

The metrics used by most works were found to be recall and accuracy when they were analyzed to compare results with this proposal. In terms of the scores obtained, we can conclude that if we compare them to the results obtained from the tests performed on the total set of data, this proposal improves on the metrics obtained by these previous works by two percentage points in terms of accuracy and 10 percentage points in recall.

Research on LinkedIn endorsements is oriented towards assessing the veracity of endorsements on LinkedIn profiles. [30] propose a framework for assessing the trustworthiness of endorsements. The work of [31] proposes to measure the trustworthiness of job candidates based on their skills and endorsements. Based on this research, one line of future work is to incorporate the validation of endorsements into the RS to determine their veracity when taking them into account for recommendations.

In order to evaluate system behavior for multiple domains, the ontology could be updated with new instances associated with new domains. In order to evaluate system behavior for multiple domains, the ontology could be acted on with new instances associated with new domains, and the use of ontologies already built could also be evaluated, together with new ways to represent areas of knowledge and job performance sectors.

With regard to information sources and given the continuous changes, the use of other SNSs should be evaluated, as well as different platforms such as, RocketReach and DataLead, from where professional profiles can be obtained.

Online applications could also be offered for course recommendations, not only for online evaluation of the system, but also to determine opportunities for improvement based on user suggestions.