Strategy for the Evaluation and Monitoring of Competencies in Engineering Programs to Improve Students’ Learning and Quality of Education

Abstract

The Sustainable Development Goals (SDGs) and the Education for Sustainable Development (ESD) framework propose the concept of competencies as a key cognitive, attitudinal, and procedural aspect aimed at the integral development of students, which implies a challenge in the way of evaluating them. Thus, the traditional monitoring of students’ progress through their grades is not enough, and monitoring competency is becoming more important. This research proposes a system for monitoring competencies in engineering programs. The system identifies the expected learning outcomes (LOs) of each course and cross-references them according to the ponderation of each evaluation planned. Subsequently, it links each LO with the competencies of the course, allowing the student to be monitored throughout their career. The strategy was applied to a civil engineering course. The students’ results, according to the course competencies, were obtained, linking them correctly with the LO and the grades obtained. The evolution of these competencies was evaluated until the end of the semester in which the students were taking the course. The analysis of the results shows the differences between the monitoring by grades versus by competencies, evidencing that there were cases in which a student passed the course by grades but failed to develop the expected competency.

1. Introduction

Improving the quality of and access to education is one of the aspects present in the Sustainable Development Goals (SDGs) proposed by the United Nations [1]. Accordingly, Education for Sustainable Development (ESD) suggests the need to foster structured processes to develop competencies in students that promote sustainable development [2]. This is based on the principles of continuous improvement and integral education, in which learning is not only based on “to know”, but extends to “to be”, “to do”, “to live together”, and “to transform” [3].

Competency-based education (CBE) seeks a comprehensive training of students, achieving a balance between theory and practice, by developing behavioural skills according to different levels of achievement defined for each specific course [4,5]. Thus, in the university context, competency-based educational programs seek to establish a set of competencies to be developed throughout a career, promoting the integration of knowledge, skills, and attitudes in the contexts of different disciplines [6], as well as student learning opportunities in accordance with the sustainable development goal of quality education [7]. In this sense, the concept of competency in CBE refers to a learning model or framework in a certain area, in this case education. In contrast, the idea of “competence” is associated with a particular ability or skill that can be measured individually in students based on a specific behaviour [8,9].

To evaluate the development and achievement of competencies throughout the training period, it is necessary to connect them with the contents and evaluations of each course [10]. The concept of learning outcomes makes it possible to assess the degree of appropriation of competencies through the more direct measurement of various achievement indicators [11]. Learning outcomes are measurable statements of what a student should know, understand, and apply in a curriculum; that is, they weigh knowledge (to know) with behaviour (know-how and know how to be) in specific contexts of interest [12]. Universities have been adopting competency-based education for years; however, monitoring both competencies as such and the learning outcomes of individual students is not a concept that is widely applied [13]. The monitoring of students is conducted through grades that are direct results of evaluations of different types, but not of learning outcomes or competencies as such [11,14].

Following the continuous improvement trends and the evaluation parameters of different international accreditation entities, this paper provides a general description of the aspects of the CBE model, giving a practical approach through its application to a real-life case study in the field of engineering. It is necessary to investigate concepts, such as learning outcomes (LOs), which play a fundamental role in the model to evaluate competencies [4,15]. Their relevance directly affects the appropriation of the competencies that emanate from the declared graduate profile. The integration of concepts, such as those mentioned, will reveal a way to measure the progress of students throughout their training process, identifying the achievement of competencies, which brings together aspects of knowledge, know-how, and knowing how to be [16,17].

In this article, the application and analysis of a case study are presented, through which it is possible to identify, evaluate, and measure the competencies of a specific course in the second year of studies, with which the initial basic training of an engineering career ends. The applied model allows a systematic review of the progress of the students and the level achieved by each student in a course, as well as what is expected at that level of study in the program, thus establishing a comprehensive evaluation system for the continuous improvement of engineering training processes.

2. Research Methodology

Our research methodology is organised in two stages: (1) concepts and (2) case study. Figure 1 shows the stages and specifies the research tools and activities, along with the results of each stage.

In the first stage, a literature review was carried out to identify the characteristics and key concepts of student learning outcomes and competencies in engineering programs. In order to identify these characteristics, it was necessary to recognise some keywords, which were selected based on relevance. The search was carried out in the Web of Science and Scopus libraries, with the incorporation of manuals and technical reports from recognised entities in the area and university accreditation organizations. Documents were selected from the year 1998 to the present. The following search topics were considered: (a) Competency-based education, (b) learning outcomes in CBE, and (c) continuous improvement cycle. Later, based on the general bibliographic support and a preliminary proposed method by the research team in [18], an “Assessment System for Learning Outcomes and Competencies in Engineering Programs” was described.

In the second stage, based on the system described, a case study was developed. The system was applied to part of a civil engineering program, and to the fourth semester course Dynamic Mechanics, to exemplify the application of the system. Two analyses were carried out: one at the course level, where the results of 3 semesters were studied (104 students), reducing the analysis to a systematic sample (probabilistic method, which is based on the principle of equiprobability). Then, an analysis at the program level was carried out, with the aim of exemplifying the impact of the application of the system proposed by the authors in the follow-up of the students. The results were visualised and analysed with a focus on identifying the advantages of the assessment system in engineering courses and programs.

3. Background

3.1. Competency-Based Education

For several decades now, competency-based education has been investigated in the teaching field, and it has its origin in the United States, where it was initially called performance-based teacher training. From the first approaches to the concept, in the 1960s, competency-based education was characterised by providing details of behavioural aspects of a professional development task, that is, considering procedural and attitudinal aspects in addition to conceptual notions [8,19,20]. This approach is intended to promote student learning from a more comprehensive perspective that considers the application of concepts in a more practical way, developing behavioural skills and aiming at the fulfilment of certain levels of achievement defined for certain courses [6,15].

The definition of the learning trajectory for the achievement of the relevant competencies must safeguard the adequate integration of situations or learning experiences, in which it is possible to demonstrate aspects of performance in realization of activities, as well as ensure meaningful learning in students that allows for an adequate connection between theory and practice [13,21]. These theoretical constructs result in three principles: learning has to be situated in a recognizable and meaningful context; it is necessary to connect theory and practice, because it lets students acquire experience and lets them reflect on these experiences; and knowledge, skill, and attitude should be integrated in learning trajectories [16,18].

These principles make it possible to establish that the design of competency-based education programs (CBE) is an approach that, by its nature, is increasingly closer to the educational field, and especially to the university field, which has chosen to base the processes of curricular redesign under this approach, and analyse assessment alternatives to best evaluate the competencies defined in the curricular plans of study [13,22].

3.2. Learning Outcomes in CBE

The competence-based education model implicitly considers the concept of learning outcome (LO), which allows for a more direct assessment of the degree of appropriation of skills, since its definition points to the development of learning experiences composed of elements that allow a more direct identification and evaluation of the desired achievement indicators [23]. Although the definition of learning results and their levels of achievement are more well known within the educational environment, there are various sources associated with this concept, among which are some imposing aspects, recommendations, and results of good practices [24]. To mention some: learning outcomes are statements of what a student is expected to know, understand, and apply in a given learning experience; learning outcomes allow a short-term measurable representation of the competencies associated with a study program, or the level of achievement of the learning results is a way of verifying the appropriation of competencies by students [25,26].

In this way, in the field of higher education and the training of professionals, the evaluation of learning results is a method that integrates different aspects in its assessment and measurement, allowing the assignment of levels of achievement [27] that are not easily identifiable if the assessment is based only on the contents of a course; the evaluation allows knowledge (know), performance (know to do), and behaviour (how to be) to be associated in its application to certain situations, allowing for a direct appreciation of the integral progress in the training of students [23,28].

3.3. Education for Sustainable Development (ESD)

Implementing the SDGs in the educational field requires an adequate integration of different elements that allow students to appropriate the knowledge, skills, and attitudes to solve the global interdisciplinary challenges facing today’s society. In this sense, the framework for Education for Sustainable Development (ESD) for the year 2030, coordinated by UNESCO, recognises Sustainable Development Goal (SDG) 4 on quality education as a key enabling agent of all the other SDGs [2].

In this sense, studies have been carried out that provide guidelines to incorporate these aspects in the design of educational curricula, delivering initiatives to reorient university training to manage sustainability. This has suggested that it is not only necessary to define key competencies for the SDGs, but also that this approach to ESD brings with it, in addition to knowing, knowing how to do, and knowing how to be, learning to live together and learning to transform, aspects that will undoubtedly lead us to rethink the terms of competence in the educational field [3,29].

3.4. Continuous Improvement

The commitment to ensuring and make visible the quality of training in higher education institutions is a priority task for universities [4], but also becomes a transversal axis in the approach to the sustainable development goals, considering that education is the basis for achieving them [7].

Most of the accrediting entities, for engineering careers, follow a common pattern: (1) publicly demonstrating through concrete mechanisms that a certain study program makes sure to educate the professionals in the graduation profiles, i.e., how the training objectives are achieved and made visible; (2) publicly demonstrate the processes that allow continuous improvement in a certain study program, and how improvement actions are carried out that benefit and add value to undergraduate training; (3) publicly demonstrate coherence and cohesion in the training process through institutional objectives (university level), the objectives of the study program (degree or program level), learning outcomes (course or subject level), how the contents are structured, how the assessment or evaluation of learning is carried out by teachers, and how feedback is given to students [30,31,32].

Table 1. Some accrediting entities for engineering programs by country or region.

Engineering Accreditation Agency (Country or Region)	QTE	CI	PE	PA	LO	S/G/OC	E/PO	EM	FM
ABET: Accreditation Board of Engineering and Technology (USA) [33]	x	x	x	x	x	x	x	x
CEAB: Canadian Engineering Accreditation Board (Canada) [34]	x	x	x	x	x	x	x	x
ARCU-SUR: Regional Accreditation System for University Careers (South America) [35]	x	x	x	x	x	x	x	x
ENAEE: European Network for Accreditation of Engineering Education (Europe) [36]	x	x	x	x	x	x	x	x	x
CEEAA: China Engineering Education Accreditation Association (China) [37]	x	x	x	x		x	x	x	x
JABEE: Japan Accreditation Board for Engineering Education (Japan) [38]	x	x	x	x	x	x	x	x
NBA: National Board of Accreditation (India) [39]	x	x	x	x		x	x	x	x
IEAust: Institution of Engineers Australia (Australia) [40]	x	x	x	x	x	x	x	x	x
ECSA: Engineering Council of South Africa (South Africa) [41]	x	x	x	x	x	x	x	x	x

shows key concepts that can be identified in different accrediting entities for engineering by territory. Improvement (CI); Program Evaluation (PE); Program Assessment (PA); Learning Outcome/Learning Objective (LO); Student/Graduate Outcome/Competence (S/GO/C); Educational/Program Objective (E/PO); Evidence Making (EM); Feedback Making (FM).

However, none of these entities propose a specific mechanism for how to achieve continuous improvement or how to measure the level of appropriation of the competencies declared in the graduation profiles, which ABET defines as assessment and evaluation [33]. Achieving continuous improvement in higher education means permanently detecting opportunities for improvement in all teaching–learning processes of a study program or faculty, in order that, both externally and internally, it is possible to make visible that all efforts go in the direction of delivering the best educational service possible, which means that all the actors in an institution are aligned with the same commitments and objectives [42].

4. Design of the Assessment System for Learning Outcomes and Competencies in Engineering Programs

The proposed evaluation system has been structured according to the five competencies cluster presented by Education for Sustainable Development: learning to know, learning to be, learning to do, learning to live together, and learning to transform [3]; to meet the challenge of educational programs (curricula and teaching methods), universities must innovate and improve to sensitise students to sustainable development [2].

The assessment system is structured for engineering programs that understand or define learning outcomes as objectives in undergraduate courses, and student outcomes, attributes, or competencies as educational objectives. Figure 2 illustrates the components of the assessment system, the actors, and the interrelationship between them.

This system methodology allows assessment, evaluation, and improvement at course and program levels. The program level assessment is defined by the average of all the courses involved in the evaluation strategy or, if applicable, by previously defined weightings (for example, by SCT). For these purposes, it recommends a unified database in which teachers can enter, save, and modify the parameters they use in their assessment process.

Table 2. Assessment system procedure. Obtaining data for evaluation and improvement.

Stage	Formula & Parameters	Description
Stage 1 Identification and Definition of Default Input Variables	${\forall }_j \in \mathbb{N} \left\{1,2,\dots ,n\right\}\to L{O}_j=“Learning Outcome j”$ ${\forall }_x \in \mathbb{N} \left\{1,2,\dots ,k\right\}\to C_x=“Competence x”$ ${\forall }_i \in \mathbb{N} \left\{1,2,\dots ,m\right\}\to A{A}_i=“Assessment Activity i”$ ${\forall }_i \in \mathbb{N} \left\{1,2,\dots ,m\right\}\to P_i=“Weight of Assessment Activity i”$ $${\displaystyle \sum _{i=1}^m}{P}_i=100$$ (1)	Should be defined in course syllabus; otherwise, teacher must define these variables
Stage 2 Definition of System Configuration Input Variable 1	$B_{i,j}=“Weight of Learning Outcome j at A. Activity i”$ $T_j=“Total Weight of Learning Outcome j at course”$ $$P_{i }={\displaystyle \sum _{j=1}^n}{B}_{i,j} ; T_j={\displaystyle \sum _{i=1}^m}{B}_{i,j} ; {\displaystyle \sum _{j=1}^n}{T}_j=100$$ (2)	Teacher must define these variables
Stage 3 Assessment and Calculation of Achievement Factor of Learning Outcomes	${\forall }_s \in \mathbb{N} \left\{1,2,\dots ,r\right\}\to s=“Student s”$ $Q_{s,i}=“Student s Qualification in Assessment Activity i”$ $MQ=“Maximun Qualification”$ $AF{\left(LO\right)}_{s,j}=“Student s Achievement Factor of Learning Outcome j”$ $$AF{\left(LO\right)}_{s,j}=\frac{100}{MQ\ast T_j}{\displaystyle \sum _{i=1}^m}{Q}_{s,i}\ast B_{i,j}$$ (3)	Using qualifications of assessment activities, achievement factors of learning outcomes are possible to calculate
Stage 4 Definition of System Configuration Input Variable 2	$E_{x,j}=“Relation between L. Outcome j and Competence x”$ $W_x=“Total number of Learning Outcomes in Competence x”$ $$E_{x,j}\in \left\{0,1\right\}; {\displaystyle \sum _{j=1}^n}{E}_{x,j}=W_x\ne 0$$ (4)	“E” variables are the relation between a learning outcome and a competency. To define these, the teacher must read both definitions and link them: 1 for relation, 0 for no relation
Stage 5 Calculation of In-Between Parameters	$D_{i,x}=“Weight of Competence x in Assessment Activity i”$ $$D_{i,x }=\frac{100}{W_x}{\displaystyle \sum _{j=1}^n}\frac{E_{x,j}\ast B_{i,j}}{T_j}$$ (5)	“D percentages” parameters are analogous to “B percentages”; they are the result of adding the “E” variables
Stage 6 Assessment and Calculation of Achievement Factor of Competencies (Course Level)	$AF{\left(C\right)}_{s,j}=“Student s Achievement Factor of Competence x”$ $$AF{\left(C\right)}_{s,x}=\frac{1}{MQ}{\displaystyle \sum _{i=1}^m}{Q}_{s,i}\ast D_{i,x} ,$$ (6)	Using qualifications of assessment activities, achievement factors of competencies are possible to calculate. Must not be confused with the accumulated achievement factor of competencies
Stage 7 Calculation of Achievement Factor of Competencies (Program Level)	$Z_x=“Total courses where Competence x is developed”$ $AAF{\left(C\right)}_{s,j}=“Student s Accumulated Achievement Factor of Competence x”$ $$AAF{\left(C\right)}_{s,x}=\frac{\sum AF{\left(C\right)}_{s,x}}{Z_x}$$ (7)	If this process is conducted in a certain number of courses, accumulated achievement factors of competencies are possible to calculate. These factors make possible competencies measurements

summarises the procedure of the proposed assessment system, including formulas and descriptions. The procedure is separated into seven stages: (1) identification and definition of default input variables; (2) definition of system configuration input variable 1; (3) assessment and calculation of achievement factors of learning outcomes; (4) definition of system configuration input variable 2; (5) calculation of in-between parameters; (6) assessment and calculation of achievement factors of competencies (course level); and (7) calculation of achievement factors of competencies (program level).

In the first stage, the variables that the system considers must be defined. These are learning outcome, competence, assessment activity, and weight of assessment activity. Generally, these variables are in the course syllabus. Otherwise, the teacher must define these variables. In the second stage, the defined variables should be correlated, assigning a weight of the learning outcomes for each activity and the total weight of the course’s learning outcomes. The teacher must define these variables. In the third stage, using qualifications of assessment activities, it is possible to calculate achievement factors of learning outcomes. The relationship between a learning outcome and competency is obtained (variable E). To define them, the teacher must read both definitions and relate them to one another (1 for a relationship, 0 for no ties). In the fifth stage, the weight of each competency in each of the evaluation activities is calculated. In the sixth stage, using qualifications of assessment activities, it is possible to calculate achievement factors of competencies (not to be confused with the accumulated achievement factor of competencies). Finally, if the process is undertaken in a certain number of courses in the seventh stage, it is possible to calculate accumulated achievement factors of competencies. These factors make it possible to measure competencies.

Figure 3 shows all the components of the proposed evaluation and improvement strategy, which consist of three main stages: input of variables to obtain data (using the assessment system), obtaining data for evaluation, and obtaining and interpreting information to detect opportunities for improvement.

The competencies to be quantified and evaluated must be defined, together with the courses where the development of the competencies is declared. It should be noted that it is not necessary to cover all courses, but only those that are considered key or capstone. The teachers in charge of the courses must be aligned with the strategy and follow the procedure described in Figure 3 step by step, since their effort will determine the validity of and confidence in the data obtained, which, in turn, will serve as feedback for their students and the structure of their courses in order to make improvements. At the end of the semester, after having carried out all the assessment activities, it will be possible to calculate achievement factors of learning outcomes and partial achievement factors of competencies to realise the level of acquisition of learning outcomes by students.

As the semesters progress, and only the last course remains to be evaluated, it will be possible to monitor the development of the defined competencies, making explicit the strengths and weaknesses of the students and providing a way for possible improvement actions such as remedies, reinforcements, or adjustments in order to achieve an adequate level of appropriation with what is declared in the graduation profile. In addition, it will be possible to obtain information on the types of assessment instruments that were used through a training period, and the entire study program will be coherent in terms of content, assessment activities, learning outcomes, courses, and competencies.

Once this has been specified and applied over time, engineering programs that opt for the strategy will be able to publicly demonstrate that they have reached a cycle of continuous improvement.

5. Study Case

5.1. Implementation Context

The application of the competence assessment model presented above was carried out in the Civil Engineering section of the Faculty of Engineering at the Pontifical Catholic University of Valparaiso (PUCV). It initially considered a study of a specific course, and then at the level of the study program to see the level of advancement of a particular competency, until the end of the fourth semester of study, considered the first cycle of initial training in engineering careers.

The study plan of the degree as mentioned above has a total duration of 11 semesters, and each course contributes to different competencies, which are classified into fundamental (FC), disciplinary (DC), and professional (PC) training competencies.

The competencies described and numbered in

Table 3. Competencies of the civil engineering degree of the PUCV.

Type	Nº	Description of Competence
Fundamental (FC)	1	It recognises the transcendent dimension of human existence, and Christian anthropology as a valuable response to the meaning of life.
	2	It acts ethically, illuminated by the Christian proposal, in real contexts, with autonomy and respect for others, seeking the common good, the promotion of human rights and the realization of the human person, in a context of diversity.
	3	Communicates in a clear and coherent manner through their mother tongue in an academic context.
	4	It uses information and communication technologies as tools for academic and professional development.
	5	Demonstrates capacity for analysis, abstraction, synthesis, and critical reflection with the aim of solving problems, building knowledge, and developing self-learning, both individually and in interdisciplinary teams.
	6	It communicates orally and in writing in the English language, in order to facilitate their insertion and participation in multicultural and interdisciplinary contexts.
	7	It recognises reading, relationships with others, physical activity, healthy living, environmental care, art, and culture as sources of integral personal development.
	8	It participates in democratic instances, committing its formation in a local, national, and international context.
Disciplinary (DC)	1	It uses the knowledge of basic sciences to understand, pose, and solve mathematical models associated with physical phenomena and processes related to the field of civil engineering.
	2	It demonstrates logical-deductive thinking that allows the student to methodically face multidisciplinary problems that require the analytical capacity of an engineer.
	3	The student masters the conceptual basis and the analysis tools of the engineering sciences area to study and solve problems of civil engineering and those that transcend the scope of the specialty.
Professional (PC)	1	It has the tools that allow it to understand the social, economic, cultural, and environmental contexts to design and develop engineering projects.
	2	The student masters the techniques and procedures relevant to the management and direction of civil engineering projects, in order to optimise the use of resources for their development.
	3	It works in interdisciplinary teams by generating integrated and efficient solutions related to civil engineering works and systems.
	4	It identifies infrastructure deficiencies and proposes solutions that are technically feasible, economically viable, and responsible to society and the environment, in the field of application of civil engineering.
	5	It designs civil works applying principles and methodologies of analysis, design criteria, and current regulations, to respond to the needs of society with an innovative vision.
	6	Makes informed decisions while protecting the community and the environment in the formulation and management of civil engineering projects.
	7	Leads, manages, and directs civil engineering works and systems, ensuring the proper use of economic, human, and environmental resources to meet the objectives of a project.

and

Table 4. Competencies of Dynamic Mechanics course.

Semester	Nº	Course	PUCV Credit	Hours per Semester	Fundamental (FC)							Disciplinary (DC)					Professional (PC)
					1	2	3	4	5	6	7	8	1	2	3	1	2	3	4	5	6	7
1	1	Challenges in Civil Engineering	4	162		x	x		x		x										x	x
	2	Introduction to Geometry	3	121.5									x	x
	3	Fundamentals of Mathematics	5	202.5		x	x							x
	4	Introduction to Physics	5	202.5			x						x	x
	5	Oral and written communication	3	121.5		x	x				x
Accumulated Total			20	810	0	3	4	0	1	0	2	0	2	3	0	0	0	0	0	0	1	1
2	6	Applied Programming	4	162			x		x		x		x	x	x
	7	Linear Algebra	4	162									x	x
	8	Calculus 1	4	162									x	x
	9	General Mechanical Physics	3	121.5									x	x
	10	Modelling and Topography Tools	4	162				x	x
	11	Christian Anthropology	2	81	x	x
Accumulated Total			21	850.5	1	4	5	1	3	0	3	0	6	7	1	0	0	0	0	0	1	1
3	12	Static Mechanics	5	202.5					x				x		x
	13	Engineering Materials	4	162			x		x		x		x	x	x
	14	Calculus 2	4	162									x	x
	15	General Physics Thermodynamics and Waves	3	121.5					x				x	x
	16	Applied Chemistry	4	162					x				x	x
Accumulated Total			41	1660.5	1	4	6	1	7	0	4	0	11	11	3	0	0	0	0	0	1	1
4	17	Dynamic Mechanics	4	162					x				x		x
	18	Differential Equations	4	162									x	x
	19	Calculus 3	5	202.5									x	x
	20	General Physics Electromagnetism	3	121.5									x	x
	21	Construction Processes and Techniques	4	162		x		x	x						x		x				x
Accumulated Total			61	2470.5	1	5	6	2	9	0	4	0	15	14	5	0	1	0	0	0	2	1

show details of the civil engineering courses until the fourth semester (initial cycle of the career). In the case of this study, the analysis at the course level is carried out in the fourth semester subject “Dynamic Mechanics” (No. 17), belonging to the area of engineering sciences whose professor is one of the authors of this article. For each course, the PUCV credits are detailed (1 PUCV credit is equivalent to a dedication of 40.5 chronological hours per semester), as are their contributions to career competencies.

5.2. Application of the Proposed Methodology

In a general context, Dynamic Mechanics is a theoretical–practical course that aims at having students develop and assimilate the fundamental principles of the laws of dynamics and achieve their correct application in problems associated with civil engineering. At the end of the course, students are expected to develop the following Learning Outcomes (LOs):

• LO1: Understands the basic principles of the dynamics of particle systems and rigid bodies for the study of their movement behaviour.
• LO2: Applies the fundamental concepts of the dynamics of particle systems and rigid bodies to the analysis of their motion and interaction with other bodies from the conception of mathematical models.
• LO3: Describes and analyses vibrations of systems with one or more degrees of freedom in simple harmonic or damped regimes for the study of the behaviour of simple systems and structures.

In addition, according to Table 3 and Table 4, the course contributes to the competencies of a civil engineering career:

• DC1: It uses the knowledge of the basic sciences to understand, pose, and solve mathematical models associated with physical phenomena and processes related to the field of civil engineering.
• DC3: Manages the conceptual basis and analysis tools of the engineering sciences area to study and solve civil engineering problems and problems that transcend the field of specialty.
• FC5: Demonstrates capacity for analysis, abstraction, synthesis, and critical reflection in order to solve problems, build knowledge, and develop self-learning, both individually and in interdisciplinary teamwork.

For the achievement of the learning results, the course considers three typologies of assessment activities: (1) quizzes, (2) tests, and (3) laboratory.

Table 5. Description of assessment activities.

Nº	Activity	Description	Quantity	Total Weight	Weight for Each One
A	Quizzes	Specific activity, to evaluate knowledge of direct application.	4	20%	5%
B	Tests	Global activity, to finalise a series of studied topics, cumulative, and their combined applications.	3	45%	15%
C	Laboratory	Practical activity, of empirical application of the topics.	1	35%	35%

shows the description of each of these types of activities, along with the number and weighting of each in the course (as indicated by Equation (1) in Table 2).

Figure 4 shows the triad: competencies, learning outcomes, and assessment activities. It is possible to see the allocation of the weights to the learning outcomes for each activity, along with the correspondence between the learning outcomes and the competencies (as indicated by Equation (2) in Table 2).

The application to the course mentioned above was carried out in three semesters between 2018 and 2019, from which a study group was obtained. A sample was to be identified to determine the monitoring of competence at the program level. Of the periods considered for this study for the subject of Dynamic Mechanics, there were 104 students, of whom 30 were from semester one in the year 2018, 34 from semester two in the year 2018, and 40 from semester one in the year 2019. The scale assessment is from 10 to 70, and, upon approval of the course, a grade greater than or equal to 40.

5.3. Course Level: Results

With the input data obtained from Table 5 and Figure 4, it was possible to calculate the percentage of achievement for both learning outcomes (as indicated by Equation (3) in Table 2) and related competencies (as indicated by Equation (4) in Table 2). Figure 5 shows the distribution of course grades. The quizzes, tests, and laboratory results are shown in green tones, and the final averages in red. Figure 6 shows the distributions of learning outcomes (orange tones) and competencies (blue tones).

To perform an analysis of the data, a systematic sample was taken. It uses a probabilistic method based on the principle of equiprobability, meaning that all the individuals in the selected sample have the same probability of being chosen. This method assures us that the extracted sample will be representative. This type of sample has the following characteristics:

• There is no discretion available to the researcher (subjective intervention of the sampler).
• Items are selected by mechanical rules.
• Sample error is considered.
• The probability of inclusion is known.

To carry out this type of sampling, the sample size that we want to calculate must be initially determined according to Equation (8):

$$n=\frac{N\ast Z^2\ast p\ast q}{d^2\ast \left(N-1\right)+Z^2\ast p\ast q}$$

(8)

where n is the sample size that we want to calculate, N is population size, p the probability of success, q the probability of failure, d the maximum permissible error, and Z the confidence coefficient for a given confidence level.

Table 6. Parameters and values considered in the study.

Acronym	Parameters	Values
N	Population size	104 students
Z	Confidence coefficient	Z = 1960 for a 95% confidence level
P	Probability of success	We consider the maximum: 0.50
q	Probability of failure	It will be 1.00 − 0.50 = 0.50
d	Maximum permissible error	We consider 3%

shows the values of the selected parameters.

With the values of Table 6, according to Equation (8), the sample size of n = 25 is calculated. Then, the selection factor (k) is calculated by Equation (9):

$$k=\frac{n}{N}$$

(9)

We obtain k = 4.217. Finally, we determinate A, which represents a random number between 1 and k; in this case, A was 3. Therefore, the students in this sample are determined from the indexes of the ordered list with the sequence A, A+K, A+2K, A+3K … all rounded to the nearest integer. With this, we obtain a sample of 25 individuals with representation proportional to the number of total students, and select individuals from a systematic sampling of the members of the course under study (presented in

Table 7. Selected sample data.

Nº	Quizzes					Tests				Lab.	Final Average	Learning Outcomes			Competencies			Final Status
	Q1	Q2	Q3	Q4	QA	T1	T2	T3	TA	LA		LO1	LO2	LO3	FC5	DC1	DC3
2	22	29	55	10	27	29	26	21	25	10	20	35	34	19	30	27	35	F
6	44	54	44	35	34	39	31	27	33	10	25	50	40	25	38	37	45	F
10	49	40	58	50	43	19	28	28	25	19	27	40	44	35	40	38	42	F
15	18	29	41	28	26	33	30	37	33	25	29	42	43	41	42	41	43	F
19	62	66	60	60	50	26	26	40	30	21	31	50	50	43	48	47	50	F
23	65	52	61	31	46	37	25	40	34	22	32	55	47	41	48	48	51	F
27	56	70	52	47	49	24	26	33	28	33	34	49	53	49	51	49	51	F
32	36	49	53	58	41	33	53	36	41	26	35	54	60	45	53	49	57	F
36	22	52	49	52	38	35	48	45	43	31	37	52	61	53	55	53	57	F
40	60	66	57	57	53	20	43	34	32	37	38	52	64	54	57	53	58	F
44	44	70	57	56	53	34	25	39	33	39	39	55	58	58	57	56	56	F
48	64	10	70	38	52	38	27	38	34	44	41	57	54	59	57	58	56	P
53	61	50	53	65	61	33	29	58	40	33	42	57	56	62	59	60	57	P
57	64	60	55	61	59	20	32	55	36	43	43	53	63	69	62	61	58	P
61	28	46	55	52	42	40	54	40	45	45	44	60	70	63	64	62	65	P
65	37	64	70	59	53	34	38	69	47	38	45	58	69	70	66	64	64	P
69	60	10	43	59	57	32	33	43	36	51	45	56	57	71	61	63	56	P
74	67	61	63	61	69	35	22	55	37	45	46	62	61	71	65	66	61	P
78	52	56	57	64	64	31	50	42	41	46	47	60	71	66	66	63	66	P
82	58	60	60	61	67	33	35	55	41	47	48	61	68	72	67	67	65	P
86	63	58	53	61	68	33	41	55	43	48	50	63	69	73	69	68	66	P
91	62	51	60	67	70	43	40	53	45	50	52	69	70	75	71	72	70	P
95	65	62	63	61	69	52	34	57	48	52	53	75	70	78	74	77	73	P
99	65	68	67	67	76	41	39	60	47	53	55	71	76	81	76	76	73	P
103	65	68	67	68	74	52	47	61	53	57	59	80	81	85	82	82	81	P

) and ordered according to the resulting average, where P is pass and F is fail.

Undertaking a general review of the data and investigation, it is possible to verify one of the premises of this research in terms of average and real scope of learning results and skills. If we look at students numbered 65 and 69 on the list, it is possible to verify that both have an average of 45; therefore, they pass the course, but their individual achievements are not the same. Among the most significant, there are differences of 8, 12, and 6 percentage points between LO1, LO2, and DC3, respectively, for the same final course result.

Along with the above, it is also possible to group the data according to grade ranges to facilitate analysis of the results, as shown in

Table 8. Data grouped according to grade ranges.

Grade Ranges	Average	Learning Outcomes			Competences
		LO1	LO2	LO3	FC5	DC1	DC3
R1 (25–29)	27	44	42	34	40	39	43
R2 (30–34)	32	51	50	44	49	48	51
R3 (35–39)	38	53	61	53	55	53	57
R4 (40–44)	42	57	61	63	60	60	59
R5 (45–49)	46	60	65	70	65	65	62
R6 (50–54)	52	69	70	75	71	72	70
R7 (55–59)	59	80	81	85	82	82	81

, with a stratification corresponding to every five points of variation in terms of averages. In this way, it is possible to identify the following summary data from the sample under study.

5.4. Program Level: Results

The proposed monitoring of competencies can also be focused on the level of the study program, which allows monitoring of the progress of each competency at different stages in the training process. Understanding that it is a gradual process, we consider, in Figure 7, the development of DC3 competence up to the fourth semester for the 25 students in the sample analysed in this case study. It is possible to visualise the trend of the formation of that competency. The lines in the red zone correspond to those students who, for the indicated semester, have not surpassed 60% of the achievement of the competence that they should have fulfilled in the corresponding courses. If students achieved 100% competency in DC3 in the fourth semester, they would have made 20% progress in this competency regarding the total study program (11 semesters).

6. Analysis and Discussion

6.1. Course Level

Based on the observed results, it is possible to affirm that, although an average of grades greater than or equal to 40, with which a subject is passed, does not reflect a direct relationship with the degree of fulfilment of the learning results or the appropriation of competencies associated with the course, this level of demand (average of grades greater than or equal to 40), associated with the applied mime of the grade scale, is 60%; however, this could be adjusted, since this requirement does not imply a change in the grade itself, but in what is expected in terms of skills by students. Thus, according to the stratified summary of the previous table, students being in Section 4 (40–44), the first within the “Passed” range, does not imply that they are above what is expected in terms of the development of certain skills. Competency-based assessment aims to highlight the importance of facts such as this, allowing a greater degree of knowledge of the real learning of students instead of just one grade.

From a graphic perspective, in Figure 8, it can be seen that the central sections are where somewhat undefined situations may arise in relation to a clear approach to the real achievement of learning outcomes and competencies. On the minimum required grade, behaviour is clearer and aligned towards the expected achievement; thus, this type of analysis makes more visible the cases in which a student passes, complying with the required grade, but has not necessarily passed the grade of appropriation of the skills considered important for a given course.

Finally, from the individual purpose of a course and the level of achievement of its competencies, there could be aspects that have no major relevance in the training process. However, there is a relationship between the different courses that make up a complete study program, so more background is required to have an overview of the true impact of the achievement of a competence, understanding that the real appropriation of this is a gradual process, and that focusing on a specific time, such as during a course, is not enough to undertake a more comprehensive analysis.

6.2. Program Level

It is possible to project the specific analysis of the course we analysed to the rest of the subjects that make up a career curriculum. Thus, it is possible to know exactly the students’ progress in the specific competencies in each of the stages of the formative path. It is also possible to see the progress of the students in the appropriation of the competencies. It is important to understand that this is a gradual process throughout the curriculum training process.

Because the level of achievement reached by approximately 48% of the students in the course is under the requirement of 60%—y delving into the graph of DC3 (Figure 7)—different analyses can be discussed. Level of achievement could show that students who do not progress according to expectations may continue in the career. If viewed from a numerical perspective, by calculating their average grades, the students can achieve the minimum passing grade, despite the real appropriation of competencies analysed at this moment out under the expected level. This situation would indicate that, for the following courses of the study program, where this competence or another related one is presented, the students would face certain weaknesses in skills, knowledge, and attitudes expected at this level. Since it is understood that student learning is a progressive and cumulative process, any deficiency evidenced at a given moment may face future situations where a student is affected by not having the necessary competencies to acquire new knowledge, and is therefore not responding to the expected results.

7. Conclusions

This work presents a proposal that considers the main aspects of defining a competency assessment system with a simple application approach but, at the same time, significance in terms of results obtained that are related to what grades represent upon completion of a course. From this point of view, the proposal presented provides a mechanism that focuses on safeguarding the quality of education, having equality as its main axis and, in this case, mainly in the measurement of the students’ learning achievements, guaranteeing equity in the formative process.

In addition to the above, it allows for the investigation of different ways of evaluating a course and, therefore, the competencies it pursues, providing more representative data. Additionally, by allowing a progressive application throughout a career, it is possible to generate reinforcement actions in case they are required, and adequately review new instances of evaluation that manage to incorporate the concept of competencies in the courses.

After applying the method to a course of study, differences in the monitoring of students by marks and by competencies were observed. Thus, it was observed that, although there were students who passed the course by grades, some competencies were not acquired correctly. That is to say, although the students exceeded the pass mark, some competencies (when looked at in isolation) did not exceed the established achievement percentages. This analysis highlights an essential aspect that teachers and those in charge of continuous improvement in education need to consider. Although university education systems tout the application of competency-based education, monitoring systems such as the one presented in this research manage to show the actual achievement and acquisition of competencies.

This problem, which can be analysed at the course level, can be expanded during the progress of a student’s education (semester by semester). When follow-ups throughout a student’s years of study are carried out, the deficiencies in competence acquisition are cumulative.

Finally, considering that education is a continuous improvement process, it is worth mentioning that one way to enhance future work is to consider some variations in the proposed model. Incorporating each course’s weighting within the curriculum would provide more relevant information about a given course. Thus, a more representative and meaningful approach could be obtained in terms of its results for the students themselves and the related teachers. Additionally with the above, it is necessary to incorporate aspects of the Education for Sustainable Development (ESD) model, whose framework is an activity that has begun to be reviewed through international organizations, where it is intended to establish general guidelines to incorporate the competencies that allow safeguarding the compliance with the SDGs that have been elucidated. In our particular case for the study carried out, this would imply making adjustments in the proposal so that it focuses on the criteria set out in this framework to facilitate quality education.