Multi-Criteria Evaluation Method in the Field of University Education: Application to a Course on Energy Markets

Abstract

The Bologna Plan adopted by European universities ended the hegemony of an evaluation system exclusively based on the performance of traditional examinations. In this area, with a view to revitalizing grading models in university education, a wide range of evaluation mechanisms has been developed in recent years. Using them, teachers may evaluate the learning levels of their students, including both the specific competences of the taught subject and the transversal competences that help students further develop their professional careers. This article presents a methodology based on a multi-criteria procedure through which students could be evaluated from different points of view, based on different types of evaluation mechanisms that are diversely weighted. Therefore, their levels of learning could be assessed more objectively. This article shows a practical case of applying this methodology, which has been used for the last five years in a course on energy markets taught as part of the Degree in Energy Engineering at the UPV.

1. Introduction

Until the implementation of the Bologna Plan, which began in 2010, university education did not generally contemplate any other form of evaluation other than a final exam, whether oral or written [1]. Since then, different evaluation methods, some of which are currently used in university education, have been developed and implemented to a greater or lesser extent. Although it is true that none of them has managed to dethrone the traditional exam, there is a great variety of tests, many of which have had a more pronounced development thanks to the application of new technologies. However, despite the fact that there are studies that show that students particularly appreciate that teachers use new technologies in assessment tests, the percentage of teachers who use them effectively is small [2].

Spanish universities have addressed evaluation based on multi-criteria mechanisms in their operating rules. Thus, the new Regulations for the Academic Regime and Student Evaluation of the Polytechnic University of Valencia, which is in the review phase, includes, in Article 15, the condition that no act of evaluation can exceed 40% of the final grade of the subject [3], something that is already implemented in some schools.

Multi-criteria methods have been widely used to evaluate programs and projects [4]. In higher education, multi-criteria methods have been also used to evaluate the achievement of the 2030 Agenda and the Sustainable Development Goals in Universities [5]. In general, multi-objective decision models allow a balanced type of analysis to be carried out of all the facets that affect the planning of a project [6]. Moreover, these kinds of methods help to assess problems that could be complex and could entail conflicting criteria [5]. In the particular case of evaluating university students, using a multi-criteria method allows for analyzing the degree of student learning with a greater degree of independence concerning the evaluation technique that is used. There will always be students who find it easier to solve an objective test (i.e., a multiple-choice test) than an open-development test, without necessarily implying that they have a higher level of knowledge, who find it easier to transmit their understanding orally, or who prefer to put their understanding in writing. Therefore, the choice by teachers of one type of test or another may be biasing the ability of students to be evaluated as objectively as possible. The use of a multi-criteria method helps to solve this problem, given that the variety of evaluation techniques to which students are subjected helps to alleviate any deficiencies of a specific evaluation system. Thus, students are allowed to express their degree of learning in the way they feel most comfortable, compensating for the students’ skills and assessing the knowledge and skills that have been acquired based on various criteria. This also requires greater involvement and effort on the part of the teacher, who need to design balanced evaluation tests that allow their students to demonstrate the degree of real learning they have achieved.

According to [7], there are four main multi-criteria evaluation methods, commonly used in engineering and investment projects: checklist methods; multi-attribute utility methods; the analytic hierarchy process; and concordance analysis. For evaluating students in higher education, a checklist method has been adapted in this paper. According to this method, information is organized in a matrix in order to compare the considered criteria. In this case, such criteria are the different evaluation techniques [8] that can be used to distinguish the level of achievement of the different learning results related to the course that a student is taking.

There are few examples of application of multi-criteria methods for students’ evaluation, but some teachers have documented their experiences on this topic. Thus, in [9], Marín-García et al. applied a multi-criteria method based on the analytic hierarchy process to analyze the performance of 10 master’s degree students according to 25 dimensions. The authors of the present work have also applied a multi-criteria method for the evaluation of a reduced group (between 8 and 12 students) at the master’s degree level [10]. In this context, this article presents the multi-criteria evaluation methodology designed for the course on energy markets taught in the third year of the Degree in Energy Engineering at the Polytechnic University of Valencia (UPV), Spain. In this case, the group of students for this course is larger (around 70 students), so the method drafted in [10] has been further developed here.

This article is structured as follows: Section 2 presents the objectives of the work carried out, which are developed in detail in Section 3, where the methodology used is described. Section 4 shows the results of the practical case of application to the course on energy markets. Finally, the conclusions of this work are included in Section 5.

2. Objectives

The general objective of this work focuses on developing a multi-criteria method for the evaluation of students receiving a university technical education, which is applied to the particular case of the course on energy markets taught as part of the Degree in Energy Engineering at the UPV. In this area, the specific objectives of the work are as follows:

• That the evaluation method designed allows for assessing the levels of student learning as objectively as possible, without being linked to a specific type of evaluation technique;
• That the evaluation method designed helps students learn through their mistakes, offering continuous feedback throughout the course, which leads to a more consistent learning process [11];
• That the evaluation be carried out continuously throughout the entire course, so that students assume greater responsibility, which favors their learning process [12].

3. Methodology and Development of the Innovation

The methodology proposed for the design of a multi-criteria evaluation system is outlined in Figure 1. First, it is necessary to select the learning objectives that are going to be evaluated with each of the evaluation techniques. This is based on the principle that these objectives have been established adequately under educational taxonomy principles [13]. For example, Bloom’s Taxonomy can be used to structure such learning objectives according to six hierarchical levels, so that learning objectives reached at higher levels are dependent on the skills acquired in lower levels [14]. Next, the most appropriate evaluation technique is chosen for each of the learning objectives to be evaluated. Table 1 collects different evaluation techniques, as collected in [8]. Each test is classified according to the type of information it offers.

The number of circles in Table 1 (zero, one or two) indicates if such dimensions as knowledge, abilities and attitudes are properly evaluated by means of the chosen technique (not evaluated, partially evaluated and fully evaluated, respectively).

Once the technique is chosen, the evaluation factors that are considered most suitable can be analyzed. For example, in an objective test, it needs to be determined if a “True/False” model or a multiple-choice model is more convenient and, in the latter case, how many questions are there, what score does each of the questions have, what is the penalty in case of error, etc. Some tips for this process can be obtained from [15].

Next, constraints that may make the test successful or not (depending on the learning outcome that it is intended to evaluate) should be assessed. Suppose the chosen test does not comply with the identified constraints. In that case, it should be reviewed again, to determine if the evaluation technique is suitable for the learning outcome to be evaluated, or if the evaluation factors have to be adjusted. In case of compliance, the characteristics of the chosen technique should be specified, by repeating the previous steps for the following techniques.

Table 1. Evaluation techniques [8].

Technique	Knowledge	Abilities	Attitudes
Oral exam or oral presentation	••	••	••
Open-ended written test	••	•
Multiple-choice objective test	••
Conceptual map	••	•
Academic assignment	••	•
Minute questions	••	•
Diary		••	••
Portfolio	••	••	••
Project	••	••	••
Problem	••	••	••
Case	••	••	••
Essay	••	•	•
Discussion	•	••	••
Observation	•	••	••

Table 1. Evaluation techniques [8].

Technique	Knowledge	Abilities	Attitudes
Oral exam or oral presentation	••	••	••
Open-ended written test	••	•
Multiple-choice objective test	••
Conceptual map	••	•
Academic assignment	••	•
Minute questions	••	•
Diary		••	••
Portfolio	••	••	••
Project	••	••	••
Problem	••	••	••
Case	••	••	••
Essay	••	•	•
Discussion	•	••	••
Observation	•	••	••

Once all the selected evaluation techniques have been specified, their assessment must be normalized, so all the evaluation tests are scored from zero to 10, to make the grades obtained by students in the different proves comparable [16]. Next, a weight is assigned to each evaluation technique according to the significance that each of them must have in the course’s final grade, since some criteria are more relevant than other ones in the teacher’s opinion [6]. The weights are expressed as a percentage, and the sum of all of them should be equal to 100%. In some cases, it can be considered appropriate for some evaluation technique to have an additional evaluation of the final score of the course, with which students could recover points lost on other tests to improve their final grade and. thus, increase their motivation to obtain a final good grade in the course [17]. In that case, it should be specified that said evidence would be considered an “additional grade”, and its weight is not included in the previous sum.

The last step, once the evaluation system has been designed, consists of the design of the evaluation matrix, where the quantitative mechanism related to the students’ evaluation is reflected. The matrix is a double-entry table, where each of the rows is a learning result, and each of the columns is one of the criteria (assessment techniques) by which each of these results is evaluated. Within the matrix, the weights associated with each of the criteria for each learning outcome are entered. The structure of the evaluation matrix, designed from [6], is shown in

Table 2. Evaluation matrix.

Learning Result (i)		Evaluation Technique (j)
		1	2	3	…	j
Result	1	W11	W12	W13	…	W1j
Result	2	W21	W22	W23	…	W2j
Result	3	W31	W32	W33	…	W3j
	$⋮$	$⋮$	$⋮$	$⋮$	$\ddots$	$⋮$
Result	i	Wi1	Wi2	Wi3	…	Wij

.

When the methodology is applied for the first time to a course, it is necessary to make adjustments according to the obtained results. Therefore, based on the first case of application and the successive ones, the evaluation matrix has to be updated according to the results obtained.

4. Case of Application

The methodology described in the previous section has been applied to the design of the multi-criteria evaluation system for the course on energy markets taught as part of the Degree in Energy Engineering at the High Technical School of Industrial Engineers at the Polytechnic University of Valencia. The syllabus of said course and the learning outcomes associated with each of the topics are shown in

Table 3. Syllabus and learning results of the course on energy markets.

Educational Unit	Lesson	Learning Results
Unit 1. Introduction to Energy Markets	1. Characteristics of Energy Markets	LR1. Explain how energy is bought and sold
		LR2. Describe the main characteristics of energy markets
	2. Basic Concepts of Microeconomics	LR3. Distinguish between regulated markets and competitive markets
		LR4. Calculate the surpluses of the participating agents in a particular market session
		LR5. Calculate own and cross elasticity of demand
		LR6. Distinguish between a monopoly and an oligopoly in an energy market
	3. Energy Contracts	LR7. Classify the types of contracts existing in an energy market
		LR8. Compare the types of contracts of an energy market according to their characteristics
Unit 2. Electricity Markets	4. Electric Sector Structures	LR9. Identify the agents of an electricity market and the infrastructures associated with them
		LR10. Analyze the operating strategies in power systems
		LR11. Classify the structures of the electricity sector in the four market models
	5. Risk Management	LR12. Identify the types of risk to which the different agents of an electricity market are subject
		LR13. Explain the characteristics of electricity prices
		LR14. List short-term and long-term energy price prediction models
	6. Electricity Transactions	LR15. Calculate the economic dispatch in a single-area electrical system
		LR16. Calculate the joint economic dispatch in a multi-area power system
		LR17. Calculate the result of the market in a consortium with a single price and without a single price
	7. Short-Term Markets in the Iberian Market of Electricity	LR18. Enunciate the operating principles of the Iberian electricity market
		LR19. Classify the MIBEL market types
		LR20. Deduct the daily market price from the generation and purchase offers
Unit 2. Electricity Markets	8. Long-Term Markets in the Iberian Market of Electricity	LR21. State the operating principles of the futures market
		LR22. Classify existing products within the futures market
	9. Operation Markets	LR23. Classify types of electrical system adjustment services
		LR24. Identify the concepts that are part of the final price of electricity
	10. Electricity Invoicing	LR25. Identify the concepts that are part of a consumer’s electricity bill
		LR26. Calculate the terms of the bill of an electricity consumer in Spain
		LR27. Calculate the terms of the access tariff of an electricity consumer
Unit 3. Natural Gas Markets	11. Sector Agents	LR28. Identify the agents of the gas system and their associated infrastructures
		LR29. Explain how gas is introduced and removed from the system
	12. Contracts and Invoicing of Natural Gas	LR30. Identify the concepts that are part of a consumer’s gas bill
		LR31. Calculate the terms of the bill of a gas consumer in Spain
Unit 4. Emissions Markets	13. International Protocols	LR32. Enunciate the international protocols that govern the emission markets
	14. CO2 market in Spain	LR33. Enunciate the operating principles of the CO2 market in Spain

.

The techniques chosen to evaluate the subject’s learning outcomes are included in Table 4.

As indicated in the methodology, the set of tests that are chosen allows for the evaluation of different aspects related to the students’ knowledge, skills, and attitudes in an appropriate way. The indicated evaluation techniques are used as follows.

Table 4. Evaluation techniques used in the course on energy markets.

Technique	Knowledge	Abilities	Actitudes
Open-ended written test	••	•
Multiple-choice objective test	••
Academic assignment	••	•
Portfolio	••	••	••
Problem	••	••	••

Table 4. Evaluation techniques used in the course on energy markets.

Technique	Knowledge	Abilities	Actitudes
Open-ended written test	••	•
Multiple-choice objective test	••
Academic assignment	••	•
Portfolio	••	••	••
Problem	••	••	••

• Open-ended written test: There are four open-response written tests, two at the middle of the course and two at the end. Each written test is weighted as 10% of the final grade of the course. They are used to assess learning outcomes related to application, analysis, synthesis, and evaluation.
• Multiple-choice objective test: Two multiple-choice tests are carried out, one at the middle of the course and the other at the end. Each test consists of 20 multiple-choice questions with four possible answers, only one of which is correct. According to the methodology presented in [18] for the design of multiple-choice tests, each correct answer adds 1 point to the test; a wrong answer subtracts 1/3 point; and unanswered questions neither add nor remove points. Each multiple-choice test is weighted as 15% of the final grade. They are used to assess learning outcomes related to knowledge and understanding.
• Portfolio: This test is used to evaluate laboratory practices. Students have to keep a portfolio with the follow-up of their activities during the practices, which they have to document and solve correctly. Three laboratory practices are carried out in a computer room. The portfolio is evaluated at the end of the course and is weighted as 10% of the final grade.
• Problem: During the course, students are presented with four problems corresponding to the different thematic units, which they must solve. The issues are different for each student, since the statement is particularized with the student’s ID number (national ID, passport, etc.). Problem statements are posted on a specific date, which students are notified of on the first day of class. From the statement’s publication to the delivery deadline, 10 days elapse, within which students have to deliver the solved problem. If a student is late in the delivery, they receive a penalty of 0.1 points per day of delay. To deliver the solved problem, they are provided with an electronic template where they must indicate the results. Within 2 or 3 days from the delivery of the solved problem, the student receives their grade and feedback with the correction of their exercise by email. To do this, the teacher uses an explicitly designed computer tool, as detailed in [19]. Each problem is weighted as 5% of the final grade.
• Academic Assignment: In addition to the previous tests, with a resulting grade of 100%, students have the possibility of doing a voluntary academic assignment, for which they can obtain up to a 5% extra score to complement their final grade. Being an additional test, it is not part of the evaluation matrix. The topic to carry out the academic work is agreed upon with the teacher during the first month of the course, and it is related to one of the topics being discussed during the course in which the student has a greater interest. The realization of the academic assignment is subject to continuous monitoring throughout the course, through tutorials by the teacher.

The resulting evaluation matrix with the weights related to each of the techniques used for each learning outcome is shown in

Table 5. Evaluation matrix of the course on energy markets.

Learning Results	Evaluation Techniques
	Open-Ended Written Test	Multiple-Choice Objective Test	Portfolio	Problem
	40%	30%	10%	20%
LR1		1.25%
LR2		1.25%
LR3		1.25%	1.00%
LR4	4.00%			2.50%
LR5	6.00%			2.50%
LR6		1.25%
LR7		1.25%
LR8		1.25%
LR9		1.25%	1.00%
LR10		1.25%
LR11		1.25%
LR12		1.25%
LR13		1.25%
LR14		1.25%
LR15	2.00%		1.00%	1.00%
LR16	4.00%		2.00%	1.50%
LR17	4.00%			2.50%
LR18		1.25%
LR19		1.25%
LR20		1.25%	2.00%
LR21		1.25%
LR22		1.25%
LR23		1.25%
LR24		1.25%
LR25		1.25%
LR26	5.00%		3.00%	2.50%
LR27	5.00%			2.50%
LR28		1.25%
LR29		1.25%
LR30	5.00%			2.50%
LR31	5.00%			2.50%
LR32		1.25%
LR33		1.25%

. Finally, the time schedule for carrying out each evaluation technique throughout the course is shown in Figure 2.

5. Results and Discussion

The aforementioned procedure was applied to the grading of students during the 2021–2022 academic year. The obtained result for each of the evaluation proves is shown in Figure 3.

During that academic year, 68 students were enrolled in the course. As can be seen in Figure 3, grades obtained in each of the evaluation techniques, weighted by the factors mentioned above (30% for multiple-choice tests, 40% for open written tests, 20% for problems, and 10% for the practices portfolio) allow for calculating the final grade for each student. By using this method, students are not tied to any specific evaluation technique, so that the obtained grading can be considered more objective. For this application, the extra grading obtained in the voluntary academic assignment has not been considered.

There is another interesting aspect to consider when this method is applied, which is highlighted in the results of student 25 in Figure 3. In this case, if just the rest of the evaluation techniques were considered, the final grade for this student would have been higher by 5 points, and the student would have passed the course. However, both the attendance of the practices and the elaboration of the portfolio are mandatory. Consequently, the final grade obtained by the student is “not attendant”, since the teacher has no evidence to evaluate the student on the mandatory proves, so the course was not passed.

The values obtained by using the multi-criteria evaluation method are compared with the results obtained by just considering the traditional exams, as shown in Figure 4. For this purpose, the grades according to the traditional method were obtained by considering only the weighted scores on the multiple-choice and open-written tests.

According to the results shown in Figure 4, it can be seen that, in general, the final grades of students improve when a multi-criteria method is used. The students with the highest grades do not have any significant variations, which could mean that excellent students are not affected by the kind of evaluation test that is used. However, for the majority of students, their obtained grades improve between 5% and 10%. Only five students would have a better grade according to the traditional method of evaluation. As an average, the grades of students improved by 5.04% when the multi-criteria method was used, which also meant that the number of students failing the course was reduced by 50% (from 16 to 8).

Finally, Table 6 shows the evaluation matrix for the average grade obtained in each of the evaluation techniques, which illustrates the weight of each learning result within the final grade of the students. Obtaining this table for each student would represent the specific evaluation of each of them according to the learning results acquired during the course.

If the methodology proposed here is compared to other cases where a multi-criteria method was used, some improvements could be detected for the future. For example, in [9], where smaller groups were considered (10 students), a mixed procedure where both the teacher and students made part of the evaluation. Advantages of peer-evaluation methods for larger groups have been also considered in other cases [20], even when multi-criteria methods were not contemplated. Therefore, a multi-criteria procedure incorporating peer-evaluation is an alternative to be investigated in future applications of this methodology.

Table 6. Evaluation matrix of the course on energy markets.

Learning Results	Evaluation Techniques
	Open-Ended Written Test	Multiple-Choice Objective Test	Portfolio	Problem
	40%	30%	10%	20%
Marks	7.18	4.44	7.27	8.87
LR1	-	0.06	-	-
LR2	-	0.06	-	-
LR3	-	0.06	0.07	-
LR4	0.29	-	-	0.22
LR5	0.43	-	-	0.22
LR6	-	0.06	-	-
LR7	-	0.06	-	-
LR8	-	0.06	-	-
LR9	-	0.06	0.07	-
LR10	-	0.06	-	-
LR11	-	0.06	-	-
LR12	-	0.06	-	-
LR13	-	0.06	-	-
LR14	-	0.06	-	-
LR15	0.14	-	0.07	0.09
LR16	0.29	-	0.15	0.13
LR17	0.29	-	-	0.22
LR18	-	0.06	-	-
LR19	-	0.06	-	-
LR20	-	0.06	0.15	-
LR21	-	0.06	-	-
LR22	-	0.06	-	-
LR23	-	0.06	-	-
LR24	-	0.06	-	-
LR25	-	0.06	-	-
LR26	0.36	-	0.22	0.22
LR27	0.36	-	-	0.22
LR28	-	0.06	-	-
LR29	-	0.06	-	-
LR30	0.36	-	-	0.22
LR31	0.36	-	-	0.22
LR32	-	0.06	-	-
LR33	-	0.06	-	-

Table 6. Evaluation matrix of the course on energy markets.

Learning Results	Evaluation Techniques
	Open-Ended Written Test	Multiple-Choice Objective Test	Portfolio	Problem
	40%	30%	10%	20%
Marks	7.18	4.44	7.27	8.87
LR1	-	0.06	-	-
LR2	-	0.06	-	-
LR3	-	0.06	0.07	-
LR4	0.29	-	-	0.22
LR5	0.43	-	-	0.22
LR6	-	0.06	-	-
LR7	-	0.06	-	-
LR8	-	0.06	-	-
LR9	-	0.06	0.07	-
LR10	-	0.06	-	-
LR11	-	0.06	-	-
LR12	-	0.06	-	-
LR13	-	0.06	-	-
LR14	-	0.06	-	-
LR15	0.14	-	0.07	0.09
LR16	0.29	-	0.15	0.13
LR17	0.29	-	-	0.22
LR18	-	0.06	-	-
LR19	-	0.06	-	-
LR20	-	0.06	0.15	-
LR21	-	0.06	-	-
LR22	-	0.06	-	-
LR23	-	0.06	-	-
LR24	-	0.06	-	-
LR25	-	0.06	-	-
LR26	0.36	-	0.22	0.22
LR27	0.36	-	-	0.22
LR28	-	0.06	-	-
LR29	-	0.06	-	-
LR30	0.36	-	-	0.22
LR31	0.36	-	-	0.22
LR32	-	0.06	-	-
LR33	-	0.06	-	-

6. Conclusions

This article highlights the advantages of using a multi-criteria assessment system, since, among its other aspects, it allows students to be assessed without being tied to a specific technique. This favors students by evaluating them more objectively, as it is common for each student to be more comfortable with some specific evaluation technique; by limiting the evaluation to a single type, some students are harmed.

The use of a multi-criteria method also makes it possible to combine the usual techniques (open-response exams or objective tests) with other types of tests, through which a continuous evaluation of the students can be carried out. In this way, the combination of various types of tests allows for a more exhaustive assessment of the degree of development of the learning objectives of the course, while allowing the teacher to choose a greater or lesser weight of each learning objective within the final grading of the course. This selection should be done according to the significance of the different learning results in the curriculum of students, based on the teacher’s teaching experience and the definitions of some key performance indexes, which will be the objective of future research.

The methodology designed and shown here has been successfully applied to the course on energy markets taught as part of the Degree in Energy Engineering at the UPV, where both teachers and students have positively evaluated its implementation, given the advantages it presents with respect to the traditional exams that were used previously. In the case of the students, grading has improved between 5% and 15% in most cases, based on the more objective evaluation of the learning results achieved during the course.

Currently, this methodology is in the process of being applied to other undergraduate and master’s degree courses taught at the High Technical School of Industrial Engineering, where it is expected to achieve equally satisfactory results.