New Methodological Approach to Classify Educational Institutions—A Case Study on Romanian High Schools

Abstract

Since 2021, the National Evaluation exam in Romania (the exam aimed to assess 14- to 15-year-old students’ knowledge at the end of lower secondary education and just before high school) has presented a novel examination structure that resembles PISA tests. The current investigation analyses the 2021 National Evaluation exam results compared to the results obtained in the previous two years (2019–2020) as an evaluation of upper education institutions’ effectiveness in Romania. The results put forward the same conclusions as proposed by extant literature on Bucharest high schools. Even though the educational institutions show apparent progress and great adaptability to change, a more in-depth analysis reveals great inequality between educational institutions. As in the case of Bucharest, nationally there are only a small number of top-performing high schools in Romania, with the majority of high schools ranking in the lowest category as conceptualised in the study. The current investigation puts together a novel methodology for classification based on the main instruments proposed in literature: a letter grade classification and Turner’s f-index. The results and the methodological proposal are especially relevant considering the latest PISA (2018) conclusions on Romania characterising the national educational system as underperforming.

1. Introduction

The new paradigm in education is to shift the focus from class-taught material to critical analysis and towards first-sight text analysis. To this end, the Romanian 2021 high school admission introduces a novel approach; 2021 was the first year when pre high school evaluation (i.e., the National Evaluation) presented a different examination structure, intended to be similar to the PISA tests (The OECD Programme for International Student Assessment of performance in reading, mathematics, and science). Since the results from the National Evaluation represent 80% of high school admission grades, an investigation of the public data regarding high school admission would be a proxy for the impact of this progressive approach to students’ evaluation. Adding to previous research at the Bucharest level [1], we proposed a national investigation of high school admission results. Our findings sustain discussions concerning the adaptability and the effectiveness of the Romanian lower secondary education (grades 5–8) and the first half of upper secondary education (9–10 grades; [2]).

The Romanian compulsory educational system comprises 11 school years: primary education (grades 0–4), lower secondary education (grades 5–8), and the first half of upper secondary education (grades 9–10 or the first two years of high school; [2]). Students’ knowledge assessment, through the National Evaluation, at the end of lower secondary education addresses the same age group as PISA exams. Therefore, besides the novel structure of the exam, the implementation at this level provides proper criteria for comparison considering the European educational standards and targets.

This discussion is urgent, especially considering the latest PISA Romania results from 2018 [3]) revealing that Romanian students are still behind the OECD average, having only slight improvements from the previous assessment. When compared to the mean share of top performers above the OECD average, 29.8% of Romanian students’ results in all subjects were below the average, with 4.1% being above average in at least one subject. This trend is reversed for countries considered top performers, as more of their students are above the average and less below. Second, 40% of Romanian students are considered functionally illiterate, far behind the OECD and EECA average. Although Romanian students show improvements on the long-term in the mathematics assessment, the curve is levelling, ranking 47 upper and 56 lower in all countries’ results.

As emphasised in the OECD and UNICEF report, the Romanian National Education Strategy requires several reforms with the aim of levelling the educational inequity [4].

Although the Romanian education standard has improved significantly in the last 10 years, there are still major improvements to be made [5]. In addition to the international disparities, the Romanian education system also fails to provide a nationally consistent standard: while some high schools are top performers, with last admission grades as high as 9.84, others are consistently delivering sup-prime results, with last admission grades as low as 4.74 [6].

This kind of initiative can only start with proper system evaluation that might provide clarity in answering questions such as: How can we best identify the under-performing institutions? What are the factors that influence high school performance in terms of admission grades? To what extent can the education system maintain its performance under legislative shocks?

2. Brief Description of the Romanian Education System

The Romanian National Education system is regulated by the Ministry of Education, in collaboration with various other Governmental institutions. In terms of legal subordination, the Education System subscribes to the Constitution, the Organic Law of National Education, as well as ordinary Government ordinances [7].

The openness and pluralistic ethos of Romania’s National Education System are its two key foundations in terms of overarching ideals. If they satisfy certain requirements, students may transfer from one institution to another or from one specialisation to another. Romania’s educational system consists of both public and private institutions and offers instruction in the country’s official language (Romanian) and in the languages of the major minority groups. The state fully funds public education, giving Romanian residents access to nearly-free education up until the last year of high school [7].

The Romanian Government requires 11 years of obligatory education, beginning with primary school (grades 0–4), lower secondary school (grades 5–8), and the first two years of high school (grades 9–10), which make up the first half of upper secondary education [2].

The Romanian National Education System provides free access to all stages of education: early education (kindergarten, up until 5 years old, not mandatory), primary education (primary school, grades 0–4, mandatory), secondary education (gymnasium: grades 5–8 and high school: grades 9–12/13), tertiary non-university education (professional and technical qualification, not mandatory), and higher education (limited nationally subsidised spots, not mandatory [7].

3. High school Enrolment Procedure

The Romanian secondary education system includes [8]:

• National colleges and high schools: national colleges are the most prestigious high schools, well-equipped, and internationally connected secondary institutions in Romania. Usually, a high school’s name is indicative of its academic focus: theoretical high schools, economic high schools, etc.;
• Military colleges: administered by the Ministry of Defense, Ministry of Justice and other such ministries, alongside the Ministry of Education;
• Economic and technical colleges: both offer academic programs geared towards technical/service industry training;
• School groups: two schools that have teamed up, generally a high school offering technical or service industry programs and a school of arts and trades.

In terms of the educational offering, high schools can be divided into three branches, which are further divided into specialisations:

• Vocational branch: pedagogy, theology, music, visual arts, and sports;
• Technological: natural resources (agriculture), services (commerce, tourism and food industry, and economic), and technical (electric, mechanic, installations, and construction);
• Theoretical: Humanities (social sciences and philology) and Sciences (natural sciences and mathematics-informatics).

Since the present study tackles the issue of high school admission and institutional performance, we have further discussed the necessary qualifications and enrolment procedure.

The first step in acceding to high school is graduating from the previous educational cycle-gymnasium, which means obtaining an average final grade of at least 5. This grade will contribute to a student’s high school admission grade [2].

Secondly, after finishing eighth grade, students undergo the National Evaluation Exam. This exam is a summative tool for evaluating students’ knowledge and abilities from their four years of gymnasium (grades 5–8) on the outside. There are two subjects on the test:

• Romanian/Maternal language and literature: depending on whether the student undertook classes in Romanian or is a member of a national minority, they undergo a 120 min test meant to evaluate their abilities to understand written text, to adequately and efficiently utilise language in verbal/written form, and to express personal linguistic and cultural identity in a national and international context.
• Mathematics: all students undergo a 120 min test meant to evaluate general and specific field competencies: data identification and processing, measurement, mathematical relationships and operations, and conclusions and solution steps in mathematical language.

The exams take place simultaneously for all students in Romania, regardless of whether they are enrolled in private or public institutions. The Ministry of Education (2022) created the examination’s legal framework, but from an institutional standpoint, the National Evaluation is organised and managed by a specialised commission made up of a president, three vice presidents, a secretary, and four to six other members. Through the County Inspectorates, each of the 41 counties in Romania oversees the implementation of the directives that are issued at the national level.

After obtaining the results from the National Evaluation Exam, students are ready to undergo the high school enrolment procedure. For a student to enrol in high school, they must prepare the application file, which includes the results obtained in the National Evaluation and their average gymnasium grade. These scores are then computed into a final admission grade with proportions of 80% and 20%, respectively.

After preparing the admission file, students submit a list of preferred high schools and specialisations based on their own affinities as well as their perceived chances of success: if their admission grade is relatively low, it would not be wise to only enlist high schools that historically had very high admission grades. If a student fails in being admitted to any of their enlisted high schools, they will automatically be redistributed by the state, based on the remaining spots. The high school admission procedures uphold early mobility standards: any student can enlist into any high school in the country. However, in order to apply for a high school that is not located in your city of residence, students must submit their application documents to a specific registration centre.

The documents are submitted to the high school registration centre, by the student, together with a parent or guardian. The list has a nationally standardised format, where students simply have to write down the identification code unique to their desired institution and specialisation in the desired order. Based on the list of priorities and the admission grade, students are distributed into high school classes.

4. Previous Educational System Evaluations

The results of previous research conducted on high school admissions in Bucharest [1] align with the PISA conclusions. The disparity in the effectiveness of educational institutions in Bucharest is underlined by the existence of a handful of top-performing high schools, together with a majority of under-performing high schools. Even though there seemed to be apparent progress considering only the National Evaluation results, in 2020, the vast majority of high schools fit in the A and B brackets of the letter grade system used, while in 2021 the majority of high schools fit in the C, D, and F brackets. The results showed that no significant differences exist in the median of the first admission grade between 2019 and 2021, while significant differences exist between 2020–2019 and 2020–2021. This offers evidence-based policy support for the need to examine both ends of the high school admissions performance range.

Further investigation approached high school performance in terms of branches, profiles, and specialisations. The Romanian education system is structured around three branches called ‘Vocational’, ‘Theoretical’, and ‘Technological’. The Theoretical branch is split into two profiles, those of ‘Sciences’ and ‘Humanities’, with each of the profiles being split further down into two specialisations: Sciences–Mathematics-Informatics and Natural Sciences; Humanities–Philology and Social Sciences. The Technological branch is split into three profiles, and the Vocational is split into five profiles.

The average first admission grade was tightly concentrated around the median score of the profiles from the Theoretical branch, varying between years by a margin of 0.10 points. In 2021, with the changes from National Evaluation, more than 50% of the scores from Humanities landed above the median, while only a third of the scores from Sciences landed above the median, suggesting more stability among Sciences. The last admission grade for the Humanities ranged between six and nine, while for Sciences, the last admission grade was concentrated towards the highest grades. For Natural Sciences, there is a four-to-six-point difference in the percentage of 2019–2020 admission average grades. For the Technological branch, the between-years distribution of admission grades presented a greater variation in the median score, which once again tended to support the presence of an underlying systemic change.

Overall, the Sciences profile registered the highest final admission grade among the five profiles in the theoretical discipline, followed by the Humanities profile. It is also worth noting that, except for in 2020, the last admittance grade in the Theoretical field was persistently lower than the otherwise passing mark (5/10).

5. New Approach to Educational System Evaluations

The study on Bucharest high schools [1], with the above-mentioned results, proposed a letter grade system classification stemming from the Anglo-Saxon school grading system with letters instead of numerical grades (i.e., A, B, C, D, F).

Table 1. Number grade conversion.

Numeric Grade	Standard Grade	Grade Point Average
90–100	A	4
80–89	B	3
70–79	C	2
60–69	D	1
<60	F	0

Source: NCES—National Centre for Education Statistics.

presents the correspondence between the numerical score obtained by a student and the letter grade classification.

For intermediary grading, pluses and minuses may be assigned to grades B and C, whereas grade A can be coupled with a minus, and grade D can be coupled with a plus.

In order to further develop the grade classification system described above, we employed two alternative classification metrics based on an f-index as proposed in extant literature on high school recruitment [9]. The index is defined as the number of feeder high schools whose former students enrolled within the higher education institution ≥ f. The f-index is attributed to an institution on a yearly basis when f out of the H feeder high schools have at least f students enrolling, whereas the rest of the high schools have less than f students each. The f-index and the potential relationship between the f-index and a high school’s position in the letter grade classification are further described in the Section 6.1. Both metrics are to be employed at the national level.

6. Methodology and Dataset

6.1. Methodology

Our approach expands on two measurements proposed in the literature: (1) a high school letter grade classification based on last admission grade and last admission rank, as proposed by Firescu et al. (2022) [1] and (2) Turner’s f-index. For the first metric, the letter grade classification is based on the last admission grade and last admission rank for the admission criteria (i.e., National Evaluation and average gymnasium grade) as presented in the ‘High school enrolment procedure’ section. As for the Turner’s f-index, the metric is an adaptation of the Hirsch index (2005) for recruitment data [9].

We commenced with the national rating (letter grades A–F) of high schools concerning average admission grades and the last admission rank. We established this classification at the county level. We limited the number of observations to 1642 (the cartesian product between the number of high schools and the number of theoretical field specialisations). This classification revealed the high schools with at least one A-specialisation and underlined national patterns that are to be further discussed. For the second metric, we employed Turner’s f-index to classify the remaining high schools and assess whether the number of feeder schools influences the classification by last admission average grade and last admission rank relative to specialisation.

We proposed a multinomial logistic regression to compute the probabilities of a student being admitted to high school at a certain academic specialisation within the hierarchical scale (i.e., A, B, C, D, or F) given his admission grade and by county and specialisation.

The multinomial model [10] is an extension of the logistical regression by relaxing the binary outcome assumption to multiple classes. The model is expressed in Equation (1):

$$\mathrm{Pr}(Clas{s}_i=\mathrm{K}-1)=\frac{e^{{\beta }_{k-1}{X}_i}}{1+{\sum }_{k=1}^K e^{{\beta }_{k-1}{X}_i}}$$

(1)

where:

• Pr—probability of category k from variable Class to be correctly classified;
• K—number of unique categories in the dependent variable, in this case the classification of high schools by last admission grade;
• $\beta$—regression coefficient obtained by using the logarithm function;
• $X_k$—the independent variable, i.e., the admission grade of the student.

Model performance was measured through Akaike Information Criterion [11] expressed in Equation (2), residual variance [10], Equation (3), and classification accuracy [10], Equation (4):

$$\mathrm{AIC}=2\mathrm{k}-2\log (\widehat{L})$$

(2)

where:

• AIC—Akaike Information Criterion;
• K—number of categories in our dependent variable;
• $\widehat{L}$—likelihood function;

$$\mathrm{RDev}\left(\beta \right|Class)=-2log(L(\beta |Class))+2log(L(Class|Class))$$

(3)

where:

• RDev—residual deviance;
• $\beta$—regression coefficient;
• Class—dependent variable;
• L—likelihood function;

  $$\mathrm{CA}=\frac{Number of categories correctly classified}{Total number of categories classified}\times 100$$

  (4)

  where classification accuracy (CA) is obtained from a confusion matrix with an equal number of rows and columns, each column representing a unique category from the dependent variable; the number of categories correctly classified represents the sum of the main diagonal, and the total is given by the sum of the margins.

6.2. Dataset Description

The original dataset consists of 91,315 observations, where each observation represents students’ admission data to a high school specialisation. Admission data consisted of grades obtained at the National Evaluation written tests. From the public dataset, we extracted the variables presented in

Table 2. Variables employed.

No.	Variable Name	Variable Definition	Observations
1	County	Administrative division in Romania
2	High school	Name of the high school at which the student was admitted (superior secondary education)
3	School	Name of the student’s origin school (secondary lower education)
4	Specialisation	Name of the academic specialisation in theoretical academic field
5	Average admission grade	Average between grades obtained at National Evaluation	Derived from the original data
6	Admission ranking	Students ranking at the county level based on average admission grade	Derived from the original data
7	Classification by last admission average grade	Letter grade classification of high schools specialisations based on last admission average grade (interval cut-offs)	Derived from the original data
8	Classification by last admission rank	Letter grade classification of high schools specialisations based on last admission rank (deciles cut-offs)	Derived from the original data
9	f-index	Classification of high schools specialisations based on equal rank-value number of the origin schools for admitted students.	Derived from the original data

concerning the theoretical field specialisations (i.e., Mathematics-Informatics, Natural Sciences, Philology, and Social Sciences) which account for 57,474, or 63%, of total observations.

7. Results

7.1. National Classifications-Letter Grade and f-Index Classifications

Figure 1 and Figure 2 present the count for the theoretical high schools in Romania by specialisation (i.e., Mathematics-Informatics, Natural Sciences, Philology, and Social Sciences) according to letter grade classification by last admission average grade and last admission rank.

The figures show that high schools cluster differently depending on the classification criteria. By considering the last admission grade, high schools follow an almost uniform distribution, slightly skewed to the right. The classification by last admission rank generated highly skewed data—60% of students are clustered in the F category.

Mathematics-Informatics and Natural Sciences fields encompass more A category specialisations, which implies greater competitiveness among students.

High schools are almost uniformly distributed among counties, except for counties including large cities with high population concentrations such as Bucharest, Cluj, Iași, Constanța, and Dolj. Bucharest alone accounts for 10% of the national population and attracts 15% of students in high school with at least one specialisation in the theoretical field. In Bucharest, the number of high schools is the same as the number of high schools in the four neighbouring counties, combined.

When counting the number of high schools with at least one A specialisation (classified both by last average admission grade and last admission rank) different national patterns emerge, as illustrated in Figure 3 and Figure 4.

When considering the last admission grade, Mathematics-Informatics and Natural Sciences specialisations reveal greater spatial variability than high schools with at least one specialisation in Philology and Social Sciences. This is congruent with the greater competitiveness identified in the case of Mathematics-Informatics and Natural Sciences, as suggested in Figure 1.

For the last admission rank, the conclusions are considerably different. We can observe a more evenly distributed number of high schools across the country. This can be explained through the proportion of admitted students in each county—a small population of students can artificially increase the ranking of a high school when using deciles as an underlying classifier. This effect is most apparent in the Philology and Social Sciences specialisations.

The f-index classification criteria for high schools account for the number of feeder schools—the previous schools attended by current high school students.

Figure 4 illustrates the mean number of feeder schools per county and specialisation. Nationally, the mean number of feeder schools is uniformly distributed, with an average number of two feeder schools per high school by specialisation. Figure 5 shows that the differences between specialisations are marginal. The only noteworthy difference is revealed at the county level, as seen in Figure 6.

Figure 7 illustrates the relation between the two metrics employed: (1) the high school classification based on last admission grade and last admission rank and (2) the f-index. The flow patterns are similar for both relations, although a smaller number of high schools with A specialisations are present when using classification by last admission grade (ClassificationLAG) when compared with classification by last admission rank (ClassificationLAR). This could be described in terms of the underlying methods used for both classifications, namely deciles for ClassificationLAR and grade cut-offs for ClassificationLAG. We can observe that high schools with an f-index of two or three are somewhat homogeneously distributed across both systems of classification and account for two-thirds of the total number of high schools by each letter classification.

7.2. Multinomial Logistic Regression

We employed several multinomial logistic regression models. The performance is presented in Appendix A providing Akaike information criterion, residual deviance, and classification accuracy obtained from the confusion matrix.

Due to low performance in terms of classification accuracy (i.e., ~20%), when using the whole dataset with no additional information except the admission grade at high school, we opted for exploratory analysis with models including county and specialisation. The initial model’s low performance can be attributed to high data heterogeneity-due to great variability, from county to county, among high schools with different classes of performance. The average increase in performance when broken down by county was ~60%.

The explanation can stand on a two-fold argument. Firstly, the dependent variable was built using fixed non-overlapping intervals for all high schools. For example, a student with a high grade can be admitted to a specialisation as F; this implies non-homogenous groups of admitted students. To control for this, additional information is required, such as the average historical performance of the student. In Appendix A, Tulcea county supports this observation with a model scoring accuracy of 100% for the Mathematics-Informatics specialisation due to the non-overlap between average grades used at admittance in solely two available classes of high schools, B and F.

The main result of the multinomial analysis is instantiated only for Bucharest, as in Figure 8, where we present the probabilities of being admitted to a specialisation within a performance class, given a student’s average admission grade.

The probabilities for admission given an admission grade are homogenous across academic specialisation and classification of high schools by last admission average grade. This allows us to identify an interval pattern for assessing a student’s chances for admission. For example, a student with an average grade between 9 and 10 has a more than 50% higher chance to be admitted to an A-category high school than a student with a grade between 7 and 8, which, most likely, will be admitted in a category C high school. As shown in Figure 8, a spike in probability is registered in categories C for Natural Sciences and Philology and D for Social Sciences due to the large number of observations that fall in these categories.

8. Discussions

The multinomial models proposed to predict where a student is likely to be admitted have an average performance of 60% by considering only admission criteria predictors.

Although high schools are uniformly distributed across Romania (except for large cities), when classifying these educational institutions, some notable differences do emerge. First of all, when the letter grade classification is employed, counts follow the different statistical distribution and geographical distributions across the country. This underlines a trend observed previously in studies concerning Bucharest—there is considerable inequality between high- and low-performing educational institutions [1].

Nationally, we identified large differences in what concerns students’ admission grades and admission ranks between Mathematics-Informatics and Natural Sciences, on one side, and Philology and Social Sciences, on the other. While the former specialisations are well represented nationally, with high geographical variability, the latter presents scarcer spatial representation, especially when considering the last average admission grade.

The f-index, which accounts for high school students’ background, namely for feeder schools from which students apply to the secondary form of education, underlines no apparent difference across the country, at least when analysing high schools in terms of letter grade classification by admission grade and admission rank. Further exploring the spatial relations between f-index and students’ admission performance by specialisation could provide more insight. However, using last admission rank classification, with an underlying decile representation, can present some challenges given that the student population between two counties can vary greatly—a lower number of students can artificially increase the ranking of one or more high school specialisations in specific low-population counties. Therefore, this classification is most suitable for within-county comparisons.

As previously elaborated, this discrepancy is vividly observed within Romania’s results in the latest PISA results from 2018 (OECD, 2019), where the country scores below the OECD average but delivers both 4.1% of top performers in at least one subject and 29.8% of students with low performance in all tested subjects. The Trends in International Mathematics and Science Study (TIMMS), a standardised European test similar to PISA, underlined similar results, with Romanian 15-year-olds being less competitive than European students [12].

UNICEF also encourage Romania to prioritise progress in education and to double its investments in education [13]. Similarly, OECD recommends strategic planning in what concerns the Romanian educational system by aligning system monitoring to educational priorities [14].

As of June 2021, seven out of ten Romanians indicated that the Romanian educational system needs major reform [15]. The acute disparity between educational institutions can be explained by the fact that middle schools that prepare students for this examination perform inconsistently, based mainly on socio-economic factors. Ultimately, the utmost indicator of poor management in Romanian schools is suggested by the high level of school dropout in Romania that occurs from secondary education to post-high school and university [16].

Romanian experts [17] identified four main vulnerabilities of the educational system, namely ineffectiveness, irrelevance (within the new economy), inequality (in what concerns the capacity to create equal opportunities), and poor endowment with teachers and trainers. The authors argue for correcting these issues to reduce development disparities between Romanian regions and to encourage innovation and true competition. This is aligned with the international literature on school management that applies the philosophy of service organisations on improved learning outcomes, continuous and sustainable improvement, and empowered personnel [18]. Moreover, systemic problems within educational institutions backlash into micro and macroeconomic implications such as unemployment that further impacts the subjective perception of life satisfaction and the social exclusion percentage [19].

Even though the current paper succeeds in the descriptive endeavour to assess the efficacy of Romanian education by exploring the results from high school admissions, a systemic change should target lower education. Our evaluation identifies a problem at the secondary level that, as suggested by the international experts, should be targeted at lower levels since the progress from primary and secondary education to higher education is only truly functional when regarded as a continual process [20].

9. Conclusions, Limitations, and Future Work

Putting together measurements proposed by prior methodological studies, namely the letter grade classification and the f-index, we provided some insights on high school institution effectiveness and equality in Romania. The public data, consisting of students’ grades from the National Evaluation, was used to assess how competing high school specialisations tend to attract students. Mathematics-Informatics and Natural Sciences were revealed as the most competitive academic specialisations comprising most A-rated specialisations. The f-index metric, accounting for students’ educational background as feeder schools for high schools, underlines a uniform distribution across the country.

Our multinomial model excludes factors outside the educational system that may be of high relevance, such as students’ subjective preference regarding high schools and socio-economic variables such as diverse costs regarding education or transportation. This only underlines the ecological interdependency between education and the rest of the social environment—economy, governance, and general development—and brings an important addition to what has been studied so far, regarding admission, besides topics like educational mobility and school choice by analysing the heterogeneity in students’ demands [21,22,23,24]. It can be stated, moreover, that few studies are conducted on data from Romania regarding the options and preferences of students when it comes to high schools [25], and much less are related to rankings and classifications [26]. Additionally, most of the research in this area focuses mainly on prediction and classification models for admission to colleges [27,28], not high schools. Thus, this study further clarifies the situation in secondary education.

Our investigation expands and discusses the broader applicability of extant measurements, and future research should focus on adding more information to these models. As above mentioned, a more accurate assessment should include factors external to the education system, external factors such as socioeconomic factors and subjective preferences or attitudes towards educational institutions. Furthermore, future evaluation may extend the analysis to other age groups and levels of education. Besides public data regarding the National Evaluation taken by 15 years old students, there is also public data regarding the exam results at the end of high school (Baccalaureate examination).

Even though a more complex evaluation would reveal more sophisticated patterns that are crucial for public policy regarding education, the differences revealed by an inexpensive basic analysis are a very important starting point. Even under rudimentary instruments, the Romanian educational system reveals inequalities and institutional underperformance within secondary education.