Broader Terms Curriculum Mapping: Using Natural Language Processing and Visual-Supported Communication to Create Representative Program Planning Experiences

Abstract

Accreditation bodies call for curriculum development processes that are open to all stakeholders, reflecting viewpoints of students, industry, university faculty, and society. However, communication difficulties between faculty and non-faculty groups leave an immense collaboration potential unexplored. Using the classification of learning objectives, natural language processing, and data visualization, this paper presents a quantitative method that delivers program plan representations that are universal, self-explanatory, and empowering; promoting stronger links between program courses and curriculum development open to all stakeholders. A simple example shows how the method contributes to representative program planning experiences and a case study is used to confirm the method’s accuracy and utility.

1. Introduction

Changing times impose new and radical challenges for societies, urging Higher Education Institutions (HEIs) to rethink their educational offer. Curriculum development is the process by which changes to educational offers are conceived and, to be successful, this process needs to create opportunities for active and consequent reflection. To create these opportunities, stakeholders’ participation is essential. This is the unanimous opinion of researchers and accreditation bodies [1,2,3], who defend making curriculum development open to all stakeholder groups, expressing the viewpoints of students, industry, university faculty, and society.

Focusing on program planning, a core process that lays at the heart of curriculum development, open principles are typically associated with interviews and focus group sessions, where non-faculty stakeholders are asked to express their views. Faculty, who hold the central managing role [4], are responsible for processing the data collected in these informational touchpoints, and it is faculty who participate in program planning discussions.

Faculty’s central role in program planning is not only a traditional functional attribute. Program planning requires scientific and pedagogic skills essential to—and therefore mastered by—faculty, that are not essential to non-faculty stakeholders. This opens an important communication gap, and the absence of dialogue between faculty and non-faculty groups [5] leaves immense collaboration potential unexplored.

Yet, the challenges imposed by a rapidly changing society, recognizing that responsive and effective program plans are more likely with the participatory (not just informational) involvement of all those concerned, forces HEI to reach out and find ways to integrate external contributions, valuing non-faculty stakeholders as experts of their own experience.

To achieve this objective, to bridge communication gaps, better representations of the program plan are essential. This paper proposes the use of a broader terms curriculum mapping method to deliver program plan representations that are universal, self-explanatory, and empowering.

The core objective of this paper is the presentation of the broader terms CM (curriculum mapping) method. Because this method uses a combination of information and data science techniques, a significant part of the paper is dedicated to the step by step description—using a simple example—of these techniques, and how they contribute to representative program planning experiences. To verify the method’s accuracy and utility, a case study is used.

But, before proceeding to the broader terms CM method presentation, it is important to explain what is meant by curriculum mapping, how it has been used previously, and what changes are required to make it a vehicle for representative program planning. This is the topic of the next section.

2. The Context: Curriculum Mapping

According to Burns [6], curriculum mapping is a process for recording what content and skills are taught in a study program. The recording relies on a visual medium, typically a chart, table, or map, depicting the building blocks of the study program and how these blocks relate to one another. Because different types of building blocks could be used, there are different types of curriculum mapping.

When individual courses are the building blocks, curriculum mapping provides a snapshot of existing learning pathways considering the available courses, helping students navigate the study program. These course mappings use the calendar year as an organizer to depict vertical (from year to year) and horizontal (within a year) relations between courses [6], and are usually represented as flowcharts. Meij and Merx [7] provide an example of a course mapping published online, with course-specific scientific and pedagogic details available as hyperlinked content.

For accreditation bodies, the grouping of contents and skills per course is not as relevant as ensuring these contents and skills results in expected learning outcomes [2,8]. For this reason, for accreditation purposes, learning outcomes are the building blocks, and learning outcomes mappings are used to show that the study program yields the expected learning outcomes. These mappings are typically represented as tables that align program learning outcomes and accreditation standards. Examples of learning outcomes mappings are given in Dyjur and Lock [9].

Learning outcomes mappings’ purpose goes beyond reporting the alignment with accreditation standards. This type of curriculum mapping is used to communicate accreditation bodies’ vision of transparency, accountability, and scientific curriculum development [5]. A vision that becomes reality with HEI adoption of outcomes-based education [10,11] and constructive alignment principles ([12] p. 99). Curriculum mapping is used, therefore, as a tool to shape HEI processes, particularly program planning.

Willcox and Huang [13] describe another type of curriculum mapping: the concept mapping. Concepts are, in this case, used for building blocks, with a concept denoting “the main idea underlying a (typically small) unit of content covered in a course” ([13] p. 9). These units of content are linked to Knowledge Concepts defined by [14], and Willcox and Huang [13] use concept mappings to provide insight into the relations between learning outcomes and between courses, helping faculty with the precise program plan navigation. Examples of this type of mapping are given in Seering et al. [15], Willcox and Huang [13], or Varagnolo et al. [16]. These authors use circular ideograms [17] and/or network graphs [18] to detail concepts’ precedence relations. The visual outputs presented by these researchers are very successful and efficient in conveying visual meaning to the complex relations found in study programs.

The analysis of the three types of curriculum mapping reveals important characteristics. Curriculum mapping is used to shape HEI processes, and this ability is valuable for opening program planning discussions to non-faculty groups. Curriculum mapping uses visual-supported communication to represent and discuss study programs, and the developments taking place in the field of information visualization can be used to bridge communication gaps, helping stakeholders to articulate their expert (non-verbal) knowledge. However, with regard to the choice of building blocks, if the objective is to increase non-faculty groups’ participatory involvement, broader (not detailed) concepts, requiring fewer scientific and pedagogic skills, should be preferred.

A curriculum mapping method that builds on the practices already available but tailored for non-faculty groups’ participation in program planning discussions is described in the next section.

3. The Method: Broader Terms Curriculum Mapping

This section presents a curriculum mapping method designed for representative program planning. A method that empowers all stakeholders.

A flowchart representing the method steps, respective inputs, and outputs, is presented in Figure 1.

The method considers four steps, detailed in the following subsections: (1) classification of course learning objective statements into broader terms; (2) use of natural language processing (NLP) to convert broader terms into quantitative frequencies of key program concepts; (3) visualization of key program concept frequencies and mappings with links between key concepts and/or courses; (4) discussion, considering the participation of all stakeholder groups, of the method’s visual outputs and decision to reclassify or review course learning objectives.

To illustrate how these steps apply, a simple example is considered.

Table 1. Learning objectives and respective broader terms classification for 5 courses. Notes: (1) A non-truncated version of this table can be found in [19]. (2) Data in this table model style and scope variability frequently found in learning objective statements and consider different levels of detail in broader terms selection.

Course	Learning Objectives	Broader Terms ${}^{\mathbf{a}}$
Mathematics (C1:MATH)	Recognize a real-valued function of a real variable; (…) Recall the concept of derivative of a real function and explain its geometric interpretation; (…)	Function of a real variable; Differential calculus; Integral calculus; Linear algebra; System of linear equations
Applied Physics (C2:PHY)	List fundamental concepts in mechanics and understand their importance to engineering; Use the international units (…)	Physics; Mathematics; Calculus; Mechanics; Thermodynamics; Fluid flow
Logistics and Oper. Manag. (C3:LOGOP)	Identify logistic activities in a generic organization; Explain the role of contemporary logistics; (…) Distinguish components of supply chain management (…)	Logistics; Supply chain management; Business; Production Economics; Operations management; Linear programming; Lean manufacturing; Process Control (…)
Energy Manag. (C4:ENER)	Discuss the efficient use of energy in industry, buildings and transports; Recognize applicable legislation and defend energy efficiency as (…)	Energy efficiency; Organization; Buildings; Facility management; Logistics; Production planning and control; Solar water heating (…)
Financial Manag. (C5:FIN)	List fundamental financial management concepts and functions; Recognize and explain financial statements; Contrast the economic and the financial analysis (…)	Financial management; Accounting; Economics; Finance; Organization; Business governance; Corporate law; Trade; Return on invested capital (…)

${}^{\mathrm{a}}$ Obtained with the Wikipedia index [20].

presents data for this example.

Table 1 includes the learning objectives for five courses—Mathematics (C1:MATH), Applied Physics (C2:PHY), Logistics and Operations Management (C3:LOGOP), Energy Management (C4:ENER), and Financial Management (C5:FIN)—of a bachelor degree in Technology and Industrial Management (also used in the case study section). For example, the first learning objective statement in the Mathematics (C1:MATH) course is: “Recognize a real-valued function of a real variable”.

The third column of Table 1 presents broader terms derived from the courses’ learning objectives. Broader terms can be multi- or single-word tokens and their identification is very much dependent on the methodology used. The fact that the broader terms CM method implements an iterative process allowing the reclassification of broader terms and the review of course learning objectives—see Figure 1—allows for the participated selection of the broader terms. The next subsection describes how broader terms presented in Table 1 were selected.

3.1. Step 1: Classification of Course Learning Objectives

To characterize courses and the program degree, the broader terms CM method uses course learning objectives (LOs). According to (Felder and Brent [8] p. 19), course LOs are defined as “statements of observable actions that serve as evidence of the knowledge, skills and attitudes acquired in a course”. These statements define key program concepts and, through these key concepts, the intricate web of course relations is revealed. Course LOs provide, therefore, access to the “mechanics” behind a program plan.

The problem of using course LOs is that they presume tacit understanding of concepts specific to disciplinary and scientific sub-areas, and this renders LO-statements seldom clear and unequivocal [21,22]. Even when LOs are written according to specific rules (e.g., considering Bloom’s taxonomy, [8,23,24]), the variability in style and scope results in a heterogeneous set, including statements that are often too abstract or too detailed [25,26].

To disclose their latent information and for effective communication, LO-statements would benefit from techniques used by library and information science professionals in resource classification. Resource classification indicates what a resource is about, and to achieve this goal, a control vocabulary, a set of broader terms (concepts or subject headings, [27]), and supporting classification has to be agreed upon. Control vocabularies are usually chosen among bibliographic classification schemes (such as the Dewey Decimal Classification), lists of subject headings [28], and thesauri [29,30,31]. More recently, for its comprehensiveness and up-to-dateness, the Wikipedia index [20] is also used (see [32,33] for a discussion of the advantages of using Wikipedia’s index as control vocabulary).

This paper considers principles of resource classification to classify course LOs. Concepts from the Wikipedia index matching course LO-statements are used to define broader terms. To illustrate how this is done, consider the excerpt of LO statements for Mathematics (C1:MATH) in Table 1: “Recognize a real-valued function of a real variable; Recall the concept of derivative of a real function and explain its geometric interpretation”. Using the Wikipedia index, the first statement could be classified according to the Wikipedia concept, “Function of a real variable”; a matching concept almost identical to the original LO. However, broader concepts could be chosen. For example, the second LO statement could be classified (with exaggeration) by the broader Wikipedia concept, “Differential calculus”. The third column in Table 1 includes results of course LO-statements classification for the five courses.

The classification example provided in the previous paragraph, and the comparison of columns two and three in Table 1, show a significant reduction in the vocabulary used to characterize the courses. Because the reduction in vocabulary could entail an important loss of information, it follows the importance of the conceptual analysis of LO-statements [27]. The importance of determining what the LO statement is about—the “aboutness”—and of the translation into (the selection of) specific broader terms.

Because program planning typically relies on the declared curriculum, with LO-statements written by university faculty, the classification of LOs is frequently performed by faculty in collaboration with curriculum planners [13,15,16]. An alternative procedure consists of an initial classification by an information science professional, subsequently revised by faculty [21]. For large database classification, automated machine learning techniques are also used [34,35]. In this paper, an initial draft classification of course LOs is made by a small multidisciplinary team of university faculty. Once visual outputs derived with the broader terms CM method are available, a reclassification is made with contributions from all stakeholders (see Step 4 in Figure 1).

For effective communication of the program plan, having courses associated with a small subset of broader terms selected from a control vocabulary is an important advantage. Key concepts found in courses can be identified, paving the way to their quantification and to the analysis of the relations between courses, i.e., to the analysis of information flows, such as topics covered, which assessments relate to which topic, and so on.

The next section describes in detail the method used in the quantitative processing of broader terms.

3.2. Step 2: Processing of Broader Terms

This paper uses natural language processing (NLP, [36,37]) to convert broader terms assigned to courses into quantitative data, i.e., into frequencies of words. It will be assumed that these words—these tokens as they are called in the NPL literature [36]—extracted from broader terms, still carry conceptual meaning and can still be used to characterize courses and the program-degree. For this reason, in this paper, token and key (program or course) concept, K, are used as synonyms.

NLP applies a sequence of processing functions to an original set of broader terms. Tokenization, the first of these functions, identifies words in broader terms that are included in a corpus; in a dictionary of tokens. Recalling Section 3.1’s example of obtaining broader terms for Mathematics (C1:MATH)—“Function of a real variable” and “differential calculus” were the resulting broader terms—and, considering the corpus of English words, after tokenization the following set of tokens $\left\{\mathrm{function},\mathrm{of},\mathrm{a},\mathrm{real},\mathrm{variable},\mathrm{differential},\mathrm{calculus}\right\}$ characterizes the Mathematics course.

But the above set includes tokens (i.e., “of” and “a”) that add no value to the course characterization; therefore, these tokens, known as stop-words, as well as any punctuation signs and numerals, should be removed. Moreover, words written with capital letters and different conjugations of the same word should be replaced by an adequate “stem-word” (in a process known as stemming, [36]).

Denoting the stemming and the purging of meaningless tokens as normalization, if a study program has N courses, after tokenization and normalization of course $\mathrm{C}i$ broader terms ($i\in \{1,2,\dots ,N\}$), a multiset ${\mathcal{K}}_i$ (allowing multiple instances of the same token) of $m_i$ tokens ${\mathcal{K}}_{i,k}$ is obtained (with $k=1,2,\dots ,m_i$). For the program-degree as a whole, a set $\mathbf{K}$ (no repetitions) with a total of $M=\left|{\bigcup }_{i=1}^N{\mathcal{K}}_i\right|$ tokens is obtained.

With $\mathrm{K}j$, the $j\mathrm{th}$ token in set $\mathbf{K}$, and the frequency of this token in course $\mathrm{C}i$ is found from

$$b_{i,j}=\sum _{k=1}^{m_i}{\delta }_{i,k}\left(\mathrm{K}j\right),$$

(1)

with

$${\delta }_{i,k}\left({\mathrm{K}}_j\right)=\left\{\begin{array}{cc}0,& {\mathcal{K}}_{i,k}\ne \mathrm{K}j\\ 1,& {\mathcal{K}}_{i,k}=\mathrm{K}j\end{array}\right..$$

(2)

Equation (1) represents the elements of an $N\times M$ matrix $\mathbf{B}$ of token frequencies per course.

Considering the broader terms for the five courses in Table 1 (column three), after tokenization and normalization the resulting course-token matrix is (Reference [19] provides the R programming code [38] used in this section):

(3)

where, given the large number of identified tokens (70), only the columns for the six most frequent are shown.

Observe how this matrix attaches quantitative information to courses based on token frequency. Observe, for instance, the link that emerges between courses C1:MATH and C2:PHY via token K2:calculus. Matrix ${\mathbf{B}}_{5\mathrm{C}}$ shows that this token is found twice among the tokens associated with course C1:MATH, and once among those associated with course C2:PHY (see also the underlined words in Table 1).

This ability to describe a study program quantitatively is an important breakthrough and a way to bridge the gap created by tacit understanding and unclear LO-statements. But, at the same time, notice how unpractical the analysis of the data in the matrix format is.

To achieve a clearer understanding of the quantitative data emerging from NLP, an alternative to matrix or tabular representations of data is essential.

3.3. Step 3: Visualization of Quantitative Data

An important result from NLP is token frequencies, the column-wise sum of the elements in the course-token matrix. A convenient visual representation of these frequencies is obtained with word clouds. Figure 2 presents a word cloud from data in matrix ${\mathbf{B}}_{5\mathrm{C}}$ (Equation (3)).

Figure 2 identifies the most frequent key program concepts—manag[ement], calculus, control, energy, linear, logistic—represented with a larger font size in a central position.

Figure 2 is adequate for identifying the relative importance of different key concepts, but provides no information concerning the relations between these or between courses. To represent these relations, researchers can choose among several alternatives; one that captures all data in the course-token matrix and makes patterns and descriptive statistics visible is presented in Figure 3a). It is the circular ideogram representation [17] of the data in matrix ${\mathbf{B}}_{5\mathrm{C}6\mathrm{K}}={\mathbf{B}}_{5\mathrm{C}}$[1:5;1:6], a submatrix including the first six columns of matrix ${\mathbf{B}}_{5\mathrm{C}}$ (Equation (3)).

The outer circumference in Figure 3a displays the five courses $\mathrm{C}i$ on the right side and the six tokens $\mathrm{K}j$ on the left side. This circumference specifies the number of links (see scale) between courses and tokens. For example, courses C3 and C4 have the largest number of links (eight and seven, respectively) to tokens. Token K1 has the largest number of links (six) to courses.

The advantage of the circular ideogram comes, especially, from the inner circle in Figure 3a, and from the stripes that link courses and tokens. The inner circle in Figure 3a emphasizes the (previously mentioned) link between courses C1:MATH and C2:PHY via token K2:calculus (see purple stripe). But much more is revealed: for example, while course C2:PHY has no further associations, course C1:MATH is also related to course C3:LOGOP through K5:linear (green stripe). The width of the stripes—the strength of the links—connecting C1:MATH to K2:calculus and K5:linear is also larger than the width for the stripes connecting these tokens to courses C2:PHY and C3:LOGOP. Given that “calculus” and “linear” are mathematics-related tokens, these results were expected, and the expert analysis of the five courses’ LO-statements (in Table 1) should result in identical conclusions. But in Figure 3, the combination of stripes’ curvature, color, and width renders the analysis universal, self-explanatory, and empowering, uncovering latent information and helping the verbal articulation of expert (non-verbal) knowledge.

Another equally useful visual representation of data in matrix ${\mathbf{B}}_{5\mathrm{C}6\mathrm{K}}$ is presented in Figure 3b. Consider the $\mathrm{C}i$ and $\mathrm{K}j$ in the outer circumference of Figure 3a as vertices $V=\{v_{\mathrm{C}1},v_{\mathrm{C}2},\dots ,v_{\mathrm{K}1},v_{\mathrm{K}2},\dots \}$, and the inner circle stripes as edges $E=\{e_{\mathrm{C}1\text{-}\mathrm{K}2},e_{\mathrm{C}1\text{-}\mathrm{K}5},\dots \}$ of an undirected multigraph $G=\langle V,E\rangle$. Figure 3b represents this multigraph with course and token vertices laid out in a way that communicates vertex centrality, i.e., where the number (the cardinality) of vertex links determine the vertex position [42]. Notice how vertices C3 and C4—with the largest number of links—shape a central cluster, while vertices C1 and C2 protrude to the periphery. Moreover, vertex centrality is emphasized through the course vertices’ diameter; with larger diameters representing courses with a larger number of incident links.

In matrix format, the multigraph in Figure 3b for the five courses and six most frequent tokens is

(4)

a square biadjacency matrix obtained from ${\mathbf{B}}_{5\mathrm{C}6\mathrm{K}}$ (superscript T denotes matrix transpose).

Figure 3a,b provide important insights into how key concepts and courses interrelate. However, a simpler and yet very useful representation would consist of the direct links between courses and between tokens.

Observing Figure 3b and matrix ${\mathbf{A}}_{5\mathrm{C}6\mathrm{K}}$, we conclude that elements of the biadjacency matrix represent the cardinality of 1-walks between consecutive vertices—with a k-walk defined as the sequence of k edges $\left(e_1,e_2,\dots ,e_k\right)$ joining $k+1$ vertices $\left(v_1,v_2,\dots ,v_{k+1}\right)$ [18]. For example, matrix ${\mathbf{A}}_{5\mathrm{C}6\mathrm{K}}$ shows that, between vertices $v_{\mathrm{C}1}$ and $v_{\mathrm{K}2}$, there are two 1-walks, $v_{\mathrm{C}1}\stackrel{2·e_{\mathrm{C}1\text{-}\mathrm{K}2}}{\to }{v}_{\mathrm{K}2}$. Between vertices $v_{\mathrm{K}2}$ and $v_{\mathrm{C}2}$, there is one 1-walk, $v_{\mathrm{K}2}\stackrel{1·e_{\mathrm{K}2\text{-}\mathrm{C}2}}{\to }{v}_{\mathrm{C}2}$. This is confirmed in Figure 3b.

A direct link between vertices $v_{\mathrm{C}1}$ and $v_{\mathrm{C}2}$ could be conceived as two 2-walks joining these vertices, represented as $v_{\mathrm{C}1}\stackrel{2·e_{\mathrm{C}1\text{-}\mathrm{K}2\text{-}\mathrm{C}2}}{\to }{v}_{\mathrm{C}2}$, with $2·e_{\mathrm{C}1\text{-}\mathrm{K}2\text{-}\mathrm{C}2}$ denoting the two available options to go from C1 to C2.

For the five courses example, using matrix algebra, the number of 2-walks between course vertices and between token vertices is found from the 2^nd power of the biadjacency matrix, with the diagonal elements of the resulting matrix made equal to zero [18]. With ${\mathbf{L}}_{5\mathrm{C}6\mathrm{K}}$ denoting the 2-walk matrix, it follows that ${\mathbf{L}}_{5\mathrm{C}6\mathrm{K}}={\mathbf{A}}_{5\mathrm{C}6\mathrm{K}}^2-diag\left({\mathbf{A}}_{5\mathrm{C}6\mathrm{K}}^2\right)$, and replacing ${\mathbf{A}}_{5\mathrm{C}6\mathrm{K}}$ gives

(5)

Submatrices ${\mathbf{L}}_{5\mathrm{C}}={\mathbf{L}}_{5\mathrm{C}6\mathrm{K}}$[1:5;1:5] and ${\mathbf{L}}_{6\mathrm{K}}={\mathbf{L}}_{5\mathrm{C}6\mathrm{K}}$[6:11;6:11] in Equation (5) represent the number of possible 2-walks between consecutive courses and consecutive tokens, respectively.

To confirm the results discussed previously for the direct link between vertices $v_{\mathrm{C}1}$ and $v_{\mathrm{C}2}$, notice the value 2 found in matrix element ${\mathbf{L}}_{5\mathrm{C}6\mathrm{K}}\left[1;2\right]$ (or ${\mathbf{L}}_{5\mathrm{C}6\mathrm{K}}\left[2;1\right]$, because the graph is undirected).

Using submatrices ${\mathbf{L}}_{5\mathrm{C}}$ and ${\mathbf{L}}_{6\mathrm{K}}$, representations of the direct links between courses and between key concepts are presented in Figure 4a,b, respectively. The strength of the links—the cardinality of possible 2-walks—is given both by numbers and by edge widths. Moreover, as for Figure 3b, vertices layout and vertex diameter provide a suggestive visual depiction of core and peripheral courses/key concepts.

Figure 4a shows that the largest number of possible 2-walks (12) occurs between courses C3:LOGOP and C4:ENER. This value can be verified in Figure 3b. The way key concepts influence links between courses is clearly reflected in courses C2:PHY and C5:FIN locations. Although C2:PHY and C5:FIN both have a single key concept among the six most frequent—K2:calculus and K1:manag, respectively (see Equation (3))—the fact that “manage[ment]” is more common than the mathematics-related concept pulls C5:FIN closer to where core program courses lie, whereas C2:PHY is pushed to a peripheral location.

Figure 4b confirms the peripheral role played by mathematics-related concept, K2:calculus, and it is interesting to contrast this graph’s discriminating potential with that of the word cloud in Figure 2. Indeed, no evidence is found in the word cloud as to differences between tokens K2 to K6 (because the number of edges incident on vertices $v_{\mathrm{K}2}$ to $v_{\mathrm{K}6}$ is the same for these tokens—three).

Figure 3b and Figure 4a provide visual evidence of course C2:PHY detachment from the remaining courses. Obviously, reasons for this should be discussed; in particular, the absence of an (expected) link between C2:PHY and C4:ENER.

Results from this section show that visual outputs from the broader terms CM method provide evidence-based details on weaknesses (and strengths) in program plans; namely, related to key program concepts and to the interrelations between these and/or courses.

3.4. Step 4: Discussion of the Visual Outputs

With the adoption of the broader terms CM method, the focus of program planning discussions is shifted from the discussion of written statements of course LOs—seldom clear and unequivocal—from atomized discourses about the links between courses, to the interpretation of quantitative data communicated visually in a way understandable to all.

Because of the universal, self-explanatory quality of its visual outputs—of the mappings—the broader terms CM method empowers all stakeholders, allowing participatory involvement of non-faculty groups in program planning discussions. Because of its quantitative nature, the broader terms CM method nurtures constructive critique, effectively addressing disciplinary and scientific boundaries, hierarchical and functional differences, and atomized discourses. With the objective identification of program plan weaknesses, it is possible to unfreeze [43] long-established beliefs, preparing the agreement for change with contributions from all stakeholders (see the review feedback loop in Figure 1).

Some weaknesses identified in the mappings may derive from course LO classification. Section 3.1 stated that an initial draft classification was made by a small multidisciplinary team of university faculty. During the discussion step, with the help of mappings, classification problems are easily identified, justifying the reclassification feedback loop in Figure 1.

As mentioned in Section 3.1, this reclassification carries some subjectivity. Different broader terms could be chosen to classify an LO-statement, and there could be LO-statements for which an adequate broader term is not included in the control vocabulary. However, the technical nature of control vocabularies and of the classification task makes the selection of broader terms distinctly less subjective than the head-on discussion of LO-statements.

Using the mappings for the five courses example, we elaborate on relevant discussion topics that would benefit from the participatory involvement of all stakeholders.

Considering frequencies and links between key program concepts, in Figure 3a and Figure 4b, stakeholders (namely, industry and society groups) could contribute with their experience to identify important key concepts, essential links, considering not only scientific and pedagogic arguments, but also the mission of HEI in the context of rapidly changing technological, economical, societal and political environments. With respect to the links between courses, and the links between courses and key concepts, in Figure 3b and Figure 4a, student and graduate groups could contribute with their experience to contrast the differences between the declared and the enacted curriculum [16,44].

Concerning the lack of an expected link between C2:PHY and C4:ENER, mentioned at the end of the last subsection—recall Figure 4a—given that Applied Physics and Energy Management syllabuses are typically linked by thermodynamics, heat transfer, and fluid flow topics, to express the importance of this link, stakeholders could use handwritten notes to communicate a desirable change, as depicted in Figure 5.

Figure 5 demonstrates the ease with which stakeholders take possession of the mappings. The dashed lines show the preferred location of C2:PHY (and C1:MATH), closer to core courses. Text in square brackets points to broader terms justifying the link between C2:PHY and C4:ENER. Figure 5 could be the starting point for the revision of these courses’ LOs, perhaps considering another forum and using detailed concept mappings.

To conclude this section, notice that having used a simple example with only five courses, it is not possible to verify whether the method identifies the key program concepts and whether the links between these and/or courses are accurate. To assess the accuracy of the broader terms CM method, the next section presents a case study.

4. The Case Study: Bachelor Degree in T&IM

To evaluate the accuracy of the broader terms CM method, results obtained with this method should be confirmed by actual observations. For this purpose, this section uses a bachelor degree—Technology and Industrial Management (T&IM)—assessed by the Portuguese accreditation agency (A3ES) in 2013. The section starts with the generic presentation of the T&IM study program and with the presentation of the recommendations issued by A3ES (the results of the assessment). Afterwards, the broader terms CM method is used to generate mappings from courses’ LO-statements. To evaluate the accuracy of the method, the mappings are compared to the recommendations, which are deemed accurate. The section ends with a discussion of this comparison.

4.1. T&IM Bachelor Degree and the Portuguese Accreditation Agency Recommendations

The bachelor degree (180 ECTS credits) in Technology and Industrial Management (T&IM, [45,46,47]) was conceived in 2006 at the College of Engineering of Instituto Politécnico de Setúbal, a Portuguese public HEI. The degree targeted mature students working in the industry sector in the region of Setúbal. Considering the characteristics of the students—mature blue color workers with formal and informal skills in their area of professional expertise—and the advanced technological settings provided by the employing organizations (which include automotive, aeronautic and ship repair industries), the 2007–2012 program plan emphasized managerial contents at the expense of engineering and mathematics. This emphasis on management topics is made clear in Figure 6, a circular dendrogram representing T&IM courses and respective departments. Out of the 38 program courses, 18 belonged to the Business Sciences Department.

Another important characteristic represented in Figure 6 is the dispersal of T&IM core and elective courses among six departments.

Six years after it began, the Portuguese accreditation agency assessed the T&IM bachelor degree [48]. Study program data reporting for the 2007–2012 period were gathered, a self-assessment report was delivered by the HEI, and an independent panel of experts (representing A3ES) visited and met with HEI stakeholders.

Regarding the program plan, the A3ES produced the following recommendations:

• Increase program-degree mathematical content;
• Steer programming skills towards high-level languages with practical use;
• Strengthen the program plan with important applied industrial management content, namely, in operations management, supply chain management, and operational research;
• Excessive number of courses, some with little additional content;
• Poor integration of topics taught in the different courses.

Concerning these recommendations, note that: (1) these are considered an accurate expression of weaknesses in the 2007–2012 T&IM program plan; (2) the non-prescriptive (and somewhat vague) style of the recommendations results from accreditation criteria allowing program-degrees to adjust to different HEI missions, to student demographics, and to available resources.

4.2. T&IM Mappings

Using LO-statements from the T&IM courses (2007–2012 program) and the methodology described in Figure 1 (excluding the feedback loop), after courses’ LO classification and broader terms NLP, a total of 256 program tokens (no repetitions) were obtained. Figure 7a presents a word cloud with the 200 most frequent key program concepts. Using the program biadjacency matrix ${\mathbf{A}}_{\mathrm{T}\&\mathrm{IM}}$, graphs with direct links between the most frequent key program concepts and with direct links between courses were obtained—Figure 7b and Figure 8, respectively.

Note that, out of all 38 courses in Figure 6, three (ETH, NET, and CAD) were not taught and were excluded; Internships I&II were also excluded, justifying the analysis of only 33 courses (for the meaning of the course acronyms, please refer to Figure 6).

With a larger font size in Figure 7a and at the center of Figure 7a,b, lay tokens “manag[ement]” and “busi[ness]” were the most frequent key concepts found in the program broader terms. Besides “manag[ement]” and “busi[ness]”, other key concepts lay in the vicinity of the graphs central region, namely, “econom[y]”, “resourc[es]”, “account” (related to management), and “engi[neering]” and “design” (related to engineering). Because only 28 (out of 256) of the most frequent tokens are represented in Figure 7b, all tokens exhibit a fair number of links. The way key concepts are linked in Figure 7b defines two distinct groups—or clusters—of key concepts: the management cluster, found towards the the top of the figure, and the engineering cluster at the bottom. Abstract key concepts, such as “process”, “perform[ance]”, and “analysi[s]” (“system” or “indic[es]”), are also found (mostly) in the interface between the management and engineering clusters.

Figure 8 presents the links between program courses in two planes. The background (gray) plane is used to show three ultra-peripheral courses with no links: MECHT, MULT, and INNOV. The forward (white) plane provides detail on the courses laying closer to the graph core region. In this detail, all courses are linked. Distant from the graph center lie courses PHY and STAT; at an intermediate distance lie MATH, MAIN, CTRLP, PROG, DRAW, ENVECON, ECON, GLOB, and ENG; the remaining 19 courses lie in the central region. A divide similar to the one identified previously between managerial and engineering concepts is also present in Figure 8, with managerial courses clustered to the (upper) left and engineering courses clustered towards the (lower) right of Figure 8 detail.

4.3. Comparing T&IM Mappings with A3ES Recommendations

Comparing Figure 7 and Figure 8 with A3ES recommendations (in Section 4.1), it is possible to evaluate, for each recommendation, whether the meaning conveyed in writing has a visual equivalent. The comparison of written and visual meaning is used to verify the accuracy of the broader terms CM method.

Consider item (i) of the A3ES recommendations—increase program-degree mathematical content. The visual equivalent of this recommendation is the (relative) absence of mathematics-related tokens in the mappings. Indeed, Figure 7a includes very few mathematics-related tokens (e.g., mathemat, algebra, theorem), with the small font size of these tokens confirming the detachment of mathematics from core concepts taught in the T&IM degree. The position of the “mathemat” token in Figure 7b, distant from central key program concepts, is also consistent with this analysis. Figure 8 provides further evidence that some action should be taken concerning mathematics contents. Courses MATH and STAT’s relative positions and the small number of links to other program courses translates into insufficient integration of mathematics content.

With regard to item (ii) of the A3ES recommendations—steer programming skills towards high level languages with practical use—the visual equivalent should be the absence of links between programming and applied key concepts. An analysis similar to the previous one shows few programming-related tokens in Figure 7a, with none among the 28 most frequent in Figure 7b. As regards the PROG course, its location in Figure 8 confirms it is among those with fewer links to central and applied courses.

As for item (iii)—strengthen the program plan with important applied industrial management content, namely, in operations management, supply chain management, and operational research—from Figure 7b, tokens “oper[ational]”, “logist[ics]” are found among the 28 most frequent. Figure 7a includes additional concepts related to the mentioned courses, such as “suppli”, “chain”, “optim”. Comparing font sizes in Figure 7a, these latter key concepts are less frequent than generic managerial key concepts “resourc”, “financ”, “account”, which could be subjectively deemed less important in a Technology & Industrial Management program plan. A detailed quantitative analysis of token frequencies and of token connections could be made, contributing with relevant insights to the constructive discussion of this recommendation.

Using a similar line of inquiry, item (iv) in A3ES recommendations—excessive number of courses, some with little additional content—would benefit from the detailed analysis of token frequencies per course and from the equivalent to Figure 3b with data from the T&IM study program. This detailed analysis and the graph are obtained with ease from matrix ${\mathbf{A}}_{\mathrm{T}\&\mathrm{IM}}$, using the methods and tool considered in supplementary material [19]. However, from Figure 8, it is possible to sort courses based on their connectivity (close to core or peripheral location). This figure depicts ultra-peripheral courses (MECHT, MULT, INNOV) in the background plane with no links. These courses are obvious candidates for detailed scrutiny; a scrutiny that should be extended to courses closer to the graph central region but, nevertheless, showing a small number of links (e.g., GLOB, ENVECON, ECON).

Finally, concerning item (v)—poor integration of topics—as stated previously in Section 4.2, Figure 7b and Figure 8 denounce the clustering of managerial and of engineering concepts. In addition, courses more detached from the graph central region and with fewer links in Figure 8 (already identified in the previous A3ES recommendation, item iv) are once more obvious candidates for detailed scrutiny.

In light of the above, and considering A3ES recommendations,

Table 2. Comparing A3ES recommendations with evidence-based visual meaning conveyed from T&IM mappings.

A3ES Recommendation		Evidence from Mappings ${}^{\mathbf{a}}$
i.	Increase program-degree mathematical content	Small number of mathematics-related key concepts and poor integration of mathematics-related courses
ii.	Steer programming skills towards high-level languages with practical use	Very small number of programming-related key concepts and detached location of the programming course
iii.	Strengthen the program plan with important applied industrial management content, namely, in operations management, supply chain management and operational research	Comparison of frequencies of applied industrial management key concepts with frequencies of generic managerial key concepts sheds light on the relative weight of each group in the program plan
iv.	Excessive number of courses, some with little additional content	Identifies and sorts courses with few (and with no) links to core program courses.
v.	Poor integration of topics taught in the different courses	Divide between engineering and management, visible both in key program concept and in course mappings

${}^{\mathrm{a}}$ Note these results are obtained exclusively from courses’ LO-statements, whereas A3ES recommendations consider a visit by an independent panel of experts, interviews and focus group sessions, among other inputs.

(second column) summarizes the evidence-based visual meaning obtained from T&IM mappings.

4.4. Discussion

From Table 2, for recommendations (i), (ii), and (v), mappings provide detailed visual evidence supporting these recommendations. For recommendations (iii) and (iv), the style (the vagueness) of A3ES statements prevent an objective comparison of visual and written meanings. Yet, these latter recommendations are useful for highlighting the striking difference between an evidence-based analysis—possible with the mappings—and the subjective interpretation—relying on tacit understanding—of A3ES written statements.

Because the mappings provide evidence supporting the majority of the A3ES recommendations, it is concluded that the broader terms CM method provides an accurate depiction of T&IM program plan weaknesses. Because all T&IM mappings rely on key program concepts, it is also concluded that these key concepts—and the broader terms CM method—are useful in program planning.

Three additional notes are worth mentioning. Firstly, despite the large number of program courses (33), classification, NLP, and visualization steps were concluded quickly and with ease, posing no particular difficulty. Secondly, Figure 8 shows that a holistic experience of the T&IM program plan, considering interrelations between the 33 courses, is possible. Lastly, unlike the course mapping of Meij and Merx [7], the detailed concept mappings of Seering et al. [15], Willcox and Huang [13] or Varagnolo et al. [16], visual outputs from the broader terms CM method do not aim at the tracing of the available learning pathways or at the tracing of detailed precedence relations between program concepts. Indeed, Wang [49], based on views derived from Gilles Deleuze and Félix Guatarri, discusses the distinction between mapping and tracing in the context of curriculum mapping. This researcher supports that the current practice of curriculum mapping in higher education is, actually, tracing. Current curriculum mappings represent fixed routes with a linear tree-like structure and an objective model of the curriculum, and this is an example of tracing. Maps have different topological characteristics. Like rhizomes, maps do not aim at guiding to a main road or familiar destination, but to represent the mesh of nodes and the patterns that emerge through the multitude of connections between nodes. The broader terms CM method provides, therefore, maps identifying clusters of key concepts, or courses; maps with multiple undirected links between courses and/or concepts. These maps’ aim is to provide a representation of the program plan that is understandable to all stakeholders, allowing, through successive iterations, the participatory involvement of non-faculty groups without imposing predefined models or fixed routes. In this sense, broader terms curriculum mapping is not a replacement for other curriculum mapping methods; it is meant as a complement to (preceding) other program planning tools.

5. Conclusions

Addressing the curriculum development process is of paramount importance. This process has profound consequences, being responsible for the preparation of future professionals and for laying the foundations for dynamic knowledge transfer systems affecting local and global realities. At the heart of curriculum development lies program planning. Program planning is of immense strategic value. The effort put into program planning propagates through all levels and subprocesses of teaching and learning, imprinting the values, intentions, and expectations that will guide stakeholders; shaping HEI educational outcomes.

To improve program planning, more participatory touchpoints to non-faculty groups (i.e., students, industry, society) are needed. Creating these touchpoints and contributing to representative program planning was the motivation behind this paper.

An important impediment to representative program planning lies in the communication gap between faculty and non-faculty groups. Curriculum mapping has been used to promote better communication between faculty and shape program planning. This paper collected practices available from different types of curriculum mapping and, using information and data science techniques, tailored a curriculum mapping method for non-faculty groups’ participation in program planning discussions. The resulting method—the broader terms CM (curriculum mapping) method—was illustrated with the help of a simple example—the five courses example. The following conclusions were found:

• (Section 3.1) Classification replaces the head-on discussion of subjective course LO-statements with the much more objective task of selecting broader terms from a control vocabulary.
• (Section 3.2) Natural language processing allows the quantitative analysis of the program plan, providing a way to cut across disciplinary and scientific boundaries, hierarchical and functional differences, and atomized discourses.
• (Section 3.3) Mappings render quantitative results’ interpretation universal and self-explanatory, empowering stakeholders with evidence-based details on weaknesses (and strengths) in the program plan.
• (Section 3.4) The discussion of visual outputs with non-faculty groups allows representative program planning, with these groups’ voices being heard on reclassification and review of course LO-statements.
• (Figure 1) The iterative nature of the method ensures program planning using quantitative elements and stronger links between courses’ LO-statements, allowing a holistic approach to curriculum development.

Despite the relevance of the above conclusions—related to the participatory involvement of non-faculty stakeholders—the simple five courses example was unable to answer the question of the broader terms CM method’s accuracy and, therefore, of the method’s utility.

To evaluate the method’s accuracy, a case study—the T&IM bachelor degree—was used. Mappings for the case study were obtained and compared with observations from an independent panel of experts. From this comparison, the following was concluded (Section 4.4):

• Mappings provide evidence supporting the observations, and the broader terms CM method provides an accurate depiction of T&IM program plan weaknesses.
• Key concepts obtained from course LO-statements—and the broader terms CM method—are useful in program planning.

Considering the benefit of non-faculty groups’ participation in curriculum development processes, and considering the progress made in information systems and relational databases [50,51,52], the merger of techniques used in the broader terms CM method and HEI information systems would help bring the method’s benefits into HEIs’ everyday reality; for example, with the inclusion of mappings in information systems’ summary dashboards. This merger is just one potential topic for further explorations in this rich and challenging research area that joins education, information, and data sciences.