Towards a Successful Post COVID-19 Transition of Monitoring, Evaluation, and Learning in Complex Sustainability Science Research-to-Policy Projects

Abstract

There is an emerging body of literature focusing on the COVID-19 livelihoods and environmental impacts, as well as the effects of the pandemics on evidence generation. However, little attention has been paid to how COVID-19 has—and is likely to continue to—affect monitoring, evaluation, and learning (MEL) systems, specifically in the context of large sustainability science research-to-policy project consortia. Here, we provide a conceptual framework of MEL responsiveness to COVID-19 effects and discuss the specific pathways to successful MEL transition. Using the UKRI GCRF TRADE Hub as a case study, we provide some examples of possible adjustments within the new context.

1. Introduction

The novel coronavirus disease (COVID-19) is a pathogenic viral infection, which was first reported in Wuhan, China in December 2019 [1,2]. As of 31 December 2020, over 82 million people worldwide have been infected with COVID-19 and almost 1.8 million have died as a result of the disease [3]. The unprecedented crisis has compounding consequences on critical response mechanisms, further exacerbating the impacts of current socio-economic and environmental phenomena, such as climate change, conflicts, and other crises in different parts of the world and requires rapid collaborative remediation [4]. If not mitigated, COVID-19 may also have potentially disastrous implications for the success and funding of projects at the research and policy interface, as well as the project management and monitoring, evaluation, and learning (MEL) mechanisms that underpin such projects.

Without being anticipated in original risk frameworks and MEL strategies, different funders allowed flexibility in projects as an adaptive measure. However, without providing additional resources for running costs, some projects were not fully successful in adjusting to respond to the new context. For example, many of the National Aeronautics and Space Administration (NASA) sponsored earth science projects, such as the Dynamics and Chemistry of the Summer Stratosphere (DCSS) and the S-MODE project (which investigates poorly understood ocean “eddies” that play a key role in transporting heat between the surface and the deep ocean) have all experienced substantial impediments as a result of the COVID-19 [5]. These projects have suspended their spring field work and anticipate delays in data delivery (ibid). Elsewhere in the United Kingdom, a survey carried out by a leading dementia research charity has shown that due to COVID-19, 95% of researchers have had research projects or clinical trials delayed, and one in five have had research projects cancelled completely [6].

In this continuously evolving context, it is imperative to thoroughly shift the traditional project management and MEL approaches to accommodate innovative techniques and ways of working that ensure that anticipated project results remain unaffected. In this commentary we provide a brief discussion regarding the COVID-19 induced changes “from old to new” MEL strategies and offer a set of potential solution pathways to mitigate against likely negative COVID-19 impacts on the delivery of large research-to-policy projects.

2. MEL Responsiveness to COVID-19 Effects

The conceptual model for MEL responsiveness to COVID-19 effects (Figure 1) is based on a triple phased approach with the business as usual (BAU) era coming first, followed by a “context responsive” phase and a “leveraging new MEL frontiers” phase. The first phase is characterized by regular implementation of a combination of traditional and improved MEL practices with tidbits of an impending pandemic. During this phase, project MEL work continued consistently with only slight modifications to ensure constant field presence and assessment of the appropriateness of interventions in a rapidly evolving context [7]. Consequently, projects continued to utilize real time MEL evidence to regularly test results frameworks and facilitate adaptive programming and timely course correction [8].

Prior to this phase, an ever-growing amount of data required by donors and generated from MEL practices resulted in a slow-paced digitization process in which organizations considered the costs and benefits of using increasingly available IT-based MEL tools and platforms. As the situation rapidly changed with COVID-19 spreading beyond borders, there followed a need for a corresponding adjustment in response to the immediate impacts, as well as beneficiary accountability mechanisms and spot checks all of which were dependent on physical field presence [9]. While the development and potential application of a COVID-19 vaccine might affect this process, it is unlikely that MEL practices would see a return to BAU anytime soon.

3. Tinkering with the New Normal

Phase two “context responsive MEL” was induced by the presence of a novel global pandemic, which gradually gravitated towards the “new normal” [10]. As the context quickly evolved and the real impact was felt across the world, the MEL community jostled to develop new and improved project management approaches and rules of engagement in a world where travel and physical contact restrictions were introduced, and virtual working arrangements became the norm [11]. With this shift came increased and inevitable investments in information and communication technology (ICT) platforms to enable limited but meaningful contact with field level activities, partners and beneficiaries. Such investments left heavy tolls on budgets since the culture of working entirely in virtual teams had not previously been fully embraced. These developments reshaped MEL systems and practices from paper-based and semi digitized to fully technology driven, albeit sometimes limited to second best solutions due to a cost of a full migration, which include subscription costs and license purchases [12]. ICT and digitization played a critical role in ensuring continuity of MEL activities to some extent. The lack of physical connection to the reported field progress, virtual backstopping and trainings on new software, inconsistencies in meeting attendance, and reporting by partners and other stakeholders all affected the quality of engagement and MEL products [13].

Consequently, communities and households were exposed to a varying scale depending on their endowments, and the most affected were workers entrenched in poverty, with small and overcrowded housing, and limited flexibility rendering working from home extremely difficult [14]. The impact of the pandemic invariably extended to gender-based roles and has exacerbated the plight of female experts whose paid economic opportunities were slowing down with a potential to reduce remunerated working hours at the same time increasing the unpaid care economy [10].

4. Leveraging New MEL Frontiers

Finally, we anticipate that leveraging the post pandemic experience to recalibrate and reposition projects back to a functional state and developing or activating a crisis modification plan will be instrumental. With funders allowing an increased flexibility in modification of projects and use of resources to adjust to the COVID-19, rendering reallocations of budget lines and the need to effectively manage new funding streams are inevitable. In essence, MEL functions are becoming central and integral in adaptive management [15] and will have to adjust in order to keep in close contact with implementation partners and beneficiaries, through leveraging online consultations, establishing hotlines, migration to online management information system (MIS) platforms and reinforcing beneficiary accountability mechanisms for them to be relevant and accommodative of the new normal [9]. There is an urgent need to continuously revisit results frameworks, testing theories of change as well as monitoring the external environment (e.g., climate change impacts), which is likely to compound negative impacts in the post COVID-19 era and may undermine the likelihood of projects’ impact and outcomes to be evident without compromising the original focus [7]. New approaches, skills and knowledge will be key in reinforcing adaptive and absorptive capacities on MEL practitioners.

Leveraging on systems put in place in the second phase, gathering new evidence from the current implementation will support internal adaptive management towards enhancing impacts of current initiatives and capacity to respond to new lines of funding for more comprehensive multi-dimensional donor response packages as a result of COVID-19. Projects should have internal mechanisms and protocols for data security and safety, and minimizing exposure risks and guaranteeing consent since the ICT platforms may result in collection and management of sensitive beneficiary data. To enhance sustainability, use of knowledge and research skills, it is prudent to capacitate communities through participatory MEL approaches which not only galvanize sense of ownership but also allow them to be independent in tracking the projects’ impact using local knowledge and relevant local-level indicators [8].

5. Pathways to Successful MEL Transition

Therefore, in response to the COVID-19 related changing MEL environments, several broad solutions pathways will be necessary to successfully adjust to the new context. These include greater reliance on conceptual and theoretical tools and frameworks, use of innovative online platforms and tools, and finally, an opportunity to re-focus towards most vulnerable populations. With regard to the first point, greater reliance on concepts and theories will be useful to understand the changing causal pathways and drivers of change within different layers of the MEL results frameworks. For example, the systems thinking, a term originally coined in 1987 by Barry Richmond [16] can be helpful to disentangle the evolving complex interrelationships and causal loops between different parts of research-to-policy projects. A social network analysis approach can be used to map out the changing roles of actors who are both involved in the project delivery and affect the outcomes of the project. In addition, generation of and regular revision of the evidence-based structured impact pathways, in addition to the theory of change, will allow integrating the past and anticipated disturbances caused by COVID-19.

Let us consider an example within the UKRI GCRF Trade, Development, and the Environment Hub, a 5 year £20 million project contributing research and policy solutions to make trade sustainable for people and nature, commenced work in 2019. Within the UKRI GCRF TRADE Hub, a recent revision of all planned strategic impact pathways has been undertaken to incorporate the current and anticipated effects of COVID-19 (see Figure 2 as an example of such pathways).

While the shift from more traditional processes (e.g., site visits, paper-based questionnaires) to the use of online software within MEL practices has been apparent for a number of years [17], the COVID-19 impact on research-to-policy development projects and the global shift to home-based working have amplified the uptake of MEL-specific software by project managers and MEL managers. Data collection software, such as form-based SurveyMonkey and Mobenzi, has allowed projects to collect information from international partners and stakeholders in the absence of field visits. Such a shift may also allow for an increased number and diversity of stakeholders reached for MEL purposes, particularly where mobile applications are used to collect data (e.g., KoBo Toolbox and Open Data Kit). The economic and carbon savings of these changes are also likely to be significant as well as increased geographic inclusivity [18]. Data management software, including TolaData and Airtable [19], allows for the collation and organization of data with increased accuracy and transparency, also enabling real-time, simplified, and often automated report generation for donors. As part of digital solutions COVID-19 dashboards and educational websites were created to help track the pandemic, generate awareness, and help behavioral change [20]. For the large projects, such as the UKRI GCRF TRADE Hub, moving to a tabular system meant greater visibility of partner inputs and increased control of permissions. Further features such as linking records between tables have allowed for communications and impact monitoring of project activities and the production of “project health” dashboards using applications including graphs with interactive displays of data.

However, this shift to digital tools also comes with challenges. Emerging literature suggests that the pandemic has contributed to increasing the divide between those who are able to telework and those who are not and has presented negative challenges such as techno stress [21]. Consequently, the inability to fully access the Internet and use some of the innovative technology tools effectively is viewed as creating disparity and segregation and other negative coping strategies such as compensatory overworking [21]. To mitigate this, organization of access to ICT at the community or village level might be useful to ensure that the group as a whole is not disadvantaged. Moreover, targeted trainings must be provided to ensure participation and appropriate capacity of data collectors, MEL leads, and end users of online tools [22]. Reaching increased audiences may also raise concerns regarding informed consent and the protection of personal data [22] and the loss of engagement quality associated with in-person meetings [23]. Addressing these challenges in the era of COVID-19 and ensuring technical, cultural and linguistical responsiveness will likely leave project MEL systems more resilient and adaptive to future disruptions to project systems. It is also likely to reduce the issue of potential data forging as the digitalization of MEL systems is likely to make systems and processes more transparent, and thus increase efficiency and accountability.

Finally, adjusting MEL approaches and operations provides an opportunity to re-focus towards the most vulnerable populations within the programmatic (and longer-term policy) objectives. As emerging evidence suggests, the impact of the COVID-19 was greatest amongst the communities already at risk due to different health, socio-economic, and environmental factors [2]. This has been particularly the case in developing countries, where poor or non-existing social welfare systems are unable to mitigate the negative livelihood effects of the pandemic. Within Tanzania’s sugar value chain context (Figure 2), the restricted movements during the pandemic affected the ability of many households to participate in income-generating activities. For instance, the results of a pilot survey conducted by the Sokoine University of Agriculture in September 2020 (SUA, 2020) [24] on randomly sampled sugarcane farmers from Morogoro Region revealed that 86.5% of the households (n = 160) could not offer labor to other farmers. This in turn impacts the earnings and the purchasing power of the households and may have negative consequences of individuals’ food security and health status. Results of the same survey [24] also showed that 76.4% of the interviewed farmers reported that the problem was exacerbated by the increase in prices of many of the industrial goods needed at the households. Initial results suggest that the prices of the inputs used in sugarcane and other crops production (e.g., agrochemicals and fertilizers) significantly increased. This is in line with the official inflation trends, and specifically the increase of the non-food inflation from 2.2% in March 2020 when COVID-19 was initially reported in the country, to 3.2% in June. The consumer price index in Tanzania also changed from 118.92 to 120.74 in the same period [25].

In this context, re-adjusting the logframe contents, in particular at output and outcome levels can contribute to alleviating otherwise disastrous COVID-19 impacts. Within the UKRI GCRF TRADE Hub country-level workplans and logframes, activities, and indicators were reviewed to ensure an optimal focus on the livelihoods of the communities most affected by the pandemic. Similarly, additional environmental impacts (such as unmanaged or poorly managed wildland fires) were taken into account to adjust the project’s response to the newly emerged challenges.

6. Conclusions

This article provides an overview of the key MEL-related changes, challenges and opportunities resulting from the COVID-19 pandemic focusing on large research-to-policy projects. Using existing evidence and the UKRI GCRF TRADE Hub as a case study, we proposed specific programmatic and data management areas which will likely contribute to a successful MEL transition towards the post COVID-19 period.

To enable such transition, it is critical that large consortia projects invest sufficient resources in modern data collection tools and data management systems. Staff training should be a priority, especially in the area of ICT. In order to re-adjust risk registers, logframes, and impact pathways, participatory approaches should be used involving all stakeholders, and these can take place through well planned virtual MEL workshops and use of online exchange platforms. Regular cross-checking of MEL components, including indicators, should be undertaken in the context of newly emerging literature and feedback received from local communities and policy-makers.