Harnessing the power of evaluation to build better partnerships

1000x563_Blog_Harnessing the power of evaluation.png

A young person has a life changing experience through a student exchange. A researcher combines data with their peers around the world to advance their field. A collaborative educational programme creates a powerful space for students across jurisdictions to share learning.

These are just three examples of the benefits international partnerships between universities can provide. Yet, our understanding of how such partnerships perform can be quite limited. We often don’t know if a partnership has made a difference over and above what can be achieved by a university independently. Evaluation gives us the evidence to understand the value of partnerships — and to use that evidence to inform decision-making.

In response to this need, the ERASMUS+ EVALUATE project has delivered a framework for the evaluation of international university partnerships that can help more universities generate that evidence for their own benefit, and that of the wider field.

This framework is a product of the EVALUATE project, titled "Developing a Framework for Evaluation of International University Partnerships." The project was funded under ERASMUS+ Key Action 2 and ran from 2019 to 2022. The EVALUATE project consortium members are the University of Copenhagen in Denmark, University College Dublin in Ireland, the University of Edinburgh in the United Kingdom, the University of Helsinki in Finland, Leiden University in the Netherlands, and the University of Sydney in Australia. These institutions collaborated to develop this comprehensive guide to assist in evaluating international university partnerships effectively.

Three core question categories of the evaluation framework provide a set of tools that could be applied to diverse evaluation contexts:

The Evaluation and Its Context

The framework provided for evaluation is designed to assist you both in crafting your evaluation approach and in conducting follow-up actions once the evaluation is complete. The primary emphasis here is on the evaluation process itself. This part of the guidance includes critical questions that probe the timing and reasons behind the evaluation, such as "Why evaluate now?" These questions are essential as they help frame the purpose and urgency of the evaluation, ensuring that it is timely and effective.

The Central Evaluation Question

This section is specifically geared towards offering insights and guiding questions that can shape the course of an evaluation. Unlike the previous section that focuses more broadly on the evaluation process, here the concentration is on the content being evaluated. This includes an examination of the dynamics between the partnership under review and its broader strategic context. It also delves into specific phases of the partnership, questioning their effectiveness and alignment with the overall objectives. This approach helps to pinpoint areas of strength and potential improvement within the partnership framework.

The Partnership and Its Context

In this part, the intention is to stimulate reflection on the historical development and objectives of a partnership, as well as its interconnections with other policies. Although this section does not directly address the evaluation process, the information gathered here is vital for any substantive evaluation. It raises questions concerning the partners involved, their goals and ambitions, the activities undertaken by the partnership, and the expectations held by each party. This background is crucial for understanding the partnership’s operational dynamics and its potential impact, providing a rich context for evaluating its effectiveness and alignment with broader goals.

Scholars and professional services join forces

This evaluation framework was developed by a combined academic and professional services team using a co-creative approach to make sure that the product was embedded in both theory and practice. Scholars from Leiden and Edinburgh universities accompanied staff in international offices across the EVALUATE project through the evaluation of specific strategic partnerships to create case-studies of evaluation in action. The insight gained from the creation of the case studies was then used to develop the evaluation framework and an accompanying set of methodologies.

Evaluation gives us the evidence to understand the value of partnerships — and to use that evidence to inform decision-making.

Lessons for embedding and mainstreaming evaluation

Of course, evaluation does not happen in a vacuum. Alongside the evaluation framework the project identified and synthesised lessons learned from the process of developing the case studies:

Lesson #1 Know your stakeholders and their (naturally diverse) interests

Identify the key stakeholders and their key responsibilities/mandates in relation to the evaluation, and create a roadmap for the partnership together.
There can be lots of individual and unconnected conversations between key stakeholders when it comes to international partnership. It is important to have a coordinated approach.
How does each partner think about success in relation to strategic partnerships? The definition of success is not self-evident.
What are the broader objectives of the strategic partnership for each university?

Lesson #2: Integrate evaluation with existing data and systems

Look for opportunities to embed evaluative methods to existing processes and practices within the partnership – for example into the management of joint degree programmes; or the governance of strategic partnerships.
Consider where evaluation fits within the wider context of priority projects. There might be competing institutional priorities.
Think about the evaluation as a capacity building exercise more than just collecting and analysing information.
To ensure that an evaluation is meaningful, the ownership of it needs to sit in the appropriate place within each institution.
Universities need to consider what data will be available that will assist with the evaluation of the partnership.
Tangible KPIs or outputs can be established in all areas of collaborative activity, including education, student mobility and research.
In our experience it is often more challenging to identify and benchmark against qualitative KPIs. The risk of this "what gets measured gets managed" approach, however, is that less measurable yet important factors can be underplayed.

Lesson #3: Get to know evaluation methods

Workshop-based interactive and participative methods of stakeholder engagement in evaluation practice (e.g. in planning how to carry out the evaluation, and deciding on the methods together) can produce meaningful results in form of common understanding which enhances a feeling of ownership for the participants.
Many outcomes often identified as being important by a university are too broad or too indirect to be attributable to a particular partnership or indeed, suite of partnerships.
An often-overlooked criterion relates to the structural robustness of the partnership. For example, higher education institutions might make it their objective to build the resilience necessary to allow the partnership to survive the departure of key personnel on either side, or leadership shifts.
It is important to consider who will conduct the evaluation of a specific project and to adjust expectations with stakeholders and decisionmakers accordingly.
Expect change in your thinking about evaluation as you delve more into the topic.
Ensure that those involved have the time and resources they need.

Get started with the handbook for practitioners

As well as the evaluation framework, ERASMUS+ EVALUATE published an accompanying handbook with cases studies of evaluation action; lessons learned from the experience of conducting evaluations; and a comprehensive literature review. The handbook can be accessed by practitioners on an independent basis and presents the data about the framework in a ‘self-service’ way so individuals can work their way through the methodology to up-skill themselves. The handbook enables staff to put theoretical concepts of partnership evaluation into practice within their own institutions.

Our aspiration is for people to be able to use the handbook and evaluation framework to drive evidence informed monitoring and decision making in the running of international university partnerships so that these partnerships enhance the quality and efficiency of their higher education institutions.

Happy evaluating!

Acknowledgements

EVALUATE was funded under ERASMUS+ Key Action 2 from 2019 - 2022. The University of Edinburgh was the project coordinator.