M3.06. Preparing the evaluation of the mentoring process

4. Developing Evaluation Questions, Measurement Framework and Evaluation Plan

Evaluation is a key tool that will be used throughout the lifetime of a mentoring programme, therefore it is essential that the tools used for evaluation, allow the process to be effective.
Using a logic model results in effective design of the effort and offers greater learning opportunities, better documentation of outcomes and shared knowledge about what works and why. More important, the logic model helps to ensure that evaluative thinking is integrated into your evaluation design and implementation.

A logic model is a graphic representation of the theory of change, illustrating the relationship between resources, activities, outputs and short-, intermediate- and long-term outcomes.

A logic model helps break down the programme implementation in a systematic way by highlighting the connections between the needs, programme resources, activities, outputs, outcomes and the long-term impact of a mentoring programme. By doing this, evaluation questions will be addressed, such as:

  • How should the programme work?
  • What are the differences between the planned process for implementation, versus reality?
  • Where are there gaps or unrealistic expectations in the implementation strategy?
  • Which areas of the logic model seem to be yielding the strongest outcomes or relationships to one another?
  • Which areas of the model are not functioning in practice?
  • Are there key factors that have not been embedded in the programme strategy that should be?

By organising evaluation questions based on the logic model, you are better able to determine
which questions to target in an evaluation.

There is another consideration in developing evaluation questions. The questions also depend
on the type of evaluation you want to conduct - a performance evaluation, a process or formative evaluation or an outcome or summative evaluation.

A performance evaluation is concerned with:

  • Ensuring accountability
  • Demonstrating that resources are used as intended and are managed well
  • Monitoring and reporting on progress toward established goals
  • Providing early warning of problems to management

A process or formative evaluation is concerned with:

  • Understanding if a programme is being implemented as planned, and according to schedule
  • Assessing if the programme is producing the intended outputs
  • Identifying strengths and weaknesses of the programme
  • Informing mid-course adjustments

An outcome or summative evaluation is concerned with:

  • Investigating whether the programme achieved the desired outcomes and what made it effective or ineffective
  • Making mid-course adjustments to the effort
  • Assessing if the effort is sustainable and replicable

A logic model is a graphic representation of the theory of change, illustrating the relationship between resources, activities, outputs and short-, intermediate- and long-term outcomes.