End-of-project evaluation – Norad cooperation agreement 2017-20

Summary

• Purpose: The Norwegian Red Cross (NorCross) seeks to conduct an end-of-project evaluation of the impact, efficiency, and sustainability of projects funded through the 2017-2020 cooperation agreement with Norad. In addition, NorCross is interested to learn around the following issues: approaches to risk management, program delivery and partnership as well as NorCross added value in achieving results
• Audience: The main audience of the evaluation is NorCross and relevant National Society partners as well as Norad. The results of the evaluation will constitute an important component in NorCross’ 2017-2020 final report to Norad
• Reports to: NorCross Coordinator for Evaluation and Learning, Øivind Hetland
• Timeframe: The deadline for submitting proposal is 16 November 6 am with estimated start-up 23 November. Tentative dates for key deliverables are:
  • Inception report: 9 December
  • Data collection: mostly in November and December, all by end January
  • First draft report: 28 February
  • Final draft report: 31 March
• Methodology summary The technical proposal should include the outline of a suitable methodology an appropriate data collection methods adapted to the purpose of the evaluation.
• Location: Geographically, the evaluation focuses on Burundi, Guatemala, Pakistan and Palestine. It is expected that the proposal takes into consideration current travel restrictions related to Covid-19 to ensure that data collection in countries of study is possible

Background
Within the framework of the 2017-2020 cooperation agreement with Norad, the Norwegian Red Cross (NorCross) seeks to conduct an evaluation of projects funded through Norad agreements.

The current cooperation agreement between Norad and NorCross for the project named “GLO-0604 QZA-16/0386 NorCross Cooperation agreement 2017-2020” is valid for the period January 2017 to December 2020. The overall goal, as formulated in the agreement, is community resilience, while the overall outcomes are divided into two: 1) Improved health for vulnerable people and 2) Strengthened ability of the National Red Cross/Red Crescent Societies to deliver countrywide services to vulnerable people. Based on the outcomes, the main planned outputs of the project are: 1) Organisational development of National Red Cross/Red Crescent Societies and 2) Increased access of target groups to health systems, and water, sanitation and hygiene (WASH) and increased knowledge of target groups on health prevention. The original budget of the grant agreement amounted to NOK 308 million (77 million per year).

Nine countries are included in the agreement – Burundi, Colombia, Guatemala, Honduras, Lebanon, Pakistan, Palestine, Somalia, and South Sudan – as are two thematic programs with a global scope focusing on health and organisational development (1)

The main modality of implementation at country level is through partnerships with the respective Red Cross or Red Crescent National Societies. A key principle in NorCross approach to partnerships isalignment with the auxiliary role of the National Society, the partner’s strategic priorities and contribution to RCRC Movement coordination. The thematic programs with a global scope have funded the International Federation of Red Cross and Red Crescent Societies (IFRC) Secretariat to provide technical standards within health and organisational development the national societies.

Evaluation purpose, criteria, and scope
In accordance with the requirements in the grant agreement between Norad and NorCross (see annexe 1), the main purpose of the evaluation, is to

• Evaluate the impact, efficiency, and sustainability of the projects in view of both the countryspecific goals and outcomes and the overall goal and outcomes of the grant agreement

In addition, NorCross is interested to learn around the following issues:

• Review NorCross and partners’ approach to and implementation of risk management, based on requirements in the grant agreement
• Identify learnings that NorCross and partners can use to improve program delivery, the partnership approach and to inform thematic and methodology development
• Review NorCross added value in achieving results at impact and outcome level and provide recommendation on possible improvement

The thematic and geographical scope is as follows:

• Burundi: The community health interventions implemented by the Burundi Red Cross with NorCross support from 2010-2020, funded by the Norad frameworks between 2013 and 2020. The focus of the evaluation is on 2017-2020
• Guatemala: The community health interventions implemented by the Guatemalan Red Cross with NorCross support between 2010 and 2020, mainly funded through Norad. The focus of the evaluation is on 2017-2020
• Pakistan: The community health (and risk reduction) interventions implemented by the Pakistan Red Crescent with NorCross support in the period 2011 to 2018, mainly funded through Norad framework agreements. The focus of the evaluation is on 2017-2020
• Palestine: The community health interventions implemented by the Palestine Red Crescent with NorCross support in the period 2010 to 2019, mainly funded through Norad framework agreements. The focus of the evaluation is on 2017-2020
  The country level evaluations should be informed by the strategic evaluation on community health interventions conducted by NorCross in 2018. The main focus of the evaluation is the period 2017-2020.

The main audience of the evaluation is the Norwegian Red Cross and the relevant National Society partners as well as Norad. The results of the evaluation will constitute an important component in NorCross’ 2017-2020 final report to Norad. The evaluation report will be published on Norad’s evaluation database (2).

Evaluation criteria and questions
A tentative list of evaluation questions and sub-questions should be suggested in the technical
proposal and a final list agreed with NorCross during the inception phase. It is expected that revised international evaluation criteria and their principles for use adopted in December 2019 (3) inform the design of the proposal, including the evaluation questions. The below focus areas and questions are indicative of the types of questions to be addressed within the framework of this evaluation.

• Impact (the extent to which the intervention has generated or is expected to generate significant positive or negative, intended or unintended, higher-level effects): Evaluate in particular results at goal and outcome level with focus on impact for target groups; considerations around attribution and contribution is important as is comparing baseline and endline data
• Efficiency (the extent to which the intervention delivers, or is likely to deliver, results
  in an economic and timely way): In considering the conversion of inputs (funds, expertise, time, etc.) into outputs, outcomes and impacts as compared to feasible alternatives in the context, particular emphasis should be put on support to core, indirect and direct project costs to NS partners
• Sustainability (the extent to which the net benefits of the intervention continue, or are likely to continue Important components to consider when evaluating sustainability (non-exhaustive list):
  • How can the sustainability of the intervention and its effects be assessed? Can the achieved results of the projects be considered sustainable in a medium- to longterm? To what extent were the projects built on and contributed to develop existing local capacities?
  • What were the major factors which influenced the achievement or nonachievement of sustainability of the project

In addition, NorCross invites the evaluation team to suggest how to cover the criteria of relevance, effectiveness and coherence in the technical proposal.

When it comes to evaluation of risk management under evaluation question 2, main focus shall indicatively be on the implementation of NorCross risk management policy, alignment between NorCross and partner’s approach to risk management as well as on how NorCross and partners address issues related to the identified areas of focus. In terms of learnings, under evaluation question 3, it is important to compare across regions and countries.

Evaluation methodology and data collection methods

The technical proposal (see below) should include the outline of a suitable methodology and appropriate data collection methods adapted to the purpose of the evaluation. It is free to suggest any additional methodological approaches and data collection methods and questions that are not mentioned in the ToR.

The evaluator must adopt a consultative and participative methodological approach. The approach should strive to include the following elements:

• Field visits and data collection from women, men and children living in target communities
• Data collection from NorCross and NS staff at national and local level as well as with volunteers
• Data collection from relevant stakeholders such as authorities and other national and local actors
• Baseline and end line study comparison
• Comparative analysis across the four country-level evaluations

It is expected that the proposal takes into consideration current travel restrictions related to Covid-19 to ensure that data collection in countries of study is possible.

Relevant written documents will be provided by NorCross as well as the relevant National Societies. Key sources of written information include:

• Project documents from partners (assessments, logical frameworks, budgets, indicator tracking tables, baseline and endline studies, annual reports)
• NorCross result framework and country results framework
• Annual internal NorCross result reports
• Annual report from NorCross to Norad
• Annual project audits and audits of NorCross
• Relevant background and strategy papers (from NorCross and partners)
• Grant agreement between Norad and NorCross and correspondence related to grant management
• Internal reviews and external evaluations, including Norad’s organisational review of NorCross (from 2019)

Deliverables and timeline

The proposal should include, but not be limited to, a plan for the following deliverables:

1. An inception report with a detailed description of methodology to be used, a data collection plan, overview of information sources, a timeline for deliverables and solutions to identified challenges
2. Feedback workshop with each Red Cross Red Crescent National Society involved in the evaluation based on preliminary findings and recommendations
3. Draft consolidated report in English including background, findings and conclusions, lessons learned and recommendations
4. Feedback workshop in Oslo or online to present draft consolidated report before revision and final approval of the report
5. The final report shall include one report for each of the 4 countries being evaluated and one overall report summarising findings from country evaluations at grant agreement level (including comparative analysis). The main body of the text for each of the sub reports should be maximum 15 pages (excl. executive summary and annexes) while the overall report should be maximum 25 pages (excl. executive summary and annexes). The reports should as a minimum include the following:
   1. Executive summary
   2. Background
   3. Evaluation methodology and limitations
   4. Findings and conclusions
   5. Lessons learned
   6. Recommendations
   7. Appropriate appendixes

The deadline for submitting proposal is 15 November with estimated start-up 23 November. Tentative dates for key deliverables are:

• Inception report: 9 December
• Data collection: mostly in November and December, all by end January
• First draft report: 28 February
• Final draft report: 31 March

Evaluation quality and ethical standards

The evaluators should take all reasonable steps to ensure that the evaluation is designed and conducted to respect and protect the rights and welfare of people and the communities of which they are members, and to ensure that the evaluation is technically accurate, reliable, and legitimate, conducted in a transparent and impartial manner, and contributes to organisational learning and accountability. Therefore, the evaluation team should adhere to the evaluation standards and specific, applicable process outlined in the IFRC Framework for Evaluation. The IFRC Evaluation Standards are:

• Utility: Evaluations must be useful and used.
• Feasibility: Evaluations must be realistic, diplomatic, and managed in a sensible, cost effective manner.
• Ethics & Legality: Evaluations must be conducted in an ethical and legal manner, with particular regard for the welfare of those involved in and affected by the evaluation.
• Impartiality & Independence; Evaluations should be impartial, providing a comprehensive and unbiased assessment that takes into account the views of all stakeholders.
• Transparency: Evaluation activities should reflect an attitude of openness and transparency.
• Accuracy: Evaluations should be technical accurate, providing sufficient information about the data collection, analysis, and interpretation methods so that its worth or merit can be determined.
• Participation: Stakeholders should be consulted and meaningfully involved in the evaluation process when feasible and appropriate.
• Collaboration: Collaboration between key operating partners in the evaluation process improves the legitimacy and utility of the evaluation.

It is also expected that the evaluation will respect the seven Fundamental Principles of the Red Cross and Red Crescent: 1) humanity, 2) impartiality, 3) neutrality, 4) independence, 5) voluntary service, 6) unity, and 7) universality. Further information can be obtained about these principles at: www.ifrc.org/what/values/principles/index.asp

Evaluator(s) qualifications

The competency requirements for the evaluator(s) are:

• University degree(s) at post-graduate (master) level in relevant fields of study (health in humanitarian settings) and relevant work experience in health/humanitarian/development programme
• Familiarity with trends and developments in humanitarian work in areas of health preferred
• Minimum of 7 years of experience in evaluation and social science, or similar, methodology required
• Demonstrated experience in planning and implementing final project evaluations required, inclusive of competency and track record in managing quantitative and qualitative data collection methods and analysis
• Experience from the relevant geographical areas required
• Knowledge and experience of working with the Red Cross Red Crescent Movement preferred
• Excellent written and spoken English, French and Spanish skills required
• Excellent writing and communication skills required in the team
• Strong interpersonal and organisational skills required

Annex 1 Norad reporting requirements
The purpose of the evaluation is based on the following detailed reporting requirements of the grant agreement between Norad and NorCross:

• Annual progress reports (annual reports for year 1-3) shall, as a minimum, include “…an account of results so far by the project, using the format, indicators and targets of the approved result framework. The overview must: 1) Show delivered outputs compared to planned outputs; 2) Show the project’s progress towards achieving the outcome; 3) If possible, describe the likelihood of impact being achieved.” In addition to this, there are reporting requirements linked to risk management, both how these have been managed as well as how identified risks related to climate and environment, gender and equality, corruption and other financial mismanagement as well as human rights
• The final report shall, as a minimum, include
  • All points listed in the previous bullet point (requirements for progress reports)
  • An assessment of the project’s effect on society (impact)
  • A description of the main lessons learned from the project (learning)
  • An assessment of the sustainability of the results achieved by the project (sustainability)
  • Furthermore, it has to show delivered outputs compared to planned outputs
• In response to the Joint Annual Report 2018, Norad furthermore emphasised the need to clarify baseline values in order to highlight the impact of the project activities over time

Notes
(1) Their respective goals are: 1) Health: “RCRC develops and promotes the evidence base for RCRC health programmes”; 2) Organisational development: “There is a global, coordinated and evidence based National Society Development approach”
(2) https://norad.no/en/toolspublications/publications/evaluationreports/, accessed 21 October 2020
(3) https://www.oecd.org/dac/evaluation/revised-evaluation-criteria-dec-2019.pdf, accessed 21 October 2020

How to apply:

Application procedure

Proposals will be ranked according to how they fulfil the three following requirements:

• Team qualifications (50%)
• Methodological approach (25%)
• Price (25%)

Interested candidates should submit their application material by 16 November (6 AM) to:

• Øivind Hetland oivind.hetland@redcross.no (Coordinator Evaluation and Learning) with copy to
• Lars-Andrè Skari Lars.Skari@redcross.no (Head of Strategy and Results Unit)

If you have questions related to the ToR, please contact Øivind Hetland. All questions should be submitted before 2 November 10 AM.

The budget for the evaluation is in the range NOK 3-500,000.

Application materials should include:

1. Technical proposal not exceeding five pages that a) demonstrates an understanding and interpretation of the ToR, b) outlines a methodology to meet the requirements in the ToR, c) proposes a timeline for deliverables, including time for NorCross and partners to comment on deliverables
2. Cover letter clearly summarizing your experience as it pertains to this assignment, your daily rate, and three professional references
3. A brief description of your firm or institution (for applicants other than individual
   consultants)
4. Curricula Vitae (CV) for all members of the team applying for consideration
5. Financial proposal itemizing estimated costs for services rendered (daily consultancy fees), accommodation and living costs, transport costs, stationery costs, and any other related supplies or services required for the evaluation
6. At least one example of an evaluation report most similar to that described in this TOR

Application material are non-returnable, and we thank you in advance for understanding that only short-listed candidates will be contacted for the next step in the application process.

According to Norwegian tax law, NorCross can only enter consultancy contracts with companies registered as legal entities with their respective tax authorities. For individual consultants, temporary employment contracts are signed.