Consultant - Real-Time Evaluation of Ukraine CASH Consortium - Remote with travel

Background:

Mercy Corps is a leading global organization powered by the belief that a better world is possible. In disaster, in hardship, in more than 40 countries around the world, we partner to put bold solutions into action — helping people triumph over adversity and build stronger communities from within.

This evaluation Terms of Reference (TOR) is a plan for preparing for, conducting, and facilitating learning from an evaluation for the UCC emergency response program, particularly focused on the first phase of the emergency scale up. It conveys clear directions to the evaluation team, defines roles and responsibilities, and includes details needed to recruit for, implement, and learn from the RTE evaluation.

The Ukraine Cash Consortium (UCC) comprises Mercy Corps, Norwegian Refugee Council (NRC), and People in Need (PIN). The Consortium’s programmatic work currently consists of the implementation of a $100 million, 17-month USAID/Bureau for Humanitarian Assistance (BHA) award of the same name. The program’s goal is to meet emergency basic needs while enhancing the protection environment for approximately 320,000 internally displaced people (IDPs), returnees and other conflict-affected people (CAPs) in Ukraine.

Purpose / Project Description:

The primary purpose of this evaluation is to learn from the initial emergency response phase and to identify lessons, successes, and points for improvement for the continued implementation of this program. The RTE will look at the implementation done by the UCC partners between May 2020 and March 2023.

The evaluation team members will examine the scale-up and coordination of the emergency response, explore to what extent the consortium structure and established processes hindered or improved the program’s adaptability, timeliness, and overall response, and evaluate UCC’s overall emergency response approach. The RTE will help UCC and partners improve its future programming through lessons learned and best practices generated through this program. Response personnel are the primary audience for this evaluation and will use the learning from it to modify the continued response implementation as appropriate.

Indicators will be agreed upon with UCC and the Evaluator and will be defined in the Analytical Framework, as agreed upon by both parties.

Evaluation Scope and Questions

The evaluation should be organized around evaluation criteria as follows, with suggested evaluation questions provided, the list will be refined with the consultant.

Relevance and Effectiveness:

1. How effective and timely was the UCC BHA emergency response?
2. How timely was the UCC scale-up to the new emergency conditions compared to other cash actors in Ukraine?
3. To what extent did the different implementation approaches (working through partners; doing direct registrations; or online self-registration) meet the needs of vulnerable communities in a safe and equitable manner?
4. Was the local partnership model of some of the consortium members successful in delivering its intended outcomes? What were the challenges and successes of this approach?
5. To what extent has the flexible modality approach helped UCC partners meet the needs of those most in need during the emergency response?
6. What were the comparable advantages of the three different modalities used by partners: working through partners, online self-registration, and doing direct registrations with potential participants?
7. To what extent did consortium members consistently implement activities in line with the program design, international norms, and standards?
8. What were the main implementation challenges program and operational teams faced while implementing the emergency response?
9. Is the BHA program design still fit for purpose or do we need to rethink and redesign it?
10. What are the main takeaways from the UCC emergency response operation? What went well and what could have been done differently? How can we improve the UCC program in the future and the UCC consortia approach?

Efficiency/Consortia model:

1. What was the added value of Consortia partnership during the initial emergency response operation?
2. What were the challenges of having to set-up/working in a consortium at the same time as scaling-up the emergency response?
3. What are the key takeaways from working through Consortia in the Ukraine emergency response phase?
4. Were parts of the response strengthened by the fact that partners were working in consortium rather than individually (e.g. economies of scale)? Were parts of the response hindered by having to work in Consortia? How can this be improved in the future?
5. Was the Consortium Model able to achieve any greater impact compared to a single agency emergency response, if so, how?
6. To what extent did the Consortium Model enhance the timeliness and quality of response?
7. What level of cohesiveness and complementarity in delivering/monitoring assistance has UCC achieved across the different consortium partners?

Coordination:

1. How effective was the UCC management unit at establishing internal communication and coordination structures in meeting programmatic needs?

2. What internal coordination problems has the consortium faced and how have they been addressed?

3. What have been the biggest successes in coordination? What were the biggest gaps?

4. How can we address these gaps in the future?

5. How effective/well-coordinated were the UCC activities and de-duplication with central and local authorities in the Area of Operation (AoO), clusters (e.g. cash working group), and other cash actors?

Learning:

1. To which extent were consortium members able to share best practices, and how did this contribute to the success of the project?
2. Did the Consortium Model offer the opportunity for learning across partners, if so, in what manner and form did this take?

Evaluation Project Tasks

1. Refine the evaluation objectives and primary research questions in consultation with UCC management team.
2. Devise and test a methodology and evaluation tools to address the specific objectives and individual research questions of the evaluation and obtain sign off from UCC management team on these.
3. Conduct secondary data collection and desk research, including using UCC’s existing project monitoring data, to identify gaps in data coverage and knowledge.
4. Collect primary data and through various stakeholders in Poland and Ukraine (or other locations) to address the specific research questions of the evaluation.
5. Provide a draft report to UCC management that will be incorporated into ongoing Program planning and evaluation, as well as recommendations for maximizing social impact.
6. Facilitate an online workshop to validate the findings of the RTE with UCC and partners staff and other stakeholders.
7. Incorporate UCC feedback into a draft report and prepare a final report. The final report should both describe the results of the evaluation, and provide actionable recommendations for improving UCC’s Program
8. Provide an English version of the RTE report, and English and Ukrainian versions for the Summary/External report.

Methodology

UCC recommends a qualitative approach to gather the RTE data through semi-structured interviews, Desk review, and Key Informant Interviews (KII). The participants include relevant stakeholders, and data is analyzed thematically to identify patterns and trends. The findings are used to make recommendations for program improvement and to inform decision-making. The evaluation process is iterative and involves continuous communication and feedback with stakeholders to ensure the validity and reliability of the data collected.

The following methodology outlines the qualitative research methods/tools that could be used.

1. Key informant interviews: the team is expected to interview a variety of staff for program implementation and support (M&E, finance, operations), cash coordination group members, donor representatives, representatives of the affected population, and civil society/local actors. The team will annex the interview guide or guides to the inception report and to the draft and final reports. Targeted roles for interview will include:

• Mercy Corps and UCC Partner Staff: Head of Programme, Thematic Advisors, Program Managers, MEL, program staff, operations, and finance, Country Directors/Steering Committee.
• DART Representatives (Planning Coordinator, Team Lead, Deputy Team Lead).
• Key members of the cash coordination group - Ukraine Cash Working Group Lead and Co-Lead, Other BHA consortia, CCD, etc.
• Civil Society/local actors - local governments, other INGOs, local civil society.
  • Central government representatives.

2. Desk review: Review program documents, including program reports, monitoring and evaluation reports, and other relevant documents to gain a better understanding of the program's implementation, management, and monitoring processes. The document review could also provide insights into the program's design, implementation challenges, and lessons learned. Existing Program Information Sources include:

• Proposal
• Logframe and M&E plan
• Situation Reports, market research conducted at various periods of the crisis
• Contingency Plans/Workplans
• Post Distribution Monitoring Data/Reports
• Aggregated (non-sensitive) CARM Data
• Learning Documents/lessons learnt workshops
• Risk and Issues Register
• Minutes of meetings
• Donor communication

3. Data analysis: Analyze the qualitative data collected from KIIs, document review, case studies, and participatory approaches using thematic analysis. This analysis could identify key themes related to the program's implementation, targeting process, impact, and sustainability.

Triangulation: Triangulate the qualitative data with other sources of data, including quantitative data and program monitoring data, to provide a more comprehensive understanding of the program's performance.

Overall, the above qualitative research methods can provide valuable insights into the experiences and perceptions of program participants and stakeholders, and help to identify strengths, weaknesses, and areas for improvement for the Ukraine Cash Consortium program.

INCEPTION PHASE

Before the start of data collection, the evaluation team will do the following:

1. Hold a short planning meeting with all members of the evaluation team including the UCC MEL Director, RTE Steering Committee (RTE SC) and relevant Program teams who will provide a brief about the Program and agree on the documents to be included in the desk review.
2. Review key internal and external documents. reports will be made available to the evaluator(s) along with other relevant project documentation.
3. In partnership with the UCC MEL Director, refine and finalise the specific evaluation questions to be explored from the scope described above.
4. Propose to the UCC MEL Director and RTE SC the appropriate methodology to be developed for the Ukrainian context to evaluate the BHA Program and address the OECD evaluation criteria.
5. Prepare an outline of the data collection methods that are required and the relevant survey templates and participatory data collection guides to be used for data collection.
6. Develop a work plan consisting of key milestones required for data collection.
7. The inception report and the data collection tools will be reviewed by UCC, and feedback provided to the evaluator.
8. Finalize the inception report and data collection tools.

PRIMARY DATA COLLECTION

The qualitative approach, feeding into the findings around the objectives outlined above, will aim to assess the effectiveness of the project management, systems, and processes established by the projects, including strategies to improve gender equity both at the participant and project management level, protection issues, and its implementation, performance monitoring, and conflict sensitivity. The qualitative evaluation will also capture lessons learned and best practices. The KII will be collected in person, if the security situation allows.

DATA ANALYSIS

Qualitative data should also be analysed and should primarily focus on developing a deeper understanding about the relevance of the Program, and providing recommendations for improving or strengthening the effectiveness, efficiency, and sustainability of the results of the Program. The analysis will be conducted using specialized software such as MAXQDA, NVivo, or ATLAS, among others. These software applications provide tools to facilitate data organization, coding, categorization, and retrieval, as well as visualization and reporting of findings.

PRESENTATION AND DOCUMENTATION OF FINDINGS AND RECOMMENDATIONS

This consultancy will take place between (May - June 2023).

The findings of the evaluation must be shared with UCC in the following formats:

• Validation of findings workshop with UCC and partners. (in-person, if security situation allows)
• Final presentation workshop  with UCC staff and partners to present findings and get feedback (ideally, in-person):
  • Agreed lessons learned and best practices that can be incorporated into relevant sectors’ programming
  • Agreed recommendations that will inform and improve UCC’s future programmatic strategy, with agreed action points and deadlines
• Draft Evaluation Report submitted to UCC MEL Director and RTE SC for feedback and comments after conclusion of field visit. UCC will submit feedback and the consultant will have to incorporate UCC feedback into the RTE Report.
• Final RTE Report- The report must be clear and concise and the following sections must be included as a minimum: Executive Summary, Literature Review of all BHA responses in Ukraine, Methodology, Analysis of Findings, Limitations and Challenges, Recommendations, Annexes: ToRs, a timeline of the response, a list of individuals interviewed, statistical outputs, templates of data collection tools used, a description of the methods employed, a summary of survey results (if appropriate) and any other relevant materials.
• UCC requests that an external version of the evaluation be prepared for external use to share key findings with other actors in the Ukrainian response, including BHA.
• A Ukrainian version of the RTE Summary/External report is to be submitted to UCC after the finalization of the English RTE report.

ETHICAL CONSIDERATIONS AND DATA PROTECTION

The evaluation team will make clear to all participating stakeholders that they are under no obligation to participate in the evaluation study. All participants will be assured that there will be no negative consequences if they choose not to participate. The evaluation team will obtain informed consent from the participants. The research team will ensure prior permission is received for taking and use of visual still/ moving images for specific purposes, i.e., research report and presentations. The evaluation team will assure the participants’ anonymity and confidentiality and will ensure the visual data is protected and used for agreed purposes only. In particular, the evaluation team will employ robust data security measures to further ensure participants’ confidentiality and anonymity. All beneficiary and community members identifying information must remain confidential. It is the responsibility of the contracted Consultant to anonymise all datasets or raw data and ensure that identifying personal information is not shared with UCC Program staff nor any external party.

The Consultant will ensure that the preservation of respondent anonymity and confidentiality is prioritized during data collection, management, storage and reporting. Respondent data will not be shared with third parties without prior consent from Mercy Corps. The Consultant is expected to have clear data protection protocols and policies that should be shared with UCC during the procurement process and inception phase.

The evaluation team must also ensure compliance with the General Data Protection Regulation (GDPR). The GDPR sets standards for the collection, storage, and use of personal data in the European Union (EU) and should be followed to ensure the protection of personal data.

The evaluation team is responsible for determining whether or not their proposed methodology would require Institutional Review Board (IRB) clearance and will be responsible for clearing the process and training if such approval is required.

ASSUMPTIONS AND REQUIREMENTS

• Evaluators will have access to all documentation and can take part in relevant meetings.
• UCC will communicate with Local Authorities regarding the Evaluation scope and dates to facilitate access for the Evaluation team.
• Evaluators will have access to key partner staff who will provide them with adequate information.
• Evaluators will take confidentiality and objectivity into consideration during the process.
• Security concerns could impact the timing and the scope of the evaluation. It is important for the team to remain flexible. They must be open to making changes to the schedule and itinerary such as visiting alternate sites, conducting remote reviews and interviews, etc.
• COVID-19 situation doesn’t prevent or restrict data collection.
• All data (primary or secondary) gathered by the evaluators will be fully transferred to Mercy Corps ownership and eliminated from evaluators’ possession.

COVID-19 MITIGATION MEASURES

The spread of the novel COVID-19 virus stands to impact every facet of humanitarian and development program design and implementation. With new emergency aid and modifications of ongoing aid and development programing, adjustments and adaptations to monitoring and evaluation approaches are imperative. External Evaluators must do their part to limit human-to-human transmission of COVID-19 and ensure protection of staff and others by taking preventive measures. The evaluator is to ensure the following:

• The evaluation methodology is adapted to include evaluation approaches that correspond to the current COVID-19 pandemic.
• Detail the measures to be taken to minimize the risk of COVID-19 transmission to consultancy staff, beneficiaries and other stakeholders.
• Demonstrate the effectiveness and efficiency of proposed adapted methodology in achieving evaluation objectives.

Proposed Governance Structure

RTE steering committee will be established to oversee the overall implementation of the evaluation and conducted in an ethical and transparent manner, and the findings and recommendations are implemented effectively.

The RTE steering committee should consist of key stakeholders such as partners (program or M&E) leads. The committee should coordinate regularly to review progress, provide guidance, and make decisions on any issues that arise.

The RTE Steering Committee should have the following roles and responsibilities:

1. Provide overall guidance and support for the evaluation process
2. Review and approve the evaluation plan, including the methodology, data collection tools, and analysis plan
3. Review and approve the findings and recommendations
4. Formulate a management response to the RTE findings and recommendations
5. Ensure that the management response is implemented effectively

The consultant/s should report to the UCC MEL Director for all the contract-related matters and aspect of the consultancy. And will report to the RTE Steering Committee on the following:

1. Conducting the evaluation according to the approved evaluation plan
2. Ensuring that the evaluation is conducted within the agreed timeframe
3. Providing regular progress reports to the RTE Steering Committee
4. Presenting the evaluation findings and recommendations to the RTE Steering Committee and other relevant stakeholders
5. Responding to feedback and comments from the RTE Steering Committee and other relevant stakeholders
6. Collaborating with the RTE Steering Committee in formulating the management response to the RTE findings and recommendations.

Timeframe / Schedule:

May 5, 2023 – Contract signing

May 12, 2023 – SUBMISSION: Draft Inception Report and Instruments

May 19, 2023 – Client Feedback on Draft

May 23, 2023 – SUBMISSION: Final Inception Report and Instruments

May 29, 2023 – Data Collection (multiple consultants can deploy for more efficient collection)

June 5, 2023 – Data Analysis and Report Writing

June 9, 2023 – Presentation of the Key Findings

June 12, 2023 – SUBMISSION: Draft Evaluation Report

June 19, 2023 – Client Feedback on Draft

June 22, 2023 – SUBMISSION: RTE Report (English Version)

June 30, 2023 – SUBMISSION: RTE Report Summary/External (Ukrainian/English Versions)

REQUIRED QUALIFICATIONS

For the purposes of this evaluation, UCC welcomes international and national evaluators to apply.

The profile of the lead consultant is:

1. 7-10 years of technically relevant work experience.
2. Individuals in academia, social research, or humanitarian evaluation with a background in humanitarian aid, research methods, development economics, development studies, or other related fields. Preference will be given to consultants with Cash/Food security evaluation experience and Real Time Evaluation experience.
3. Prior experience with conducting Real Time Evaluations or After-Action Reviews.
4. Extensive experience of conducting evaluations along DAC OECD evaluation criteria, ideally leading an evaluation team and experience of designing evaluation methodology/tools, data analysis etc. in an emergency setting.
5. Experience of working in humanitarian contexts and good understanding of humanitarian response work and conflict sensitivity – both in Programs and operations.
6. In-depth knowledge of qualitative research methods.
7. Competent in using statistical packages for qualitative analyses.
8. Use of national consultants is essential for work within Ukraine due to security constraints and local and contextual knowledge.
9. A gender balance of field enumerators is essential to conduct data collection with both male and female beneficiaries.
10. Excellent presentation and writing skills.
11. Capacity to work collaboratively with multiple stakeholders.
12. Excellent analytical and writing skills in English and ability to translate the final external report into Ukrainian.

**Diversity, Equity & Inclusion
**Achieving our mission begins with how we build our team and work together. Through our commitment to enriching our organization with people of different origins, beliefs, backgrounds, and ways of thinking, we are better able to leverage the collective power of our teams and solve the world’s most complex challenges. We strive for a culture of trust and respect, where everyone contributes their perspectives and authentic selves, reaches their potential as individuals and teams, and collaborates to do the best work of their lives.

We recognize that diversity and inclusion is a journey, and we are committed to learning, listening and evolving to become more diverse, equitable and inclusive than we are today.

**Equal Employment Opportunity
**We are committed to providing an environment of respect and psychological safety where equal employment opportunities are available to all. We do not engage in or tolerate discrimination on the basis of race, color, gender identity, gender expression, religion, age, sexual orientation, national or ethnic origin, disability (including HIV/AIDS status), marital status, military veteran status or any other protected group in the locations where we work.

**Safeguarding & Ethics
**Mercy Corps team members are expected to support all efforts toward accountability, specifically to our stakeholders and to international standards guiding international relief and development work, while actively engaging communities as equal partners in the design, monitoring and evaluation of our field projects. Team members are expected to conduct themselves in a professional manner and respect local laws, customs and MC's policies, procedures, and values at all times and in all in-country venues.