Final Program Evaluation for Strengthening Resilience in Agriculture, Livelihood and Markets

Project/Consultancy Title

Final Program Evaluation for Strengthening Resilience in Agriculture, Livelihood and Markets (STREAM)

Project Location(s)

Mundri East and Mundri West Counties of Western Equatoria (Legacy Amadi) State and in Koch County of Unity State

Duration

45 working days (9 weeks) - from 21st of December 2022 to 31st of January 202****3

Please apply early: Applications considered on a rolling basis.

**Application deadline: 26 December 2022, 23.59
**
Please address your applications to “ss-pr@mercycorps.org while copying “lgideon@mercycorps.org “& kaguret@mercycorps.org

Background.

Western Equatoria (WES) and Unity State (US) are among the states in South Sudan mostly affected by recurrent conflicts. These states face a multitude of challenges resulting from the combined effect of prolonged violence and economic shocks which have resulted in displacements, loss of livelihoods, loss of property and assets, destruction of infrastructure and dysfunctional markets among others. These shocks have also been seen to increase inter-communal tensions as there is increased pressure exerted on available limited resources. In response to some of these shocks, households have been forced to resort to negative coping mechanisms which continue to undermine both their immediate and long-term economic opportunities.

1)    Program to be Evaluated

Strengthening Resilience in Agriculture, Livelihoods and Markets through Local Institutions in Mundri East & West and Koch (STREAM), SWISSS development cooperation (SDC) and Agence Française de Développement (AFD) Funded is a three phase seven years’ program, with the first phase starting as of 1st December 2020 to November 30th, 2022, upon completion this phase, phase two was to start as of 1st December 2022 to 30th November 2025 and Third phase starting as of 1st December 2025 to November 2027

2)    Purpose of the Final Evaluation

Mercy Corps is seeking a qualified consultant to conduct a final program evaluation of SDC & AFD funded Strengthening Resilience in Agriculture, Livelihoods and Markets through Local Institutions in Mundri East & West and Koch (STREAM), SDC and AFD Funded, 1st December 2020 to November 30th, 2022. Hence the Terms of Reference has been developed to attract interest from consultants to undertake a final program evaluation. This final evaluation is being undertaken with funding from SDC and AFD and will be conducted to determine post-program conditions from which a comparison can be made and for the purpose of informing future programs, monitoring and evaluating performance of the program, contributing to learning and research and measuring changes realized because of the program implementation. It will be essential to gain in-depth understanding of the challenges related to food security, livelihoods, and markets. Below are the purposes for program evaluation survey using (Market system development (MSD) Lenses:

The overall objective of final evaluation is to collect data for a set of indicators outlined in the program monitoring and evaluation plan that will assess program performance and impact against intended objectives and Mercy Corps to design future programs and influence/shape its program strategies.

Specific objectives.

The evaluation will specially serve to:

1. Assess the appropriateness of the intervention design and approaches in addressing the identified problems, considering the context, adaptability in the context adaptability in the context of conflict and displacement and reaching the interventions to the targeted groups (Relevancy)
2. Measure the extent to which the intervention has been achieved or its likely to achieve its intended, immediate results i.e., the outputs (Efficiency) and the outcomes (Effectiveness)
3. Measure the extent to which the intervention has achieved or contributed to the achievement of the program goal (Impact) and the wide scope of the program.
4. Assess the synergies between the program and other Mercy Corps interventions in the program locations. The (Coherence) of the interventions with policies and programs of partners operating the same locations and context.
5. Assess the potential for or contribution and progress made in achieving lasting solutions and behavior changes with major focus on (Sustainability) and the extent to which the net benefits of the interventions continue or are likely to continue.
6. Assess the quality of program governance and management
7. Identify key lessons learnt, best practices, challenges and draw evidence - based recommendations for future similar programs.

2.1.  Evaluation criteria and key questions

S/No.

Evaluation Element

Proposed key evaluation questions

Relevancy

1. To what extent has the program improved formal and informal institution including capacity of national partner capacity and the key relevant stakeholder’s role in the implementation
2. To extent do stakeholders demonstrate effective commitment to the objectives of the interventions (Ownership)
3. To what extent are activities and outputs consistent with the program’s results framework (RF)

Efficiency

1. To what extent have resources (funds, human resources, time, and expertise etc.) been allocated efficiently to achieve outcomes?
2. To what extent was the program implemented most efficiently?
3. To what extent has the program received technical and administrative support to facilitate achievement out the program goal.
4. How is the communication between program team, partners, and the community

Effectiveness

1. To what extent are context factors (political stability or instability, population movement considered in the program design and implementation?
2. What were the major factors influencing the achievement or non-achievement of the program’s targets and outcomes?
3. To what extent has the program created unintended positive or negative effects? Were the negative effects considered for possible risk mitigation?

Impact

1. To what extent have the outputs been achieved? And if so to what extent have outputs contributed to achievement of the outcome?
2. What changes in the lives of the households can be attributed to the program
3. What negative coping mechanism have been avoided or reduced because of the program both food security and livelihood coping mechanisms

Coherence

1. To what extent was the program coherent with the policies and program from other partners operating in the same program implementation areas
2. What were the synergies between the STREAM program and other Mercy Corps programs in the areas of program implementation.

Sustainability

1. To what extent has the program built local ownership and capacity of people and institutions in program implementation areas
2. To what extent has the private sector been sufficiently involved in contributing to the outcomes
3. Does the program have an exit strategy including schedules and guidelines for the transfer of responsibility to the government department? Is there a budget scenario for the time after the program?

Lesson learnt

1. What are the good practices and key lessons learnt from the program as well as practices in the program locations and program participants
2. What should be repeated and developed for the next phase of the program and what should not be repeated next time.

CARM

1. What mechanisms and processes were disseminated relevant to the program information to program participants and other concerned stakeholders? How effective were the mechanisms in terms of coverage and ensuring beneficiary knowledge of the program
2. Was there a process in place to receive, process and resolve complaints? What were some of the key findings and lesson learnt
3. What collaboration and coordination mechanisms were adopted during the implementation of the program and to what extent have such mechanisms added value to the program.

3)    Program Background & Relevant Documents

Mercy Corps aims to make strategic investments in Mundri East and Mundri West Counties of Western Equatoria (Legacy Amadi) State and in Koch County of Unity State. Mercy Corps’ goal for the target regions aligns with Mercy Corps’ global mission to build secure, productive, and just societies.  STREAM’s Overall Goal is that vulnerable households, youth, and women in Mundri East, Mundri West and Koch Counties in South Sudan have improved well-being making use of the skills, knowledge and enabling environment facilitated by the Program. The Phase 1 of the STREAM Program will focus on one key objective: to improve food security of 6,350 households through enhanced gender equitable market access and increased agricultural production and off farm and non-farm livelihood opportunities.

The Main Objective of the project is “Vulnerable households, youth and Women in Mundri East, Mundri West and Koch Counties in South Sudan have improved their wellbeing by making use of the skills, knowledge and enabling environment facilitated by the program”.

The main outcomes of STREAM are:

• Outcome 1: Increased food production yields among male and female smallholder farmers in 6,350 targeted farm households
• Outcome 2: Increased income for 2,000 male and female youth households engaged in non- farm livelihood opportunities and agro-processing opportunities and off-farm agro-processing opportunities
• Outcome 3: Strengthened market systems and local institutions to ensure gender equitable access to financial services, livelihood inputs and extension services for 6,350 households.

4)    Final Evaluation Design

Final Evaluation Methodology

The consultant is expected to propose and design the methodology for conducting the baseline survey using market system development lenses. Both primary and secondary sources are not limited. A mix of qualitative and quantitative methods such as surveys, focus group discussions and in-depth interviews with key informants, and observation of the system will be used to collect primary data. Surveys and interviews are expected to take place at all program sites (Mundri East and West and Koch).

1. Qualitative methods include.

Cognizant of COVID-19, the consultant is expected to conduct Focus Group Discussions (FGDs), Key-Informant Interviews (KIIs) while ensuring safety for all participants. Findings from this technique will be used to refine questions raised through the thematic reviews.

1. Quantitative Methods include.

It is important to be able to acquire data that permits the Mercy Corps and its partners to compare changes over time in different locations (such as counties) as per the indicators. Household surveys will be conducted to measure the selected indicators. This data should be collected either through face-to-face interactions or remote interviews. However, the consultant is at liberty to come up with its methodology to undertake this study not necessarily limited to the above approaches

2a. SamplingThe consultant will implement a sampling procedure which is representative of the consortium life of the project target population. It is important that the Final evaluation achieves a statistically acceptable sample size for the respective project participant categories.

The consultant will develop and abide by the approved sampling methodology. Where sampling strategy is changed for whatsoever reason, the consultant is expected to recompute to achieve representative samples with a resubmitted sampling protocol. Therefore, a clear, and unambiguous proposal on sampling methodology and sample size is expected from the consultant. Sampling method should attain representative and generalizable results for all project participant categories.

1. Data sourcing

The baseline measurement considers both primary and secondary data sources.

Primary data sources include:

• Farmers groups/cooperatives
• Households’ female and male farmers
• Women’s organizations/groups
• female and male youth (18 – 35 years)
• Financial service provider/Micro finance institution
• Honey, Lulu, carpenter, tailors, bakers etc.
• County agricultural department
• Relief and rehabilitation commission official

The secondary data sources include but are not limited to studies from baseline survey, South Sudan’s FSL-Cluster, FAO, WFP, SDC, IPC latest findings and other bilateral donors, the consultant is expected to undertake review of existing documents using appropriate document review tools and methods.

1. Data analysis.

The consultant should adopt an iterative data analysis approach. This approach allows for prompt data validation as enumerators collect data. Hence, the consultant will develop a data analysis plan. Analysis should integrate findings from the different sources of data. The consultant should corroborate, and triangulate data from different primary data sources.  Depending on access to the country and within the country, the consultant should factor in remote analysis and presentation of findings and sharing of documents in collaborative forums.

1. Data Quality Issues

The designed data collection techniques, Quality of data should not be compromised, and maximum care should be taken to avoid or at least minimize errors at all stages of the baseline measurement process. Some techniques such as, but not limited to, the following will be applied:

• Before data collection: Pilot testing the data collection tool will be required to verify the reliability and validity of the tool. This includes for both face to face and remote approaches.
• During field data collection: For household surveys, data entry will be on the spot using an electronic questionnaire. Monitoring enumerators for accuracy in doing the interview and in capturing data will be necessary. Checking through all completed responses (daily) to ensure any mistakes or inconsistencies are corrected on time will add value to the quality of data.
• Data analysis: Perform iterative data analysis which involves continuously analyzing key variables as part of data quality checks using various methods such as: frequencies or cross-tabulations or any forms of regressions.

1. Communication of findings/reflections.

Final Evaluation findings will be validated by the relevant community stakeholders and Mercy Corps staff through a validation workshop. Once the validation is completed then the final report will be produced.8. Consultant Management

Mercy Corps will support the consultant both physically and remotely throughout the baseline. Whenever the consultant is identified, the consultant will work with the Program and Project Manager, MEL Manager The team will commence the inception process to:

• Review of Final Evaluation protocols, sampling design, data collection and analysis plan, COVID-19 and EBOLA risk reduction plan.
• Finalize tools and approve final plan prior to commencing data collection
• Providing technical support and oversight during data collection process

Data analysis The Consultant on this note is required to provide a data analysis plan and share before undertaking full assessment together with the data collection tools.

1. After Action Review.

The consultant will provide a reflection on the entire baseline assessment process shall be documented and shared with the Project team and Mercy Corps MEL department so that they may be taken into consideration for future studies. The documentation of these after-action reviews will be vital for reflection, growth, and continued improvement.

1. Final Evaluation survey limitation.

The Final Evaluation Survey report should also entail the limitations identified during the survey period may include:

4) Indicator to measure during the baseline survey.

Mercy Corps - South Sudan

Strengthening Resilience in Agriculture, Livelihoods and Markets through Local Institutions in Mundri and Koch (STREAM) Programme Phase 1

Result Level

Description

Indicators

Baseline Value ??

GOAL for the seven year intervention:

Vulnerable households, youth and women in Mundri East, Mundri West and Koch Counties in South Sudan have improved well-being making use of the skills, knowledge and enabling environment facilitated by the STREAM Programme.

OBJECTIVE of Phase I (1st Dec 2020 – 30th Nov 2022

To improve food security of 6,350 households through enhanced gender equitable market access and increased agricultural production and off farm and non-farm livelihood opportunities.

1. Increase in average household farm crop yield by 30% for target households

2. Average household land area under cultivation increased by 50% for target households

3. Number of farmers reporting an increase in the number of food secure months

4. Average HH Score on the Coping Strategies Index (CSI) for target households

5. Participants report a change in attitude in practice around negative coping mechanisms

6. 30% of the target households have an alternative off farm/ non-farm means of livelihood support

OUTCOMES

1.       Increased food production yields among male and female smallholder farmers in 6,350 targeted farm households

1.1   % Of male and female smallholder farmers that have adopted improved agricultural practices

1.2   % Of increase in yield in staples due to adoption of improved agricultural practices

1.3   % Of male and female smallholder farmers that have reported reduced post-harvest losses

AFS_ARI_1

1.4   Number of smallholders with increased incomes from agricultural production

AFS ARI_2

1.5   Number of smallholders applying more agro ecological farming practices

2.       Increased income for 2,000 male and female youth households engaged in non- farm livelihood opportunities and agro-processing opportunities and off-farm agro-processing opportunities.[1]

2.1   Number of agro-based initiatives supported, and number of women and youth involved/ employed

2.2   Income earned by these agro-based initiatives

2.3   Number of SHGs given Working Capital assistance (grants)

2.4   Number of loans issued by SHGs to members for IGAs

2.5   Households reporting improved quality of life and wellbeing in response to IGA loans and subsequent income generation

2.6   Change in perception and acceptance for women to engage in traditionally male livelihoods.

IED_TRI_5

2.7   Proportion of women with positive perception on their influence on business and economic related decision making.

3.       Strengthened market systems and local institutions to ensure gender equitable access to financial services, livelihood inputs and extension services for 6,350 households[2]

3.1   Number of cooperatives engaged in seed production and details of seeds produced

3.2   Number of private agro and non-agro based service/ input providers and details services offered

3.3   % of male and female smallholder farmers benefited from and reporting satisfaction with accessing quality inputs and tillage services

3.4   # and types of improved non-farm livelihood support services and opportunities created

3.5   # of tailored and inclusive formal and informal financial services available for households engaged in agricultural and off-farm livelihood activities.

3.6   % of farmers reporting confidence in the quality of extensions/financial services provided by the local partners,

3.7   # of individuals/ households benefited from formal and informal financial services

3.8   # of functional markets and market systems

3.9   # and type of local institutions (MFI, Cooperatives, Self-Help Groups) functional and providing services to members and clients.

3.10           % of households that have diversified their source of income supported by the program.

IED_ARI_3

3.11           Number of people having access to and making use of formal financial products and services

OUTPUTS

1.1   Adoption of improved agricultural practices among male and female smallholder farmers

1.1.1.        # of functional Agricultural Cooperative Societies (both revived and formed) targeting male and female smallholder farmers

1.1.2.        # of Country Department of Agriculture extension cadre staff capacity build on climate smart agriculture

1.2.      Reduced post-harvest losses among male and female smallholder farmers

1.2.1.        % of male and female smallholder farmers who report using improved storage and post-harvest handling practices

2.1   Expansion of off-farm and agro-processing livelihood opportunities among male and female youth.

2.1.1         # of value addition facilities and enterprises such as honey collection and processing, lulu oil processing and cassava process revived or scaled up

3.1     Increased access to quality inputs and tillage for male and female smallholder farmers

3.1.1         Existence of certified quality seed dealers and or producers working closely with farmers’ cooperative societies.

3.2     Improved off-farm livelihood support services and agro-processing opportunities

3.3     Tailored and inclusive formal and informal financial services available for households engaged in agriculture and non-farm livelihood opportunities

3.3.1         Number of individuals disaggregated by age and gender reporting increased access to financial services

5) Team Composition, Roles & Responsibilities

Include for me your team members here and their roles.

S.no.

# Of Days

Activity

Responsible Person

W1

W2

W3

W4

W5

W6

W7

W8

W9

1

4

Develop an Inception Report detailing the process and methodologies to be employed to answer the evaluation questions. This should include all evaluation tools, and important time schedules for this exercise, and be presented to Mercy Corps South Sudan for review and further inputs before going to the field.

Consultant

2

5

Undertake desk review of the relevant program documents that include the Proposals, Implementation Plans, Program Implementation Reports, Mercy Corps strategy documents, Monitoring Survey Reports, and any other relevant documents

Consultant

3

1

Conduct orientation of in-country leadership to discuss expectations and logistics

Consultant & M&E

4

5

Design, develop, critique (with Mercy Corps South Sudan’s STREAM Program and MEL team) and refine data collection tools including translation to Moru/Nuer languages, when needed appropriately.

Consultant

5

5

Train Enumerators/Surveyors and Pre-test data collection instruments

Lead Consultant and Field Officer

6

8

Oversee data-collection activities

Enumerators & Field Officer

7

5

Encode and analysis data

Consultant

8

5

Prepare draft Final Evaluation report

Consultant

9

2

Conduct debriefs meeting in-country and present draft report (Validation workshop)

Consultant

10

3

Finalize Report and share with Mercy Corps

Consultant

11

1

Final approval by Mercy Corps

Director of Programs/CD

12

1

Report dissemination and follow up

Director of Programs

45

Total

6)    Description of Deliverables

Deliverables:

Successful performance of this assignment will be based on production and timely submission to Mercy Corps of the following deliverables:

• Inception report – Both for the final evaluation; setting out how they will approach the assignment, proposed methodology and timetable/work scheduling and data collection tools, including survey questionnaires include an annex highlighting how COVID-19/EBOLA safeguards will be achieved. There will also be an oral presentation to Mercy Corps and representatives from partner organizations. The research protocol and survey instruments will need to be approved before going to the fieldwork stage
• Finalize survey tools and inclusive and gender sensitive sampling design and gender framework analysis tools
• Enumerator training, tool pre- testing and data collection for both tools
• Draft report - Draft report (in English) will be shared with Mercy Corps within 15 working days of completion of field work, submitted to Mercy Corps for input for Final evaluation
• Final Report – for both gender framework analysis and main baseline survey; Five Hard Copies and a soft copy of the report consisting of input from consortium members, completed within 5 days after input is received.

The program final evaluation survey shall be conducted in maximum 45 days starting from the date of signing the contractual agreement. With the first draft submitted at least 30 days after the start of the evaluation with a report maximum of 60 pages including annexes and the summary report of 30 pages’ maximum.

1. Report Structure & Content

The final report will at the very least contain, but not be restricted to the following.

• Introduction
• Acknowledgements
• Lists of abbreviations
• Executive Summary (including main conclusions and recommendations).
• Background with analysis of the context.
• Methodology
• Limitations
• Main Findings
  • Key findings (This section should provide the answers to the evaluation questions. This should include data represented in tables, charts, graphs, and other figures. Additionally, contextual analysis of the data explaining its meaning should be also provided in the form of sentences.
  • Quantitative: comparing BL to EL in phase 1 (3 counties) with statistical tests
  • Qualitative (phase 1; 3 counties) comparing what revealed at BL to EL
  • summary using qualitative and quantitative results (and discuss how the context might have influenced this
  • Comparison BL to EL for phase 1 (3 counties) testing for statistical significance of change
  • Synthesis, recommendations, and lesson learnt (This is space for the evaluation team to think about the data and results and make concrete recommendations for current or future program improvements, pull out organization lessons learned, and generally comment on data and results.  Everything presented in this section should be directly linked back to the information presented in the Results section of the report)
• Conclusion (Summary based on the evidence and analysis)
• References
• Annexes (a revised log frame, TOR, abbreviations, persons interviewed, references, tables with survey data, questionnaire forms used, summary indicators table with baseline values, data collection tools, consent form, Statistical analyses (codes and formulas used) Raw data (both quantitative and qualitative etc.

7) Plans for Dissemination, Use & Follow Up.

As laid out in schedule section of this SOW, MEL manager will lead the STREAM Team in developing an action plan based on the findings and recommendations of the final evaluation Survey. The MEL manager will be responsible for presenting the key findings and recommendations to the STREAM. Copies of all data collections tools and information gathered during the baseline will be provided for use and dissemination among the Mercy Corps Team and for use and dissemination at headquarters level.

8) Budget/ Consultancy Fee

The total charge to be paid for the consulting assignment will depend on the technical proposal. A technical and financial proposal based on the Terms of Reference outlined above must be provided. Hence most of the payments are entirely responsibility of the Consultancy firm/Consultant and invoice Mercy Corps as per the agreement. However, below will be incurred from Mercy Corps side.

S/no

Activity

Unit

Rate (USD$)

Total

Air fare

4

275

1,100USD$

Grand Total

1,100USD$

9) Final Evaluation survey Timing

The Final survey will be conducted for a duration of 7 weeks (from 19th of December 2022 to 31st of January 2023). This will include preparation and delivery of the final report

• Whilst Mercy Corps and the other consortium members will undertake to facilitate travel and organize meetings for the consultant, it is the consultant’s ultimate responsibility to follow through and ensure that all relevant respondents are interviewed, and relevant project sites visited.
• Mercy Corps will be responsible for consultants’ accommodation in the field and travel within the project location while the consultant will be responsible for the requisite visas and travel permits.
• Travel to field and within the project locations involves risk (travelling by car and through insecure locations). Other than providing standard security arrangements and information, Mercy Corps will take no responsibility for the safety of the consultant(s) and/or of their respective property during this exercise. Personal insurance will be the responsibility of the consultant(s). This car will entirely be hired by the consultants.

10) Skills and qualification required.

• At least lead consultants with a master’s degree in food security or related area and over 10 years of significant food security-related research/work experience.
• Applied qualitative and quantitative research experience.
• Market system development approach assessment skills is an added advantage.
• Significant experience of undertaking desk reviews, as well as assessments and/or evaluations.
• Experience performing food security program surveys within South Sudan will be a major advantage.
• Proven publication record related to food security (attach at least one research sample).
• Ability to provide evidence of producing high quality assessment/evaluation reports in English and working within tight timeframes (understanding of relevant local languages will be an added advantage).

11) Terms of Payment.

Payment will be effected as follows; First installment (25%) of the total cost on submission and acceptance of inception report, 25% upon submission of first draft report and Final payment (50%) upon completion and approval of the final report. Additional information on payment terms and conditions will be included in the contract.

12) Additional Information.

• Consultants shall abide to WHO and Government of South Sudan COVID-19 SOPs.
• Consultants shall be required to sign and abide to Mercy Corps Safeguarding Policy (which includes prevention of sexual exploitation and abuse, and behavior protocols)
• Consultants shall abide to EU beneficiary data privacy/management policies

13) Ethical considerations, Confidentiality and Proprietary Interests.

• The Consultancy Firm/Individual consultant holder needs to apply standard ethical principles during the assignment. Some of these must deal with confidentiality of interviewee statements when necessary, refraining from making judgmental remarks about stakeholders.
• The incumbent shall not either during the term or after termination of the assignment, disclose any proprietary or confidential information related to the service without prior written consent by the contracting authority. Proprietary interests on all materials and documents prepared by the contract holder under this assignment shall become and remain properties of Mercy Corps

14) Communication.

For queries or need for additional information on the consultancy please contact:  rlubang@mercycorps.org & sedwardbonis@mercycorps.org

15) Application Process.

Interested individuals and companies should submit the following application documents:

• Expression of interest outlining how the consultant(s) meets the selection criteria and their understanding of the ToR and methodology
• A proposed activities schedule/work plan with time frame
• Copy of CV of the consultant(s) who will undertake the evaluation
• One recent example of similar evaluation report written by the applicant
• Legal documents for interested consultancy firms
• Provide technical proposal
• Financial proposal detailing consultant(s) itemized fees
• Mercy Corps will evaluate technical and financial proposals and award the assignment based on technical and financial feasibility.  Mercy Corps reserves the right to accept or reject one or all proposals received without assigning any reason and is not bound to accept the lowest or the highest bidder

Please address your applications to “ss-pr@mercycorps.org while copying “lgideon@mercycorps.org “& kaguret@mercycorps.org

The deadline for submitting applications is 26th of December 2022.

[1] We need to establish existence of agro based opportunities

[2] Consider some Market Systems Resilience type questions/indicators (focusing on both structural and behavioral characteristics)