Programme End Evaluation
Terms of Reference – Call for Proposals
“Integrated Community Development December 2014 to November 2017”
- Purpose of this Document
This Call for Proposals for an End Evaluation of DPA’s ICD Programme in Kampong Thom Province seeks an evaluator to design, implement and report on the project in line with the TOR set out below.
Interested parties should provide a costed proposal setting out how they would plan to evaluate the programme in line with these TOR, along with CVs of the consultants, a short document detailing their experience, and an example of a recent, similar piece of work. All submissions will be treated in confidence and will not be shared beyond those involved in the procurement process.
- Brief Overview of the Programme:
This consultancy is to evaluate the SCIAF and CAFOD funded Development and Partnership in Action (DPA) three-year programme December 2014 to November 2017 in the central of Cambodia.
The Integrated Community Development Programme is in Sandan Districts of Kampong Thom Province, in 15 mainly indigenous villages. The project works with 5,979 beneficiaries (3,016 are women or 50%) in 15 villages of two communes (Sandan and Ngon) of the district. The beneficiaries are the Kuy Indigenous poor and vulnerable and include smallholder farmers without the resource to step-up agricultural production from subsistence level; women headed and disabled households with little or no livelihood.
Key programme components: Communal Land Titling (CLT) and Community Fish Sanctuary (CFIS) protection and conservation, diversification of agricultural based livelihood and income, Disaster Risk Reduction (DRR) and Climate Change Adaptation (CCA), and advocacy.
Communal land titling (CLT) and community fish sanctuary (CFIS) protection and conservation: The project carries out activities following government guidelines on the process of CLT and CFIS. These include three steps of CLT (IP legal identification, IP legal entity, IP legal land registration) and four steps of CFIS organising process (identify situation and options for establishment of CFIS, monitoring and identification of CFIS location, official recognition on community CFIS conservation, management, protection and development of the CFIS.
Diversification of agricultural based livelihood and income: These activities involve provisions of trainings, exposure visits, and coaching on diversified agricultural productions (upland and lowland rice, crops, vegetable, animal rearing, natural fertility soil improvement, establishment of agriculture groups and an animal vaccination campaign. Villagers organise themselves to form the Agricultural Groups (AGs) of rice, crops, and vegetable, animal raising, and cash crop groups (cassava and cashew). The established AGs were trained in agricultural techniques related to types of group provided agricultural inputs. The project also supports agricultural technology trials, farmer-to-farmer learning, training on business knowledge, and support AGs to form Agricultural Cooperatives (ACs).
Disaster Risk Reduction (DRR) and Climate Change Adaptation (CCA): DPA provide training to villagers on DRR and CCA concepts and support establishment of Village Disaster Management Teams. The project enables villagers to develop and establish a community disaster management plan in case a natural disaster such as drought or flooding occurs. Farmers were provided with skills and knowledge to adapt their farming practice and inputs (crops and small scale irrigation) to changing patterns of weather.
Advocacy: DPA support and train the Village Development Actors and various committee members how to effectively enhance and strengthen their networking and advocacy strategies. In each village there were focal point persons trained on local advocacy, gender, women rights, and IP and land rights.
The programme has a participatory approach, developing strong relations between the village development associations, committees, local authorities and technical institutions through collaborative working and provision of local government technical expertise. DPA continues to work in co-operation with local authorities and co-ordinate with technical departments such as the Fishery Administration Cantonment, Provincial Department of Land Management and Urban Planning and Construction, Provincial Department of Rural Development, Provincial Department of Agriculture to provide technical and legal support to the target communities.
Participatory processes are central to the programme with the emphasis on ensuring that the target communities genuinely own the development processes and the outcomes in their communities, and through this a greater appreciation and understanding of their community assets and how best to manage and protect them.
- Integrated Community Development Programme-Kampong Thom:
In Kampong Thom, Sandan and Ngon, the Kuy Indigenous People (IP) households in 15 villages have increased income and food security all months of the year and the Kuy Indigenous People’s traditional natural resources are protected and used sustainably.
In 15 villages in Sandan and Ngon, 1376 (female 668) Kuy Indigenous People, have increased household income, food security, resilience to the impact of seasonal flooding and drought, the community natural resources are protected and used sustainably, women are leaders and domestic violence is reduced.
The total direct beneficiary number is 1376 with no duplication of numbers and includes: WHH households, 144, Agricultural Groups (AG) and Agricultural Cooperatives (AC) members 1015, 88 Community Fishing Sanctuary members and 129 beneficiaries from the flooding and drought output.
- Number of households food secure all months of the year.
- Reduction in number of IP households who require external assistance following flooding or drought
- Number of households who protect and benefit from Community Fishery Fish Sanctuary (CFiFS)
- Number of IP households with legal communal land titling
- % of target men and women who state there is increased influence of women in the community and in the home.
- % of target IPs who state they have increased influence over natural resource and land issues
- 251 Farmers (Female 102) have increased on farm income from cash crops
- 144 Women Headed Households have increased income from selling and breeding livestock
- 368 Farmers (Female 156) have increased upland and lowland rice yields
- 1015 farmers (Female 427) benefit from newly established Agricultural groups and Agricultural Cooperative
- 516 IP households (Female 630) are organized and able to respond and cope at community and individual level to flooding and drought
- 520 IP household’s (WHH 268) demarcated land and rivers are protected by legal agreements and community action
- 900 IP villagers (Female 450) actively involved in promoting and protecting the rights of women
- 700 IP villagers (Female 312) take action on land issues and illegal logging
- Evaluation Objectives:
- Determine whether the underlying programme theory of change and assumptions were valid;
- Assess and score the programme’s achievement of its anticipated outcomes and outputs as set out in the programme Logframe ;
- Assess the relevance, efficiency, effectiveness, impact and sustainability of the project outcomes and outputs;
- Assess the extent to which the project promoted gender equality and women’s empowerment, and was able to meet the originally assessed needs of the communities as they were expressed in the original needs assessment;
- Provide information on what worked well, what did not, and why; and
- Provide recommendations, for improvement for the upcoming project design.
- Evaluation approach and methods:
These evaluation Terms of Reference (TOR) have been developed by DPA and in line with the SCIAF evaluation guidelines. SCIAF’s approach to evaluations aims to combine the two purposes of learning and accountability. These terms of reference do not intend to define the methodology in detail. However, SCIAF expects the methodology to incorporate certain principles and processes, set out here. As such SCIAF will use the Bond Evidence Principles as a tool to review and quality assure the draft DPA ICDKT evaluation report.
The evaluation should:
- Ensure that all key stakeholder groups affected by the programme (both directly and indirectly) are consulted and their views considered in the evaluation. The level and form of participation by partner staff, beneficiaries and other stakeholders should if possible go beyond consultation in order to maximise learning opportunities.
- Evaluate the Programme logic, as set out in the logical framework;
- Evaluate the extent to which the programme achieved its intended objectives;
- Consider the relevance, efficiency, effectiveness, impact, and sustainability of the programme, in line with OECD/DAC definitions and guidance;
- Evaluate the extent to which the programme integrated gender and poorest into its design and implementation, and the extent to which it has promoted gender equality;
- Consider how the SCIAF cross-cutting issues of disability, environment and climate change have been addressed by the Programme;
- Ensure it is in line with the BOND Evidence Principles of:
- Voice and Inclusion: the perspectives of people living in poverty, including the most marginalised, are included in the evidence, and a clear picture is provided of who is affected and how;
- Appropriateness: the evidence is generated through methods that are justifiable given the nature of the enquiry;
- Triangulation: the evidence has been generated using a mix of methods, data sources and perspectives;
- Contribution: the evidence explores how change happens, the contribution of the intervention and factors outside the intervention in explaining change;
- Transparency: the evidence discloses the details of the data sources and methods used, the results achieved, and any limitations in the data or the conclusions.
SCIAF and DPA will formally respond to the Evaluation report’s recommendations setting out our response to the recommendations, and how we plans to address the findings in the future.).
- Evaluation Process Key Steps
- Call for Proposals;
- Review of proposals by Evaluation Committee;
- Interviews with shortlisted candidates;
- Selection and contracting with preferred candidate;
- Preparation of an inception report detailing methodology and timescale, submitted to DPA / SCIAF Evaluation Committee for review.
- A discussion with the Evaluation Committee regarding the methods proposed sampling a representative cross section of the village communities studied. Approval of plan;
- Implementation of agreed methodology;
- Feedback session with each village visited to discuss and clarify key findings and observations.
- A presentation with the staff of DPA ICD-Kampong Thom and its stakeholders to discuss preliminary findings and analyses and to reflect upon lessons learned from the evaluation.
- Submission of draft report to DPA, SCIAF and CAFOD;
- Review by DPA, SCIAF and CAFOD and comments to consultant;
- Final written report submitted and signed off by DPA, SCIAF and CAFOD; and
- Management Response prepared and all staff and stakeholders are fully briefed on the key findings, observations and recommendations the evaluation
- Evaluation Committee:
The Evaluation Committee will consist of the Executive Director, DPA Programme Managers, MEL Officer and ICD-Kampong Thom Project Manager, SCIAF Programme Officer and SCIAF Programme Manager. Committee members will be available to the Evaluator for consultation on the general direction of the evaluation and to address any issues/concerns arising during the exercise.
- Evaluation Methodology
To be provided by the consultant in their proposal
- Review all relevant documents and draft and confirm with the Evaluation Committee the objectives of the evaluation, process and methods to be used, expected outcomes, and the format for the presentation of the results.
- Provide leadership to the evaluation in developing appropriate interviewing techniques, ensuring data collectors are adequately trained and supervised, preparing for village feedback sessions, and presenting preliminary findings and lessons learned to the Kampong Thom team.
- Coordinate the activities of the Evaluation Team and keep the Evaluation Committee and funding partners informed of progress.
- Use a participatory approach that encourages discussion of the findings, and ownership of the conclusions and recommendations by all primary stakeholders.
- Upon completion of the final draft, discuss the results with the Evaluation Committee and funding partners
- Complete the final evaluation report, following the outline agreed upon with the Evaluation Committee and (funding partners)
- Include in recommendations inputs to future programming design: beneficiaries identify what is important to them for the development of their communities and families.
- Required Outputs:
The consultant will be expected to provide to DPA, CAFOD and SCIAF with:
- An inception report setting out the detailed methodology and timescale for the evaluation including sample sizes and sampling methods, developed to meet all aspects of this TOR, in line with the proposal received as part of the procurement process;
- A full Draft Report, no later than the 2nd of January 2018. The draft report should include a specific recommendations section, and an Evaluation Summary and Scoring Sheet (see annex 1) which includes a scoring of the projects performance against intended objectives.
- A Final Report will be due by the 4th of January 2018. The consultant will be responsible for writing the final report and submitting it to DPA Executive Director, SCIAF and CAFOD Programme Officers within the agreed time.
The evaluation report, which will be written in English language, should be clear, easy to understand, concise and useful for the organisations and beneficiaries. The main body of the report should usually not exceed 30 pages (A4 pages on both sides) and will include an executive summary and recommendations. Technical details should be confined to appendices that should also include a list of informants and the evaluation’s team’s work schedule. The Evaluation Summary and Scoring Sheet is considered an integral part of the evaluation report.
Background information should only be included when it is directly relevant to the report’s analysis and conclusions.
Recommendations should be recorded in a separate section and should also include details as to how they might be implemented. Each recommendation should be accompanied by a suggested date for completion and an indication of the lead actor responsible for taking it forward.
The evaluator should complete the table in Annex 1 below, and include it as an annex to the evaluation report.
SCIAF will prepare an Evaluation Management Report after the evaluation is completed.
One soft copy and one hard copy of the final report (MS word 2007 and excel format – font size: 12 and Times New Roman) will be submitted to DPA, SCIAF and CAFOD by evaluator.
The following is intended as a guide to the times estimated for each component of the work. The Team Leader will be required to prepare and submit a detailed work plan at the beginning of the evaluation. The evaluation is expected to take 16 days. The evaluation is scheduled to begin on 18th December 2017 and finished by 4th January 2018.
|Days||Description of Work|
|1 day||Meet with the Evaluation Team and Evaluation Committee, study relevant
programme documents, design evaluation process and methods, select
sample villages, develop guide questionnaires with team, finalise and
submit work plan
|8 days||Travel to Kampong Thom, meet field staff, and review field documents and Visit 3-4 villages, conduct interviews and focus group discussions, and feedback findings to villagers. Also, interview programme staff, counterparts, NGOs and local government officials, and other provincial development agencies concerned. Presentation to ICD-Kampong Thom team and return to Phnom Penh.|
|½ day||Interview DPA ICD Support team and Management staff in Phnom Penh|
|5 days||Write evaluation report and submit draft to Evaluation Committee.|
|½ day||Discuss draft report with Evaluation Committee and project leadership staff (debriefing meeting), SCIAF and CAFOD|
|1 day||Complete final report and submit report to DPA Executive Director, SCIAF and CAFOD|
The debriefing meeting to present the draft report will be organised on the 2nd of January 2018. The final report will be due by the 4th of January 2018.
Annex 1: Evaluation findings summary sheet
SCIAF Evaluation Summary and Scoring Sheet v1.0
provide a short, accessible summary of a programme / project evaluation; provide a scoring of the programme based on an assessment of the achievement of outcomes and outputs; and explain how the evaluation met with the BOND Evidence Principles.
internal to SCIAF, to be used by the programme team for management and learning purposes. The Focal point will be the relevant Programme Officer. Once signed off by the Programme Manager this will be shared with Department and possibly more widely.
the scoring sheet should be compiled separately from the evaluation report, and presented in draft form alongside the draft report. Comments from the programme team should be considered before the evaluator provides a final version of the summary and scoring.
Section 1: Evaluation Findings Summary
asks for a commentary against the OECD/DAC evaluation criteria. This should be part of the TOR for the evaluation, so this section should be a short summary of the conclusions reached in the evaluation. In addition to the OECD / DAC criteria a view should be given on the extent to which the project addressed gender issues, and that it represented value for money.
The gender section seeks an opinion on the extent to which the project was ‘gender neutral’, ‘gender sensitive’ or ‘gender transformative’. These are defined here as:
- ‘gender neutral’ – there is little or no reference to gender in the project, there is no analysis of gender relations; and the project does not address the differential interests of women and men, or address the relations between them. There is no information as to whether the project impacted on women or men differently, either positively or negatively.
- ‘gender sensitive’ – the project makes reference to gender relations and the different interests of men and women; there is some form of gender analysis; the project seeks to address the different needs of men and women but tends to focus on practical needs rather than attempting to challenge structural inequalities.
- ‘gender transformative’ – Project seeks to transform unequal power relations between men and women through empowering women, challenging norms in gender relations, policy change, claiming rights etc. These can be at a personal, household, community or societal level.
Projects frequently address a range of issues – the categorisation will most likely relate to a sense of where on balance the project lies.
Value for money is likely to represent a combination of the judgements of efficiency, effectiveness and impact.
Section 2. Score
This section draws on the DFID project scoring system. This requires an assessment of the intervention’s achievement of planned outputs and outcomes. Priority should be given to the achievement of outcomes rather than outputs.
The scoring system has five levels, from A++ to C.
Scoring may not be simple – there may be variations in the level of achievement against different outcomes and outputs, and not all outcomes and outputs are quantified or can be easily quantified.
Moreover, a programme may achieved a range of outcomes and outputs that were not included in the original plan, while at the same time not achieving everything in the original plan. This may have been the result of deliberate choices, with changes made to the plan along the way, or this may only become apparent during the evaluation.
Bearing these different issues in mind, SCIAF expects the evaluator to use their best judgement to come to a fair, balanced and objective score. We also require that the reasoning behind the scoring is set out briefly in the document.
|Score||Outcome Description||Output Description||Guide to Scoring|
|A++||Outcome substantially exceeded expectation||Outputs substantially exceeded expectation||126% or more attainment against plan indicators.|
|A+||Outcome moderately exceeded expectation||Outputs moderately exceeded expectation||101 – 125% attainment against plan indicators.|
|A||Outcome met expectation||Outputs met expectation||90% – 100% attainment against plan indicators|
|B||Outcome moderately did not meet expectation||Outputs moderately did not meet expectation||75% – 89% attainment against plan indicators|
|C||Outcome substantially did not meet expectation||Outputs substantially did not meet expectation||74% or less attainment against plan indicators|
Section 3: Evidence Principles
The BOND Evidence Principles were developed by the British overseas NGO network BOND as a standard for judging the quality of evidence used in evaluations. They are a test that can be used to assess the ‘weight’ that can be put on evaluation findings and recommendations.
They look at five key areas:
- Voice and inclusion: to what extent have the perspectives of people living in poverty, including the most marginalised, been captured and reflected in the evaluation?
- Appropriateness: has the evidence for the evaluation been generated through methods that are justifiable given the nature and purpose of the evaluation? Were the methods relevant and adequate?
- Triangulation: has the evidence been gathered using a mix of methods, data sources and perspectives in order to cross-check and validate the different information, and include all relevant perspectives?
- Contribution: has the evaluation explored how change happens, what the contribution of the intervention was to any observed changes, and allowed for factors outside the intervention to account for them?
- Transparency: is the evaluation clear about the data sources and the methods used, and any limitations to the research?
Here we ask that the evaluator provides a brief commentary against each of the five points.
The overall length of the document should be no more than a maximum of 4 pages, and ideally less.
SCIAF Evaluation Summary and Scoring Sheet
|Section 1. Evaluation Findings Summary
(Using OECD/DAC framework)
(Are the objectives of the programme valid – relevant to the identified problem – and are activities and outputs consistent with objectives and intended impacts?)
(Were the activities implemented in a cost-efficient way compared with possible alternatives?)
(To what extent were the objectives achieved and what were the major factors influencing their achievement (or non-achievement)?)
(Are the benefits of the programme likely to be maintained after the end of the project/donor funding and what has influenced this?)
(What have been the (positive or negative, intended or unintended) changes in the lives of the beneficiaries as a result of the project?)
|Was the project gender neutral, sensitive or transformative?|
|Did the project represent value for money? Please explain your thinking.|
|Section 2. Score
(Using DFID Scoring System)
|Explanation for Score||Score|
|Section 4. Quality of the Evaluation Compliance with BOND Evidence Principles:|
|Explain how the evaluation meets the BOND Evidence Principles|
|Voice and Inclusion|
Annex 2: 5. Key questions the evaluation should address:
The evaluation study shall verify, analyse and assess in detail the issues and questions outlined below referring to the five evaluation criteria endorsed by the OECD-DAC (relevance, effectiveness, efficiency, sustainability and impact)
The evaluation team is required to use their professional judgement and experience to review all relevant factors and to bring these to the attention of the partner organisations, target beneficiaries and donors (SCIAF)
Assess the extent to which the objectives of the programme are relevant and in line with the needs of the target beneficiaries. Assess the extent to which the programme design is consistent with the local situation, including the physical and policy environment.
This should include an assessment of the quality of programme preparation and design, the logic and completeness of the planning process and the internal logic and coherence of the programme design
- How did the project team ensure a fair and equitable selection of beneficiaries and that the beneficiaries selected were bona fide targets, have the most vulnerable groups been reached?
- How did the project ensure that programme was relevant to the needs of the target group and continued to be relevant against changing needs of the target group during the project period?
- To what extent were delivered services perceived by beneficiaries to be relevant (contextually and culturally, for example)?
The extent that the programme achieved results in terms of defined indicators and how have the assumptions affected programme achievements? This should include specific assessment on benefits to target groups, including women, disabled and people living with HIV and AIDS.
- To what extent have outputs (project results) led to the achievement of project outcomes and accomplishment of project purpose?
- To what extent have the outputs been achieved, and to what extent are the intended target group adapting and applying alternative practice and learning?
- What were the major factors influencing the achievement or non-achievement of outputs?
- What are the benefits of the project for different target groups?
- To what extent has the programme effected and contributed to its wider environment, the long-term social change at the community level?
- Has the project contributed to poverty alleviation of the target beneficiaries
To what extent have resources (incl. time) been used optimally in order to achieve results? Have the results been achieved at a reasonable cost?
- Has the project been cost-effective?
- Have project outputs been achieved at reasonable cost or would other approaches have led to the achieving of the same results at more reasonable costs?
- Was the right thing done for the right costs?
- How well have the activities transferred resources into the intended results in terms of quality and timeliness?
- Have financial resources been used as planned?
- Can the ‘cost per beneficiary’ be calculated, and how do they compare to other projects’ cost ratio?
Assess whether the positive outcomes are likely to continue after the programme has ended.
Key Question Addressed by the Evaluation
- To what extent are the benefits to the target communities sustainable in the longer term and what mechanisms have been put in place by the programme to reduce reliance and maximize family/community structures ownership;
- What is the ability of target communities to maintain the benefits of the programme such has higher levels of household income;
- Financial sustainability – whether communities will continue to organise themselves after the financial support of the partner has ended; and
- Replication- to what extent was /is the programme successful in establishing models of interventions which could be used for other areas.
Assess any effects arising from an intervention. This includes immediate short-term outcomes as well as broader and longer–term effects. These can be positive or negative, planned or unplanned. What significant change can be observed and attributed to the program?
- Determine the overall impact of ICDKT Project on the quality of life of target indigenous communities.
- To what extent did the programme strengthen the committees in land, river and forests NRM?
- Are there any unexpected results arising from this programme and any ways of reducing the impact of any undesirable results?
- Crosscutting themes:
Gender, Equality / inclusion and participation
- To what extent did the programme promote equality between men and women? To what extent were the concerns of these groups addressed
- To what extent did the programme ensure equitable benefits to women benefit equally? To what extent has this been effective
- To what extent has the programme empowered the grassroots community committees to represent and advocate on NRM and other local issues with stakeholders
- To what extent has the programme influenced local NRM planning and decision making at district, provincial and national
- What was the level of collaboration between community and national networks?
Adding Value and Downward accountability
- To what extent does DPA engage with its ‘beneficiaries’, builds relationships, and is accountable for results in ways that enable learning and improvement towards the achievement of the programme?
- To what extent does funding partners engage with DPA, build this relationship and is accountable for results in ways that enable learning and improvement towards the achievement of the programme?
- To what extent did funding partners add value to DPA and programming?
 SCIAF Evaluation Guidelines, Version 2, July 2014. These include a requirement to ensure, where relevant that donor guidelines are reflected in the evaluation. In this case the funder is the Scottish Government and there are no particular requirements for evaluations.
 After this point ‘programme’ is used for both programmes and projects.