THE APPROACH

This guide presents an evaluation framework that assesses the contribution to recovery associated with post-disaster interventions. It is a method focused on assessing positive and negative changes to the lives of affected people and other local stakeholders, in the medium term following a disaster event. The output of the approach is a report that presents detailed findings, an in-depth analysis drawing on the findings, and a concluding section that discusses the contribution to change generated by post-disaster interventions – including a series of summary statements.

The approach assumes that changes in people's well-being and livelihoods can be most clearly identified at a household level. Assessing change necessarily involves identifying what the situation was like for households before and after the disaster occurred, as well as the situation following a period of post-disaster recovery and intervention. We present a technique for collecting retrospective data to cover these changes, combining quantitative and qualitative data collection and analysis.

The approach requires similar levels of staffing and resources as many other types of evaluation. However, although the methodology is structured, it requires key skills in design, field data collection, and interpretative analysis that may not always be available in agency field teams. It may often be advisable to commission a specialist third party to undertake the evaluation: this approach is well suited to a group of agencies acting together to commission the work. Note that the Contribution to Change methodology is designed to complement other evaluation tools; it is not a substitute for process evaluations or audits aimed at examining whether specific project inputs and outputs have been met.

Emphasis on evaluating 'contribution'

This guide uses the concept of Contribution to Change to describe the relative importance of post-disaster interventions in aiding people's recovery. This is distinct from a focus on 'attribution', which seeks to establish what specific changes have resulted from an agency's intervention.

This is because the activities of an individual agency, and the effects of those activities, will not normally occur in isolation but rather as part of a multilayered, complex response by both local and external actors (see 'Example from the field 1'). Social, economic, and political contexts, including the effects of international markets, access to communications, past or present conflicts, and environmental factors, also have a bearing on the final outcomes for a given population following an emergency. Thus it is more realistic in such settings to consider 'contribution' to outcomes or, as some authors prefer, 'contributory impact'.

The Contribution to Change approach views recovery as depicted in Figure 1. The idea is to try to understand the relative importance of the intervention activities depicted by the bottom arrow (normally this refers to the range of interventions in a given site, though in some instances the method may be applied to a single intervention).

Another important aspect of this approach is the recognition that it is not sufficient to look only at outcomes that have been achieved. An intervention may have contributed in a major way to the change that has occurred, but if that level of change has itself been only limited, there is a danger that the value of the contribution will be exaggerated.

We therefore also need to look at the overall progress of recovery in order to calibrate the contribution that interventions have made to the extent of recovery that has actually been achieved. Both are important aspects of assessing contribution to change.

Concentrating on just the 'intervention' part of the previous diagram, we can depict this distinction as shown in Figure 2.

Assessing contribution is not an easy task. Although we present a systematic methodology for undertaking this type of evaluation, the end results of the analysis require careful interpretation of the evidence collected. We need to look at the effects of the intervention, consider other responses and contributions, and be aware of the effect of external factors and issues in shaping recovery. All these elements need to be included in the analysis and its conclusions presented in the report.

The most important output of the Contribution to Change methodology is a detailed discussion of how these elements have shaped key aspects of people's lives. This will be reflected in what we call the narrative analysis (see Part Three, Step 9) and the conclusions (Step 10), which are set out in the final report (Step 11).

However, one way to provide a shorthand interpretation of contribution is to create Contribution to Change statements (the tool for which is provided in Step 10). This tool generates simple statements about level of recovery (the overall progress of recovery) and contribution to recovery achieved (the changes that can be linked to interventions), and compares these to create a Contribution to Change statement (the contribution of the intervention towards meeting the recovery that is required).

Other defining elements of the methodology

Focus on livelihood changes at household level

The methodology is rooted in the idea of undertaking analysis at the grass-roots scale in order to reveal the most significant changes in people's lives associated with a disaster and the subsequent 'recovery' period. Recent reviews of evaluation in post-disaster settings argue that it is imperative to ensure that assessing the effects of interventions on the 'lives of affected populations' is at the heart of evaluation (Proudlock et al., 2009).

The intention of the approach is to take a holistic or multi-dimensional view of people's lives – one that looks broadly at different aspects of people's well-being. It is felt that a more holistic evaluation of the changes in people's lives is needed as all too...