Developing a methodology for measuring the impact of a development network
Authors - Jerry Adams, Simon Batchelor, Kevin McKemey, Ian Wallace
1. Abstract
2. Introduction
2.1 Aim of the project
2.2 Acknowledgements
3. The Rationale for the Impact Assessment Project.
4. Some background from the literature
5. Conceptual framework
5.1 Background principles
5.1.1 A People-centred Approach
5.1.2 A Holistic approach
5.1.3 Joint Accountability and Learning
5.1.4 General Guiding Principles:
5.2 Limits to the Framework
6. Tearfunds Proposed Approach.
6.1 Method details
6.1.1 Qualitative approach
6.1.2 Use of Teams of Consultants.
6.1.3 Participation.
6.1.4 Indicators.
6.1.5 A Rating/Scoring System
6.1.6 Selection Methodology
6.1.7 Criteria of selection.
6.1.8 Alternative Approaches to Selection
6.1.9 Management
7. Conclusions
7.1 Expected Outputs from the Project
7.1.1 Methodology
7.1.2 Baseline Readings
7.1.3 Analysis of Current Impact
7.2 Final Remarks
Tearfund is a UK based Charity with a turnover of £30million in 1998 and over 460 predominantly South based Partners. Over the last few years it has undertaken a strategic review. As part of the conclusions of this review process it has emphasised Capacity Building (CB) and Impact Assessment (IA). In 1997 it commissioned a project to measure its own impact. The aim of the project is to measure the impact of Tearfund’s work in strengthening the capacity of its partners and, in so doing, addressing the problem of poverty. This paper describes the backdrop to this impact assessment, and the development of a methodology for such a large and complex task. The paper notes the difficulty of impact assessments, particularly the choice between quantitative and qualitative data. A methodology was needed which is rigorous and feasible, which can be carried out in a short period of time and which is not dependent on extensive programme monitoring systems. In this respect the methodology which Tearfund is in the process of developing is taking a middle course following the lessons learned by others and attempting to minimise any foreseeable weak points. The intention of the paper is to share this internal review with the wider development community in order to receive feedback and creative comment.
Over the last few years Tearfund has undertaken a major strategic review. The strategic review was a major process of participation and consultation of UK staff, external related organisations and a large number of partners overseas. The result of this consultation process has been the development of a 10 year strategy. Two features relevant to this paper came from this strategic review.
First the overseas department adopted Capacity Building as a major theme of its work. Tearfund currently works with 469 partners. The majority of these partners are based in the South and for over 30 years Tearfund has been supporting organisations that are grounded and rooted in the South. Over the years this partnership has ranged from a simple and "traditional" donor/implementor relationship (Tearfund supplying finance and resources to a South based organisation), to a complex and genuine partnership where South based organisations have influenced the planning and strategy of Tearfund. With the strategic review came the decision that Tearfund would consciously strengthen its role as Capacity Builder of South based organisations, and lessen its role as a donor, simply providing finance for projects.
Secondly, and linked to the above point, was the priority for the organisation to be able to make an assessment of its "Impact". It was clear that in order to develop appropriate and effective approaches to ‘Capacity Building’ it would be necessary to first know what is the impact of project support at the present time. The Impact Assessment Programme (IAP) described in this paper aims to provide Tearfund and its partners with an assessment of the impact of the Partnerships and Projects/Programmes that Tearfund is currently involved in. It also aims to develop tools that can be utilised for developing and refining strategy of both Tearfund and its partners, at both macro and micro levels.
The aim of the project is to measure the impact of Tearfund’s work in strengthening the capacity of its partners and, in so doing, addressing the problem of poverty. In order to do this the project must:
measure change which has occurred where there have been Tearfund inputs; and,
endeavour to establish a direct causal link between the change and the inputs (if that is possible).
A large number of people have been involved in this project. It is not possible to name each one, but this paper is presented with due acknowledgement to the considerable work carried out by all those involved.
3. The Rationale for the Impact Assessment Project.
It is clear that the ability to assess impact has become a priority issue not only for Tearfund but for the NGO community as a whole. NGOs need to be able to assess the impact of their work effectively. The unsubstantiated assumptions of the early 80’s that closeness to ‘beneficiaries’ meant that they would have an impact has increasingly come into question. Many NGO’s receive substantial funds from Government/ International donors. These donors are now asking about the impact of work that they have supported. NGO’s are also more critically aware themselves of the need to find out what their impact is, both for organisational learning and strategy development as well as to inform an increasingly aware and discerning public/supporter base.
In the book ‘NGO Performance and Accountability: Beyond the Magic Bullet’ (Edwards and Hulme 1995,) the authors saw the rise in popularity of NGOs and the increase in funding channelled through them by governments as having consequences in terms of performance and accountability.
In summary the reasons for NGOs, Tearfund included, to have to be able to assess the impact of their work seemed to be:
While reluctant to admit an organisational weakness, Tearfund, like many NGOs, had not, until recently, carried out regular planned impact assessments of programmes. It regularly carried out monitoring of activities and outputs, and audited financial trails, even carried out "evaluations" which had some elements of impact assessment. However there are few examples of a focused impact assessment of achievement of objectives and the consequences of those objectives. The biggest implication of this is that most of the projects supported do not have adequate and suitable data for impact assessment.
Again it seemed that Tearfund was not alone in this situation. The issue of the reliability of Impact Assessment information was raised by the OECD/DAC Study on Impact (Kruse et al 1997) where the lack of ‘firm and reliable evidence’ on the impact of NGO development projects and programmes was very much related to the ‘paucity of data and weakness of evaluation methodologies’. This weakness of baseline data influenced the strategy for Tearfund’s organisational impact assessment.
4. Some background from the literature
At present there appear to be two main approaches to impact assessment of development interventions. The first emphasises the need for data collection and analysis, sometimes through the use of semi scientific data collection tools such as Cost Effectiveness Analysis. Whilst recognising the limitations of the scientific/ positivist approach it still holds on to some elements of that perspective and tends to judge data collection and analysis in that light. Useful examples of this approach are the Country Studies carried out by ODI on Evaluating the Impact of NGO’s in Rural Poverty Alleviation (ODI Working Papers 37,49,50,51,52). The problem with this approach is its dependence on adequate sources of data. This is clearly highlighted in the ODI research project on assessing impact of NGO’s in poverty alleviation projects (Riddell 1990 & Robinson 1991) where the assessment team found it very difficult to use the tool of Cost Effectiveness Analysis because of a lack of baseline and monitoring data.
The second approach takes a more naturalistic perspective and locates impact assessment at a field level. It puts a high value on the involvement of local people in gathering and analysing data and negotiating its meaning and validity. An example of this is the paper by Kamara and Roche, (1996) and their work in Northern Ghana. A fundamental concern of this study was that it should not be extractive in nature, but should provide insights for the communities involved on the methodologies and findings. Another important element of the study, which reflects its commitment to the naturalistic approach was that it would not follow a pre-determined or rigid path but instead be very iterative. The proposal put much emphasis on the importance of ‘the communities perspective’ and that analysis of information or data should be seen and understood through their eyes. This is seen in the way in which the research tools of participant observation, semi structured interviews, focus groups and mapping structures were cross-checked by ‘respondent validation’.
Both of these approaches have strengths and weaknesses. The problem with the first is its dependence on adequate sources of data. The second approach, values the process of carrying out the impact assessment as well as the outcomes, but is very time consuming. The dilemma here is that a totally qualitative/relationship-based approach takes time. Even here NGOs using this approach are stressing the need for good baseline studies and monitoring and evaluation systems. This as an important area to address because the current approaches have serious limitations. They either stress the need for complex data collection tools (such as Cost Effectiveness Analysis) which are difficult to monitor and evaluate and require input from specialist researchers and do not pick up on qualitative/ relationship issues. Or the naturalistic approach which, though appropriate in its approach (addressing issues of ownership and involvement) requires a lot of time, effort and resources, as well as the aquisition of new and appropriate skills by evaluators.
In their recent book Oakley et al (1998) see that NGO’s are responding to the problems of carrying out effective impact assessments by developing more sophisticated project monitoring and evaluation systems. The weakness of this approach, as they point out in an eloquent quote from Rick Davies, is that such systems ‘seem to slide inevitably from "extensive attention" to the detail of setting them up, to modest concern for data generation, to less interest in their use and ultimately to a "minimal" interest in them as instruments of evaluation’ (Davies 1995).
If Project Monitoring and Evaluation (PME) systems alone cannot be relied on to provide good data for assessing impact then the issue must be, can the problems with evaluation methodologies and tools be addressed so that they can be used to analyse and assess impact? The OECD/DAC study is important in that it provides an analysis of a number of impact evaluations.
The problems highlighted are: weak (not competent) use of evaluation tools, unverified data (lack of rigorous analysis) and lack of data (quantity and quality). It is precisely for these reasons that Tearfunds Impact Assessment Project focuses on developing a methodology which will address these issues.
There is an additional issue in relying on having adequate PME systems in place in order to assess impact. This approach will, of necessity, exclude projects and partnerships where those systems are deemed inadequate (there is an additional issue here of who deems PME systems to be adequate/inadequate!). This could well affect their ability to be funded. It also forces one into either assuming that only projects and partnerships with good PME systems are likely to have an impact or that impact methodologies reliant on adequate PME systems are unable to assess the majority of development projects!
In assessing which data collection methods would be most appropriate for the development of a methodology Tearfund needed to take into account the need for methods which were replicable and could be used under field conditions; particularly NGO Field staff who are not research specialists. The approach used is primarily with qualitative methods. At the same time this poses problems, as can be seen by the Case Study of the Catholic Development Commissions work in Zimbabwe (Brand and Wakatama, 1992). The study makes a number of points:
Within Tearfund over the last few years, impact assessment has been looked at from various different contexts (organisational assessment, community health development, evaluation) and by other commissioned researchers (child-sensitive development, gender policy, HIV/AIDS). At the start of the Impact Assessment Study there was a need to bring these principles of good practice together in a consolidated framework, drawing also on the experience of external sources. This conceptual framework therefore provides the background principles upon which methods of assessing the impact of Tearfund’s work internationally can be developed.
The framework draws heavily on Tearfund’s Mission Statement, Operating Principles, Strategy and Values and the Impact Assessment document. The approach and methodology of the process will reflect Tearfund’s Values Statement
NGOs need to be able to assess their impact using methodologies and tools which are consistent with their ethos and approach.
5.1.1 A People-centred Approach
A people-centred approach to development lies at the heart of Tearfund’s values, therefore impact assessment is essentially about people. Impact assessment looks at changes in the lives of individuals, communities and wider society, and assesses how the community has changed in its attitudes and abilities to address its situation as a result of a particular intervention.
"Impact Assessment refers to an evaluation of how, and to what extent, development interventions cause sustainable changes in living conditions and behaviour of beneficiaries and the differential effects of these changes on women and men. Impact Assessment also refers to an evaluation of how, and the extent to which, development interventions influence the socio-economic and political situation in a society".
(Oxfam and Novib Research Programme on Impact Assessment - First Phase, 1995).
The differential effects of these changes can be extended beyond a simple distinction between men and women to include other distinct groups within the community (children, different ethnic groups, levels of poverty, etc.).
Tearfund’s approach to development is drawn from a holistic perspective. (Ref Operating Principles). Impact is therefore defined in holistic terms, with an emphasis on relationships. Social change is analysed from different perspectives (political, economic, social, spiritual, etc.), recognising that the whole is greater than the sum of the parts. Impact assessment goes beyond looking at aspects of change in isolation and attempts to assess change in relationships.
5.1.3 Joint Accountability and Learning
Tearfund’s ethos of evaluation is defined in the Impact Assessment document (Ref draft format). The approach is based on the principles of joint accountability and learning: "Evaluation of the impact of activities is called Impact Assessment. The purpose of Impact Assessment is to learn from the process to better inform future decisions by everyone involved." (Evaluation Ethos document).
5.1.4 General Guiding Principles:
The objectives of the Impact Assessment necessitate the following characteristics within the project methodology:
"Development is a complex process involving the interaction of economic, social, political and cultural processes, and the attempt to resolve human and power conflicts in seeking to gain access to resources."
(‘The Impact of NGO Development Projects’ ODI Briefing Paper (2), 1996).
Social processes are complex, dynamic and inter-related. At the levels of the individual, community and society relationships between people and their environment are complicated and constantly changing. Any attempt to study those relationships is limited and cannot take into account all the processes simultaneously. There are difficulties in defining terms such as "project" and "community": none of these is clear-cut, but rather forms part of an inter-related whole.
The framework is value-laden: what we choose to measure and what we choose to leave out reflects the underlying values of Tearfund as an organisation.
The study will only give an indication of what is happening in the light of Tearfund strategy. In order to understand social processes more thoroughly, in-depth studies would need to be carried out over a much longer-time period.
The study will not be able to "prove" cause and effect in anyway, or definitively attribute impact to Tearfund. The study will only be able to make an assessment based on interpretation of data.
Ensuring an adequate balance of qualitative and quantitative
Ensuring an integrated overview, not just segmented aspects
A balance of assessing inter-relationships, not just indicators
What level of detail in indicators: participatory approach suggests can’t have blueprint indicators
How far will this project be participatory in its approach? As can be seen in the Kamara and Roche study participation takes a lot of time and resources. It will be important to understand what levels of participation are possible. The Impact Assessment Project is concerned with ‘process’ (the way in which the project is carried out) as well as the specific outputs. Therefore participation and the involvement of different stakeholders is key. The participation of the Partner in the assessment is encouraged, though it is recognised that this is not easy. Participation can vary from full involvement as a member of the team, through partial involvement to limited involvement (at the beginning and end). Stephen Biggs (1989) defined the different levels of participation as follows:
Consultative, Co-operative, Collaborative and Collegiate.
Tearfund’s mission statement is concerned with holistic transformation, which includes not only the social and physical realms, but also the spiritual realm. This raises the particularly difficult issue of how far "spiritual impact" can or should be assessed.
6. Tearfunds Proposed Approach.
A methodology is needed which is rigorous and feasible, which can be carried out in a short period of time and which is not dependent on extensive programme monitoring systems. In this respect the methodology which Tearfund is developing will take a middle course between the two current approaches, where possible taking the best points from each and minimising on their weak points.
The IA Conceptual Framework (ref) describes two phases to the project. Phase 1 has two main objectives - firstly to develop the tools from which to be able to assess impact at a later date and secondly having devised these tools, to set down a baseline. Phase 2 of the project is to return to the baselines and make assessments of impact. It should therefore be noted that Tearfund will only be able to properly understand the extent of its impact in three/five years time during Phase 2 (though some insight into current impact can be gauged now).
This paper only describes the opening process of Phase 1. Four studies have been carried out and a workable but evolving methodology has been developed.
A qualitative approach through the use of qualitative assessment tools. The main tools to be used will be Focus Groups, Meeting Key Informants, Analysis of Secondary Data and Observation. The adoption of these qualitative tools is appropriate as the purpose of the whole assessment is to value and attempt to measure/understand the impact of Tearfund and its Partners, which is primarily through relationships. The concept of ‘Triangulation’ is applied through a mix of methods/tools, use of consultancy teams, multiple data sources. The purpose of this is to give an acceptable degree of objectivity to the subjective perspectives.
6.1.2 Use of Teams of Consultants.
In developing a methodology for the selection of Partners the issue of cost and feasibility is a key factor. At the project design stage it was estimated that an assessment visit would take between two and three weeks with the involvement of three external consultants. The methodology of using small teams of consultants builds on the ‘Sondeo’ Method described by Hildebrand (1979). For each assessment a small team of two - three consultants with different disciplines is formed. Tearfund divides its work into ‘Regions’ (eg South and Central America, West and Central Africa) Ideally one of the consultants will be from the Region that the assessment has been done. The team of 2/3 work together with local Partner staff in general. The ‘mini-teams’ can be changed around during the course of the assessment to provide fresh insights.
The strength of the small team approach is in the breadth of perspectives that are included. In order to make best use of the different perspectives it is important to spend time at the beginning briefing the whole team, refining the data collection plan and assigning specific responsibilities (dependent on skills and experience). It is also essential to ensure that there is adequate time at the end of the assessment for the team to analyse the information.
The Impact Assessment Project is concerned with ‘process’ as well as the specific outputs and participation is a key part of the process. It has been essential that partners understand the purpose of the project and do not see it as an ‘imposed’ evaluation of themselves. At the same time it is inevitable that some local and Partner/Donor issues will be brought up. These are discussed with the Partner at the debriefing/ roundup session and also with the Regional Team at Tearfund UK.
In July and August 1998 a set of indicators were developed. Since this is effectively an organisation wide assessment, the indicators were based on Tearfunds Mission Statement and Operational Principles (draft 1998).
The result was nine ‘Grouping’ or ‘Meta’ Indicators: Participation, Compassion, Empowerment, Organizational Competence, Advocacy, Change in Status, Social relationships, Spiritual and Sustainability. The meta indicators were then further refined into general indicators and specific question areas. A sample is given below.
Table of Social relationships.
| Gender Sensitivity |
| 1 Balance of women employees/ beneficiaries |
| 2 Women feel free to participate in decision making |
| Sensitivity (marginalised groups: elderly etc) |
| 1 Inclusion of marginalised in decision Making |
| 2 Reference to Children/Youth |
| Cultural Sensitivity |
| 1 Reference to and respect for local institutions |
| 2 Degree to which staff work in local idiom |
One of the difficulties with developing a methodology is the need for data to be compared over time and between cases. Where the researchers are the same this is less of an issue and can be dealt with by using a software textual analysis package such as Folio Views or QSR NUD*IST. As this methodology will be used by a number of different people we have developed a 5 point scoring or rating system to cross-check data. It is intended to be a support to the narrative data and to enable/assist in comparison across reports. It is not intended to replace or overshadow the qualitative perspectives and reports, nor to be used as a checklist of questions.
Scoring is done by each consultant for each sub indicator. The scoring method has been the subject of much discussion. Originally a simple system of a 1-5 scale was proposed (and used in Peru) where 1 = very weak, 2 = weak, 3 = average, 4 = strong, and 5 = very strong. Any sub indicators deemed to be not relevant are marked N/A.
There was some concern that, given that the indicators were to be used across different teams and over a number of years, the scale was too subjective, and for instance what was weak to one person might be average to another. To address this issue, midpoints were developed for each of the subindicators. These were developed and used in the Cambodia case. In this study the team that developed the midpoints were able to use them with no particular difficulties.
However, the team for India, who had not been involved in the indicator development and were predominantly technical consultants (water engineers used to quantitative data), found the indicators and midpoints difficult to use. Although comparison between the consultants showed they had a common understanding of the scoring system, and comparison with the narrative demonstrated that the scoring system had been functional, the difficulties raised caused some reassessment of the scoring system.
After exploring close definition of each sub indicator, giving a representation of points 1,3 & 5, it was decided that it was better to go to a more flexible system rather than a tighter defined system. Semantic ends are now being developed for use in subsequent baseline assessments.
The scoring system facilitates comparison. However the scores only make sense with the support of the narrative and regular ‘in situ’ debriefings so that the consultancy team can share impressions. At the end of the assessment the team pull their findings together to get an average score for each ‘meta’ and sub indicator.
At an early meeting with the Tearfund leadership it was stated that the study did not need to be statistically significant or based on random sampling (primarily because of the cost implications). The key point was that the study should provide information which could be used to develop a baseline and to develop and test the methodology. With this in mind a different approach was developed which took a qualitative perspective. The issue in adopting a purposeful sampling approach is credibility whereas in random sampling it is generalisability. Qualitative research attempts to persuade through rich depiction and strategic comparison across cases, thereby overcoming the abstraction inherent in qualitative studies (Firestone, 1987).
In selecting partners to study the aim was for an accurate three dimensional picture which enabled capture of the diversity of situation and modes of operation.
1. Length Of Partnership
With the importance that Tearfund puts on partnership and relationships it will be important to compare the impact of short and long term partnerships. Half of the partners selected will have been partners for more than five years and half for five years and under. Only current/active partnerships will be included.
2. Church Partners / NGO or Para Church Partners
As Tearfund works with both church partners and with NGO/Para Church Partners it is essential that this (potential difference) is assessed. The partnerships selected will represent that balance.
3. Sample of Urban and Rural Projects
Urban Issues are of increasing importance to Tearfund. A representative selection of the partnerships selected will have a focus on urban projects.
4. Geographical Diversity:
Each Region will be equally represented. With the exception of the UK Region which will have fewer. This takes into account the fact that the UK Region does not yet have partnerships of longer than five years.
5. Maximum Variation.
As this is not a random sample we propose to look at partnerships which show maximum variation or perceived ‘most’ and ‘least’ impact.
6.1.8 Alternative Approaches to Selection
In carrying out the first four assessments Tearfund became aware of the need to reassess the selection criteria that has been used and to look at the possibility of using different selection approaches. Using a ‘purposive’ approach gave some in depth information but it was increasingly felt by the Leadership that a statistically valid sample was needed so that the results are not so much a snapshot of Tearfunds performance but actually allow a defensible measure of Tearfunds overall impact.
The negative side of the current approach is that it is very widely spread. Each assessment visit is costing on average £9,000. Because the visits are to different areas the guiding team have to spend a lot of time and effort on setting up study teams, communication with the Partner and briefing and debriefing. With the development of the methodology it would be possible to use selection techniques which looked at a cluster of Partners. This would give advantages in that a small group of consultants could look at several projects (in a region). This would save on both time and costs.
At the time of writing, there is ongoing discussion of using the scoring system to develop a self assessment tool. Such a tool would be introduced to partners through regional workshops and validated using the external teams. The Cambodia team used self assessment as part of their study and found that the self scores were generally half a point higher than the external teams, but that there was a good parallel over the whole. This would seem to suggest that self scoring is a possibility. This will be investigated in the second phase of the project.
Management of the process is perhaps unique to the structure of Tearfund. However, since this paper is intended to share the realities of such an approach, the following diagram is given to communicate the complexity of managing such an impact assessment.
The methodology described above provides a detailed framework within which to carry out qualitative assessments. Because the framework is detailed it enables the researchers to hone in on relevant issues and areas and to focus questioning, data collection and analysis. The methodology is not meant to exclude unstructured or ‘serendipitous’ information being gathered. It is designed to provide a clear framework from which information can be ordered and analysed. In this respect it can be seen to be holding the middle ground between the scientific and naturalistic approaches mentioned earlier. It would appear to give good quality reliable information which can enable organisations and their partners to review and refine their approaches. Participation in this development phase has been limited to consultation and co-operation. This needs to be further assessed in the next phase to see if it is possible to be more collaborative.
It is unlikely that there will be a single methodology which will be appropriate for all situations and needs. The second phase of this project will concentrate on further refining the methodology for rapid in depth field visits. At the same time the development of a self assessment technique will be explored in order to give a broad overview of the ‘whole picture’ of Tearfund’s involvement.
7.1 Expected Outputs from the Project
There are three expected outputs from the study in 1998/1999: a workable methodology, baseline readings, and an idea of current impact.
In the Terms of Reference for the IA study, the aim of the project is stated "to start building a capacity within Tearfund to assess our impact internationally." Part of this capacity is a tested methodology and rigorous rationale which can become an accepted and usable tool for assessing impact.
This methodology has been developed through 4 pilot studies being carried out in 1998. In the light of the 4 studies, the methodology will be revised and consequently reviewed by the panel in early January 1999. It will then be used to take a baseline on 8 partners in early 1999, and following these visits, be reviewed again by the panel in May 1999. The present plan is then to undertake 10 baselines field visits in the financial year 1999/2000. By the end of this process it is expected that a fully useable methodology will have been developed. Depending on the findings of the 18 studies, and following recommendations from the panel and field visit teams, the methodology could be utilised in a number of ways:
There will be two categories of baseline information gained. A specific baseline for each partner, and a broad baseline picture across the regions.
Partner Baseline. The specific baselines will be fed-back to the Regional teams via a de-briefing with the field visit Team Leader. In addition each field visit team will produce a report containing a narrative and numerical analysis of the current situation. This will be presented along with the partner’s own numerical self-assessment, using the same indicators. Any other subsequent issues arising from the field visit will be dealt with through communication by the Project Leader with the Regional team.
Tearfund Baseline. Following all 12 visits, a summary of the baselines will be produced by the mini-team, and discussed with the client and panel. Whilst the numerical analysis will assist in highlighting certain information, the challenge will be to ensure that the narrative reports are given the main attention.
7.1.3 Analysis of Current Impact
Whilst the present field visits are to set a baseline, and the remit of the visiting teams is not to undertake retrospective assessment of impact, it may be expected that teams will gain a qualitative impression of Tearfund’s impact thus far. It would be a waste of their expertise if there was no scope for using this data as a positive tool for improvement of either Tearfund’s or a partners present impact.. There may be important themes emerging which should be discussed at the relevant levels.
The aim of the project is to measure the impact of Tearfund’s work in strengthening the capacity of its partners and, in so doing, addressing the problem of poverty. This paper has described the backdrop to this impact assessment, and the development of a methodology for such a large and complex task. We have noted the difficulty of impact assessments, particularly the choice between quantitative and qualitative data. The methodology which Tearfund is in the process of developing is taking a middle course taking the lessons learned by others and attempting to minimise any foreseeable weak points. The intention of the paper has been to share this internal review with the wider development community in order to receive feedback and creative comment.
References.
Adler, P.A. (1985). 'Wheeling and Dealing' An Ethnography of an Upper Level Drug Dealing and Smuggling Community. New York: Columbia University Press.
Alkire, S. (1997). Impact Assessment: Oxfam Vs Poverty . Oxford: Oxfam. 18pp.
Blankenberg, F. (1995). Methods of Impact Assessment Research Programme, Resource Pack and Discussion. Paper for the Case Studies Phase. The Hague: Oxfam UK/I & Novib
Bryman, A. (1988). Quantity and Quality in Social Research London: Routledge.
viii + 198pp.
Dawson, E. (1996). Assessing The Impact: NGOs, Women And Empowerment Oxford, OXFAM. 18pp.
_________ (1995) Phase III West Africa: Novib And Oxfam UK/I Impact Assessment Research Programme. Oxford: Oxfam. 42 pp.
_________ (1995) Women, Gender and Impact Assessment A Discussion Paper. Oxford: Oxfam. 18pp.
De Coninck, J. (1992). ODI Working Paper 51. Evaluating The Impact Of NGO’s In Rural Poverty Alleviation; Uganda Country Study London, ODI, 119pp.
Denzin, N. (1978). Sociological Methods A Sourcebook
Edwards, M and Hulme, D. (1995). Non-Governmental Organisations Performance and Accountability Beyond The Magic Bullet London, Earthscan.
____________ eds (1992). Making a Difference: NGO’s and Development in a Changing World. London: Earthscan/SCF
Feurstein, Dr M,T. (1986). Partners in Evaluation Evaluating Development and Community Programmes With Participants. London, Macmillan.
Fielding, N. & Fielding, J. (1986) Linking Data Qualitative Research Methods Series No: 4. London, SAGE.
Fowler, A. (1997). Striking a Balance: A Guide to Making NGO’s Effective in International Development. London: Earthscan/INTRAC
Goyder, H. (1995). New Approaches to Participatory Impact Assessment. London: Action Aid
Guba, E. & Lincoln, Y. (1981). Effective Evaluation London, SAGE.
Guba, E. and Lincoln, Y. (1989). Fourth Generation Evaluation London, SAGE.
Hammersley, M. (1989). The Dilemma of Qualitative Method Routledge.
Hammersley, M. (1992). Whats Wrong With Ethnography. Methodological Explorations
Hildebrand, P.E. (1979). Summary Of The Sondeo Methodology Used By ICTA IDS Conference Paper, IDS.
Hopkins, R. (1995). Impact Assessment Overview And Methods Of Application Oxfam (UK&I) and Novib.
_________ (1995). The Experience Of Latin American NGO’s. Oxfam (UK&I) And NOVIB Research Programme On Impact Assessment. The Hague: Novib, 39pp.
Hulme, D. (1997). Impact Assessment Methodologies For Microfinance: A Review Paper for Virtual Meeting of the CGAP Working Group on Impact Assessment Methodologies. Manchester University.
Kamara, S., and C Roche., (1996). A Participatory Impact Assessment And Advocacy Project For Poverty Alleviation In Northern Ghana. Paper Prepared for the 3rd International Workshop on the evaluation of social development. Ghana & Oxford: ISODEC and Oxfam, 15pp.
Krueger, R. (1988). Focus Groups: A Practical Guide For Applied Research SAGE.
Kruse, S-E. Kylonnen, T. Ojanpera, S. Riddell, R. Vieljus, J-L. Searching For Impact And Methods NGO Evaluation Synthesis Study Volume 1 Main Report, Volume Two Appendices A Report Prepared for The OECD/DAC Expert Group On Evaluation. Min. of foreign Affairs, Finland.
Marsden, D. Oakley, P and Pratt, B. (1994). Measuring the Process: guidelines for Evaluating Social Development INTRAC.
Mason, J. (1996). Qualitative Researching London, SAGE.
Maxwell, J. ((1996). Qualitative Research Design Applied Social Research Methods Series No 41, SAGE.
Morgan, D. (1988). Focus Groups as Qualitative Research Qualitative research Methods Series No 16. London, SAGE
Newens, M. & Roche, C. (1996). Evaluating Social Development: the experience of Oxfam Paper prepared for the 3rd International workshop on the evaluation of social development. Oxford, OXFAM. 17pp.
Patton, M.Q. (1982) Practical Evaluation London, SAGE.
Patton, M.Q. (1986). Utilisation Focussed Evaluation London, SAGE.
Patton, M.Q. (1990). Qualitative Evaluation and Research Methods London, SAGE.
Riddell, R. (1990). ODI Working Paper 37. Judging Success Evaluating NGO Approaches To Alleviating Poverty In Developing Countries London, ODI, 57pp.
Riddell, R. (1995). Non-governmental Organisations and Rural Poverty Alleviation ODI, London. Clarendon, Oxford.
Robinson, M. (1991). ODI Working Paper 49 Evaluating The Impact Of NGO’s In Rural Poverty Alleviation India; Country Study London, ODI, 131pp.
Ruxton, R. (1995). Participation In Impact Assessment Oxford: OXFAM UK/I. 23pp.
Silverman, D. (1993). Interpreting Qualitative Data SAGE.
Slim, H. & Thompson, P. (1993). Listening For A Change Oral Testimony And Development Panos, London.
Vivian, J. (1994). ‘NGOs and Sustainable Development in Zimbabwe: No Magic Bullets’. Development and Change, vol 25 pp 181-209
Wehkamp, A., and F blankenberg. (1995). Report of the Workshop to Introduce the
Research Programme on Methods of Impact Assessment at CYSD, Inida Bhubabeswar/Puri. The Hague: Novib, 24 pp.
____________ (1995). Report of the Workshop to Introduce the Research Programme on Methods of Impact Assessment at ENDA Zimbabwe. The Hague: Novib, 18 pp.
Yin, R. (1990) Case Study Research Design And Methods London, SAGE.
Zadek, S. and Evans, R. (1993). Social Audit. London: New Economics Foundation
Related Pages: Tearfund Organisational Assessment Briefing Papers | Putting impact assessment into practice | Tear Fund
For more information please contact: Webpage@gamos.org
© Copyright SJ Batchelor 2000
If you have reached this page via a search engine and are not using the framed
navigator, please click here.
If you cannot support frames, click here for an older
non-frames version of the site