Summary

Partners in Development Evaluation
Learning and Accountability - 25-26 March 2003

Summary by Daniel Kamelgarn
Head of the Development Evaluation Unit
Ministry of the Economy, Finance, and Industry

The workshop held in Paris on 25 and 26 March 2003 at the French Ministry of the Economy, Finance, and Industry has been arranged in collaboration with the DAC Working Party on Aid Evaluation (renamed since the Network on Development Evaluation). This workshop was organised following requests from non-DAC members to be included in the Network’s thought processes.

Approximately 300 evaluation managers and specialists from 43 different countries, including 18 from the developing world, gathered in Paris for two days. The 180 organisations thus represented encompassed the three types of partners that conduct such evaluations:

The donor community: bilateral aid agencies, development banks, and United Nations organisations;
Civil-society organisations: NGOs (non-governmental organisations), private foundations, trade unions and trade organisations;
Various components of aid beneficiary countries: authorities, civil society, etc.

The workshop allowed various evaluation professionals to meet, get to know each other, and discuss topics and concerns of common interest. Never before has this type of exchange taken place on such a broad scale: for that reason alone, the workshop was already a success. The high participation rate and the large number of countries in attendance are proof of the growing interest in aid evaluation. Specifically, the participation of approximately 50 representatives from emerging countries shows that evaluation has started to become an established practice in these countries and that efforts to enhance evaluation capacity are producing results. The massive participation of organisations from civil society and the quality of their presentations clearly demonstrates that the evaluation of aid has become common practice amongst such organisations. The sole objective in evaluating their activities is not – or is no longer – limited to meeting donor expectations; the studies undertaken by these organisations underscore their willingness to become more effective, efficient, and transparent in the eyes of the public.

As Francis Mer, French Minister of the Economy, Finance, and Industry, reiterated in his opening speech, the evaluation of aid must help improve the effectiveness of aid through accountability for the actions undertaken and through the lessons drawn, but must also foster conditions for increasing aid, given that, "The more we are all convinced that we have effective ways of implementing and measuring the impact of the aid we give, the more we shall have the courage and boldness to increase the amount we give.” This double objective can also be met by increasing an evaluation’s audience via the expansion of its participant base. The latter also provides a response both to the concerns expressed by aid stakeholders and their legitimate requests. As stated by Rob D. van den Berg, Chair of the DAC Network on Development Evaluation, increasing aid partnership presupposes increasing the partnership of those involved in aid evaluation.

Many topics were discussed during the four sessions of the two-day workshop.

1. Evaluation practices according to community

During the first session, donors and civil-society organisations presented their activities and practices. Each of the three donor communities already has a network through which its members discuss their various practices, fostering a certain degree of harmonisation. The chairs of the three networks (the DAC Network on Development Evaluation, the Evaluation Cooperation Group, and the United Nations Inter-Agency Working Group on Evaluation) drew up an account of their members’ activities, their current preoccupations, and their future directions. While civil-society organisations do not have such networks at their disposal on an international level, they have established consultative bodies in certain countries; accordingly, three presentations were given outlining the practices of these organisations in the United Kingdom, Norway, and France. During ensuing discussions, various international NGOs and representatives of beneficiary countries complemented the foregoing reports by giving accounts of their own practices. Networks of evaluation professionals are being established in beneficiary countries as well as internationally (in particular, via IDEAS – the International Development Evaluation Association); these can help improve the exchange and development of best practices.

It therefore appears that evaluation practices have become widespread within each community, with similarities and common questions as well as differences due to the specific objectives pursued in each category. For example, NGOs have more participatory practices and a greater interest, presumably, in the learning aspect of evaluation, whereas donors tend to focus more on the question of improving aid performance and effectiveness while emphasising the notion of accountability.

The morning session also highlighted the need for everyone to become more familiar with each other’s practices while avoiding the creation of uniform standards in terms of evaluation.

2. Current issues and challenges

During the second session, subgroups discussed various issues and current challenges in the areas of development and evaluation: participatory evaluation, evaluation of aid impact (especially at the social level), ways and means of internalising (or owning) evaluation results, and the evaluation of humanitarian or conflict-related assistance.

In terms of participatory evaluation, the workshop underscored the richness and profusion of NGO practices aimed at involving aid beneficiaries in the evaluation process. This type of evaluation presents advantages in terms of the four criteria that influence the quality of evaluation: access to the public, credibility, use of results, and efficiency. The discussions and presentations also revealed that participatory evaluations require a very structured approach and a high level of expertise in order to be successfully carried out. Another idea raised, and one worth retaining, is the notion that stakeholder participation is not compulsory at each step of the evaluation process. The participatory approach of the evaluation can focus on either step: on the design phase in some cases and on implementation in other situations. However, participation in the definition of the evaluation’s terms of reference is clearly essential.

The evaluation of aid impact, particularly on the social level, was approached via three case studies highlighting the methodological difficulties involved in defining this essential aspect of the evaluation process. Aid has an impact on all stakeholders in development and not just on the countries benefiting from the aid. As a result, it is difficult to establish indicators (a task that is currently the focus of many studies), and it is also essential to exercise caution when using such indicators. The work carried out by the DAC Network on performance-based management and indicator-selection criteria was of great interest to participants, although the thought processes regarding these aspects have not yet been concluded.

The ways and means of internalising evaluation results lie at the heart of the learning process, in terms of both the formulation of policies and the structure of organisations. The internalisation is facilitated when the evaluation process integrates this objective from the start. Three organisations presented their experiences, including a Filipino trade union organisation (supported by a Danish NGO) that experienced growth through integration of the lessons of the evaluation; an international NGO that modified its management structure by adding the notion of learning; and a British fund fighting to decrease the exclusion of the poor. Each organisation demonstrated how evaluation could become a powerful lever for changing and improving practices provided that the internalisation and circulation of results is undertaken properly.

Other pertinent ideas were voiced:
a) internal evaluation and self-evaluation can be a preparation for external evaluation and a means of making it constructive;
b) evaluation must not obscure the notion that development is a process and cannot be assessed exclusively on project results;
c) learning should be capitalised in a systematic and uniform manner so as to ensure that it is an individual as well as a collective endeavour; and
d) evaluators should be selected with an eye to their skills in mediation and negotiation.

Evaluation of humanitarian assistance was developed in the early 1990s. Assessments of evaluations carried out and presented by ALNAP, as well as the debates that ensued, demonstrate that this type of assistance requires evaluation policies that differ from those required by development projects. The question of what took place often seems to take precedence over an examination of the reasons for success or failure.

3. Accountability

The third session concentrated on one of the main functions of evaluation: accountability. From bilateral and multilateral donors to NGOs, all organisations must contend with the demand for accountability, although this requirement does not always appear in the same form and the responses to it are various. Moreover, evaluation is not the only way to satisfy this requirement: charitable organisations, which are accountable to their donors, have created charters and criteria that focus on performance and transparency (particularly in the financial and commercial areas), whereas in public administrations like that of France, parliamentary control can be exercised over aid activities.

UNAIDS has established an impressive framework for monitoring evaluations: 189 countries have signed a joint agreement for the monitoring of actions launched at the international level. This framework is based on a limited number of readily accessible indicators that can be applied to the different steps of the process and which meet monitoring requirements in both global and national terms.

In addition, Robert Picciotto, Director of the Global Policy Project, reminded us that providing accountability entails an examination of all public policies and their impact on poor countries. Accordingly, development assistance (as well as other policies in developed countries, such as trade policies) should not simply be accountable to sponsors from the developed world, but should bear an even greater responsibility toward aid beneficiaries. The terms for exercising this level of monitoring have yet to be established.

The accountability of NGOs to their donors raises specific questions. The Australian experience is particularly thought provoking, as it relies on the establishment of indicators selected and discussed jointly by the NGOs and the government agency concerned. However, if NGOs are to account for the manner in which they conduct their actions, they must be allowed to exercise this accountability in their own manner, without losing their autonomy.

4. Enhancing collaboration and evaluation capacity

The fourth and final session concluded the workshop with an examination of ways in which to enhance collaboration between the various evaluation communities and to enhance evaluation capacity.

Evaluation capacity has undeniably improved in beneficiary countries during the last few years through the establishment of networks such as those supported by UNICEF. It has also developed within both large and small NGOs in developed countries through assistance and support funds, such as the F3E in France.

The most striking aspects of collaboration involve the range of experiences represented by the various development stakeholders and the pressing insistence of aid beneficiaries and civil-society organisations to be considered full partners by donors – in short, their demand for reciprocity and partnership. As such, joint evaluations should take into account the preoccupations of all those involved rather than remaining exercises to which partner countries are merely invited. As Professor Chen Zhaoying has pointed out, the current perspective remains that of the donor, and much work remains to be done before evaluations are genuinely perceived as representing the viewpoint of both donor and recipient.

As Jean-Michel Severino, Director-General of the French Development Agency, has stressed, evaluation is not a technocratic process but rather a social one, though the social engineering of this process remains to be perfected. In particular, so-called joint evaluations must be carefully rethought in terms of their methodology if they are to become true exercises in partnership. It might also be said – to echo the words of Rob D. van den Berg – that the time has come for moving from the evaluation of aid to the evaluation of development.

* *

Certain workshop topics were only lightly touched upon, giving rise to the need for further discussion. This first meeting has created expectations and generated new ideas, however, and in that respect may represent a new step in the evaluation of aid and development.

What was most striking during these sessions was the shared desire for dialogue and an awareness of the broad range of experience yet to be fully drawn upon – the recent yet significant development of evaluation practices in NGOs and the progress achieved in this domain by aid beneficiary countries, for instance.

Secondly, an expectation has arisen regarding the implementation of collaborative, equal-terms relationships amongst the various communities. There were several NGO representatives who expressed their appreciation at being treated as full-fledged partners during the workshop, a degree of respect that in turn guaranteed the event’s success. This development can also be attributed to the level of maturity that these organisations have attained, progress that renders such parity both possible and essential. From the donor viewpoint, the main lesson to be drawn from the meeting is that it is necessary for donors to overhaul their approach to the evaluation process by promoting and developing cooperation with the other institutions involved in development assistance as well as with the beneficiaries themselves. Two terms come to mind that might guide such overhaul: partnership and reciprocity. Such changes are expected, fundamental and necessary.

Finally, although the two main objectives of evaluation are accountability and learning, it is important to recall that evaluation of aid practices also helps promote development within our own societies.