2007 Program

Addresses
Presentations
Workshops
Download conference schedule (PDF: 180 KB)

Addresses

Opening keynote speech

Getting the measure of evaluation's impact
Sue Soal

It is not uncommon to hear donors remarking that they have x-thousand pages worth of evaluation in their files; and that this has cost them x-million dollars, but they wonder what the value of the exercises has been. We do know that conventional, outcomes based evaluation does generate a great deal of learning for the evaluators themselves, consolidating their expertise and marketability in the evaluation industry. But what is the contribution of evaluation to the learning of those who are evaluated? Where evaluation dwells on mistakes or deviations from the plan, it comes to play an audit role, promoted on grounds of its contribution to ensuring accountability. But in that environment, learning is not encouraged. In fact, just as mistakes are driven underground, (scuppering the quest for accountability) so too is learning, sometimes disappearing altogether. In order to show compliance and to be given a positive (or just neutral) evaluation outcome, development practice resorts to being the formulaic implementation of given plans. Facilitation of living process, intelligent responses to emerging situations, lively engagment with successes and failures - all of these take a back-seat. In this way, evaluation becomes instrumental in compromising the very good that the development agenda claims it is pursuing.

Plenary addresses

The effectiveness agenda: AusAID’s emerging response
Nic Notarpietro

This address will:

1. Review the Aid White Paper’s rationale and framework for strengthening the performance orientation of Australia’s aid program. This will include commitments to the Millenium Development Goals (MDGs), building partner countries’ capacity to Manage for Development Results, and to improve donor alignment and harmonisation (the Paris Declaration).
2. Describe how AusAID is responding through new organisational arrangements, performance assessment systems and better ‘lesson learning’ at both strategy and activity level. Emerging challenges facing AusAID in implementing the new performance agenda, as well as implications for partners and implementing agencies will also be discussed.
3. Consider, from a donor’s perspective, how communities and civil society (including both Australian and local NGOs) might contribute most constructively to the effectiveness agenda, and provide some examples of where this is already happening in practice.

The role of quantitative measurement in development effectiveness
Associate Professor Brett Inder

This talk will be based around case studies on the role of quantitative research in measuring development effectiveness. Examples will build around the broad theme of communities and development, and will illustrate the relative merits of quantitative research compared to qualitative research. We will look at the potential for using official databases, as well as some issues around primary data collection, and at commonly used specific measures of effectiveness and performance.

Does effectiveness make any difference anyway?
Chris Roche

These closing remarks at the End of Day 1 will explore some of the tactics that individuals and organisations use to resist the findings of reviews and evaluations - no matter how methodologically sound they may be - and therefore pose the question about what can be done to overcome this. It will be suggested that this has implications for both how we see the process of evaluation and learning as well as organisational context within which it takes place.

Experience vs expertise in evaluating community development
Professor Dennis Altman

This presentation will pose the question whether there is an inevitable tension between technical and community-based forms of measuring or assessing effectiveness, and how such tensions are related to questions of expertise and power. It draws largely on HIV/AIDS, as a sector where these debates are particularly current, and makes the following assertions:

1. We should question the assumptions that there is a gap between "scientific" and "community" knowledge;
2. Even if we get the methodologies that "work" their implementation depends upon political, social and cultural factors;
3. Genuine M&E demands empowerment of those at risk, and a reminder of the original links between human rights, public health and community development: this returns us to the Alma Ata declaration;
4. Therefore we need end practice of talking about 'M&E science' in isolation, and develop a far more inclusive definition of what such a 'science' might entail

Measuring effectiveness: A network perspective
Dr Rick Davies

This session will explore different approaches to describing and evaluating portfolios of relationships, and the networks of relationships they are part of. This will be done through the use of methods adapted from social network analysis, and card sorting methods. Along with a brief introduction to some useful methods Rick will also highlight the wider potential and challenges of taking a network perspective on development, where no one actor is in control and the structures that emerge can be seen as a form of collective strategy that may or may not be as effective as it could be.

Community cultural development: Effectiveness as tool for personal and social change
Elizabeth Cham

The session will tell the story of how initial funding for a small, diverse group of not-for-profit organizations led to important changes in Federal and State policy and practice. It will also critically examine the way evaluation is used by donors and nonprofits. Evaluation is critical in testing effectiveness. Research shows that non-profits have insufficient time, funding and expertise to evaluate their work and that they often end up collecting lots of dubious data at great cost to themselves and ultimately to the people they serve.

Understanding effectiveness from the perspective of NGOs working in Cambodia and what this can tell us about the challenges of measuring effectiveness.
Rhonda Chapman

Following the original research work on NGO effectiveness done by ACFID, it was agreed that understanding effectiveness as experienced and defined by the partners of Australian NGOs was an important next step in understanding effectiveness more broadly. This paper presents the preliminary results of PhD research with NGOs in Cambodia and what their practice and experience can tell us about effectiveness at the field level. These emerging findings indicate a sophisticated and intuitive understanding of effectiveness at the community level but one that is difficult to document, measure and report in the current aid structure, systems and processes within which NGOs work. It also sheds some light on the synergies between organisational values, structure and systems and their impact on effectiveness – a theme which will be explored in a later workshop and looking at the implications for Australian NGOs.

Presentations

Assessing organisational performance in promoting community development: A case study from the Peruvian Amazon
Dr Karla Wesley

In the 1990s, the World Wildlife Fund-Denmark and The Nature Conservancy implemented one of the most comprehensive community conservation and development initiatives in Latin America. Working with a population of approximately 100,000 residents spread across 5.1 million hectares, their strategies were driven by distinctively different agendas: one favoured humanitarian assistance and development, while the other promoted a more traditional protectionist conservation approach. From 1999-2004, this research assessed the issues and challenges of 10 years of community development initiatives in the Pacaya Samiria National Reserve. This paper will present insights into assessing large projects in remote rural areas, from macro and micro levels of analysis and offer strategies for overcoming key stumbling blocks to this process.

The value chain development model: Helping Armenian communities redevelop agriculture in a post-Soviet context
Jeffrey E. Engels

Value chain development unites communities through the creation of Forward and backward linkages that generate jobs, increase incomes, and grow small and medium enterprises (SMEs) in developing countries. With strategic, coordinated interventions, value chain-building joins stakeholder communities in a participatory process that puts them at the centre of their own development and contributes to aid effectiveness.

Recent research conducted in the post-Soviet republic of Armenia demonstrates the importance of value chains in private sector development to alleviate poverty and create a viable free-market economy. This paper will offer lessons learnt and insights into measuring effectiveness along the value chain using the Center for Agribusiness & Rural Development/Armenia (CARD) experience as a case study.

How has life changed? Did we make a difference? A case study from Afghanistan
Dr Deborah Storie

This case study is of an evaluation of a rural community development project in Northern Afghanistan. The evaluation process demonstrated some of the challenges of facilitating change in complex conflict situations exacerbated by adverse weather patterns and significant local, national and international disparities in power and vulnerability. One particularly disturbing finding was that the development team's commitment to achieve 'measurable results' sometimes led to interventions that were welcomed by relatively powerful community members but that had demonstrably adverse consequences for the most vulnerable community members.

Learning about learning: Critical to improving development outcomes
Dr Juliet Willetts, Dr Paul Crawford, Associate Professor Cynthia Mitchell

The term learning appears widely in the development sector. It is often used to describe one of the core aims of a monitoring and evaluation (M&E) system, and it is often an explicitly defined desirable outcome of interaction with communities. In fact implicit in the word ‘development’ is the idea of change that takes place through some sort of learning process. However the term ‘learning’ in the development field appears to be understood in simplistic terms. It is often equated with ‘provision of information’ to different actors and is least likely to lead to the kind of lasting change sought by the development sector. Deep learning is the only kind that gives lasting change.

This presentation will outline different levels of learning and propose ways to integrate this thinking into design of M&E systems and project design processes. Such ‘learning about learning’ is likely to enable more meaningful participation in project and M&E processes, and ultimately enable greater change in desirable directions in development practice.

How performance and program effectiveness is managed within countries
Chris Nelson

This presentation will look at how AusAID has developed its new approach to country strategies, performance frameworks and agency level M&E. The recent publication of the White Paper has promulgated the Office for Development Effectiveness as the agency monitoring unit. It has also encouraged AusAID country programs to take a more integrated approach to planning and performance and has led to changes in quality assurance systems. The presentation will provide an overview of how these processes have been managed in the Philippines program and what this means for the future. The emphasis will be on the pragmatic aspects of rolling out institutional change to programs operating through a larger agency structure.

How can NGOs be effective in achieving the MDGs?
Dr Matthew Clarke and Dr Simon Feeney

The international community, NGOs included, has identified eight Millennium Development Goals (MDGs) by which it will assess its success in improving the lives of the poor by 2015. Over 85 percent of NGOs are involved in activities that are aimed at promoting or achieving the MDGs. These activities may include programming interventions or advocacy activities. To achieve the MDGs, development interventions must occur at different levels of society. NGOs have the greatest capacity to impact the achievement of MDGs through programming at micro- and meso-levels and through advocacy at macro- and supramacro-levels. This paper will focus specifically on how NGOs can impact the achievement of the MDGs in both their programming and advocacy interventions, as well as providing some brief case studies to illustrate this impact. Analysis of their efficacy, efficiency and effectiveness will be discussed before efforts to increase their impact are highlighted and recommendations are made.

Staffing as a factor in effectiveness – further reflections
Dr Francois Tsafack

Effectiveness is not something that happens or can be achieved in isolation to all other things that take place within and around a program or project life cycle. Effectiveness is not limited to one factor but a range of factors including adequate project level staffing that have qualifications, technical skills and experience; a work culture that values and promotes effectiveness and work employment conditions that encourages and facilitates effective programming. Financial resources and good DM&E, among others, are very important but the value of high quality project level staff cannot be overstated. Before we ask whether we are effective in our community interventions, we should first ask whether we are providing a suitable programming environment within and around which effectiveness can be created, nurtured, maintained and promoted.

Measuring gender equality results
Kate Nethercott

Indicators for measuring gender equality remain poorly utilised, making it difficult to know if efforts are on track to achieve gender equality. There are inherent risks from not adopting a gender perspective in monitoring and evaluation - we will not know whether both women and men have benefited equitably from expanding aid budgets or whether our aid interventions have in fact exacerbated existing inequalities. Development results can be maximised only with adequate attention to the different needs, interests, priorities and roles of women and men. Paying attention to these issues through development activities, particularly in monitoring and evaluation, will improve the quality of development initiatives and the sustainability of results.

Being effective: Implications for Australian NGOs from the research to date.
Dr Linda Kelly and Rhonda Chapman

Since the original research work undertaken by ACFID in 2003 there has continued to be a focus among many NGOs on the area of effectiveness and quality. Some review of this work shows that a number of Australian NGOs have been progressing in their focus on more effective practice. New challenges are emerging however. The emerging research from the field suggest that NGOs still have much to learn from their partners. New consideration of organizational forms suggest that NGOs have to work harder to consider their structure and approach in order to develop a more effective approach to development work. This workshop will briefly outline the history and evidence so far in Australian NGO work on effectiveness. It will review some of the new emerging thinking among some NGOs, including those that are trying to change their very ways of doing business. It will then pose a series of questions for NGOs about how they might undertake their development work and to assess that work from the perspective of effectiveness.

Workshops

NGOs as learning organisations
Sue Soal

This workshop session is being held in conjunction with the Development Practice Advisory Committee of ACFID. It will contrast and compare systems and thinking developed by the Community Development Resource Association (CDRA) in South Africa with ideas developed amongst Australian NGOs.

Social Networks and Most Significant Change
Dr Rick Davies

This session will follow on from ideas raised in Rick’s address and allow participants to explore the ideas of social networking and MSC in more depth.

Active citizenship and accountability: The Humanitarian Accountability Project (HAP)
Joshua Pepall

This workshop will be run in two sections: first, a presentation on issues arising from the 2004 Asian Tsunami; and second, the demonstration of a range of practise skills and field benefits associated with increased community engagement in project management. This second section will be based on World Vision's tsunami response work in Sri Lanka, the first World Vision project to be accredited as HAP-I compliant.

Ethics and international NGOs
Chris Roche, Dr Keith Horton and Dr Linda Kelly

This workshop will explore some ideas emerging from a recent meeting in Australia between a number of moral philosophers interested in ethics and some International NGO staff. Some of the questions we may explore include:

1. Is what NGOs say about themselves and their achievements consistent with what they know to be true? Is there a need for NGOs to be more honest about their achievements and more humble about their potential to make a major difference?.
2. Is the growing corporatisation, growth and associated risk management which have become more of a feature of International NGOs leading to a shift in values and stifling creativity and innovation?
3. The work NGOs do is complex, sometimes fails and occasionally does harm to those they ultimately seek to benefit. It is also clear that what might work in one place may well not work elsewhere. At the same time there is a growing internal and external demand for NGOs to be able to demonstrate their achievements in simple, straightforward ways. How do NGOs best manage this tension?
4. No one NGO can do it all. Also NGOs acting seperately can individually do good but collectively do harm i.e. the presence of NGOs in refugee camps around Rwanda allowing rebels to regroup. What makes for a sensible division of labour between NGOs and when is it important to collaborate more?