I’ve been a little concerned that the Direct Support to Farmers (DSF) Evaluation that JMD is currently conducting In Takengon was relying too much on fairly qualitative data--that is, anecdotal evidence, stories, personal opinions, etc. in determining whether the project was (and remains) useful. Especially with the majority of publicized reports coming out of large NGOs these days that rely on statistics and charts and averages (ie quantitative data) that, while useful, really don’t get at individual realities, especially in such a wild and data-illusory place as Aceh. JMD staff have been receiving some guidance on how to implement a good evaluation, and our field officers live in the areas in which they work, and are continuously gathering data, compiling case studies, and making individual plans that further a group’s objectives by helping each farmer work on her own challenges.
So it was music to my ears to discover a guy named Michael Quinn Patton, Founder and Director of Utilization-Focused Evaluation. [“Utilization-Focused Evaluation (“U-FE) begins with the premise that evaluations should be judged by their utility and actual use; therefore, evaluators should facilitate the evaluation process and design any evaluation with careful consideration of how everything that is done, from beginning to end, will affect use.”]
His book Qualitative Research & Evaluation Methods underscores the value of in-depth inquiries, long-term studies as opposed to brief questionnaires and quick visits, and the inclusion of the participants/beneficiaries as co-evaluators.
“If you know how to listen, systematically collect, and rigorously analyze anecdotes, the patterns revealed are windows into what’s going on in the world. It’s true that to the untrained ear, an anecdote is just a casual story, perhaps amusing, perhaps not. But to the professionally trained and attuned ear, an anecdote is scientific data—a note in a symphony of human experience.”
This is all on a really great site called, cleverly, Better Evaluation (www.betterevaluation.org).
I visited Will Allen’s blog on the site called “Addressing sustainability in evaluation,” and found this rather familiar question: “When we talk about a “sustainable future” what do we mean?”
Sustainability . . . is starting to be thought of as not only about the environment, but about human rights. These rights include ensuring people have access to food, water, health, security and that they have a voice in their decision making. From this perspective we need to place a real focus on those people whose voice is often missed out or marginalized – taking particular care to ensure the input of societal sectors such as Indigenous peoples, and women. In a complex world we are not just looking at setting one path to the future, but working inclusively with a range of stakeholders to ensure a diversity of development pathways to safeguard both the environment and human rights.
Another blog, by Deborah O’Connell, discusses her 5-year research on biofuels (ring a bell?) and a report she prepared for the World Economic Forum in 2013.
She writes, “Exploring the concept of ‘sustainable biofuels ‘ required gaining an international perspective on not just the food versus fuel dimension of sustainability, but a range of different sustainability aspects, including the carbon footprint, the water use, the clearing of forests and impacts on biodiversity, and the social issues for direct and indirect effects.
What she found, like we all have, is that sustainability is a mercurial little bugger, and “Even with widely accepted definitions for sustainability and sustainable development, the central questions of ‘sustaining what, for whom, where, and for how long?’ remain laden with human values and social choices.”
She outlines, as many do, an outline for a successful and useful evaluation of sustainability—not just projects like ours but ingrained ways of economic life like palm oil production (yeoowww), and while the prose gets rather technical for me (“cumulative-longitudinal; integration at the case an context levels?” Excuse me while I cut off a finger) I agree with nearly everything she says . . . and then down she goes, into the bureaucrat’s trap (I’m assuming she wrote the entire blog, so I’m giving he credit for it).
Meeting sustainability goals is one of the most important and urgent challenges for humanity yet for many of us, thinking about sustainability in evaluation means assessing the extent to which the benefits of a project, programme or policy are maintained after formal support has ended.
Well . . . yeah. I’m not understanding the “yet” bit—as if making sure the project continues after formal support ends is wrong-headed.
Another view of sustainability, she says, “will help evaluators and commissioners of evaluation to consider the impact of the project (or other forms of intervention) on ‘development that meets the needs of the present without compromising the ability of future generations to meet their own needs’ (World Commission on Environment and Development 1987).”
See this is where I always have a problem. If she had only said she did not want to compromise the ability and the desire of future generations to meet their own needs” in the manner established by the project or program, I’d agree.
Projects rarely ask if is this a tradition or a cultural identity that beneficiaries would like to preserve, or that they would like their children to preserve. Frankly, if I were a kid in Aceh I’d want to go to college and get as far away from the coffee farm as I could. So it would be helpful to find out if this is the case. What of sustainability then? How, really, are you gonna keep ‘em down on the farm? Or in the factory, or on the shrimp boat?
For answers to those questions, I’m going back to Michael Patton, because it is, really, all about knowing how to listen.