I’ve been
a little concerned that the Direct Support to Farmers (DSF) Evaluation that JMD
is currently conducting In Takengon was relying too much on fairly qualitative
data--that is, anecdotal evidence, stories, personal opinions, etc. in
determining whether the project was (and remains) useful. Especially with the majority of publicized
reports coming out of large NGOs these days that rely on statistics and charts
and averages (ie quantitative data) that, while useful, really don’t get at
individual realities, especially in such a wild and data-illusory place as
Aceh. JMD staff have been receiving some guidance on how to implement a good
evaluation, and our field officers live in the areas in which they work, and
are continuously gathering data, compiling case studies, and making individual
plans that further a group’s objectives by helping each farmer work on her own
challenges.
So it was
music to my ears to discover a guy named Michael Quinn Patton, Founder and
Director of Utilization-Focused Evaluation. [“Utilization-Focused Evaluation
(“U-FE) begins with the premise that evaluations should be judged by their
utility and actual use; therefore, evaluators should facilitate the evaluation
process and design any evaluation with careful consideration of how everything
that is done, from beginning to end, will affect use.”]
His book Qualitative
Research & Evaluation Methods underscores the value of in-depth inquiries,
long-term studies as opposed to brief questionnaires and quick visits, and the
inclusion of the participants/beneficiaries as co-evaluators.
“If you
know how to listen, systematically collect, and rigorously analyze anecdotes,
the patterns revealed are windows into what’s going on in the world. It’s true
that to the untrained ear, an anecdote is just a casual story, perhaps amusing,
perhaps not. But to the professionally trained and attuned ear, an anecdote is
scientific data—a note in a symphony of human experience.”
I visited
Will Allen’s blog on the site called “Addressing sustainability in evaluation,”
and found this rather familiar question: “When we talk about a
“sustainable future” what do we mean?”
Sustainability
. . . is starting to be thought of as
not only about the environment, but about human rights. These rights
include ensuring people have access to food, water, health, security and that
they have a voice in their decision making. From this perspective we need to
place a real focus on those people whose voice is often missed out or
marginalized – taking particular care to ensure the input of societal sectors
such as Indigenous peoples, and women. In a complex world we are not just
looking at setting one path to the future, but working inclusively with a range
of stakeholders to ensure a diversity of development pathways to safeguard both
the environment and human rights.
Another
blog, by Deborah O’Connell, discusses her 5-year research on biofuels (ring a
bell?) and a report she prepared for the World Economic Forum in 2013.
She
writes, “Exploring the concept of ‘sustainable biofuels ‘ required gaining an
international perspective on not just the food versus fuel dimension of
sustainability, but a range of different sustainability aspects, including the
carbon footprint, the water use, the clearing of forests and impacts on
biodiversity, and the social issues for direct and indirect effects.
What she
found, like we all have, is that sustainability is a mercurial little bugger,
and “Even with widely accepted definitions for sustainability and sustainable
development, the central questions of ‘sustaining what, for whom, where, and
for how long?’ remain laden with human values and social choices.”
She
outlines, as many do, an outline for a successful and useful evaluation of
sustainability—not just projects like ours but ingrained ways of economic life
like palm oil production (yeoowww), and while the prose gets rather technical
for me (“cumulative-longitudinal; integration at the case an context
levels?” Excuse me while I cut off a
finger) I agree with nearly everything she says . . . and then down she goes,
into the bureaucrat’s trap (I’m assuming she wrote the entire blog, so I’m
giving he credit for it).
Meeting
sustainability goals is one of the most important and urgent challenges for
humanity yet for many of
us, thinking about sustainability in evaluation means assessing the
extent to which the benefits of a project, programme or policy are maintained
after formal support has ended.
Well . .
. yeah. I’m not understanding the “yet” bit—as if making sure the project
continues after formal support ends is wrong-headed.
Another
view of sustainability, she says, “will help evaluators and commissioners of
evaluation to consider the impact of the project (or other forms of
intervention) on ‘development that meets the needs of the present without compromising the ability of
future generations to meet their own needs’ (World Commission on Environment
and Development 1987).”
See this
is where I always have a problem. If she
had only said she did not want to compromise the ability and the desire of
future generations to meet their own needs” in the manner established by the
project or program, I’d agree.
Projects
rarely ask if is this a tradition or a cultural identity that beneficiaries
would like to preserve, or that they would like their children to preserve. Frankly,
if I were a kid in Aceh I’d want to go to college and get as far away from the
coffee farm as I could. So it would be
helpful to find out if this is the case.
What of sustainability then? How, really, are you gonna keep ‘em down on
the farm? Or in the factory, or on the shrimp boat?
For
answers to those questions, I’m going back to Michael Patton, because it is,
really, all about knowing how to listen.
.JPG)
