It's that time of year again. Time to talk about #failforward at Awamaki!
How to measure impact is one of the most difficult questions facing small non-profits. Every funder and most donors want to see you measuring impact, but it is pretty hard to do, and even harder with scarce resources.
For a long time at Awamaki, we measured our impact simply by measuring how much our cooperatives were earning. This is a fairly simple indicator. After all, the positive effects of women’s earnings in poor households are well documented. But as we grew, it was harder for this indicator to stand alone. Funders wanted to know how the women spent the money they earned, and whether those earnings translated into other types of empowerment in their community. Also, we wanted to know how our program affected women’s lives so that we could find ways to improve our work.
So we put a lot of effort into developing a plan to monitor our impact. This plan was based on talking to the women of our cooperatives. We designed and administered a “Well-being survey,” intended to help us understand what effects our program had in the communities. We even intended to administer it every year in order to monitor changes in the community. The survey asked about everything from how many times per week families eat meat to how women participate in household decisions.
We spent hours designing, coding, and administering the survey. Gathering the data was hard. Only 2 of our 6 cooperatives speak fluent Spanish, so we always have a Quechua-speaking staff member along. Only a few of the women in our cooperatives can read and write, so instead of distributing written/paper surveys we usually talk to each woman individually. We conducted these interviews during the cooperative’s regular meetings, where it can be tricky to talk to everyone in the limited amount of time that we’re all gathered. The whole process ended up taking months, and monopolizing the time of our small staff.
We had big plans for this data: we wanted to use it to report to donors, apply to grants, and track our own progress. We thought that next year, we might try to give the survey to women who are not part of an Awamaki cooperative, so that we can see if there is a difference between those who work with us and those who don’t. It isn’t exactly a randomized control group, but it was our best approximation! But it turned out we were getting ahead of ourselves. Our Well-being Survey was not destined to be repeated.
First, administering the survey was awful. It took a huge amount of time, and the women were clearly bored, busy and just trying to get through the questions as quickly as possible. Many of the questions had no clear Quechua translation, or the nuances of the questions were lost in translation. (For example, “Do you speak up in your family?” vs. “Do you speak up in your community?” is tricky when the word for family and community is the same.)
Second, the quality of the information we collected was bad. We had a hunch during the data collection that the answers we were collecting weren’t entirely accurate, but the proof was in the questions about income earned. Awamaki keeps good records about how much each woman and each cooperative earn from our programs. The survey included questions about how much the women’s households earned in total. Many of the women reported a lower total household income than we knew definitively that they were making at Awamaki. When compared to what we had on file for their Awamaki-generated earning, their answers didn’t make sense. In general, our artisans would underreport their income, some even reporting no income at all.
We’re not sure exactly where this discrepancy is coming from, but we have a few ideas. Perhaps the poorly estimated figures are a result of the women having no formal or consistent form of household record-keeping or budgeting within their families, which may be a product of low literacy rates. It could also be that the artisans fear that if they report a high income, Awamaki will give that cooperative fewer orders or reduce assistance in other ways. This of course is not our intention at all, but that trust can be hard to build. It’s also likely that the style of the interview itself is not conducive to accurate answers. Would you like to discuss, through an interpreter, your exact income with a distant colleague? Honestly, it can be a little awkward for everyone involved.
This wasn’t the only problem with our data. There were many issues, and we won’t be administering our lovingly crafted Well-being Survey again. We are trying a new method for next year. To measure economic success, we have started doing observational studies of the women’s homes to gauge wealth. We visit and observing the number of household appliances, the material the floor and roof are made of, whether there is a bathroom, etc.). We have also started running focus groups to approach the answers we sought from the Well-being Survey. We plan to invite a small group of women to the office, serve food, ensure a comfortable environment, and have a general conversation among the women instead of a direct interview with each individual. We think that these issues will be easier to address in a low-key group setting, rather than a rushed interview that puts each individual on the spot.
So wish us luck! We promise that when we have something interesting to report, we will share it with you. If not, we will have another excellent failure story for next year and even more lessons learned.
So many questions!