The Patricelli Center for Social Entrepreneurship awards annual seed grants to fund the launch or early stage growth of a Wesleyan-connected social enterprise, project, program, or venture. Each grant recipient reports back with blog posts and photos. Here’s the latest from Nina Gerona ’15. Read her earlier post here.

*****************************

Months ago I began designing a social metrics survey for the Indonesian coconut and cacao farmers of the Big Tree Farms organic food company. Big Tree Farms works to increase the income of the farmers by introducing organic growing practices and connecting farmers to a new, higher quality market. The company decided it was important to track changes of income and a variety of lifestyle impacts for the often poor farmers in their supply chains.

I tested the social impact survey by collecting data from 58 coconut farmers. I found wording and presenting these questions to be quite challenging as I had to work through both cultural and linguistic differences. However, in the process I gained insight into human based data collection and the household economies of rural Indonesian coconut farmers. For more details on my project, continue reading below.

I want to thank the Patricelli Center, particularly Makaela Kingsley and the seed grant committee for their amazing and generous support.

In the summer of 2013 I spent a short month designing drafts for Big Tree Farms’s social metrics strategy. My research began by examining development projects of different NGOs and socially conscious companies for inspiration. I explored project designs, logical frameworks, implementation proposals, impact measurement models and data sets for any useful recommendations or strategies that would fit my work.

While the array of development data was vast, I was surprised by the lack of consensus and scarcity of impact measurement data. Since this area has recently been gaining much publicity in the development industry world, I figured there would be more practical and globally accepted frameworks. However, impact measurements are still a recent phenomenon and often recognized for their difficulty of implementation and data collection.

The majority of development projects, especially in the past, rely on initial data from the planning and implementation of their proposals. While many scenarios are thought out during this period in order to maximize change, years after the project, when the actual impact would occur, little to no data is collected. One of my mentors, an experienced development economist who has worked on these projects in the past, stated dismay with the lack of impact measurement post completion. He says it is common for project leaders to be unable to point out specific impacts of their projects beyond repeating their initial goals. As the director of the World Business Council for Sustainable Development says, “solutions will not happen without a solid understanding of what works and what doesn’t – and having sound measurement systems in place is fundamental to obtaining this insight.” (credit: WBCSD Guide to Measuring Impact). When brining resources into a community, whether money, education or material goods, it is vital to track a number of different measurements after doing so. Since many aspects of a person’s household are interconnected, it is important that the measurements can give a clear picture to the greater impact of the development project in order to understand its successes and failures.

Many companies, from small fair trade businesses to large investment banks are recognizing the importance of impact metrics and are frequently turning to the leading tool in the business, the Global Impact Investing Network IRIS metrics. IRIS states it is a “a single source for measuring the whole performance of your investees.” However, in my opinion, has often been oversimplified to span across a wide range of industries and capacities and can thus miss very critical links between cause and effect (credit: Getting Started with IRIS). While it is useful to measure acres of organic land or number of very poor farmers, without more detailed information per household it is impossible to see the many ways in which the lives of the farmers involved were impacted by the project. For instance, perhaps the farmers who are labeled very poor do have a lower income, but they could also be more literate, healthier, and more satisfied with their daily lives than their richer counterparts. At least for Big Tree Farms, we found the overall picture more important than easily measurable details that only give a small, or potentially misinterpretable, understanding.

I began my survey with general questions for which measurements could offer insight into the farmer’s lives. Some questions pertained to basic needs like access to safe drinking water, food nutrition, illness and living situation. Other questions focused on the family’s daily activities such as: Do you children attend school full time? Which jobs contribute to your family’s income? How many hours do you work per day? Other questions were meant to gain an understanding of how the family may spend their income, particularly extra income they might gain from working with within the Big Tree Farms program. These questions focused on household assets such as televisions, motorbikes etc.

While these questions were quite exciting, it was overwhelming to realize I would have to ask thousands of questions and spend years in order to fully understand the famer’s lives. To keep the survey simple, I picked four areas to focus where Big Tree Farms has a direct impact: health, and nutrition, income, assets, and access to credit. After I had developed questions for measurements in these categories, I realized each question must be answered twice for a baseline comparison. One question would ask how the situation is currently, and one would ask how it was the year before the farmer began producing under the Big Tree Farm program. This double question strategy is important for understanding impact since we can look at change since starting work with the company. My final survey had 36 questions. Ultimately I think there were too many questions, particularly because some of them contained multiple parts and all were asked twice, making the survey a bit tedious.
We went into the field to interview coconut tree tappers and sugar producers from five villages, four of which have already been working with Big Tree Farms and one in which the farmers were just beginning their relationship with the company and transitioning into the organic program. The survey was given by five members of the company’s in-field organic certification team over a period of three days to 58 respondents.

The data results were a bit mixed. Some questions lent themselves to simple answers. For instance we found 48% of the farmers interviewed only had an elementary school education and 26% of farmers now have more electricity than they did before. However 84% of the households still have only have between 200 and 500 watts, enough for a few small appliances and lights but not much else. Interestingly 100% of the households reported owning a television although only 22% of the households owned refrigerators. All of the farmers seemed to have a reliable water source, most from either a spring or a well. About 9% of the households reported using a non-flush outhouse while the rest of the families owned indoor squat toilets.

Voltage of house now (watts)
<200	5	9%
1,300-2,200	1	2%
200-500	49	86%
500-1,300	2	4%

Big Tree Farms has the most direct impact on the farmer’s lives through their income. Thus I hoped to obtain accurate income measurements in order to draw correlations from this data piece. However, it was very difficult to obtain reliable answers on this subject. The Javanese farmers deal almost entirely with a cash based economy and very few keep financial records. Therefore when asked, many struggled to report what they earn per month, or at all. I ended up asking them to report by whichever time frame they found easiest, by day, week, month or year and then did calculations to find monthly income. I also tried a round-a-bout way to find the amount of money they earned by asking all monthly expenses, farm costs, food, utilities, education, transportation, household appliances, personal supplies, religious expenses, savings, loans and remittance, in the same open time frame manner. I figured all household expenses should be about equal to household income, yet the discrepancies were large. The average household expenses per person came to about $144.54 while the average income, including the income earned from outside jobs and coconut production came to $67.60 per household member. While there were a few households with somewhat similar expenses and incomes, there were often even greater discrepancies per household than shown by these averages. For example some farmers reported having five times more expenses than their income and vice versa. Interestingly we also asked about other agricultural products and many households said they sold vegetables, fruit or animals at the local market but never included them in their income calculations. This exclusion is just one example of these farmers understanding money very differently than I do.

I ran into another cultural difference when asking about coconut tree productivity. I figured it would be simple to ask how many trees each farmer owned and the average kilos of coconut sugar each farmer was selling per month. However, I didn’t realize the collective village nature of this work. One person may own some trees, but will then split the nectar production with their neighbors who help them tap when their trees are ready. Similarly, this same person may help and share in the product of their neighbors’ trees, thus creating a much more complex system to track than I had expected. Beyond this, the farmers were not all selling the same products. Some sold plain nectar while others boiled the product into block sugar, or granulated sugar at their house. Each of these products takes a different amount of time and labor and is thus sold at very different prices.

While there were other difficulties with the data collection process, there are many interesting points and correlations that remain unanalyzed from the 58 surveys we collected. Right now I am comparing the results between farmers who have yet to join the program, those who have been working with BTF for one year and those who have been involved for two to three years. So far it seems that the longer the farmers have worked with Big Tree, the more they have added to their household assets and improved their housing. I could look for correlations between many different data pieces including household diet and illness or between income and education but in the interest of time I will unfortunately have to limit my findings.

 

While there is much to learn from the more easily comprehensible data, even the small setbacks and surprising (or unrealistic) data will help improve future surveys and also give an important glimpse into the thought processes and lifestyles of these farmers. I see this experience as a great self led introduction into the complicated, yet essential, world of social impact metrics and will surely use this new knowledge for the rest of my life.