Localization Part 2: Implementing a Localized Evaluation & Learning Framework

“You have little power over what’s not yours” – Zimbabwean proverb 

As evaluators we’re often brought in to assess the success of programs that are implemented in vastly different cultural contexts. So, to ensure that our evaluation is relevant, effective, and useful – how do we integrate the individuals closest to and most affected by the social issue so that they are driving how an evaluation is conducted and deciding how to interpret success?

In this blog, we explore some ways we have worked to do that in our two different roles – one as an external evaluator working with funder and nonprofit clients, and the other as an internal evaluator leading a nonprofit’s evaluation strategy. In Part 1 of this blog series, we discussed the idea of “localization” in the evaluation design phase. This blog picks up from that conversation and assumes that an Evaluation and Learning Framework has already been built, and that you are now ready to engage in data collection. As you begin, here are some important things to keep in mind:

Continue to integrate local experts in method selection. To make sure our evaluation can gain insights into whether social change has occurred, it needs to be generated by and with the communities being served. In part 1 of this blog series, we talked about working with the community to co-create the theory of change and the indicators of success. This co-creation approach needs to continue in our selection of data collection methods and throughout the entire evaluation implementation phase. Local community-based experts (that may or may not self-identify as an “expert”) will optimize the evaluation process because of their intimacy with the pressing needs of their community and their understanding of how data and insights can be collected and shared with quality and integrity. These guides will help ensure data collection is appropriate and relevant and can help us think beyond our usual patterns.

When working with Princeton in Africa, TCC Group developed and refined data collection tools that could be used offline and featured open-ended questions so that program staff could work with local community members to determine what successful health services organizations looked like to them. In implementing an evaluation of the New Brunswick Healthy Housing Collaborative in New Jersey, we engaged community members from the largely immigrant neighborhoods who had received housing advocacy training. Through conversations held in Spanish, they voiced how they perceived the success of the program and where they wanted more support in the future. The tailoring of tools to be offline, translated, and in the interest of what the ultimate end-user imagines for their future was key to the success of accurately localized data because it engaged with the community members pragmatically.

When working with Princeton in Africa, TCC Group developed and refined data collection tools that could be used offline and featured open-ended questions so that program staff could work with local community members to determine what successful health services organizations looked like to them. In implementing an evaluation of the New Brunswick Healthy Housing Collaborative in New Jersey, we engaged community members from the largely immigrant neighborhoods who had received housing advocacy training. Through conversations held in Spanish, they voiced how they perceived the success of the program and where they wanted more support in the future. The tailoring of tools to be offline, translated, and in the interest of what the ultimate end-user imagines for their future was key to the success of accurately localized data because it engaged with the community members pragmatically.

Have honest discussions about evaluation constraints. Sometimes we don’t have the option to co-create and iterate on the evaluation design freely with the community. If this is the case, be honest about who or what the evaluation was prompted by and why. Instead of using language that can feel condescending (We’ve heard language used such as “this will be good for your community”), ask, “Given the limitations, how can we together best design this evaluation for the community?” Listen actively and be honest about your ability to iterate on the evaluation design as you collect data and learn together.

Test the data collection tools together. It’s important to test the questions and tools as we develop them, and working with our local experts gives us realistic feedback about their validity. It also continues to ensure the relevance of the tools to the community’s learning needs. Even if one is using a “verified indicator” or an indicator that has previously been formalized as appropriate for a given context, it’s not unusual to find that a context has changed quite a bit from the time it was verified to the time one might finally conduct the data collection. Therefore, we must remember to test our tools at various points in time and changes in context to ensure they are collecting the data that we are really interested in.

Listen and look for what happens in between the scripted questions. We’re taught as data collectors to carefully prepare our questions and to separate all dissimilarities (group by age, gender, social class, etc.) to increase comfortability within qualitative data collection settings. As practitioners, we know this is idealistic and that if we can’t adapt our plan, we will miss key sources of information. While we rely heavily on previously conducted research, this research is limited, and if previously conducted work is all we are primed to see then we have little chance of making discoveries and advancing narratives.

Be realistic about the time it takes for effective community integration and for outcomes achievement. Time and time again we’ve witnessed a push to implement evaluation on a timeline that is impractical and that sacrifices the quality of what we can learn from the evaluation. Those pushes are almost exclusively coming from audiences that are external to the context of where the work takes place. Timelines are set based on board meeting presentations, the publication of an annual report, or the budgeting cycle of the funder. They don’t consider realistic timelines for social change to occur. This risks excluding important perspectives and drawing false conclusions. Aren’t sure what a realistic timeline is? That’s why having your locally-based partners involved in the evaluation design is a critical step – one that sets up your evaluation for success.

As evaluators, it is our responsibility to be the bridge between the data sources and the ultimate end-users of the data. This means having honest conversations that center the voices and learning needs of local communities deliberately with senior leadership within our organizations and our partners. It means building on the expertise of the community itself and thinking of all the stakeholders as part of a collective effort to create lasting social change.

Thank you for joining us in these reflections on how we can localize or center community voice in our data collection practices. We hope you will join us in sharing what you are learning about localized evaluation practices as well!

Isha Varma has over 13 years of experience with international development nonprofits, intermediary funds, social enterprises, and foundations. Her expertise in Monitoring, Evaluation, and Learning has led her to build organizational strategies that optimize impact for both operational performance and ultimate social outcomes. She grounds her practice in rights-based participatory methods, human-centered design, and systems thinking. Isha has an MPH in the Measurement and Evaluation of Global Health Systems and Development from Tulane University’s SPHTM and a B.A. in Psychology from UCLA. She has spent time working in East and West Africa, Southeast Asia, and the Dominican Republic.