Our Culture of Continuous Learning: Retooling our Evaluation Tools

At TCC Group, we believe in building the evaluation & learning capacity of our clients from across the nonprofit, philanthropic, and corporate sectors to allow them to respond to the shifting normal. This practice was put to the test for many of our clients, and TCC itself, in 2020. In this blog, we profile a few projects where we retooled our Evaluation and Learning services to ensure that we ourselves were staying adaptive to the multi-faceted needs our clients had.

Click on each project to learn more:

A state-level advocacy campaign poised to seize the moment around childcare

The COVID-19 pandemic directly impacted three ongoing campaigns being supported by the William Penn Foundation and evaluated by TCC Group. While distinct, each of the campaigns sought to advance the well-being of children in Pennsylvania through multiyear efforts involving a range of community stakeholders. The shutdown of in-person programs and services for children in March 2020, combined with constantly evolving responses from local, state, and federal authorities to the needs of children and other vulnerable populations, resulted in a desire to reconsider how these campaigns should be reacting to this rapidly changing situation.

Despite having a 2020 plan in place, the Evaluation & Learning Team immediately began to consider how to restructure its approach to address unanticipated challenges, such as engaging parents who were no longer accessible at childcare sites and had taken on fulltime, at home childcare roles. To support a structured process of reflection with the Foundation and other key stakeholders, the team hosted an after-action review (AAR) in the summer of 2020 to consider what had worked well in the initial months of the pandemic response, acknowledge challenges and accomplishments, and tailor next steps to be most adaptive to the uncertain future.

For the Evaluation and Learning Team, this work included sharing data with the campaign partners and reflecting on their learnings. They also modified existing evaluation tools and procedures to be responsive to the new remote reality, while ensuring continuity with earlier data collection. Critical to how well this process has worked is the long-term, trusting collaborative partnership the Evaluation & Learning Team developed with the Foundation and the campaigns long before the pandemic occurred.

• “Covid-19 had perhaps its greatest impact on the childcare sector in the United States. Working with an advocacy campaign for many years through a capacity-building framework allowed us to document, analyze, and then build in time for shared group reflection so everyone could understand what was happening across various dimensions and where there were strengths and gaps in the approach. The reflection time in particular allowed for some meaning to emerge after a time of very high intensity.” – Beth Cain, Consultant, Evaluation & Learning

A faith-based camp looking to better understand its ties with the local community

Since the mid-1950s, Delaware-based Camp Arrowhead has offered children and youth a summer camp experience centered on how to improve educationally, socially, and spiritually. The COVID-19 pandemic and a desire to be more connected with historically underserved Delawareans led the Camp to seek out TCC Group to establish an evaluation plan that would help tell the story of the organization, develop organizational perspective around racial equity and social justice, and illuminate how new space plans would help position the Camp for the future. While TCC’s long record of successfully conducting evaluative research and learning made the firm an ideal fit for the assignment, the pandemic precluded the types of in-person leadership brainstorming and consensus building typically employed to create theories of change and evaluation plans.

Given the inability of having camp leadership in a room together, the Evaluation & Learning Team regrouped and recommended employing a far more expansive “concept mapping” to develop the theory of change around which the evaluation plan would be structured. This entailed asking stakeholders for their thoughts on what outcomes they felt Camp contributed to and then asking them to group outcomes from other stakeholders by categories.

Through this process the team was able to incorporate the voices of a far broader array of stakeholders, such as operations staff, alumni campers, members of the local diocese, and organizations that are the Camp’s seasonal partners. Camp leadership benefited from this wider range of perspectives, which helped them to better articulate their own vision and deepen their understanding of the Camp’s relationships with its stakeholders.

• “We’ve found that while our clients often offer great perspective of other actors in their community, there are groups that they may not be considering that are often doing complementary work. Inviting these stakeholders into the evaluation planning process allowed us to make sure that our client was able to think about their success using perspectives from across the ecosystem.” – Rose Konecky, Consultant, Evaluation & Learning
• “Things like empathy, volunteerism, and reflection are often lost in evaluation processes. But soft skills like these were brought up again and again during the concept mapping process and hearing that feedback from stakeholders gave us a path to elevate their status and help our client articulate what’s so special about camp in the modern era. The concept mapping helped us articulate not just the big traditional camp outcomes but a lot of the soft skills that children develop at camp.” – Charles Gasper, Senior Consultant, Evaluation & Learning

A national health-equity funding supporting a multi-year community of practice

Coming into 2020, the Evaluation and Learning Team had been working with a national foundation that created an initiative focused on developing leaders on health equity. The Foundation already relied on the team to devise an evaluation strategy related to this leadership program to ensure clarity and cohesion among its four component programs, as well as a robust understanding of how these programs serve their beneficiaries. From developmental to formative to impact evaluations, the team had been developing tools for measurement, collecting data, and making meaning through custom dashboards and reports. As the COVID-19 pandemic took hold, however, existing analyses and reporting were insufficient to provide the Foundation with the readiness to best serve its objectives and the evolving needs of grantees in this unprecedented time.

Recognizing how overwhelming the early days of the pandemic were for even the best-resourced organizations, the Evaluation & Learning Team took the initiative to mine new perspective from data already being collected for the evaluation. Among questions examined were: Which grantees are facing the greatest potential need? What might those needs be? What form should the Foundation’s support take? Should the Foundation pause recruitment for these programs while existing grantees adapt to the new reality?

With novel insights in hand, the Evaluation & Learning Team then chose to go beyond neutral guidance and instead offer the foundation a range of explicit recommendations for strategic action. The Foundation worked with the team to refine and integrate these insights, enabling it to pivot quickly to address immediate grantee needs while reinforcing the ongoing value of the Initiative.

• “Raising questions is an important part of an evaluator’s role. In this situation, our history of work with this client let us ask questions and then dive deep into that historical data to answer as best as we could. Getting to take the step towards answering without having to ask people to respond to a data request during a nascent global epidemic allowed the Foundation to be ready to move forward.” – Jared Raynor, Director of Evaluation & Learning
• “This was definitely one of those engagements where the line between evaluation and strategy was blurred. We were fortunate to have a history of work with this Foundation and to be seen not as just an outside evaluator but as a full thought partner. And the changing circumstances related to COVID-19 freed us to put on that strategy hat more often that we often do as evaluators and help our client respond to the moment.” – Lisa Frantzen, Senior Consultant, Evaluation & Learning

A foundation trying out a new model of rapid response grantmaking

The Evaluation and Learning Team worked with the United States-based arms of a global foundation as they tried a new way of awarding grants – focused on getting dollars out quickly with limited restrictions to organizations working to create a racial equity movement.

The team used a process mapping approach to examine the foundation’s new rapid response model of grantmaking. Because the foundation created different topical teams – each focused on identifying potential individual and organizational grantees  – the Evaluation and Learning Team mapped the processes that each topical team used. These process maps were comprehensive – looking at the inputs: how the teams were created, their leadership structures, how decisions were made, the extent to which teams had standardized and clear goals and the outputs: the amount of funding, the number of organizations, and the number of organizations without preexisting ties to the foundation. The Evaluation and Learning Team also worked to unpack the capacities that staff needed to contribute to this process. One finding: more junior staff appreciated the opportunity to recommend organizations for grant awards and work in ways that were outside of their typical capacity.

• “Process mapping showed us that almost every individual perceives a process differently, depending on their own vantage point. Until we started using this tool, we were missing a lot of the key questions. Once we started mapping out the process, we were able to discover where there were differences in people’s assumptions and opinions, and then work to unpack those to set the work up in a better place moving forward.” -Deepti Sood, Senior Consultant, Evaluation & Learning
• “Process mapping helped me figure out the many, many inputs that go into a foundation being ready for grantmaking. Understanding the differences in how staff at different levels responded to the new grantmaking roles that were trying on helped us deepen our thinking on what capacities staff need to be effective program officers.” – Stephanie Coker, Consultant, Evaluation & Learning

A mental health focused nonprofit looking to understand and strengthen the capacity of their chapters

The Evaluation & Learning Team was working with a chapter-based national membership organization focused on mental health when COVID-19 started.  As the pandemic intensified, the organization saw a rising urgency to address mental illness in local communities, and a need to strengthen their chapters.  The Evaluation and Learning Team would be working to build out, refine and launch the organization’s capacity assessment tool in collaboration with representatives of local chapters. Before being able to take that step, it was key to build trust between each party. A capacity building tool will ask those who take it for their insights and their time, and these insights ensure that the questions asked aligned with organizational thinking and culture.  Once trust was established, chapters could take the steps to assess their organizational readiness, and identify their strength and weaknesses.

Currently in the information-gathering phase, the Evaluation and Learning Team is already seeing valuable data from local chapters, where they have begun to determine the areas in need of more support; their human resources, fundraising and revenue generation efforts.  The community representatives are encouraged and look forward to continuing to learn ways to apply the data from this tool to help chapters strengthen and meet the growing needs for mental health in communities where they operate.

Representatives who took the survey and participated in interpretation sessions felt supported by seeing the nonprofit take an organized and data-driven approach to capacity building. After working together for several months, they, along with members of the Evaluation and Learning Team, and nonprofit staff, created a group poem during our last meeting called “You are Not Alone” which speaks of connection and empathy, and reminds them that this is why they are doing this work.

• We learned that the most important part of capacity building is building trust and buy-in. We need to ensure that we build tools that are intentional and directly align with the need of those who will be benefitting from them. We have to ensure transparency and accountability. What value will people who participate and these assessment receive? .” — Jackie Loweree, Consultant, Evaluation & Learning

Interested in learning more about our Evaluation services? Contact us at info@tccgrp.com