Amplifying Voices: Shaping the Future of Evaluation

Sometimes, when it’s hard to articulate your thoughts, art can be an excellent way to spark ideas. At the American Evaluation Association (AEA) 2024 Conference, I found myself wrestling with how to capture the conference’s transformative spirit. After three unsuccessful attempts at writing this blog post about the future of evaluation, I decided to try a technique I learned in a great AEA pre-Conference session on arts-based evaluation, hosted by AND Implementation; erasure poetry. 

Erasure poetry is a data collection method that starts with a question and a relevant text. Using a dark marker, you black out everything you feel is irrelevant to answering the question. For this blog, I chose the question: What did the AEA 2024 conference mean to me? The text I worked with was a short blurb about the conference from their website. What I ended up with was: 

After blacking out the words, the next step in this data collection process is to ask the respondent (in this case, me), “What do the remaining words mean?” 

Well, the AEA Conference got me thinking deeply about the future of evaluation, with the word “future” appearing twice in my poem. It sparked questions like: “What kind of future are we shaping now?” and “As AI and technology become increasingly prevalent, how can we use these tools responsibly to amplify voices?” 

This responsible and practical use of technology to amplify voices was central to a session I co-presented with my TCC Group colleague Charles Gasper. Entitled “Unveiling Media Narratives: A Deep Dive into Content Analysis Leveraging Technology and Machine Learning,” the session highlighted our approach to a TCC Group project evaluating the impact of a journalist fellowship program. The goal was to analyze the topics and themes fellows wrote about before and after participating in the program. The challenge? In media analyses, evaluators often sift through vast amounts of content to understand which voices and topics are being represented or excluded. We had over 60,000 pieces of media content to analyze! 

To tackle this, we used a Latent Dirichlet Allocation (LDA) topic modeling approach—a machine-learning technique that identifies themes by analyzing patterns in word usage across texts. This method enabled us to uncover insights about the voices and themes present without manually reviewing every piece. By leveraging technology, we amplified voices that would have been nearly impossible to uncover otherwise. The session was well-received, with one attendee later describing it in an email as, “the highlight of my conference experience.” 

Another phrase from my poem, “working hand in hand,” underscores the importance of participatory methods in data collection—a recurring theme at the conference. These methods bring community members, implementers, and evaluators together as partners to collect, analyze, and use data. At TCC Group, participatory methods are embedded in our work through initiatives such as community advisory groups and community-led focus groups. Other examples of these approaches were illustrated during a session by Back of the Napkin Consulting. They demonstrated how to use a game of “hot potato” to measure group cohesiveness. Participants were tasked with keeping a balloon in the air for 60 hits, both before and after an intervention. Evaluators then measured changes in group cohesion by tracking the percentage of participants involved and the time it took to complete the task. 

The word “share” also appeared three times in my poem, reflecting the selflessness I always notice at the AEA Conference. It’s not about competition but about sharing knowledge and insights to advance the field collectively. 

The 2024 AEA Conference provided meaningful glimpses into the future. As evaluators, if we continue fostering discussions on how to shape that future responsibly and with a human-centered approach, I am optimistic about what lies ahead. To learn more about our presentation, contact us here!