AI in Evaluation: Where Do We Go From Here? 

Last month, I attended the American Evaluation Association’s (AEA) annual conference in Portland, Oregon. The event brought together evaluators, researchers, and practitioners from around the world to explore emerging topics and share innovative practices. 

One of the most captivating themes of this year’s conference was the surge of interest in artificial intelligence (AI) as a tool to support evaluation efforts. From chatbots to advanced data analysis tools, AI is proving to be both a powerful asset and a complex partner in our work. 

AI and the Evolution of Evaluation 

Throughout the conference, there was palpable excitement around the potential of AI to assist evaluators in a variety of ways. Attendees were particularly intrigued by the use of chatbots designed to inform and streamline evaluation efforts. These AI-driven tools show promise in organizing and analyzing large datasets, offering quick access to insights that might otherwise take hours or even days to uncover. As evaluators, we face growing expectations to provide fast, actionable insights, and AI is beginning to meet this need. However, the consensus was that we must tread thoughtfully—AI should be an enabler rather than a replacement. 

Navigating the Advantages and Challenges of AI 

 The potential of AI in evaluation was met with a healthy dose of caution. As evaluators, we cannot simply hand over our responsibilities to an algorithm and expect it to deliver nuanced, context-sensitive results. Instead, we need to enhance our understanding of AI, recognizing both its advantages and limitations. The tools on display highlighted some of these challenges, particularly those related to the lack of the deeper understanding of cultural and situational nuances that evaluators must consider, making human oversight and engagement essential to ensure meaningful findings and recommendations. 

New Tools and Applications for Evaluation 

A variety of AI-powered tools showcased at the conference piqued the interest of many attendees. These ranged from applications that support research of previous social science studies to agents that assist in the creation of theories of change and logic models. Additionally, tools designed for both qualitative and quantitative analysis were introduced, showcasing how AI could help evaluators process large volumes of data more effectively. It was clear that while AI tools are improving at a remarkable pace, we must still consider the quality and diversity of data that we feed into them. Without carefully curated inputs, AI’s outputs can miss critical context or even skew conclusions. 

AI and Context: The Human Element in Evaluation 

In our own presentations, we emphasized that while AI can provide structure and speed, it remains a “co-pilot” rather than a driver in the evaluation process. As mentioned earlier, contextual factors, often subtle and complex, require the expertise of experienced evaluators who pay attention to the local or sector-specific intricacies at play – engaging key informants when necessary. For example, some tools claim to bring context into their analysis, but the reality is that many AI systems still struggle to account for nuanced, situational differences. Our team shared our own experiences in using AI to support our work, highlighting that even as we explore new tools, we remain committed to ensuring our evaluations reflect the voices and stories behind the data. 

Looking Forward: Blending AI and Human Insight 

Incorporating AI thoughtfully will require a balance of skill, awareness, and ongoing learning. We’re excited to continue exploring how AI can amplify our work, giving due weight to both technology and the invaluable human insights that have always defined effective evaluation.  

Portland provided a fantastic backdrop for these conversations, and I’m looking forward to seeing how the field evolves in the coming years. As we embrace new AI tools, we’ll do so with a commitment to maintaining the rigor, relevance, and responsiveness that our clients and communities deserve.