Taming the ‘Wild’ Terrain of Qualitative Analysis

Qualitative data analysis is an art and science that seeks to convey textual or visual data’s meaning. That sounds nice, but navigating the best way to do qualitative analysis can feel as chaotic as the wild west. This perceived disorder arises from the many unique approaches to qualitative analysis, making it hard to determine the best practices. For those without access to pricey or complicated tools, there are even greater levels of frustration. This is why I was so excited to share about qualitative data analysis as an instructor at the Mongolian Evaluation Association’s Second Annual Summer Bootcamp for Young and Emerging Evaluators. By providing guidance on how to conduct rigorous qualitative analysis with accessible, low-cost, easy-to-use tools, we can better ensure that all evaluators can conduct rigorous qualitative analysis, and all communities can benefit from those findings.  

The Mongolian Evaluation Association is an organization that promotes the professionalization and institutionalization of evaluation at the national level in Mongolia. In alignment with this mission, they instituted their Summer Bootcamp, which coincides with Eval4Actions’ Youth in Evaluation week. Hosting regional learning opportunities like this is key to ensuring community voices are heard, engaged, and understood. By tailoring the program to local needs and contexts, it creates a strong support system of evaluators in the area, encouraging ongoing collaboration and learning. Plus, holding the training locally avoids the physical and financial challenges of long-distance travel, making it more accessible and relevant for everyone involved. 

To better understand the benefits of discussing qualitative data analysis at this Bootcamp, let’s consider some of the varying methodologies and techniques that can lead to frustration and confusion. For instance, the tools used in qualitative analysis range widely, from the most advanced Qualitative Data Analysis (QDA) software with Large Language Model (LLM) support to the simplest tools like pen and paper. The methods used in qualitative analysis are also diverse, from advocates of coding data—where textual or visual data is distilled into concise codes, and patterns within those codes are identified to reveal deeper insights—to those who argue that it can be destructive or dangerous because it risks oversimplifying complex data, losing nuanced meaning, and introducing bias 

To make qualitative data analysis more accessible, rigorous, and fun, regardless of participants’ access to fancy tools or their prior knowledge, I focused on three key aspects during my interactive session. While I (unfortunately) did not join in person this year, I was able to ‘Zoom’ in from halfway around the world in Philadelphia and speak with the twenty-three participants who were gathered at a campground resort outside Ulaanbaatar, Mongolia. The three key aspects I spoke about were: 

1. Definitions: Establishing standard definitions is crucial for effective qualitative analysis. We can’t begin to use methods like coding without understanding their meanings, variations, and pros and cons. Additionally, expanding our definitions can help us explore alternatives to our default analysis methods. 
2. Accessible Tools: While advanced QDA software can be helpful in certain situations, it is often expensive and has a steep learning curve. This makes it impractical for many young and emerging evaluators or teams with varying experience levels in qualitative analysis. This is why my presentation focused on using tools like Microsoft Word and Microsoft Excel, or any other standard word processor or spreadsheet software that someone has easy access to. These tools offer a more accessible middle ground, enabling rigorous analysis without a steep learning curve or high cost. 
3. Process: Understanding the steps involved in qualitative analysis is essential. This includes knowing when and how to transcribe data, the benefits of creating codebooks, and innovative ways to present findings, such as through poetry and process diagrams. 

This session equipped young and emerging evaluators with practical tools and insightful strategies to tame the complexities of qualitative data, and make it less ‘wild’. It sparked lively questions amongst the participants about best practices of qualitative data analysis and the efficacy of using LLMs for this purpose. It was also a reminder of the amazing era we are in, where people from all over the world can converge, create community, and engage deeply on both lighthearted and important issues.