Seminar: POP-TEXT: Do definitions matter? LLMs, prompts and outcomes -Jessica Di Cocco (University College Dublin)
14:00-15:00 (GMT) Wednesday, January 28.
Please join our mailing list (opens in a new window)here to receive the Zoom link for this event.
Abstract: Recent advances in large language models (LLMs) offer new opportunities for large-scale text analysis in political science, while raising methodological questions about conceptual validity, prompt design, and evaluation. This project addresses these challenges in the study of populism, a field characterized by rich theory but persistent disagreement over how populist discourse should be operationalized in texts. I present an LLM-centred framework that combines theory-driven codebooks with prompt engineering to classify populism in a multilingual corpus of party manifestos, translating three conceptualizations (ideology, strategy, and narrative) into structured prompts. At the time of the workshop, prompt development is complete and LLM-based classification has been conducted for most countries. Validation using supervised machine-learning models and manually coded text segments is ongoing. The project also introduces a longitudinal extension of the corpus. Germany currently serves as the first case with expanded coverage back to the second half of the nineteenth century, coinciding with the emergence of mass politics, with similar extensions planned for additional countries. This design enables both methodological testing on older documents and exploratory analysis of long-term trends in political language, including the potential cyclical nature of populist discourse.
About the speaker: Jessica is a Marie Skłodowska-Curie Fellow at University College Dublin, working part-time in the School of Politics and International Relations under the guidance of Prof. James Cross. She is a text-as-data enthusiast studying the promises and limits of machine learning and generative AI in Political Science, with research interests in comparative politics, political communication and sophistication at the textual level, populist and niche parties, and electoral campaign dynamics. Alongside her role at UCD, she is a part-time Research Fellow at the Robert Schuman Centre for Advanced Studies at the European University Institute, where she investigates global cleavages and the politicization of inequality. She previously held a Max Weber Fellowship at the same institution.