91Ö±²¥

Title
The Challenges and Opportunities in Evaluating Generative Information Retrieval

Abstract
Evaluation has long been an important part of information retrieval research. Over decades of research, well established methodologies have been created and refined that for years have provided reliable relatively low cost benchmarks for assessing the effectiveness of retrieval systems. With the rise of generative AI and the explosion of interest in Retrieval Augmented Generation (RAG), evaluation is having to be rethought. In this talk, I will speculate on what might be solutions to evaluating RAG systems as well as highlighting some of the opportunities that are opening up. As important as it is to evaluate the new generative retrieval systems it is also important to recognize the traditional information retrieval has not yet gone away. However the way that these systems are being evaluated is undergoing a revolution. I will detail the transformation that is currently taking place in evaluation research. Here i will highlight some of the work that we've been doing at RMIT university as part of the exciting, though controversial, new research directions that generative AI is enabling.

Bio
Mark Sanderson is Professor of Information Retrieval at RMIT University where he is Dead of Research for the STEM College. Mark received his Ph.D. in Computer Science from the University of Glasgow, United Kingdom, in 1997. Mark was the first researcher show the value of snippets, a component of search interfaces which are now a standard feature of all search engines. While a faculty member at the 91Ö±²¥ Information School, Mark, with Paul Clough, co-founded in 2003 the annual imageCLEF evaluation campaign, which continues to run today. Mark was general chair of ACM SIGIR in 2004 and PC chair of ACM SIGIR 2009 & 2012; and ACM CIKM 2017. Prof Sanderson is also a visiting professor at NII in Tokyo.

Joining online
This talk is hybrid and can be joined online using the link on the