AI Reasoning, Retrieval, and Synthesis over Large-Scale Document Collections
Dr. Tomer Wolfson, University of Pennsylvania
Abstract: General purpose large language models (LLMs) have largely exhausted the pool of "publicly" available data for training. But there is a lot of other data that belongs to corporations, hospitals, academia, and individuals, that will never find its way as training data to the general-purpose models. Thus, the key role for LLMs, and the AI agents they power, will be to facilitate access to, and effective use of, external data encountered for the first time at inference time. In this talk I will present two works that address key challenges in this setting. First, I will focus on LLM-powered "agents" which surface hard-to-find information through multiple rounds of reasoning and retrieval. I will introduce MoNaCo, the first benchmark that evaluates AI agents’ ability to answer long-horizon questions that span dozens---and in some cases hundreds---of web pages. Using MoNaCo, I will show that information synthesis from large document collections remains an open challenge for current AI agents, and I will discuss promising directions for bridging this gap. Second, I will present our work for improving retrieval-augmented language models in scenarios where query-document relevance is implicit and not readily captured by surface-level textual similarity. Our approach addresses both efficiency and performance limitations in retrieval-augmented LLMs under these conditions.
I will conclude with an outlook on future directions, including the development of more capable and efficient AI agents for large-scale data analysis, as well as specialized agents for expert domains through interdisciplinary collaboration in healthcare, law, and other scientific fields.
Bio: Tomer Wolfson is a Postdoctoral Fellow at the University of Pennsylvania and a member of the Cognitive Computation Group headed by Prof. Dan Roth. His research lies in the intersection of Natural Language Processing and Data Management. He is interested in developing AI-powered systems that can retrieve, reason over, and integrate information from large-scale document collections. His work has introduced novel methods for machine question understanding and retrieval-augmented language models.
Tomer completed his PhD in Computer Science at Tel Aviv University, where he was advised by Prof. Jonathan Berant and Prof. Daniel Deutch. His research has been recognized with several honors and awards, including the Postdoctoral Fellowship in AI and Data Science from the Israeli Council for Higher Education (VATAT), the VATAT PhD Fellowship in Data Science, and the AI3 Award for outstanding work from the Allen Institute for AI.
