Paper | Corpora | Retrieval Tasks | Generation Tasks

We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation. MTRAG-UN consists of 666 tasks containing over 2,800 conversation turns across 6 domains with accompanying corpora. It focuses on four open challenges:

• UNanswerable Question — the user question cannot be answered from retrieved passages
• UNderspecified Question — the user question is ill-formed or ambiguous, lacking the information to determine a clear intent
• NONstandalone Question — the user question cannot be understood without the prior turns
• UNclear Response — the user doesn't understand or disagrees with the model answer and requires clarification

The paper describing the benchmark and experiments is available on Arxiv:

MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations
Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, Marina Danilevsky

MTRAG-UN uses the four corpora from MTRAG (ClapNQ, FiQA, Govt, Cloud) plus two new enterprise corpora:

The four shared corpora are available in the corpora folder. The Banking and Telco corpora were created by crawling ~1K web pages per domain to produce sets of inter-connected pages suitable for writing complex conversations.

Retrieval tasks are provided in BEIR format. The qrels cover the 468 answerable and partially answerable questions.

Tasks are derived from the 666 conversations by selecting one representative turn per conversation (prioritizing challenging UN-turns). For underspecified conversations, the underspecified turn is selected.