Benchmarking Agentic Search in Real-World Local Life Services

LocalSearchBench is a benchmark for evaluating LRM-based agentic search in local life services. This repository is aligned with the camera-ready paper and includes the benchmark data pointer, evaluation toolkit, and leaderboard snapshot.

• Project Website: localsearchbench.github.io
• Code Repository: github.com/localsearchbench/localsearchbench
• Artifact DOI: 10.5281/zenodo.20416849
• Paper DOI: 10.1145/3770855.3817466

Recent advances in large reasoning models (LRMs) have enabled agentic search systems to perform complex multi-step reasoning across multiple sources. However, most studies focus on general information retrieval and rarely explore vertical domains with unique challenges. In this work, we focus on local life services and introduce LocalSearchBench, which encompasses diverse and complex business scenarios. Real-world queries in this domain are often ambiguous and require multi-hop reasoning across merchants and products, remaining challenging and not fully addressed. As the first comprehensive benchmark for agentic search in local life services, LocalSearchBench comprises a database of over 1.3M merchant entries across 6 service categories and 9 major cities, and 900 multi-hop QA tasks from real user queries that require multi-step reasoning. We also develop LocalPlayground, a unified environment integrating multiple tools for LRM interaction. Experiments show that even state-of-the-art LRMs struggle on LocalSearchBench: the best model (DeepSeek-V3.2) achieves only 35.60% correctness, and most models have issues with completeness (average 60.32%) and faithfulness (average 30.72%). This highlights the need for specialized benchmarks and domain-specific agent training in local life services.

• Six service categories: Dining, Lifestyle, Shopping, Accommodation, Healthcare, and Tourism.
• Nine major Chinese cities: Shanghai, Beijing, Guangzhou, Shenzhen, Hangzhou, Chengdu, Chongqing, Wuhan, and Suzhou.
• Merchant-scale grounding: 1,354,185 final merchant profiles constructed from over 1.6M original seed records.
• Real-world geographic coverage: 10,741 landmarks, 145 districts, and 9,956,907 retrievals.
• Multi-hop QA supervision: 900 validated multi-hop QA tasks, with L3/L4 intelligence levels and 3–5 search hops.
• LocalPlayground evaluation: Integrated LocalRAG and web-search tools for measuring answer quality and trajectory quality.

The dataset file in data/ is stored with Git LFS. Please make sure Git LFS is enabled before cloning or pulling the repository.

git lfs install
git clone https://github.com/localsearchbench/localsearchbench.git

from datasets import load_dataset

ds = load_dataset("localsearchbench/localsearchbench")
print(ds["train"][0]["Question"])
print(ds["train"][0]["Multi-hop search path"])
print(ds["train"][0]["Answer"])

The following numbers follow the camera-ready paper's Table 5, under Max Round N = 5.

@inproceedings{he2026localsearchbench,
  title={LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services},
  author={He, Hang and Yue, Chuhuai and Dong, Chengqi and Tian, Mingxue and Chen, Hao and Liu, Zhenfeng and Chai, Jiajun and Wang, Xiaohan and Zhang, Yufei and Liao, Qun and Yin, Guojun and Lin, Wei and Wan, Chengcheng and Sun, Haiying and Su, Ting},
  booktitle={Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2},
  year={2026},
  doi={10.1145/3770855.3817466}
}

MIT License. Commercial and research use permitted with attribution to LocalSearchBench.