Ryo Kamoi is a Ph.D. student at Penn State University, advised by Dr. Rui Zhang. He is broadly interested in Natural Language Processing, with a specific focus on:
- LLM Reasoning
- Error Detection and Reward Modeling
- Self-Correction and Verifier-Guided Refinement
- Fact-checking, factuality evaluation, and textual entailment
- Vision-language models
[Personal Website] [Google Scholar] [Semantic Scholar]
- FoVer PRMs [models and datasets] [code] [project website]
- Paper: Generalizable Process Reward Models via Formally Verified Training Data (arXiv 2025)
- FoVer is a novel method to create training data for Process Reward Models (PRMs) without relying on human annotation
- We propose to use formal verification tools like Z3 and Isabelle to automatically annotate step-level error labels on LLM responses to create training data for PRMs
-
ReaLMistake [huggingface dataset] [code]
- Paper: Evaluating LLMs at Detecting Errors in LLM Responses (COLM 2024)
- Benchmark for evaluating error detection methods that detect mistakes in LLM responses
- Expert error annotations on responses from GPT-4 and Llama 2 70B on three tasks
-
WiCE [dataset and code]
- Paper: WiCE: Real-World Entailment for Claims in Wikipedia (EMNLP2023)
- Dataset for document-level NLI
- Fine-grained textual entailment dataset built on pairs of natural claims and evidence extracted from Wikipedia
- VisOnlyQA [project website] [huggingface dataset] [code]
- Paper: VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information (arXiv 2024)
- Dataset for evaluating visual perception capabilities of LVLMs on geometric and numerical information about scientific figures
- When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (TACL 2024)
- Paper list on self-correction of LLMs: https://github.com/ryokamoi/llm-self-correction-papers
- Shortcomings of Question Answering Based Factuality Frameworks for Error Localization [human annotation]