KITE: Korean Instruction-following Task Evaluation

KITE (Korean Instruction-following Task Evaluation) is the first comprehensive benchmark specifically designed to evaluate the Korean instruction-following capabilities of Large Language Models (LLMs). Unlike existing Korean benchmarks that focus mainly on factual knowledge or multiple-choice testing, KITE directly targets diverse, open-ended instruction-following tasks.

The instruction-following capabilities of LLMs are pivotal for numerous applications, from conversational agents to complex reasoning systems. However, current evaluations predominantly focus on English models, neglecting the linguistic and cultural nuances of other languages. KITE addresses this gap by providing a benchmark that captures the unique characteristics of the Korean language, including its complex syntax, rich morphological features, honorific system, and dual numbering systems.

• Comprehensive Evaluation: Two distinct benchmarks covering both general and Korean-specific instruction-following tasks
• KITE General: 427 instructions translated and filtered from Google's IFEval dataset
• KITE Korean: 100 instructions created from scratch to address Korean-specific linguistic features
• Verifiable Instructions: Rule-based evaluation ensuring objective and measurable outcomes
• Cultural Awareness: Instructions embedded with Korean cultural context

KITE General consists of universally applicable tasks derived from the IFEval dataset. The development process involves:

1. Automated Translation: Using GPT-4o to translate the original IFEval dataset
2. Manual Verification: Meticulous review to identify and correct translation errors
3. Contextual Filtering: Removal of English-centric instructions (e.g., capitalization, English-only responses)
4. Expert Review: Five native Korean speakers with NLP knowledge reviewed all instructions

Out of the original 541 IFEval instructions, 114 were filtered out as culturally or linguistically irrelevant, resulting in 427 high-quality Korean instructions.

KITE Korean comprises 100 instructions (25 per category) created from scratch to evaluate Korean-specific linguistic phenomena:

Korean presents unique challenges for LLMs:

• Agglutinative Structure: Combination of roots and affixes to form words and sentences
• Flexible Word Order: Post-positions (Josa) provide syntactic information, allowing flexible word order
• No Case Distinction: Unlike English, Korean does not distinguish between uppercase and lowercase
• Honorific System: Complex levels of politeness embedded in grammar
• Dual Number Systems: Native Korean and Sino-Korean numbers used in different contexts
• Cultural Context: Communication styles deeply embedded in Korean culture

Existing English-centric benchmarks fail to capture these nuances, making KITE essential for accurate evaluation of Korean LLMs.

• Python 3.9 or higher
• CUDA-compatible GPU (for HuggingFace models)
• Conda (recommended)

1. Clone the repository:

git clone https://github.com/junkim100/KITE.git
cd KITE

2. Create and activate the conda environment:

conda env create -f kite.yml
conda activate kite

3. Install the package:

cd korean_instruction_following_eval
pip install -e .

The easiest way to run evaluation is using the provided run.sh script:

bash run.sh

Edit run.sh to configure your evaluation:

# Dataset selection: 'general' or 'korean'
DATASET_TYPE="korean"

# For KITE Korean, select categories
KOREAN_CATEGORIES=('acrostic' 'honorifics' 'numbers' 'postposition')

# Model configuration
MODEL_TYPE="hf"  # Options: 'openai', 'hf', 'solar', 'clova'
MODELS=("meta-llama/Meta-Llama-3-8B-Instruct")

# Shot configuration
SHOT_NUM=0  # Options: 0, 1, 3, 5

You can also run evaluation manually using Python:

python korean_instruction_following_eval/main.py \
  --instruction_file korean_instruction_following_eval/data/culturally_aware/instruction/acrostic.jsonl \
  --response_output_dir korean_instruction_following_eval/data/culturally_aware/response/acrostic/0_shot \
  --eval_output_dir korean_instruction_following_eval/data/eval_results/acrostic/0_shot \
  --shot_num 0 \
  --verbosity -1 \
  --model_type hf \
  --model meta-llama/Meta-Llama-3-8B-Instruct

• gpt-3.5-turbo
• gpt-4o

• meta-llama/Meta-Llama-3-8B-Instruct
• google/gemma-7b-it
• yanolja/EEVE-Korean-Instruct-10.8B-v1.0
• Any other HuggingFace model with instruction-following capabilities

• SOLAR 1 Mini Chat (solar-1-mini-chat)
• HyperCLOVA X (HPX-3.0)
• EEVE Korean models

KITE employs verifiable instructions to ensure clear and measurable outcomes. Each instruction is decomposed into sub-instructions, and the model's response is evaluated for each sub-instruction using rule-based checking.

Accuracy = (Σ Σ f(s_ij)) / (Σ n_i) × 100%

Where:

• N = total number of instructions
• n_i = number of sub-instructions for instruction i
• f(s_ij) = 1 if sub-instruction s_ij is followed correctly, 0 otherwise

Performance of various models on KITE (0-shot setting):

Key Findings:

• GPT-4o demonstrates the highest performance across both benchmarks
• Korean-specific models still lag behind GPT-4 in Korean language proficiency
• Performance varies significantly across different shot settings
• Instruction-following requires specialized tuning beyond general language understanding

korean_instruction_following_eval/
├── data/
│   ├── culturally_aware/          # KITE Korean
│   │   └── instruction/
│   │       ├── acrostic.jsonl     # 25 acrostic poem instructions
│   │       ├── honorifics.jsonl   # 25 honorifics instructions
│   │       ├── numbers.jsonl      # 25 number system instructions
│   │       ├── postposition.jsonl # 25 postposition instructions
│   │       └── culturally_aware.jsonl  # All 100 combined
│   └── translated_and_filtered/   # KITE General
│       └── instruction/
│           └── relevant.jsonl     # 427 translated instructions
├── eval/                          # Evaluation logic
│   ├── evaluation_main.py
│   ├── instructions.py
│   ├── instructions_registry.py
│   └── instructions_util.py
└── main.py                        # Entry point

If you use KITE in your research, please cite:

@misc{kim2025kitebenchmarkevaluatingkorean,
      title={KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Models}, 
      author={Dongjun Kim and Chanhee Park and Chanjun Park and Heuiseok Lim},
      year={2025},
      eprint={2510.15558},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.15558}, 
}

We welcome contributions to KITE! Please feel free to:

• Report bugs or issues
• Suggest new Korean-specific instruction categories
• Improve evaluation metrics
• Add support for new models

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

• Based on Google's IFEval benchmark
• Thanks to all native Korean speakers who participated in the filtering and evaluation process
• Supported by the Korean NLP research community

• Author: Dongjun Kim, Chanhee Park, Chanjun Park, Heuiseok Lim
• For questions or feedback, please open an issue on GitHub or contact Dongjun Kim: junkim100@gmail.com