This repository provides the code and resources for the Open Korean Historical Corpus, a large-scale, openly licensed dataset of historical Korean texts.

This dataset is a large-scale collection of texts spanning 1,300 years of Korean history.

Key Features:

• Size: 17.7 million documents and 5.1 billion tokens.
• Temporal Span: 7th century to 2025.
• Sources: Compiled from 19 distinct institutional archives and public domain collections.
• Languages: Covers 6 languages, including:
  • Classical Chinese
  • Middle Korean
  • Early Modern Korean
  • Modern Korean
  • North Korean
  • Japanese
• Writing Systems: Includes under-represented scripts like Korean-style Sinitic (Idu) and Hanja-Hangul mixed script.

The full corpus is available for download on the Hugging Face Hub.

• Sample (1.3 MB): ./sample.jsonl
• Dataset (28.6 GB): https://huggingface.co/datasets/seyoungsong/Open-Korean-Historical-Corpus

This repository contains the code used for preprocessing, language and script identification, and classification (including the Idu classifier) described in our paper. Please note that the initial web crawling scripts are not included here to avoid placing an undue burden on the source institutions' servers. Researchers interested in the crawling code may request it from the corresponding authors, subject to an agreement on responsible use.

• Code: The code in this repository is released under the MIT License.
• Data: The Open Korean Historical Corpus is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) License.

@article{song2025open,
  author  = {Seyoung Song and Nawon Kim and Songeun Chae and Kiwoong Park and Jiho Jin and Haneul Yoo and Kyunghyun Cho and Alice Oh},
  journal = {ArXiv preprint},
  title   = {Open Korean Historical Corpus: A Millennia-Scale Diachronic Collection of Public Domain Texts},
  url     = {https://arxiv.org/abs/2510.24541},
  volume  = {abs/2510.24541},
  year    = {2025}
}