telecom-okg

Efficient LLM Adaptation for Opinion Knowledge Graph Construction: Lessons from the Telecom Industry.

A flexible and efficient framework for training and aligning Large Language Models (LLMs). This package supports multiple training modes, including Pre-training (PT) and Supervised Fine-Tuning (SFT), utilizing Hugging Face tools and DeepSpeed for distributed performance.

• Multi-Mode Training: Support for Pre-training (PT) and Supervised Fine-Tuning (SFT).
• Architecture Support: Compatibility with Causal and Multimodal language models.
• Distributed Training: Integration with Hugging Face accelerate and deepspeed for scalable training across multiple GPUs.
• Efficient Downloading: A unified utility for fast, version-independent model acquisition from Hugging Face Hub.
• Dataset Management: Tools for shuffling, splitting, and preprocessing datasets into training, validation, and testing sets.
• Experiment Tracking: Seamless integration with Weights & Biases (W&B) for real-time monitoring and logging.

• Python 3.12
• PyTorch
• Hugging Face Transformers, Datasets, and Accelerate
• DeepSpeed
• WandB
• Flash Attention (optional, for supported architectures)

uv sync

The project environment is initialized by sourcing the provided shell script. This activates the virtual environment and exports necessary variables, eliminating the need to prefix subsequent commands with uv run.

source load_venv.sh

The authentication process is initiated via the following command:

hf auth login

Models are acquired from the Hugging Face Hub using the official Hugging Face CLI. It utilizes snapshot_download to ensure efficient, version-independent model acquisition.

Using specific inclusion filters:

hf download <MODEL_ID> \
    --local-dir ./models/<MODEL_ID> \
    --include "*.safetensors" \
    --include "*.json" \
    --include "*.txt" \
    --include "*.md"

Alternatively, excluding specific patterns (e.g., PyTorch weight files):

hf download <MODEL_ID> \
    --local-dir ./models/<MODEL_ID> \
    --exclude "*.pth"

The authentication process is initiated via the following command:

wandb login

Distributed training is initiated using the accelerate launch command. The framework supports overwriting configuration settings via CLI arguments.

export WANDB_PROJECT="telecom-okg"
export OMP_NUM_THREADS=1

accelerate launch \
    --num_processes <NUM_GPUS> \
    --mixed_precision "bf16" \
    --config_file configs/deepspeed_zero3.yaml \
    run_train.py \
    --training_mode ft \
    --model_name_or_path ./models/<MODEL_NAME> \
    --dataset_name <DATASET_PATH> \
    --output_dir ./output

Comprehensive execution templates and scripts are available in the examples/templates/ directory.

Raw text data format:

{
    "id": "item-001",
    "text": "The content for pre-training goes here."
}

Conversational data format:

{
    "id": "chat-001",
    "messages": [
        {
            "role": "user",
            "content": "Identify the capital of France."
        },
        {
            "role": "assistant",
            "content": "The capital of France is Paris."
        }
    ]
}

The following environment variables are utilized:

• WANDB_PROJECT: Defines the project name in Weights & Biases.
• OMP_NUM_THREADS: Controls the number of threads for OpenMP. Defaults to 1 to prevent system overload.

• src/telecom_okg: Core package containing alignment logic, loaders, and preprocessing utilities.
• configs/: Directory for accelerate and deepspeed configuration files.
• preprocess_chat_template/: Path for storing custom chat templates for SFT.
• data/: Example data files for PT and SFT training modes.
• examples/: Example shell scripts for training setup.

• Estimated Release Date: August 7, 2026

If you find this work useful, please cite our paper:

@inproceedings{yang2026telecomokg,
  author    = {Yang, Nai-Chi and Hsieh, Yu-Ming and Ma, Wei-Yun and Chang, Kuo-Wei},
  title     = {Efficient LLM Adaptation for Opinion Knowledge Graph Construction: Lessons from the Telecom Industry},
  booktitle = {Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  series    = {SIGIR '26},
  pages     = {5026--5031},
  year      = {2026},
  doi       = {10.1145/3805712.3808450},
  url       = {https://doi.org/10.1145/3805712.3808450}
}