This repository contains the code for Exp Template.

• 📦 uv
  • A Python package manager replacing pip and poetry.
• 📝 pyproject
  • For project management.
• ⚙️ hydra
  • For flexible configuration management.
• 📊 mlflow
  • For experiment tracking and visualization online.
• 🌱 encourage
  • A custom library for handling LLM inference, prompt handling, and utility functions.

To initialize the environment using uv, run the following command:

uv venv --python 3.12p
uv sync

When using this template you have to declare all your configuration parameters in the conf/defaults.yaml file. Also modify the conf/model/defaults.yaml and conf/data/defaults.yaml files to fit your needs.

To run a LLM you can use config from the launch.json file. If you want to run it without it you can use the following command:

 CUDA_VISIBLE_DEVICES=1 uv run start_vllm_server_as_process.py model=qwen2-7B  

To run the execution of the model you can use the following command:

uv run src/scivqa/evaluation/execution.py

If something broke in the evaluation you can use the following command to run the evaluation again:

uv run src/scivqa/evaluation/evaluate.py

But you have to change the output_folder in the defaults.yaml to the folder where the execution results are stored.