ARIA: Autonomous Reinforcement-learning with Intelligent Abstraction
ARIA is a modular RL framework for symbolic music generation with human-in-the-loop (HIL) adaptation.
ARIA combines three agents:
- Perceiving Agent (GHSOM): clusters musical features and exposes structure.
- Generative Agent (DQN family): selects cluster-level actions via Tianshou 2.x.
- Human Agent: provides preference feedback for adaptation.
Supported algorithm variants include dqn, dueling_dqn, c51, and rainbow.
Prerequisites:
- Docker Engine 20.10+
- Docker Compose v2+
- NVIDIA Container Toolkit for GPU (optional)
# Build and start
cd docker && docker-compose build
docker-compose up -d
# Open shell
docker-compose exec music-generation-rl bash
# Verify GPU
python -c "import torch; print(torch.cuda.is_available())"
# Run training
python scripts/training/run_training.py --config configs/training.yamlNotebook profile:
docker-compose --profile notebook up -d
# http://localhost:8888pip install -r requirements.txtUse the unified pipeline to produce feature, reduction, GHSOM, and cluster-evaluation artifacts.
# Full pipeline
python scripts/preprocessing/run_preprocessing_pipeline.py \
--config configs/preprocessing/full_pipeline.yaml
# Resume from existing stage outputs
python scripts/preprocessing/run_preprocessing_pipeline.py \
--config configs/preprocessing/full_pipeline.yaml \
--skip-feature-extraction \
--features-artifact artifacts/features/raw/existing
python scripts/preprocessing/run_preprocessing_pipeline.py \
--config configs/preprocessing/full_pipeline.yaml \
--skip-feature-extraction \
--skip-dimensionality-reduction \
--reduced-artifact artifacts/features/reduced/existingMain preprocessing configs live in configs/preprocessing/.
Typical output layout:
artifacts/preprocessing/<run_id>/
extracted/
reduced/reduced/
ghsom/trained/
evaluation/evaluated/
pipeline_manifest.json
pipeline_config.yaml
# Default training
python scripts/training/run_training.py
# Custom config/overrides
python scripts/training/run_training.py \
--config configs/training.yaml \
--timesteps 200000 \
--seed 42
# Benchmark variant config
python scripts/training/run_training.py \
--config configs/benchmark/main/rainbow_cl.yaml# Interactive HIL session
python scripts/inference/run_inference_pipeline.py hil \
--checkpoint artifacts/training/run_xyz/checkpoints/final.pth \
--config configs/inference/interactive.yaml
# Sequence analysis
python scripts/inference/run_inference_pipeline.py analyze \
--sequences outputs/inference/sequences.json
# Preference-guided simulation
python scripts/inference/run_inference_pipeline.py simulate \
--checkpoint path/to/checkpoint.pth \
--inference-config configs/inference/simulation.yaml \
--scenario calm_relaxation# MIR policy evaluation
python scripts/evaluation/run_mir_evaluation.py policy \
--checkpoint <path/to/checkpoint.pth> \
--config configs/inference/mir_evaluation.yaml \
--strategy transition_smoothed \
--n-episodes 100
# MIR post-hoc analysis
python scripts/evaluation/run_mir_evaluation_analysis.py \
--input outputs/mir_evaluation/TIMESTAMP/policy_mir_comparison.json \
--mode auto
# Policy diagnosis
python scripts/evaluation/run_policy_diagnosis.py \
--config configs/inference/policy_diagnosis.yaml \
--output outputs/diagnosis/ \
--checkpoint <path/to/checkpoint.pth>For MIR compare mode, checkpoint specs must use label:path with no spaces after :.
Use split YAML configs by domain:
configs/training.yamlconfigs/preprocessing/*.yamlconfigs/inference/interactive.yamlconfigs/inference/simulation.yamlconfigs/inference/mir_evaluation.yamlconfigs/inference/content_retrieval.yamlconfigs/inference/policy_diagnosis.yamlconfigs/benchmark/**andconfigs/ablations/**
Precedence: CLI args > YAML config > code defaults.
src/ Core library (agents, envs, training, inference, evaluation)
scripts/ User-facing CLIs
configs/ YAML config families
artifacts/ Generated preprocessing/training artifacts
test/ Unit and integration tests
docker/ Container setup
python -m pytest
python -m pytest --cov=src --cov-report=html
ruff check src/ scripts/
ruff format src/ scripts/
pre-commit run --all-filesThis project is licensed under the MIT License. See LICENSE.
Please follow CONTRIBUTING.md for commit message conventions and workflow guidance.