GyLLM

Lightweight LLM environments and rollout tools for reinforcement learning.

Packages:

• gyllm: environment API + registry + wrappers (packages/gyllm).
• nanorl: rollout + RL utilities (packages/nanorl).

This is a side project built by one man and one clanker. There will be sharp edges, some parts are undertested, and the RL code isn't going to scale by design.

Still, it seems to work at a fundamental level - reward goes up.

Gyllm = Gym + LLM. Clever, right?

It's my redesign of what an RL env could look like in the era of LLMs. The central abstraction is a Request, which carries an observation, maybe a reward, and it might call for an action. Every env step returns zero or more requests.

This simple structure allows seamlessly implementing batched/vectorized environments, multi-agent environments, heterogeneously batched environments, all under a single abstraction.

NanoRL = Nano + RL. Clever, right?

It's a basic implementation of a few RL algorithms to work well with gyllm on a single GPU. Right now it has REINFORCE, PPO and GRPO.

The unusual thing about NanoRL is that it completely removes the need for separate inference and training copies of the model, removing the weight update overhead. It does so by hacking into vllm's internal model and sharing weights with the transformers-based training model.

This approach, naturally, doesn't scale to more than one GPU. I only have one GPU at home, so that's fine.

import gyllm

env = gyllm.make("openenv/echo")
requests = env.reset()
actions = {r["actor"]: "hi" for r in requests if r["needs_action"]}
requests = env.step(actions)

If you clone the repo, you can run a some RL training scripts with uv run:

uv sync
# PPO on tic tac toe
uv run scripts/train_ppo_agent.py --config scripts/configs/ppo_ttt.yaml

# GRPO on MATH
uv run scripts/train_grpo_agent.py --config scripts/configs/grpo_math.yaml

Note: by default, the training scripts (and nanorl) require Weights & Biases (wandb). To run without logging, set WANDB_MODE=disabled (or WANDB_MODE=offline) in your environment.

gyllm-web is a lightweight local UI for exploring envs. It runs a FastAPI server that lists available environments, lets you create sessions, reset, and step them with manual actions while streaming the resulting requests/rewards. It's handy for debugging env dynamics, prompt formatting, and reward signals before you train.

uv pip install "gyllm[web]"
uv run gyllm-web --host 127.0.0.1 --port 8000

Or run it straight from the workspace without installing extras:

uv run --with "gyllm[web]" gyllm-web --host 127.0.0.1 --port 8000

Then open http://127.0.0.1:8000 and start a session for any registered env.

Python 3.12+.

Using uv:

uv venv --python 3.12
source .venv/bin/activate
uv pip install gyllm nanorl

Use the Spark-specific wheels and extras:

uv venv .venv --python 3.12
source .venv/bin/activate

uv pip install --prerelease=allow \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  "vllm @ https://github.com/vllm-project/vllm/releases/download/v0.14.0/vllm-0.14.0+cu130-cp38-abi3-manylinux_2_35_aarch64.whl" \
  "nanorl[spark]"

Notebook that mirrors scripts/ttt_reinforce.py:

uv venv --python 3.12
uv sync
# For DGX Spark:
uv sync --extra spark

uv run python -c "print('hello')"
uv run scripts/train_grpo_agent.py --config scripts/configs/grpo_gsm8k.yaml

MIT. See LICENSE.

OpenEnv envs in gyllm are adapted from https://github.com/meta-pytorch/OpenEnv (BSD 3-Clause).