Back to Blog
Production

LoRA Training Problems For Video Models

Fix the most common LoRA training problems on LTX-2.5: CUDA out of memory, frame count errors, Gemma mismatches, bad validation samples and slow runs.

LTX Team
Production
LoRA Training Problems For Video Models
Key Takeaways
  • Fix CUDA out of memory with the low VRAM config, checkpointing, 8-bit encoder and quantization.
  • Resolve frame-count, Gemma mismatch and split-pack errors with one-line config fixes.
  • Check the validation recipe before retraining, then tune rank, steps and checkpoints.

Most LoRA training failures on LTX-2.5 fall into five buckets: GPU memory, frame and resolution alignment, mismatched checkpoint components, a validation recipe that doesn't match the checkpoint, and speed. Each one has a documented fix in the LTX trainer, usually a single config line or command flag. This guide maps the error you see to the change you make, so you can fine-tune LTX without guessing.

Prerequisites before you debug

LTX-2.5 LoRA training through the ltx-trainer package expects Linux with CUDA (CUDA 13+ recommended), because the trainer depends on triton, which is Linux-only. The standard config is sized for an NVIDIA GPU with 80GB VRAM. For 32GB cards such as the RTX 5090, the trainer ships configs/t2v_lora_low_vram.yaml, which enables INT8 quantization and other memory optimizations. All model paths must be local files. If you only need inference rather than training, the hosted LTX API avoids the hardware requirement.

How do you fix CUDA out of memory?

CUDA out of memory is the most common LoRA training problem on video models, because every frame adds tokens. Work through the fixes below in order, cheapest first.

Start from the low VRAM config

The low VRAM config is the fastest fix on a 32GB GPU. Copy t2v_lora_low_vram.yaml and change only your paths and dataset before touching anything else, so you start from a combination the trainer team already sized for that card.

Five memory switches

Memory switches in the trainer config, each documented in the troubleshooting guide:

optimization:
 enable_gradient_checkpointing: true
 batch_size: 1
 gradient_accumulation_steps: 4   # effective batch size = 4
 optimizer_type: "adamw8bit"
acceleration:
 load_text_encoder_in_8bit: true
 quantization: "int8-quanto"      # also int4-quanto, fp8-quanto

Gradient checkpointing trades speed for memory and is recommended for most runs. The 8-bit text encoder roughly halves the VRAM the Gemma encoder holds while caption embeddings are computed, saving about 8 GiB, but it requires bitsandbytes and passes the full-precision weights through host RAM first, so budget around 26 GB of system RAM for LTX-2.5. Gradient accumulation keeps a larger effective batch while each step holds a single sample.

OOM only during validation

OOM that appears only when validation videos are sampled usually means AdamW state and the VAE decoder cannot coexist on the GPU, which is typical of full fine-tunes and high-rank LoRA runs. Set offload_optimizer_during_validation: true under acceleration. The offload happens once per validation interval, not per step. If the crash happens even earlier, during preprocessing, lower the resolution or frame count in --resolution-buckets.

Frame count and resolution errors

Frame count errors show up as frames must satisfy (frames - 1) % T == 0. With the default VAE, T is 8, so valid counts include 1, 9, 17, 25, 33, 41, 49, 57, 65, 73, 81, 89, 97 and 121, while 24, 32, 48, 64 and 100 fail. Width and height must be multiples of 32. The same rules apply to validation.video_dims, so a valid dataset can still fail at the first validation step.

Resolution buckets are also your best memory dial. The transformer sequence length is (H/32) * (W/32) * ((F-1)/8 + 1), so a 768x448x89 bucket gives 24 * 14 * 12 = 4,032 tokens. Halving the frame count roughly halves that number, while dropping from 960x544 to 512x512 cuts the spatial term by about half. The documentation's own guidance: larger spatial size with fewer frames for detail, smaller spatial size with more frames for motion. If you preprocess with several buckets, set batch_size: 1, because samples of different shapes cannot share a batch.

Checkpoint and text encoder mismatches

Checkpoint mismatches produce errors that look scary but have one-line fixes.

Gemma version mismatch

Gemma version mismatch means the checkpoint and text encoder root are incompatible. LTX-2.5 requires the LTX-specific fine-tuned Gemma 4 (for example gemma4-12b-ltx-v1), not Google's vanilla Gemma 4, and never the Gemma 3 root from an LTX-2.3 run. The checkpoint's gemma_source_checkpoint metadata names the right one.

Split pack carries no VAE

Split pack errors ("is the transformer of a split checkpoint pack and carries no video VAE") mean model_path points at a transformer-only file. Name every component:

model:
 model_path: "/path/to/diffusion_models/ltx-2.5-22b-dev-transformer-bf16.safetensors"
 text_encoder_path: "/path/to/text_encoders/gemma4-12b-with-proj-ltx-2.5-bf16.safetensors"
 video_vae_path: "/path/to/vae/ltx-2.5-video-vae-bf16.safetensors"
 audio_vae_path: "/path/to/vae/ltx-2.5-audio-vae-bf16.safetensors"

Preprocessing needs the same components through --video-vae-path and --audio-vae-path.

Moving a dataset from LTX-2.3

Dataset reuse after switching from LTX-2.3 to LTX-2.5 fails because the caption features in conditions/ come from the Gemma model and are not interchangeable across versions. Preprocessing skips existing .pt files by default, so stale features survive silently. Preprocess into a fresh directory, or rerun process_dataset.py with --overwrite. Two smaller setup errors: "No module named 'ltx_trainer'" means you ran scripts outside uv run, and "Model path does not exist" means you passed a URL instead of a local path.

Why do validation samples look bad?

Validation samples that look bad do not always mean the LoRA is bad. Check the validator before you retrain. The trainer's validation runner is one-stage, and for an LTX-2.5 distilled checkpoint the documented starting recipe is 8 steps, CFG scales of 1.0 and STG scales of 0.0. Running distilled weights with the general example values (30 steps, video CFG 3.0) can make a healthy LoRA look broken. The production two-stage distilled workflow lives in ltx-pipelines, not in the training-time validator.

Once the recipe matches, the documented fixes are:

• Condition the validation: a first_frame image gives a more reliable read than pure text-to-video.

• Fix captions: LTX models prefer long, detailed captions covering visuals and audio. Auto-generated captions can hallucinate, so review dataset.json before preprocessing.

• Check target modules: for audio-video training, use patterns like "to_k" that match both branches instead of "attn1.to_k", which only hits video self-attention.

• Verify the data itself: scripts/decode_latents.py decodes the precomputed latents (add --with-audio for audio) so you can compare them with the originals.

LoRA rank, steps and overfitting

LoRA rank sets capacity. The documented typical range is 8 to 128, with alpha usually equal to rank, and the troubleshooting guide suggests trying 64 or 128 when outputs lack detail. Learning rate typically sits between 1e-5 and 1e-3 (the example config uses 1e-4), and the training guide notes that lower learning rates often produce better results.

LoRA overfitting is easiest to catch in validation samples, which is why the training guide recommends watching them closely. Use the checkpoint settings to give yourself a way back: with checkpoints.interval: 250 and a generous keep_last_n, every saved lora_weights_step_XXXXX.safetensors pairs with a validation sample, and you can ship the step just before the samples start copying your training clips. Start with a few hundred steps on a small dataset to confirm the pipeline works, then scale toward the 2,000 to 3,000 step range the docs use in examples.

Why is LoRA training slow?

LoRA training speed is the cost of the memory fixes above. If you have headroom, set enable_gradient_checkpointing: false first, since checkpointing is the documented trade of speed for memory. Then add torch.compile through Accelerate:

uv run accelerate launch --config_file configs/accelerate/ddp_compile.yaml \
 scripts/train.py configs/t2v_lora.yaml

Multi-GPU runs use the DDP or FSDP configs in configs/accelerate/, and the effective batch size becomes batch_size × num_processes. Turn memory optimizations off one at a time while watching nvidia-smi, so you know which switch is buying back speed.

A pre-flight checklist

A pre-flight check catches most of these problems before a long run:

1. Preprocess a small subset first, with --decode to inspect the decoded videos.

2. Confirm every frame count and dimension sits on the VAE grid.

3. Match the text encoder to the checkpoint metadata.

4. Set the validation recipe to match the checkpoint type.

5. Run watch -n 1 nvidia-smi during the first validation interval.

The LoRA training run setup guide, the LoRA dataset guide and the guide to running and monitoring LoRA training cover the steps around this checklist in depth.

Summary

LTX-2.5 LoRA training problems usually trace to memory, frame alignment, mismatched checkpoint components or a mismatched validation recipe. CUDA out of memory responds to the low VRAM config, gradient checkpointing, the 8-bit text encoder, quantization and the 8-bit optimizer. Frame counts must satisfy (frames - 1) % 8 == 0 with the default VAE, and dimensions must be multiples of 32. LTX-2.5 needs its LTX-specific Gemma 4 text encoder and caption features preprocessed with that encoder, and distilled checkpoints should start validation at 8 steps with CFG 1.0 and STG 0.0.

Ready to train? Open the LTX Trainer, start from the low VRAM or standard config that matches your GPU, and run the pre-flight checklist before your first full LoRA training run.

Table of Contents: