DEV Community: Francisco Booth

Why HF_HOME Points To Ephemeral Storage On RunPod By Default And How To Fix It

Francisco Booth — Sun, 11 Oct 2026 14:23:40 +0000

If you have ever run a HuggingFace model download on RunPod and come back to find your cache gone, this is why.

The default is wrong for rented GPU pods

HuggingFace sets HF_HOME to ~/.cache/huggingface when no environment variable overrides it. On a standard Linux machine that is fine your home directory persists.

On RunPod, your home directory is on the container volume, which is ephemeral. When the pod stops, it is gone.

RunPod has two storage layers: the container volume, which is ephemeral, and the network volume mounted at /workspace, which persists across pod restarts. HF_HOME defaults to the wrong one.

This means every model download, every tokeniser, every dataset you pulled via datasets.load_dataset() gone on pod restart. You pay to download them again on the next run.

On large models that is not a minor inconvenience. A 7B model download at pod startup adds minutes and real cost every single time.

The fix is one line in your startup script:

export HF_HOME=/workspace/.cache/huggingface

/workspace is RunPod’s persistent volume mount. One line and your cache survives every restart.

Why this is easy to miss

There is no error message. HuggingFace downloads the files successfully to the ephemeral path. Your training run completes. The problem only surfaces when you restart the pod and the download happens again and even then it looks like a slow startup, not a misconfiguration.

Silent failures are the hardest to debug because you are not looking for them.

Automating the check

If you want this caught automatically before every job rather than relying on remembering to set the variable, ComputeFence checks it as part of its pre-flight scan:

uvx computefence doctor

It flags HF_HOME pointing to ephemeral storage and prints the exact export command for your platform. It also checks GPU visibility, Accelerate config, checkpoint directory persistence, and disk headroom the other silent failures that cost compute budget without a log entry.

The two-line pre-flight for every RunPod job

export HF_HOME=/workspace/.cache/huggingface
uvx computefence doctor

The first line fixes the cache. The second confirms everything else is configured correctly before the GPU bill starts.

GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence

My GPU Training Job Ran for 20 Hours and Produced Nothing. There Was No Error Message.

Francisco Booth — Tue, 06 Oct 2026 20:12:49 +0000

I was fine-tuning the Exolio AI classifier that detects machine-generated texts. The classifier can be seen at huggingface.co/FranciscoBooth1/exolio-ai-detector.

Between January and April 2026, I ran dozens of training jobs using Vast.ai. Cost of runs: approximately £1,000.

All of the eventually identified failed runs had one thing in common: none of them gave me an obvious sign that they had failed.

The run that ran all night and gave no output

For Exolio v15, I was switching from DeBERTa-v3-large to ModernBERT. I rented a machine, uploaded the training script, installed dependencies, and initiated the run.

Everything in the terminal looked exactly as expected:

Uploading training script...
Installing Python dependencies (this may take several minutes)...
[Dependencies installed successfully]
Starting training in tmux session 'ssh_tmux'...
Training started successfully! Instance will auto-destroy when complete.

The script returned to my local terminal and training started running in tmux on the remote machine. There was no way to monitor what happened inside that tmux session. Everything looked good. I went to sleep.

The next day I tried to SSH back in:

ssh: connect to host [instance IP] port 55157: Connection refused

The instance was gone. The log files disappeared along with it. I checked my HuggingFace repository to see if there were any results:

Last modified: 2026-04-12 01:20:33+00:00
.gitattributes
config.json
deberta_3class_latest.zip
model.safetensors
tokenizer.json
training_args.bin

There was no modernbert_latest.zip. The last modification time was April 12 which was the previous DeBERTa run. No new files had been added.

That was the only sign that something had gone wrong. Not an error message. Not a bad benchmark score. Absence of new files.

I could not recover the logs the instance auto-destroyed and took everything with it so I cannot identify the root cause. Twenty hours had passed, the instance was gone, and no new model had appeared in my HuggingFace repository.

I ran a training job, nothing obviously failed, and after twenty hours the infrastructure disappeared and there was no artifact left. I was not able to tell exactly why.

Another failure: training on CPU for hours without knowing

Earlier in the same period I rented what seemed to be a good machine and started fine-tuning DeBERTa. The loss function was printing. The iteration counter was counting. Everything was going as expected.

Except the speed was wrong. I saw approximately 24 seconds per iteration. On working instances the same workload had been around 0.4 seconds per iteration.

I ran nvidia-smi. GPU utilisation: 0%.

My script resolved the device to CPU because torch.cuda.is_available() returned False no exception thrown, just CPU. The machine had CUDA installed but my PyTorch environment could not see the GPU. Training continued on CPU with no errors while I was paying full GPU rates.

I stopped the run. Rented a different machine. Restarted.

The pattern

In both cases there was no error message that I could act on. In the first case there was no way to know anything had gone wrong until I tried to SSH back in. In the second case I noticed a strange iteration speed and stopped the job.

In the second case the problem was visible before python train.py. torch.cuda.is_available() returning False is visible before the job starts. GPU utilisation at 0% is visible within seconds of launch.

Additionally and not as an explanation of the overnight run, since I had no logs there are other silent failure modes that are also detectable before the training job starts. At least some of the conditions surrounding these failures were detectable before python train.py if I had known to check them:

Whether PyTorch could see the GPU: torch.cuda.is_available() returns False before the job starts
Whether the HuggingFace cache was configured to use a potentially ephemeral path: an environment variable check
Whether the checkpoint output directory was under a known instance-local prefix: a path check

None of these checks were complicated. I just did not perform them systematically on each new machine.

What I built

After enough of these I decided to build a pre-flight checker. One command before each training job:

pip install computefence && computefence doctor

Or without installing the package:

uvx computefence doctor

Real output from uvx computefence doctor on my local Mac — on a real GPU pod the PyTorch and volume checks run against actual hardware:

ComputeFence v0.2.6 — Pre-flight diagnostic
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

5 WARNINGS · 0 BLOCKERS · 1 PASSED

Environment
✓ Python 3.14.3
⚠ PyTorch not found — skipping GPU checks
Fix: pip install torch torchvision torchaudio

Storage
⚠ HF_HOME is not set. HuggingFace may cache to instance-local storage
that is not persistent across pod termination.
Fix: Set HF_HOME to a persistent volume path before training
⚠ No mounted persistent volumes detected at /workspace, /runpod-volume, or /vast
Fix: Mount a network volume at /workspace before launching your job
⚠ Root disk (/) — 12.1 GB free of 460.4 GB (below 20 GB)
Fix: Free up disk space or move checkpoints to a larger volume

Dataset
⚠ No dataset path provided — skipping dataset checks

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
5 warning(s) found. Review before launching.

Checks:

1) GPU and CUDA visibility - fails if torch.cuda.is_available() returns False; reports relevant CUDA and PyTorch version information and warns on detected version skew

2) HuggingFace cache path — warns if HF_HOME is not set or points to a path known to be instance-local on supported providers.

3) Accelerate configuration — warns if there is an inconsistency between the configured number of processes and the GPUs visible in the environment.

4) Disk headroom — warns when less than 20 GB is available; blocks if less than 5 GB is available.

5) Checkpoint output directory — warns if the path is under known instance-local prefixes that may not survive pod termination.

6) Dataset — optionally checks for duplicates and conflicting labels with --dataset.

Checks that it does not do: training script correctness, model architecture, learning rate safety, runtime behaviour during training, or GPU utilisation during training. ComputeFence is a pre-flight check. It does not monitor the running job.

A passing check does not guarantee that the training job will succeed.

The pre-flight check took under 30 seconds on my test pod.

The ask

ComputeFence is free, open source, and MIT licensed.

If you run training jobs on Vast.ai, RunPod, or any other rented GPU, try uvx computefence doctor before your next run. If it finds something interesting, paste the output in the comments below.

GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence

24 seconds per iteration instead of 0.4. I paid for six hours of GPU compute and trained on CPU the entire time.

Francisco Booth — Sun, 06 Sep 2026 10:22:30 +0000

Failure 1 — CUDA silently fell back to CPU

My training job launched on Vast.ai and ran to completion. Iteration time was 24 seconds instead of 0.4 seconds. CUDA had fallen back to CPU silently. PyTorch logged nothing. I had been billed for six hours of GPU compute while training on an unaccelerated CPU thread the entire time.

Failure 2 — HF_HOME on ephemeral disk

Every fresh pod re-downloaded base model weights to /root/.cache — the ephemeral container disk wiped on pod shutdown. Same download, same cost, every run. The fix is one line. I did not know it for weeks.

export HF_HOME=/workspace/.cache/huggingface

Failure 3 — Accelerate config mismatch

My accelerate config had num_processes: 2. The pod had one GPU. Training launched, appeared to run, and produced garbage output. No error thrown. The configuration simply did not match the hardware.

Failure 4 — Dirty dataset

28,432 duplicate rows. 312 conflicting labels. Loss collapsed to 0.693 on step one — the exact cross-entropy value for random guessing on a binary classification problem. I spent three days debugging model architecture and learning rates before I scanned the dataset.

The pattern

None of these printed an exception. All of them were visible before python train.py if I had known what to check.

I spoke to 13 ML engineers on RunPod, Vast.ai, and AWS. Nine had lost checkpoints to ephemeral disk. Eight had shipped silent garbage with no log error.

An ML engineer at a major German industrial company told me his team maintains a five or six part manual bash script that they run before every GPU job because no standardised tool exists.

When enterprise teams are hand-rolling bash scripts, the problem is real.

So I built ComputeFence.

Standard install:

pip install computefence
computefence doctor

Zero install on a fresh pod:

uvx computefence doctor

30 seconds. Every warning prints the exact fix command.

ComputeFence v0.2.5 — Pre-flight diagnostic
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2 WARNINGS · 0 BLOCKERS · 3 PASSED

Storage
⚠ HF_HOME is not set — model weights will cache to ephemeral disk
Fix: export HF_HOME=/workspace/.cache/huggingface
⚠ Root disk (/) — 14.3 GB free (below 20 GB)
Fix: Free up disk or move checkpoints: df -h to check usage

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2 warning(s) found. Review before launching.

It checks GPU and CUDA visibility, HuggingFace cache path persistence, Accelerate GPU count versus what is actually on the instance, disk headroom for checkpoints, and checkpoint output directory persistence via --output-dir.

It does not check training script correctness, learning rate safety, or anything that only fails during the run. HuggingFace cache and your checkpoint output directory are separate paths — fixing one does not fix the other. ComputeFence catches configuration mistakes before the GPU starts billing.

Add it to your pod startup script. Exit code is 0 on warnings and 1 only on hard blockers so training will still launch on warnings:

pip install computefence && computefence doctor && python train.py

Does it catch anything real?

Three operators have run it before paid training jobs on RunPod.

One caught HF_HOME writing to ephemeral disk and an Accelerate config mismatch on a RunPod A100. He fixed both before launch.

One confirmed the storage warning matched real pod behaviour and said he would not have caught it without the tool.

Aaron, an ML engineer who ran ComputeFence on a RunPod A40, said the Accelerate warning would have made him stop and investigate before launching.

Still very early.

If you run it before your next paid job on RunPod, Vast.ai, Lambda Labs, or any bare metal instance — paste your computefence doctor output in the comments or open an issue on GitHub. I want to know which checks fire on real setups and which are noise.

Free. MIT licensed. Works on any bare metal GPU provider.

GitHub: github.com/Francisco-Booth/ComputeFence