Rethink cfg.DATA as a generic multiprocessing cache #1978

Summary

cfg.DATA currently mixes two responsibilities:

1. a generic runtime cache for lazily loaded OGGM data
2. a container intended to support multiprocessing reuse of cached state

This was pragmatic historically, but it now creates unclear semantics and likely avoidable overhead.

I think OGGM should stop using cfg.DATA as a generic shared cache for heavy read-only objects and instead move toward typed, per-process lazy caches plus narrowly scoped shared state where needed.

Background

cfg.DATA seems to have started as a general replacement for special-purpose globals such as demo metadata. Later, it was extended to support multiprocessing reuse for cached reference datasets.

Today it stores a mix of objects, including:

• pandas DataFrames
• pandas Series
• lists and sets
• nested dicts
• DEM grid objects

That makes it a broad mutable global with mixed semantics.

Why this is still questionable on Linux/fork

Even on Linux, where major OGGM runs typically use fork, the current pattern is not obviously the right one for heavy read-only cached data like geodetic MB tables.

There are three different designs to compare:

1. preload the DataFrame in the parent, then fork workers and let them inherit it read-only via copy-on-write
2. let each worker maintain its own local lazy cache
3. store the DataFrame in a manager-backed shared dict such as cfg.DATA

For large read-only tables, option 3 is often the weakest of the three, even under fork:

• pre-fork loading avoids repeated file reads and avoids proxy overhead on each access
• per-worker local caching avoids proxy overhead and keeps semantics simple
• manager-backed caching adds IPC and pickle/unpickle overhead to every top-level read

So the main concern here is not just spawn. It is that a manager-backed global cache is a weak fit for large read-only scientific tables even in the fork case.

Secondary note on spawn

This issue is not Mac-specific. However, the same design looks even less attractive under spawn-based multiprocessing, where worker startup and state reconstruction are already more expensive.

Problems

1. Manager/proxy-backed access is expensive for large cached objects

If cfg.DATA is manager-backed, each top-level read goes through proxy IPC plus pickle/unpickle of the stored object.

That is a poor fit for cached payloads like:

• geodetic mass-balance DataFrames
• temperature-bias DataFrames
• DEM grid bundles

A quick local benchmark on macOS gave roughly:

• manager dict read of a large DataFrame: about 5 ms per access
• manager dict read of a medium list: about 0.9 ms per access
• local dict read: effectively negligible

So this only helps if it avoids a much more expensive repeated file load. Under fork, it may also lose to simply loading once before worker creation and letting workers inherit the object read-only. For repeated lookup-heavy use, it can become a net loss.

2. Mutation semantics are easy to misuse

Values fetched from a manager dict are copied out. In-place mutation does not propagate back unless the caller reassigns the value.

For example:

cfg.DATA[key].append(value)

is not safe if the stored value comes out of a proxy-backed dict.

This makes cfg.DATA look like a normal dict while not fully behaving like one.

3. It is likely inferior to simpler alternatives for geodetic MB data

For geodetic MB specifically, the most natural alternatives are:

• load once in the parent before forking and rely on copy-on-write inheritance
• use an explicit per-worker lazy cache

Both options keep read access local and avoid repeated manager round-trips.

That makes the current cfg.DATA pattern hard to justify for a large read-only reference table.

4. The abstraction is too broad

cfg.DATA currently mixes together unrelated concerns:

• reference/calibration tables
• DEM/grid helper state
• lookup lists
• demo metadata
• workflow caches

That makes it hard to reason about:

• what is cached
• who owns it
• whether it is safe to mutate
• whether it should be shared
• whether cache invalidation matters

Proposal

A. Stop using cfg.DATA as a generic cache for heavy read-only multiprocessing data

cfg.DATA should no longer be the default place for arbitrary runtime-loaded objects that we want to reuse across workers.

B. Move heavy read-only caches to per-process lazy caches

For helpers like:

• get_geodetic_mb_dataframe
• get_temp_bias_dataframe
• similar file-backed reference tables

use explicit per-process lazy caching instead, e.g.:

• functools.lru_cache
• module-level dicts keyed by file path / dataset / options

This fits OGGM's multiprocessing model well because workers are typically long-lived, so each worker can load once and then reuse cheaply.

This is attractive even on Linux/fork, because it keeps reads local and avoids manager-proxy traffic.

C. Keep shared state narrow and explicit

If some state truly must be synchronized across workers, keep it separate and narrowly scoped.

Roughly:

• shared coordination state: maybe manager-backed
• large read-only datasets: local per-process cache
• nested mutable objects: avoid storing them in manager-backed dicts

D. Prefer typed caches over one generic global bag

Instead of cfg.DATA, prefer caches owned by the relevant module, for example:

• reference-data cache in downloads/utilities
• DEM/grid cache in DEM-related code
• workflow lookup cache in workflow code

Suggested first steps

1. Move get_geodetic_mb_dataframe to a dedicated per-process cache keyed by file path and regional.
2. Move get_temp_bias_dataframe to the same pattern.
3. Move DEM grid loading away from cfg.DATA['dem_grids'] into a dedicated lazy loader.
4. Audit current cfg.DATA[...] writes and classify them as:
   • read-only local cache
   • true shared coordination state
   • legacy mutable global

Benefits

• lower overhead in multiprocessing hot paths
• clearer ownership of caches
• less risk of silent bugs from proxy-fetched mutable values
• easier maintenance and testing