An fsspec filesystem for
Lore, Epic Games' open-source version control
system — the Lore equivalent of fsspec's built-in GitFileSystem.
It lets any fsspec-aware tool (pandas, pyarrow, zarr, dask, fsspec.open, …) read
files out of a Lore repository at an arbitrary branch or revision via lore://
URLs, backed by the official lore Python bindings (liblore).
Lore is a centralized VCS — the server is the source of truth — so the
natural address of a repository is its server URL (lore://host:port/repo,
the same URL lore clone takes). Point the filesystem at that URL and it
maintains a managed bare local instance (metadata only, under
~/.cache/lore-fsspec; override with cache_dir=) and streams file content
lazily from the server as you read; fetched content is retained locally, so
repeat reads don't refetch. Writes commit and push back to the server.
Attaching to an existing local clone also works, exactly like fsspec's
GitFileSystem — the clone is Lore's client-side session handle either way
(the lore client API is instance-rooted; there is no clone-less remote mode).
import fsspec
# Remote-first: address the repo by its Lore server URL.
fs = fsspec.filesystem("lore", path="lore://vcs.example.com:41337/my-project", ref="main")
fs.ls("Content/Maps")
with fs.open("Content/Config/Game.ini", "rt") as f:
print(f.read())
# URL form: lore://<host>:<port>/<repo>[:<ref>][@<in-repo-path>]
with fsspec.open(
"lore://vcs.example.com:41337/my-project:main@Content/Config/Game.ini", "rt"
) as f:
print(f.read())
# Or attach to an existing local clone (`lore clone ...`), GitFileSystem-style.
# URL form: lore://<clone-path>[:<ref>][@<in-repo-path>]
fs = fsspec.filesystem("lore", path="/path/to/clone", ref="main")lore exposes a real coroutine API, so LoreFileSystem is an fsspec
AsyncFileSystem — usable directly from asyncio for concurrent reads:
fs = fsspec.filesystem("lore", path="/path/to/clone", ref="main", asynchronous=True)
data = await fs._cat_file("Content/Config/Game.ini") # awaitable
many = await fs._cat(["Content/A.ini", "Content/B.ini"]) # runs concurrentlyWrites go through a transaction that maps onto Lore's native stage→commit flow — all files in the block land as a single revision (or none, on error):
fs = fsspec.filesystem(
"lore", path="lore://vcs.example.com:41337/my-project", ref="main", writable=True
)
with fs.transaction(message="Import baked lighting"):
fs.pipe_file("Content/Lighting/Baked.bin", data)
fs.pipe_file("Content/Config/Game.ini", ini_bytes)Branch topology is exposed as explicit methods (not folded into the write
transaction), mirroring fetch(). A transaction can target a branch so the
revision lands there in isolation; merging back is a separate, explicit step
because it can conflict:
fs.branches() # ['main']
fs.create_branch("import-job", checkout=True) # like `git switch -c`
# Commit onto a branch in isolation, then restore the original branch on exit.
with fs.transaction(message="Import", branch="import-job", create=True):
fs.pipe_file("Content/Lighting/Baked.bin", data)
# Clean merges land as one revision; a conflicting merge is aborted and raises
# with the conflicting paths (auto-resolution is intentionally never performed).
fs.merge("import-job")End-to-end usecases, distilled from the integration benchmark
(tests/test_benchmark_parquet.py). They show the library doing real
data-engineering work, not just listing files.
Upload a multi-file parquet dataset as one Lore revision (all files land
together, or none on error), then read it straight back with two engines —
DuckDB (SQL) and Polars (DataFrame) — through the lore:// protocol:
import duckdb
import polars as pl
from lore_fsspec import LoreFileSystem
fs = LoreFileSystem(path="/path/to/clone", writable=True)
# Atomic import: every shard becomes part of a single revision.
with fs.transaction(message="import events dataset"):
for shard in ("part-000.parquet", "part-001.parquet", "part-002.parquet"):
fs.put_file(f"/local/events/{shard}", f"warehouse/events/{shard}")
# Query with DuckDB by registering the filesystem under the 'lore' protocol.
con = duckdb.connect()
con.register_filesystem(fs)
rows, total = con.sql(
"SELECT count(*), sum(amount) "
"FROM read_parquet('lore://warehouse/events/*.parquet')",
).fetchone()
# Or read DataFrames directly via fs.open file objects (column projection works).
df = pl.concat(
pl.read_parquet(fs.open(p), columns=["region", "amount"])
for p in sorted(fs.glob("warehouse/events/*.parquet"))
)
by_region = df.group_by("region").agg(pl.len()).sort("region")Ingest a new partition on an isolation branch, audit it without exposing it on
main, then publish it atomically by merging — the classic
"write-audit-publish" pattern:
fs = LoreFileSystem(path="/path/to/clone", writable=True) # starts on 'main'
# WRITE: commit the new partition onto a fresh branch, not main.
with fs.transaction(message="ingest Feb partition", branch="ingest-2026-02", create=True):
fs.put_file("/local/part-new.parquet", "warehouse/events/part-new.parquet")
assert fs.ref == "main" # the transaction restored the original branch on exit
# AUDIT: main is untouched; read the new data back by targeting the branch ref.
fs.ls("warehouse/events") # main: existing partitions only
fs.ls("warehouse/events", ref="ingest-2026-02") # branch: includes part-new
audited = pl.read_parquet(
fs.open("warehouse/events/part-new.parquet", ref="ingest-2026-02"),
)
# PUBLISH: a clean merge lands as one revision; a conflicting one aborts and raises.
fs.merge("ingest-2026-02")uv sync # install deps (fsspec, lore-vcs)
uv run pytest # run testslore ships a platform-specific liblore shared library; tests skip when the
current platform's library isn't available.
MIT (matching Lore).