LaminDB - A data framework for biology

Makes your data queryable, traceable, reproducible, and FAIR. One API: lakehouse, lineage, feature store, ontologies, LIMS, ELN.

Highlights:

• lineage → track inputs & outputs of notebooks, scripts, functions & pipelines with a single line of code
• lakehouse → manage, monitor & validate schemas; query across many datasets
• feature store → manage features & labels; leverage batch loading
• FAIR datasets → validate & annotate DataFrame, AnnData, SpatialData, parquet, .h5ad, zarr, ...
• LIMS & ELN → manage experimental metadata, ontologies & markdown notes
• unified access → storage locations (local, S3, GCP, ...), SQL databases (Postgres, SQLite) & ontologies
• reproducible & auditable → auto-version & timestamp execution reports, source code & environments; attribute records to users
• integrations → vitessce, nextflow, redun, and more
• zero lock-in & scalable → runs in your infrastructure; not a client for a rate-limited REST API
• extendable → create custom plug-ins based on the Django ORM
• production-ready → used in BigPharma, BioTech, hospitals & top labs

If you want a GUI, you can connect your LaminDB instance to LaminHub and close the drylab-wetlab feedback loop: lamin.ai.

Copy summary.md into an LLM chat and let AI explain or read the docs.

Install the lamindb Python package:

pip install lamindb

Create a LaminDB instance:

lamin init --storage ./quickstart-data  # or s3://my-bucket, gs://my-bucket

Or if you have write access to an instance, connect to it:

lamin connect account/name

Track a script or notebook run with source code, inputs, outputs, logs, and environment.

import lamindb as ln

ln.track()  # track a run
open("sample.fasta", "w").write(">seq1\nACGT\n")
ln.Artifact("sample.fasta", key="sample.fasta").save()  # create an artifact
ln.finish()  # finish the run

This code snippet creates an artifact, which can store a dataset or model as a file or folder in various formats. Running the snippet as a script (python create-fasta.py) produces the following data lineage.

artifact = ln.Artifact.get(key="sample.fasta")  # query artifact by key
artifact.view_lineage()

You'll know how that artifact was created and what it's used for (interactive visualization) in addition to capturing basic metadata:

artifact.describe()

You can organize datasets with validation & annotation of any kind of metadata to then access them via queries & search. Here is a more comprehensive example:

To annotate an artifact with a label, use:

my_experiment = ln.Record(name="My experiment").save()  # create a label record
artifact.records.add(my_experiment)  # annotate the artifact with the label

To query for a set of artifacts, use the filter() statement.

ln.Artifact.filter(records=my_experiment, suffix=".fasta").to_dataframe()  # query by suffix and the ulabel we just created
ln.Artifact.filter(transform__key="create-fasta.py").to_dataframe()  # query by the name of the script we just ran

If you have a structured dataset like a DataFrame, an AnnData, or another array, you can validate the content of the dataset (and parse annotations). Here is an example for a dataframe.

With a large body of validated datasets, you can then access data through distributed queries & batch streaming, see here: docs.lamin.ai/arrays.