Skip to content
sanjaynagiPublic

About

A python-based, AI-native scientific workflow orchestrator

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Ginkgo

License Python Tests Quality Documentation

Ginkgo is a scientific workflow orchestrator built for the 21st century.

  • @flow and @task() — define workflows in plain Python, no DSL to learn
  • An intuitive, aesthetic API built specifically for data science and bioinformatics
  • natively dynamic workflows — expand workflows during runtime from resolved tasks
  • content-addressed caching — never recompute what hasn't changed
  • isolated environments — pixi or containers, per task
  • agent-friendly — built from the ground up for workflows to be built and operated by AI agents
  • cloud-native I/O — stream inputs directly from S3, GCS, or Azure and stage outputs back, without local copies
  • deep observability — provenance and CLI tooling

See it in action

A bacterial GWAS, from 60 assemblies to resistance-associated unitigs: dynamic task graphs, pixi environments, caching, provenance, measured resource usage, and resuming after a failure. The workflow is in examples/bacterial_gwas.

Watch the walkthrough: a bacterial GWAS run with Ginkgo

Documentation

📖 sanjaynagi.github.io/ginkgo

It covers installation, quickstart, core concepts, environments, notebook tasks, caching, CLI usage, and a canonical example workflow.

Installation

Quick install (curl)

Install the ginkgo CLI in one line. Requires uv to be installed:

curl -LsSf https://raw.githubusercontent.com/sanjaynagi/ginkgo/main/install.sh | sh

This installs the ginkgo-workflow package from PyPI into an isolated environment via uv tool install. Re-run the same command to upgrade. The package is ginkgo-workflow; the command and the import are both ginkgo:

pip install ginkgo-workflow

Plugins

The installer gives you core plus ginkgo-pixi, which runs tasks' bare env="name" Pixi environments. Other optional integrations ship as separate plugins in this repository, each one an extra of ginkgo-workflow:

  • ginkgo-aws: the s3:// object store and the mountpoint-s3 FUSE driver
  • ginkgo-gcp: the gs:// object store, the gcsfuse FUSE driver and the GCP Batch executor
  • ginkgo-oci: the oci:// object store and the rclone FUSE driver
  • ginkgo-pixi: Pixi environments for tasks' env="name", and ginkgo env
  • ginkgo-k8s: the Kubernetes executor
  • ginkgo-notebook: @task("notebook"), which runs marimo notebooks
  • ginkgo-report: the ginkgo report command, for HTML run reports
  • ginkgo-slack: Slack run notifications

To install ginkgo with every first-party plugin:

uv tool install --force "ginkgo-workflow[all]"

Or name only the plugins you want:

uv tool install --force "ginkgo-workflow[pixi,k8s]"

ginkgo plugins lists what is installed. A remote_file() or remote_folder() URI needs the plugin for its scheme; without it, ginkgo names the plugin to install.

Pixi

For local development:

pixi install
pixi run test
pixi run typecheck

The default environment installs every first-party plugin editable, and pixi run test runs core's tests and the plugins' (tests/ and plugins/). To check that core works with no plugin installed, as CI does:

pixi run -e core test-core

After pixi install, build the uncoded symbol index used by AI coding tools (regenerated automatically by the pre-commit hook, but useful to seed up front):

pixi run uncoded sync

If your workflows use Pixi-backed task environments, pixi must also be available on PATH when you run them.

Run the CLI with either:

pixi run python -m ginkgo.cli --help

or:

ginkgo --help

Editable install

If you prefer a plain Python environment:

pip install -e .

Minimal Example

A population-genetics workflow that filters a VCF, computes per-population allele frequencies, and renders a summary marimo notebook (the notebook task needs the ginkgo-notebook plugin).

from pathlib import Path

import numpy as np

from ginkgo import file, flow, shell, task
from ginkgo_notebook import notebook

POPULATIONS = ["YRI", "CEU", "CHB"]
NOTEBOOKS_DIR = Path(__file__).resolve().parent / "notebooks"


# shell task — runs bcftools in a subprocess
@task("shell", env="genomics_tools")
def filter_snps(vcf_path: file, min_maf: float) -> file:
    """Filter to biallelic SNPs above a minor-allele-frequency threshold."""
    output = "results/filtered.vcf.gz"
    return shell(
        cmd=(
            f"bcftools view -m2 -M2 -v snps -i 'MAF>={min_maf}' "
            f"{vcf_path} -Oz -o {output} && bcftools index {output}"
        ),
        output=output,
    )


# python task — uses scikit-allel, fanned out per population via .map()
@task()
def allele_frequencies(vcf_path: file, population: str) -> file:
    """Compute per-SNP alt-allele frequencies for one population."""
    import allel

    callset = allel.read_vcf(str(vcf_path), fields=["calldata/GT"])
    ac = allel.GenotypeArray(callset["calldata/GT"]).count_alleles()
    freqs = ac.to_frequencies()[:, 1]  # alt allele frequency

    output = f"results/af_{population}.npy"
    np.save(output, freqs)
    return file(output)


# notebook task — runs a marimo notebook and renders it to HTML
@task("notebook")
def population_structure(af_files: list[file], populations: list[str]) -> file:
    """Render an HTML population-genetics summary notebook."""
    return notebook(NOTEBOOKS_DIR / "population_structure.py")


# flow
@flow
def main():
    filtered = filter_snps(vcf_path="data/chr22.vcf.gz", min_maf=0.05)
    af_results = allele_frequencies(vcf_path=filtered).map(population=POPULATIONS)
    return population_structure(af_files=af_results, populations=POPULATIONS)

Run it with:

ginkgo run flow.py

All three tasks above share one file, and a task's cache identity covers the whole module it is defined in — so editing any one of them invalidates the cached results of all three. That is what makes an edit to module-level state such as POPULATIONS invalidate the tasks that read it. Once a step is expensive enough that you mind re-running it, give it its own module; see the caching guide.

Canonical Example

The docs and examples are centered on examples/bioinfo, which demonstrates:

  • Pixi-backed shell tasks
  • a container-backed shell task
  • .map() fan-out across samples
  • a local Python aggregation task that returns a DataFrame

Run it with:

cd examples/bioinfo
ginkgo run

Core CLI Commands

To confirm a workflow is wired correctly, run ginkgo run --dry-run: it previews the plan for the entrypoint you actually run without executing any task body. Validation workflows a project keeps under tests/workflows/ are workflows too, so they run by path: ginkgo run tests/workflows/smoke.py. See the CLI guide.

  • ginkgo run
  • ginkgo doctor
  • ginkgo debug
  • ginkgo init
  • ginkgo inspect workflow
  • ginkgo runs (runs ls, runs show)
  • ginkgo history
  • ginkgo query
  • ginkgo export (export events, export manifest)
  • ginkgo report (from the ginkgo-report plugin)
  • ginkgo why
  • ginkgo secrets (secrets list, secrets validate)
  • ginkgo cache ls
  • ginkgo cache clear
  • ginkgo cache prune
  • ginkgo cache explain
  • ginkgo env ls (from the ginkgo-pixi plugin)
  • ginkgo db (db path, db check, db prune, db vacuum)

For full details on every command, see the CLI reference.

Repository Layout

ginkgo/
├── core/
├── runtime/
├── envs/
└── cli/
  • core/ contains the user-facing DSL
  • runtime/ contains evaluation, scheduling, caching, provenance, and value transport
  • envs/ contains execution backends
  • cli/ contains the ginkgo command-line interface

Ginkgo is licensed under the Apache License, Version 2.0. See LICENSE.

About

A python-based, AI-native scientific workflow orchestrator

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages