Programmable data realization for DAG workflows on Kubernetes.
Status: research prototype. The API is stable enough to run real workloads on a k3s cluster; expect rough edges in tooling. This release covers one-shot DAGs (ODAGs); streaming DAGs are future work.
A DAG edge A → B says that B needs an output of A. It says nothing about
how that output gets there: whether it goes through a shared store or directly,
which node serves it, how many copies exist, or how long they live. Workflow
engines such as Argo, Tekton, and Kubeflow Pipelines fix all of that up front by
routing every intermediate through an artifact store and tying its lifetime to
the tasks that produce and consume it.
Wayline separates the two. Every intermediate output is a named object, and each object has a physical realization: its copies, its serving point, its movement, and its lifetime. The realization is live, revisable state that an external policy can patch while the run executes, without changing the DAG or moving any task. A per-node data agent holds objects and moves them peer to peer; the controller reconciles the data plane toward the desired realization and starts a task pod only once its inputs are already on its node.
On an 8-worker edge testbed with links that degrade, disconnect, or fail:
- revising an object's serving point after a producer's uplink degrades cuts traffic through the choked link by 55–63% and completion time by up to 43%, with one policy patch and no task restart;
- retaining an object on a relay node that runs no task completes every workflow across two contacts 20 s apart, where fixed direct delivery completes none;
- risk-aware replication gives the survival of always-on replication at 16–77% less replica storage-time;
- with task placement and per-node order held fixed, changing only the data path moves an 18-task AI City pipeline from 195.5 s (store-mediated) to 163 s (direct), and costs seven scientific-workflow structures 1.03–1.45×.
The evaluation lives in eval/experiments; see
Reproducing the evaluation.
- How it works
- Repository layout
- Prerequisites
- Quick start
- Writing tasks
- Warm runners
- ODAG reference
- Revising a realization at runtime
- Plan-driven copies and recovery
- Schedulers and policies
- CLI reference
- Web UI
- Build & deploy reference
- Cluster setup
- Reproducing the evaluation
- Troubleshooting
- License
wayline apply -f template.yml kubectl patch odag <run> ...
wayline run <template> (external policy: spec.realization)
│ │
▼ ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ wl-system namespace (control plane) │
│ │
│ odag-controller ui-server :8080 │
│ • scheduler: heft | random | saga/<algo> | http://… • K8s watch cache │
│ • dispatch: a task pod starts only when every • SQLite history │
│ input object is installed on its node • REST + SSE │
│ • reconciler: converges copies / serving point / • React frontend │
│ eviction toward spec.realization (live) │
│ • profiler: EMA runtime + dataSize per (task, node) │
└──────────────────────────────────────────────────────────────────────────┘
│ creates task pods; asks agents to push / alias / evict
▼
┌──────────────────────────────────────────────────────────────────────────┐
│ workers: task pods + data-agent DaemonSet (hostPort 8082) │
│ │
│ ┌──────────┐ 1. PUT output → LOCAL agent (temp → fsync → rename) │
│ │ producer │ 2. agent pushes to each consumer's node (digest-checked) │
│ │ (WlTask)│ 3. pod may exit; the object outlives it on the agent │
│ └────┬─────┘ │
│ │ agent → agent install (content-addressed, idempotent) │
│ ▼ ┌───────┐ │
│ ┌──────────┐ │ relay │ a node that runs NO task can hold a copy │
│ │ consumer │ │ agent │ (temporal relay, replica, cache) and │
│ │ (WlTask)│ └───────┘ serve consumers from it │
│ └──────────┘ recv() = local file read │
└──────────────────────────────────────────────────────────────────────────┘
Objects. Within a run, an output's identity is <run, producer, name>; the
data-plane key is producer for the default output or producer.name for a
named one. A task can emit several named outputs, each with its own consumers
and its own realization.
State the controller sees. Each data agent tracks two signals per object:
- task lifecycle:
Pending → Running → ComputeDone → Failed; - per-destination transfer state:
Pending → Transferring → ReadyRemote | Failed, plus a node-local installed marker (.wl-ready, with.wl-sha256).
The agent is the only writer of .wl-ready; installs are atomic and
digest-verified, and every agent verb (install, push, alias, evict, cancel) is
idempotent, so the controller can retry any step after a crash. The wire protocol
and on-disk layout are in docs/architecture.md.
Safety rules the reconciler enforces. A consumer runs only when every input
object is installed on its node, regardless of how the bytes arrived. A node may
not appear in both copies and evict of one entry. The last installed copy of
an object is never evicted while a consumer may still need it. A newer revision
supersedes a pending one; a partial object is never exposed.
wayline/
├── api/v1/ # CRDs: odags.wl.io, odagtemplates.wl.io
├── cmd/
│ ├── odag-controller/ # controller: scheduling, dispatch, realization reconciler
│ │ ├── realization.go # spec.realization → copies / servingCopy / evict
│ │ ├── planmodel.go # the plan's cost model, kept per run
│ │ ├── copypolicy.go # plan-aware copies (spec.realizationPolicy)
│ │ ├── nodehealth.go # node-loss detection
│ │ ├── recovery.go # re-planning after node loss or link degradation
│ │ ├── linkstate.go # link evidence from transfers vs. the plan
│ │ ├── datavertex.go # pod-less data vertices and serving-point rebinding
│ │ ├── saga.go, heft.go # SAGA bridge and built-in HEFT
│ │ └── profiler.go, cache.go # EMA profiler, cross-run reuse (cacheKey)
│ ├── data-agent/ # per-node DaemonSet: object store + peer-to-peer transfers
│ ├── ui-server/ # REST + SSE + embedded React UI
│ └── cli/ # `wayline` CLI (cobra, kubectl-style)
├── pkg/scheduler/ # scheduler interface
├── sdk/python/wl/ # Python SDK: `from wl import WlTask`
├── ui/ # React + Vite frontend
├── deployments/ # namespace, RBAC, Deployments, DaemonSet
├── examples/ # ODAG examples (dag-pipeline, named-outputs, wide-pipeline, …)
├── eval/
│ ├── experiments/E0…E8/ # the paper's evaluation: scripts, policies, committed results
│ ├── policies/ # scheduling-policy comparison (HEFT, CPoP, MinMin, OLB, …)
│ ├── tools/fault.py # fault injection: crash, partition, link throttle
│ ├── functional/ # functional checks for copies, recovery, links
│ └── … # earlier microbenchmarks and baselines
├── docs/ # architecture, getting started, SDK, bring-your-own-scheduler
└── Makefile # build / image / deploy targets
| Tool | Purpose |
|---|---|
kubectl |
Cluster access |
docker |
Building images |
go ≥ 1.23 |
Building the Go binaries |
node ≥ 20, npm |
Building the React UI |
| a k3s cluster | Wayline targets k3s; ~/.kube/config configured |
python3 |
Policies, evaluation scripts, and the SAGA scheduler sidecar |
# 1. Install CRDs, namespace, and RBAC
make install
# 2. Build all images and push to the local registry
make push-all
# 3. Deploy the data-agent, controller, and UI
make deploy
# 4. Build the CLI
make build # produces bin/waylineRun the bundled example pipeline (generate → transform → output):
make example-odag # or: bin/wayline apply -f examples/dag-pipeline/odag.yml
bin/wayline get odags
bin/wayline status dag-pipeline
bin/wayline logs dag-pipeline generate
bin/wayline delete dag-pipelineTemplates are the usual way to run: wayline apply -f examples/named-outputs/template.yml
registers an ODAGTemplate, and wayline run named-outputs creates a run from it.
The UI is at http://<master-ip>:30080. A longer walkthrough is in
docs/getting-started.md.
Tasks are ordinary container images. Inside, use the wl SDK; the controller
injects all peer/topology configuration as WL_* environment variables.
from wl import WlTask
task = WlTask() # reads WL_* env vars
inputs = task.recv_all() # {dep_name: payload}; local file reads
result = process(inputs)
task.send(result) # default output, delivered to every successorA task may emit several named outputs, each an independent object with its
own consumers and its own realization. Names must be declared in the task's
spec.outputs, and consumers pick one in spec.inputs:
task.send(result) # default output
task.send("alert", alert) # named output <run, task, alert>
task.send_raw("features", features) # bytes, no JSON copy
alert = task.recv("infer.alert") # a named object of an upstream tasksend / recv accept any JSON-serializable value; send_raw / recv_raw
move bytes. See docs/sdk-quickstart.md and
examples/named-outputs.
# Build from the repo root:
# docker build -f examples/my-dag/tasks/my-task/Dockerfile -t <registry>/my-task:latest .
FROM python:3.11-slim
WORKDIR /app
COPY sdk/python/wl ./wl
COPY examples/my-dag/tasks/my-task/task.py .
CMD ["python", "task.py"]A task normally runs in its own pod, which costs about a second of pod startup per task (E9). For short tasks that overhead dominates. A task can instead run as a call on a warm runner: a long-lived process on each node that executes invocations without creating a pod. Add one field:
- name: resize
image: <registry>/my-runner:latest # the runner's image
runner: images # run on the "images" runner
function: resize # optional; defaults to the task name
dependencies: [capture]Nothing else changes. The scheduler places the task as usual, the controller invokes it on the runner of the chosen node with the same environment the pod would have had, and the task uses the same SDK and data agent (inputs, named outputs, pushes, timings). Pod and warm tasks can be mixed in one DAG.
A runner is any pod labeled wl.io/runner=<name> that serves the runner
API on port 8090 and mounts /data/wl-outputs from the host. The SDK
provides one. Either register functions:
import wl
@wl.function
def resize(task): # receives a ready WlTask; close() is automatic
img = task.recv_raw()
task.send_raw(shrink(img))
wl.serve()or serve an existing task script unchanged:
python -m wl.runner --script task.py --preload numpy --cpus 7.4The runner forks a zygote once all imports are done. The zygote forks one
child per invocation, so a call starts in milliseconds, CPU-bound calls run
in parallel, and a crash fails only that task. Calls are admitted by their
CPU requests, as the kubelet admits pods: a call starts once its request
fits beside the running ones within --cpus, and the rest queue. A
DaemonSet is the usual way to run one per node; see
eval/experiments/E12/runner.yml.
--preload imports run with one OpenMP/BLAS thread: OpenMP (torch's CPU
kernels) is not fork-safe, and a thread pool started while preloading
(building a model runs parallel kernels) would leave every forked call
waiting on it forever. Each call then computes with as many threads as CPUs
it requested, unless it sets its own.
Limits today: a task's command is ignored in warm mode.
Every task has a CPU request: its own resources.cpu, else the template's
defaults.resources.cpu, else 1 CPU. The pod requests exactly that, and
schedulers plan with it: a node's capacity is its free CPU (allocatable
minus what non-Wayline pods request), and tasks share a node while their
requests fit. The controller admits pods and warm calls against one account
per node, and a task's CPU is released when the task reports it has
finished, not when the kubelet later marks its pod Succeeded. Wayline's built-in heft and every SAGA scheduler plan this
way, so a 1-CPU task and a 3-CPU task can run side by side on a 4-CPU node,
and a schedule's times assume they do.
That keeps each task's runtime known only if a node runs as fast with many
tasks as with one. See docs/limitations-and-future-work/cpu-clock-under-load.md.
An ODAGTemplate is a reusable spec; wayline run <template> creates an ODAG
run from it. Both share the task schema below.
apiVersion: wl.io/v1
kind: ODAGTemplate
metadata:
name: my-dag
namespace: wl-system
spec:
scheduler: heft # random | heft | saga/<algo> | saga/<pkg.Class> | http://host:port
schedulerConfig:
spreadEpsilon: 0 # HEFT tie-break: spread parallel layers (0 = off)
retryPolicy:
maxRetries: 2
defaults: # fallbacks for tasks that omit the hints
runtime: 3
dataSize: 1MB
retention: # garbage collection of old runs and their data
maxRuns: 10
data:
policy: keepLatest # immediate | delayed | keepLatest | none
keepRuns: 2
tasks:
- name: produce
image: 192.168.1.163:5000/my-produce:latest
command: [python, task.py]
dependencies: []
runtime: 30 # seed estimate; refined by the EMA profiler
dataSize: 300MB # default output size, for the scheduler's cost model
outputs: # optional named outputs
- { name: alert, dataSize: 1KB }
- { name: features, dataSize: 50MB }
constraints:
nodeNames: [anrg-3] # restrict placement
- name: actuator
image: 192.168.1.163:5000/my-actuator:latest
command: [python, task.py]
dependencies: [produce]
inputs: # which object of the producer; default output if omitted
- { producer: produce, object: alert }
resources: { cpu: "200m", memory: "128Mi" }
- name: stage # a pod-less data vertex: holds / forwards bytes, runs no container
type: data
dependencies: [produce]
- name: analyze
image: 192.168.1.163:5000/my-analyze:latest
command: [python, task.py]
dependencies: [stage]
cacheKey: analyze-v3 # opt-in cross-run reuse of this task's output| Task field | Meaning |
|---|---|
type |
compute (default) runs the container; data is a vertex the controller realizes through the agent (alias + push) with no pod. A data vertex has exactly one dependency and at least one successor. |
outputs[] / inputs[] |
Named outputs and which one a consumer takes. Producers listed in dependencies without an inputs entry supply their default output. |
runtime, dataSize |
Scheduler hints; the profiler refines them per (task, node) across runs of a template. |
constraints.nodeNames |
Placement restriction. |
cacheKey |
If an earlier run produced this task's output under the same key and the copy is still installed, the task does not execute: the controller pins it to the node holding the copy, aliases the bytes under this run's name, and serves consumers from there. Requires a retention policy that keeps run data. |
| Status field | Description |
|---|---|
status.phase |
Pending → Scheduling → Running → Succeeded / Failed |
status.makespan |
Wall-clock makespan in seconds |
status.tasks[] |
Per-task phase, agent-reported state, node, podName, startTime, completionTime, retries, cachedFrom |
status.objects[] |
Per revised object: copies[{node, state}] (Transferring / Installed / Evicted) and servingCopy, as actually reconciled |
The full schema, with every field's description, is in
api/v1/odag-crd.yml and
api/v1/odagtemplate-crd.yml.
spec.realization on a live run is the policy interface. Each entry names an
object and states the desired copies, serving point, and evictions; the
controller converges the data plane toward it using only agent verbs, and reports
the result in status.objects. The DAG, the pods, and task placements are never
touched.
kubectl -n wl-system patch odag my-dag-run-abc12 --type merge -p '{
"spec": {"realization": [
{"object": "produce", "copies": ["anrg-7"], "servingCopy": "anrg-7"},
{"object": "produce.features", "copies": ["anrg-7","anrg-8"], "evict": ["anrg-3"]}
]}}'| Field | Semantics |
|---|---|
object |
Producing task, or task.output for a named output |
copies |
Nodes that must hold a valid copy. Additive: copies elsewhere are left alone unless named in evict. |
servingCopy |
Node whose copy serves future consumer installs and data-vertex execution. Empty keeps the producer's copy. |
evict |
Nodes whose copy must be removed. Refused for the last copy while consumers may need it. |
origin |
Who wrote the entry: empty for a user or external policy, plan for the plan-aware policy, recovery for node-loss recovery. The controller rewrites only its own entries. |
What the controller does with it:
- Copies. For each missing copy it picks a source (the serving copy if its bytes are valid, else the valid copy with the fastest link to the target under the run's bandwidth model, with degraded links priced at what they deliver now) and asks that node's agent to push. A pending or active transfer is polled, not re-posted; a failed one is retried on a later pass, which is how a copy lands when a contact window reopens. Each run has one reconcile worker; a dispatch pass wakes it, so a copy of an object starts as soon as its producer finishes. Lost nodes are neither sources nor targets.
- Serving point. A data vertex whose input names a
servingCopyexecutes from that node instead of its assigned one, and the old node's outbound transfers of the object are cancelled, so a revised path does not compete with the flows it replaces. Consumers are gated on any valid copy on their node, so a copy that arrived from the new serving point satisfies them. - Eviction. Per-object delete on the named agents, subject to the last-copy rule.
A revision can be issued before the producer runs, while transfers are in
flight, or after the producer has exited. Sample policies (a few dozen lines of
Python each, driven by measurements or signals) are in
eval/experiments/E1/policy.py (serving-point rebinding on a degradation
signal), E2 (temporal relay across contacts), and E3/policy.py (risk-aware
replication on a health signal).
The plan that places a run's tasks prices every task on a node and every transfer on a link. The controller keeps that model for the whole run and uses it for three more decisions: which outputs get an extra copy and where, where a task goes when its node is lost, and what to do when a link turns out slower than planned.
spec:
scheduler: heft # any scheduler; the policy reads its plan
realizationPolicy:
mode: planAware
threshold: 2 # avoided delay per second of copy transfer required
minGainS: 1 # ignore copies that avoid less delay than this
podStartupS: 3 # dispatch-to-start of a pod task that must run again
warmStartupS: 0.1 # the same for a warm call
maxCopyBytes: 500MB # optional budget per runFor every consumer, the policy considers losing the consumer's planned node
just before it starts. The consumer would move to the node where it finishes
earliest (its fallback), and each input would arrive there from the nearest
surviving copy or, if every planned copy was on the lost node, after its
producer runs again. The delay this adds beyond the consumer's slack is what a
copy on the fallback avoids. A copy is written into spec.realization (origin
plan) when the avoided delay is at least threshold times the copy's transfer
time. The copy starts once the producer's own pushes to its consumers are done.
status.realizationPolicy.decisions records every (object, consumer) pair with
the delay, slack, copy time, and the reason for the decision.
With push delivery, a consumed object already has a copy on its producer's node and on each consumer's node. A lone copy remains where the plan co-located a producer with all its consumers, and those are the objects the policy protects. An input whose producer cannot run anywhere else (a sensor capture pinned to its node) is always copied: without the copy, losing the node ends the run.
The controller probes the data agents of nodes that active runs use, each
second. A node is lost when its agent has not answered for
WL_NODE_LOSS_GRACE_S seconds (default 5) or its Ready condition is not True.
For each affected run:
- every unfinished task on the lost node moves to the surviving node where it would finish earliest, counting where its inputs already have copies (so a plan copy draws the task to its node);
- its pod or warm call on the lost node is removed and it runs again there;
- each input it lacks comes from a surviving copy (a
recoveryentry inspec.realization); only when no copy survives does the producer run again, recursively; - a task that can run on no surviving node fails the run, with the root cause
in
status.message.
Every repair is appended to status.recovery (lost nodes, moved, restarted
and re-run tasks, copies ordered). A returning node takes new placements again;
repaired runs do not move back. WL_NODE_LOSS=off disables detection.
A transfer that runs past twice its predicted time plus 2 s, or a completed one that took that long, marks its link degraded for 60 s. The prediction includes the contention the plan expected, so a fan-out that shares a link as planned is not evidence. While a consumer that has not started waits on a degraded link, the run is re-planned (at most every 5 s): the consumer moves where its input already is when that finishes clearly earlier, or takes the input from another copy over a healthy link.
eval/tools/fault.py crashes a node (k3s-killall), restores it, partitions
it (processes keep running), heals it, and throttles or cuts its links.
eval/functional/ holds checks that each run once and assert behavior:
ft_recovery.py (a two-task chain: plan copy, crash with and without the
policy, partition), ft_gdtm.py (the GDTM tracker, losing a sensor node
mid-run), and ft_link.py (a link throttled after planning). They need the
testbed and a sudo password in SUDO_PASS.
Each object is either durable (fsynced to storage on every node that holds it, so it survives a crash) or not (written atomically, so no reader ever sees a partial object, but a power loss can lose it and its task reruns). Syncing every object is expensive on edge storage: on the testbed's eMMC, one sync can wait for other files' large writes, and E13 measured 7 to 11 s of stalls on a run's critical path.
spec.durability sets the policy:
| value | durable objects |
|---|---|
auto (default) |
a run's final outputs (objects nothing consumes) and outputs with a cacheKey |
all |
every object |
none |
none |
node |
each data agent's --sync default decides |
tasks[].durable and tasks[].outputs[].durable override the policy for
one task or one output. A running policy can also make an object durable
after the fact:
spec:
realization:
- object: fuse.tracks
durable: true # sync every installed copy now; later copies inherit itThe decision travels with the object: the SDK marks each install, and the agents forward the mark on every copy they push.
spec.scheduler accepts four forms:
| Value | Behaviour |
|---|---|
heft, random |
Compiled-in schedulers |
saga/<name> |
A built-in SAGA algorithm (heft, cpop, peft, minmin, maxmin, sufferage, …) run in the SAGA sidecar |
saga/<pkg.Class> |
Any importable saga.Scheduler subclass, so a scheduler validated in simulation runs here unchanged (WL_SAGA_PATH / WL_SAGA_EXTRA_PACKAGES on the sidecar) |
http(s)://host:port |
Any service implementing the scheduler contract, in any language |
The controller passes the scheduler task and node properties, profiled costs,
output sizes, network rates, and current data locations, and enacts the returned
placement and per-node order. On an external scheduler's failure it falls back to
built-in HEFT. See docs/bring-your-own-scheduler.md.
Placement (which node runs a task) and realization (how its outputs are served) are separate decisions in separate spec fields; a policy may control either or both.
wayline apply -f <file> Create/update an ODAG or ODAGTemplate (kind auto-detected)
wayline get [odags|templates] [-n] List resources
wayline status <name> [-n] Detailed ODAG status (per-task phase, node, timing)
wayline logs <odag> <task> [-n] Stream logs from a task pod
wayline delete <name> [-n] Delete an ODAG + its pods/services
wayline delete template <name> [-n] Delete an ODAGTemplate
wayline run <template> [-n] Create a new run from an ODAGTemplate
wayline runs <template> [-n] List runs of a template
wayline show <template> [-n] Template detail + profile summary
Global flag: --kubeconfig <path> (default: $KUBECONFIG or ~/.kube/config)
The legacy verb groups wayline odag … and wayline template … remain available
as hidden aliases. Realization revisions are plain kubectl patch calls on the
run object (see above); the CLI does not wrap them.
Served by ui-server on NodePort 30080.
| Page | URL | Description |
|---|---|---|
| ODAG list | / |
All ODAGs with phase, makespan, age |
| ODAG detail | /odags/{ns}/{name} |
Graph view, tasks table, run-history chart |
| Templates | /templates |
ODAGTemplates and their runs |
| Batch | /batch |
Multi-ODAG submission with a combined Gantt chart |
| Cluster | /cluster |
Per-node task counts and utilization |
Live updates arrive via Server-Sent Events (/api/events).
For local UI development see docs/local-dev.md.
make build # all Go binaries into bin/ (incl. bin/wayline)
make ui-build # React UI into ui/dist/
make test # Go unit tests
make push-all # build + push control plane + example images
make push-controllers # build + push only control-plane images
make install # CRDs + namespace + RBAC
make deploy # data-agent DaemonSet + odag-controller + ui-server
make rollout # restart control-plane deployments (pick up :latest)
make example-odag # submit the dag-pipeline example
make clean-deploy # remove control-plane resources (keeps CRDs)
make clean-all # remove everything incl. CRDs and namespaceOverride the registry/namespace per invocation: REGISTRY=myreg:5000 make push-all.
One-time setup for a fresh k3s cluster (master + workers).
# 1. Local registry on the master
docker run -d -p 5000:5000 --restart=always --name registry registry:2
# 2. Trust it from Docker on the master
echo '{"insecure-registries":["<master-ip>:5000"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
# 3. Mirror it from k3s on EVERY node, then restart k3s
# write to /etc/rancher/k3s/registries.yaml:
# mirrors:
# "<master-ip>:5000":
# endpoint: ["http://<master-ip>:5000"]
sudo systemctl restart k3s # master
sudo systemctl restart k3s-agent # each workerThe data-agent DaemonSet binds hostPort 8082 on every node and writes
node-local objects under /data/wl-outputs. The prototype data-agent image is
x86-64 only.
Every experiment in the paper is a directory under
eval/experiments with its harness (*_paper.py or
*_pilot.py, run from the control node), the network treatment it applies
(*_net.sh, contacts.sh: tc shaping on the sender's egress), the policy it
exercises, an analysis script, and the committed per-run results (runs.csv,
run objects, flow records, controller logs, provenance).
| Directory | Capability shown | Results |
|---|---|---|
E0/ |
Clean-testbed characterization: goodput and RTT for every node pair | results/ |
E1/ |
Live serving-point revision under a degraded producer uplink (fixed, adaptive-late, adaptive-early, static oracle; B/2 … B/16) | results-paper/ |
E2/ |
Temporal relay across disjoint contacts; the relay runs no task | results-paper/ |
E3/ |
Risk-aware replication before source-copy loss | results-paper/ |
E4/ |
Per-object control with named outputs: equal-sized objects respond differently to rebinding | results-paper/ |
E5/ |
Policy enactment: HEFT, MaxTP, and OLB placements executed with per-node order fixed, direct vs store-mediated; the broader six-policy sweep and the Argo + MinIO referents are in eval/policies |
results-paper/ |
E6/ |
Applications: AI City multi-camera pipeline (Part B) and seven WfChef scientific-workflow structures (Part A), schedule-matched direct vs store-mediated | results-paper/, partA/results-paper/ |
E7/ |
Reconciliation under faults: controller and agent restarts, superseding and conflicting revisions, last-copy eviction | results-paper/ |
E8/ |
Control-plane overhead and scaling: idle cost, object and copy count, concurrent transfers, concurrent runs | results-campaign/ |
All measurements need the 8-worker x86 k3s testbed (an AI City clip manifest for
E6 Part B). Analysis scripts (analyze_*.py, E8/plot.py) regenerate the tables
and figures from the committed results without a cluster.
eval/policies holds the scheduling-policy comparison, and the
older eval/e0-microbench, eval/mcmt, eval/synthetic-dags, and eval/stress
directories hold the earlier data-plane microbenchmarks and baselines
(make repro-figures regenerates those).
Controller pod is Pending. The control plane targets the master node, which
is often SchedulingDisabled. The Deployment carries a toleration for
node.kubernetes.io/unschedulable:NoSchedule and a nodeSelector for the master
hostname; adjust both to your cluster (deployments/odag-controller/deployment.yml).
Task image pull fails. Confirm the registry container is up
(docker ps | grep registry) and that /etc/rancher/k3s/registries.yaml exists on
the worker and k3s-agent was restarted after writing it.
An ODAG hangs / a task never starts. A task pod starts only when every input
object is installed on its node. Check the controller, the relevant data agent,
and, if a revision is in flight, status.objects:
kubectl logs -n wl-system deployment/odag-controller --tail=40 | grep -e realize -e vertex
kubectl logs -n wl-system -l app=data-agent --tail=40
kubectl -n wl-system get odag <run> -o jsonpath='{.status.objects}'
kubectl get pods -l wl-odag=<name>A revision does not converge. status.objects shows a copy stuck in
Transferring: the push toward that node is failing (no route, agent down). The
reconciler keeps retrying a failed transfer every 5 s and logs
refusing to evict last copy or node … in both copies and evict when it
declines an entry.
MIT — see LICENSE.