Skip to content
GthulhuPublic

About

eBPF-based workload orchestration platform, built for Cloud Native ecosystem.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

411 stars

Watchers

2 watching

Forks

Latest commit

Β 

History

405 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Gthulhu: From Kubernetes Resource Allocation to Linux Task Scheduling

cncf landscape ebpf landscape

LFX Contributors Go Portability Go Report Card

logo

DRA chooses what and where; Gthulhu controls how it actually runs.

Gthulhu is a cloud-native runtime scheduling platform built around Kubernetes, eBPF, and Linux sched_ext.

Kubernetes can decide whether a workload may run, which node it runs on, and increasingly which GPU, NIC, CPU set, NUMA domain, or other device it receives. But those allocation decisions do not by themselves guarantee that the workload's critical Linux threads receive the CPU service, locality, responsiveness, or isolation needed to meet an SLO.

Gthulhu focuses on that execution gap.

Kueue / Workload API
  admission, quota, fair sharing
            β”‚
            β–Ό
kube-scheduler / DRA
  Node + device + topology allocation
            β”‚
            β–Ό
Gthulhu Runtime Plane
  workload / claim β†’ Pod/cgroup β†’ TGID/TID
            β”‚
            β–Ό
sched_ext + eBPF
  runtime policy + verification
            β”‚
            β–Ό
Delivered workload SLO
  latency / throughput / jitter / utilization

The long-term direction is Claim2Core: connect actual Kubernetes allocation to Linux task scheduling without reimplementing kube-scheduler, DRA, or Kueue. See the 2026 Claim2Core roadmap.

Demo video: https://youtu.be/Cyjrh9cW1a8

What Gthulhu Does Today

Scheduling observability

Gthulhu uses eBPF to collect scheduler behavior at process/task level and aggregate it into Kubernetes-aware metrics, including:

  • CPU runtime and scheduler wait time;
  • voluntary / involuntary context switches;
  • run count and scheduling frequency;
  • CPU migrations;
  • workload-level scheduling pressure.

These metrics can be exported to Prometheus/Grafana and used by KEDA for autoscaling.

Distributed scheduling intent

A central Manager resolves workload-level policy and distributes intent to a per-node Decision Maker. Each Decision Maker resolves local processes/tasks and exposes node-local strategies to the Gthulhu daemon.

Custom CPU scheduling with sched_ext

On Linux 6.12+ with sched_ext, Gthulhu can apply workload-aware CPU scheduling policy without patching the kernel.

Current policy primitives include:

  • priority treatment;
  • custom time slices;
  • CPU locality / affinity hints;
  • preemption-oriented behavior where supported by the scheduler path.

TID-aware node policies

Linux schedules tasks/threads, not only process leaders. Node policies therefore scan /proc/<tgid>/task/<tid> and can match non-leader worker threads by comm.

This matters for workloads where the critical work happens in named worker threads, such as:

  • LLM engine/decode workers;
  • NCCL/RDMA progress threads;
  • DPDK or packet-processing workers;
  • host-side helper threads.

A node policy resolved to a worker thread is keyed by TID, so the scheduler can target the entity Linux actually dispatches.

Preview and verify node policies

Send an authenticated POST /api/v1/node-scheduling-policies/preview to the Manager with an inline rule, for example:

{"policy":{"nodeNames":["gx10"],"commandRegex":"^EngineCore(_DP[0-9]+)?$","priority":5,"executionTime":20000000}}

Alternatively, send {"policyId":"<existing policy ID>"}. The Manager resolves the same node selectors as policy creation and asks each matching node's Decision Maker to scan its live task table. The response data includes the rule, matchedNodes, matchedTasks, conflicts, and warnings. An unavailable node produces a warning, so an empty match list does not necessarily mean every node was scanned. Decision Makers also expose this endpoint directly for a single-node preview.

Each match includes the node, TGID, TID, comm, per-task startTime (clock ticks since boot), raw cgroupPath, and podUID if discovered. Conflicts list the contenders and the winner for the evaluated set. An inline rule has no persisted ID or generation, so previewing a saved policy by ID gives more reliable precedence information. Preview does not change policies, scheduler state, or BPF maps.

To record benchmark provenance, save the preview and the matching nodePolicyMatch records from the Decision Maker's GET /api/v1/scheduling/strategies. The latter identifies the winning rule and generation and its intended BPF key. In kernel mode, collect the daemon's node_policy_apply and node_policy_remove logs for the same policy ID, generation, node, TID, and start time. A verified apply confirms the key and values were read back from the priority maps; a verified removal confirms the key was absent from both maps. These are point-in-time observations of map state, not proof that the task received CPU time. A replacement strategy can keep the same TID in the maps, so do not treat replacement as key removal.

Priority semantics across scheduler modes

A strategy with Priority > 0 is a boost. Priority == 0 is non-boosting.

  • In user-space mode, a TID-specific strategy is preferred, with TGID fallback for group-wide Pod policy behavior. A slice-only rule can still carry a custom time slice without jumping the queue.
  • In kernel mode, non-boosting strategies are not inserted into the priority BPF map. This avoids interpreting priority 0 as the highest preemptive level. Kernel mode does not yet provide full slice-only parity with user-space mode.

These semantics reflect the merged TID-aware scheduler work in the main repository and Gthulhu/plugin.

Claim2Core Direction

The next architecture step is not "use ResourceSlice as another selector". The intended boundary is:

  • ResourceSlice = device inventory;
  • ResourceClaim.status.allocation = workload's actual allocation;
  • Gthulhu = runtime execution plane after allocation.

The target lineage is:

Workload / PodGroup UID
  β†’ Pod UID
  β†’ ResourceClaim UID + generation
  β†’ allocated driver / pool / device
  β†’ NUMA / PCIe / network topology
  β†’ Pod cgroup
  β†’ TGID / TID / starttime
  β†’ sched_ext DSQ / BPF-map entry
  β†’ observed runtime metrics
  β†’ workload SLO

Near-term implementation order:

  1. correct DRA / ResourceSlice semantics;
  2. add a read-only ResourceClaim observer;
  3. add Claim-to-Task preview / provenance;
  4. introduce a bounded static DRAExecutionPolicy;
  5. validate one workload adapter (free5GC/UPF or LLM serving);
  6. add closed-loop runtime control only after static policy and provenance are trustworthy.

Hard boundaries

Gthulhu is not a GPU scheduler. sched_ext schedules Linux CPU tasks; Gthulhu does not directly schedule CUDA kernels, GPU SMs, MIG, or NIC hardware queues.

Gthulhu can improve host-side execution for GPU feeder/launch threads, tokenizers, NCCL/RDMA progress threads, DPDK pollers, packet-processing workers, data loaders, checkpoint workers, and similar tasks.

CPU DRA, cgroups, kubelet, and other resource managers remain authoritative for the CPU/resource envelope. Gthulhu must optimize inside that envelope, never outside it.

Architecture

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                         Gthulhu Control Plane                       β”‚
β”‚                                                                    β”‚
β”‚  User / Web UI / CRD ──▢ Manager ──▢ MongoDB / Kubernetes API     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                               β”‚ scheduling intent / runtime config
                  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                  β–Ό                         β–Ό
         Decision Maker              Decision Maker
            (Node 1)                    (Node N)
             β”‚   β”‚                        β”‚   β”‚
             β”‚   β”œβ”€β”€ eBPF metrics        β”‚   β”œβ”€β”€ eBPF metrics
             β”‚   └── task resolution     β”‚   └── task resolution
             β–Ό                            β–Ό
        Gthulhu daemon               Gthulhu daemon
             β”‚                            β”‚
             └──────▢ sched_ext / BPF β—€β”€β”€β”€β”˜
                         β”‚
                         β–Ό
                   Linux scheduler

Prerequisites

Metrics / monitoring

  • Go 1.22+
  • LLVM/Clang 17+
  • libbpf
  • Linux kernel with eBPF/BTF support

Custom CPU scheduling

  • Linux 6.12+ with sched_ext / CONFIG_SCHED_CLASS_EXT

The eBPF monitor does not require sched_ext. Custom CPU scheduling does.

See the installation guide for deployment details.

Build

make dep
git submodule init
git submodule sync
git submodule update
cd scx
cargo build --release -p scx_rustland
cd ..
make build

Cross-build for arm64:

make build ARCH=arm64

Lint:

make lint

Test

make test

Or select a kernel version:

make test KERNEL_VERSION=6.12

Gthulhu runs daily portability tests across supported Linux 6.12+ kernels.

Run with schedkit

Install schedctl from schedkit, then run:

sudo schedctl run gthulhu

Run with Docker

docker run --privileged=true --pid host --rm ghcr.io/gthulhu/gthulhu:latest /gthulhu/main

Debugging

See Scheduler optimization notes for the cgo hot-path design, adaptive-priority investigation, measured results, and reproduction commands.

Adaptive wakeups (default in user-space scheduling)

Application wakeup preemption is enabled by default in user-space scheduling; the former -adaptive-wakeup flag has been removed. Kernel mode is unchanged. Every task still goes through Go. A wakeup qualifies only if its known previous full runnable burst and its final execution segment are both positive and at most 1 ms. BPF reports the previous full burst through exec_runtime on wakeups, rather than the newly reset current burst. Reenqueues retain current-burst runtime reporting.

Go reduces a qualifying task's dispatch deadline by one default slice, flooring nonzero deadlines at 1 and preserving explicit priority 0. This adjustment occurs after selection from the plugin heap; it does not change the heap's ordering. BPF only preempts an ordinary task still running on the selected CPU if its deadline is later. Scheduler-process threads remain protected. There is no minimum-current-runtime gate: rejecting a one-shot request on that basis had no retry and could delay an eligible wakeup until a later scheduling event. Single-CPU tasks retain their allowed CPU when idle selection returns ANY; all dispatches still undergo affinity validation.

The scheduling loop uses direct, nonblocking ringbuffer consumption instead of a polling goroutine and channel handoff. Completed Go dispatches are drained by BPF before scheduling additional policy work. Once the Go pool is empty, the loop uses Linux sched_yield to hand the CPU back to the BPF dispatcher; it does not spin while holding a Go P or block inside asmcgocall. The BPF fallback/scheduler slice uses scheduler.slice_ns_min in this mode, while slice_ns_default remains the Go fairness/deadline budget. Kernel mode retains its existing queue and slice behavior.

Before becoming the default, with both experimental flags, builtin_idle: false, and diagnostics disabled, three repeated schtest --filter adaptive_priority runs measured final nonzero ratios 0.24, 0.22, and 0.21, below the required 0.50. Each run also measured a sub-0.50 ratio in its >=5-second batch; this is not based solely on the test runner's zero-valued convergence return. These are host-specific results, not a latency guarantee. Validate fairness and throughput on the target workload.

Repeating with the repository's monitor-enabled configuration measured 0.21, 0.22, and 0.22. The fairness test also passed with the experiment enabled (0.98) and disabled (0.96).

After making adaptive wakeups the default, three monitor-enabled runs with neither experimental flag measured final nonzero ratios 0.22, 0.19, and 0.21; their >=5-second batches also remained below 0.50. Default-mode fairness passed at 0.98. -usersched-preempt is not required for these results.

sudo ./main scheduler -config config/config.yaml
sudo schtest/target/debug/schtest --filter adaptive_priority

Experimental scheduler-loop wakeup preemption

sudo ./main scheduler -config config/config.yaml -usersched-preempt -trace-scheduler-cpu

-usersched-preempt is off by default and only applies to the user-space scheduler. The pinned loop registers its Linux TID after attachment and clears it on exit. When that thread wakes with pending work and no idle CPU is found, BPF can head-insert it on its assigned, affinity-allowed CPU and request preemption, globally limited to once per millisecond. Other scheduler-process threads and slice-expiry reenqueues retain the existing policy; throttled wakeups fall back to SCHED_DSQ. The idle-CPU path is unchanged. This is not a latency guarantee: higher scheduling classes and runtime waits can still delay the loop. The early startup window before TID registration also uses the old policy.

Compare runs with/without this flag under the same workload and diagnostic settings. Measure kernel runnable-to-running delay, slow-call frequency, and application throughput/latency; a lower scheduler delay alone is not a win if application performance regresses. CPU-time tracing itself adds overhead.

sudo bpftool prog list
sudo bpftool map list
sudo cat /sys/kernel/debug/tracing/trace_pipe

To diagnose user-space scheduling stalls without changing the cgo bridge:

sudo ./main scheduler -config config/config.yaml -trace-scheduler-loop

This opt-in diagnostic reports each call's cumulative count, average, maximum, and count taking at least 1 ms every second, plus the active call and its elapsed time. p50_le, p99_le, and p999_le are cumulative histogram percentile upper bounds (power-of-two nanosecond buckets), not exact percentiles. Like max, they cover the entire run, not just the previous second. It adds timing/atomic overhead; leave it off for performance comparisons. Durations include time descheduled, GC pauses, and lock waits, not just CPU time. BlockTilReadyForDequeue can legitimately wait while idle; a long wait alone does not prove a deadlock. Correlate it with queued work and a goroutine dump (pprof is available on port 6060 when debug: true). The reporter is a Go goroutine: missing reports can indicate runtime or system starvation and cannot identify the blocked call by themselves. This diagnostic does not measure BPF dispatch latency or replace kernel tracing.

For SelectCPU CPU-time attribution, enable the more intrusive diagnostic:

sudo ./main scheduler -config config/config.yaml -trace-scheduler-cpu

This implies loop tracing and adds two CLOCK_THREAD_CPUTIME_ID clock syscalls per selection. The startup log includes the pinned loop's Linux tid. SelectCPU_thread reports average thread CPU time and a paired sample for the largest wall-time measurement: worst_wall, worst_cpu, and worst_non_cpu_estimate. A large wall time with small CPU time points toward descheduling or blocking, not CPU-intensive selection. The difference is an estimate because the wall/CPU clock boundaries differ and include diagnostic overhead. CPU time includes user and kernel execution on this thread; it does not isolate BPF execution or distinguish GC, lock waits, and OS scheduling. Only the slowest sample is retained; there is no per-call logging.

To correlate GC/runtime scheduling, set debug: true and capture a short trace from the local pprof endpoint while reproducing the workload:

curl --fail 'http://127.0.0.1:6060/debug/pprof/trace?seconds=10' -o /tmp/gthulhu-runtime.trace
go tool trace /tmp/gthulhu-runtime.trace

For kernel scheduling evidence, substitute the logged numeric TID below:

sudo perf sched record -t <tid> -- sleep 10
sudo perf sched timehist -t <tid>

Runtime tracing does not fully describe syscalls made via bare asmcgocall; use kernel scheduling evidence to verify off-CPU attribution. These captures add overhead and should be short; neither is enabled by default.

After changing BPF callbacks, rebuild with make build. The wrapper includes qumun/main.skeleton.h, so the root build generates the skeleton there (not in the repository root) and forces a Go rebuild to avoid cached external C archive/header contents. A successful Go link alone does not establish that the updated BPF program was embedded.

Queued tasks must be counted before bpf_ringbuf_submit publishes them. The current plugin checks nr_queued after dequeuing a record; publishing first can make it consume and discard a task while the counter is still zero, leaving a stranded count after the producer increments it. TestQueuePublicationOrder reproduces this interleaving with the actual plugin without requiring BPF privileges.

The qumun test TestBlockTilReadyForDequeueRequeueBlocksWithWaitingProducer reproduces the current channel requeue deadlock without loading BPF: a waiting producer fills the slot released by the receive, so requeueing blocks even after cancellation. It is a diagnostic reproduction, not a fix or proof that a particular watchdog exit was caused by this path.

Related repositories

  • qumun β€” Go framework for building custom Linux schedulers with eBPF / sched_ext.
  • plugin β€” user-space scheduling strategy implementation.
  • docs β€” official documentation published at gthulhu.org.
  • gtp5g-operator β€” telecom / GTP5G integration work.

Research / workload directions

Two immediate validation paths are especially useful:

  • CPU DRA Γ— Gthulhu Γ— free5GC/UPF β€” fastest path to a credible end-to-end runtime scheduling demo with p99/p99.9 latency and jitter.
  • GPU + RDMA + CPU DRA Γ— phase-aware LLM scheduling β€” higher research upside around TTFT, ITL, GPU idle gaps, and NCCL/RDMA progress.

Longer term, Gthulhu is also exploring device-local execution domains, scheduling provenance/replay, shared-device interference control, device-health-aware degradation, and cooperative resource reclamation.

Meaning of the Name

The name Gthulhu is inspired by Cthulhu and Golang. The many tentacles represent coordinating complex distributed runtime behavior.

The underlying framework qumun is named after the Bunun word for "heart", reflecting the scheduler's role at the heart of the operating system.

Contributing

See CONTRIBUTING.md and the official contribution guide.

Useful contributor areas include Kubernetes/DRA, eBPF, sched_ext, NUMA/topology, scheduling provenance, testing, free5GC/UPF, and accelerator workloads.

Community

License

Apache License 2.0.

Special Thanks

About

eBPF-based workload orchestration platform, built for Cloud Native ecosystem.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

411 stars

Watchers

2 watching

Forks

Releases

Packages

Used by

Contributors

Languages