Add external execution capacity behind a disabled feature flag - #749

The flag gates admission of external objects and publication of capacity demand,
so an integration that is only half deployed cannot dispatch work. It defaults to
off, which keeps every existing deployment on the native path.

Two companion settings come with it. The observation max age backstops the
validUntil a provider sets for itself, and the workspace resync shortens the
fifteen minute reconcile backstop that would otherwise let a stale observation
stay counted as available capacity.

Co-authored-by: Cursor <cursoragent@cursor.com>

Implements the demand, plan, claim, get and release calls SaFE makes against an
external capacity controller, with service mTLS and a bounded per-call timeout.

Three refusals are deliberate. A response carrying the contract mock header is
rejected outside local tests, because a synthetic reservation reaching the
dispatcher would place pods on capacity that does not exist. An unknown field in
a reply is an error rather than an ignored value, so a profile this build cannot
honour cannot pass as a supported one. Redirects are not followed, since that
would carry the service identity to an address the configuration never approved.

Error codes are exposed as predicates: IsQueueable marks the answers that mean
the workload keeps waiting rather than faults worth retrying.

Co-authored-by: Cursor <cursoragent@cursor.com>

Workload admission skips the workspace quota check for an external workspace. That
check compares the request against status.totalResources, which stays empty until
the provider publishes a node, so applying it would reject every submission while
the workspace is idle -- the scale-from-zero case the mode exists for. The budget
is arbitrated by the provider at claim time, and request shape is still bounded by
validateResourceEnough against the node flavor.

Node admission replaces the private IP, SSH secret and port requirements with the
provider allocation reference, and keeps the flavor and taint checks on both paths.
The hostname stays required because mutateMeta derives the object name from it and
node_k8s_controller matches the execution cluster node by that name.

The workspace label, the lifecycle mode and the allocation reference are immutable.
Flipping any of them would move a live object between two reconcile paths that make
opposite assumptions about whether a host exists behind it.

All three creation paths refuse external objects while the feature flag is off.

Co-authored-by: Cursor <cursoragent@cursor.com>

processNode dispatches external nodes to a path that performs no host operation:
no SSH, no hostname or DNS change, no addon install, no kubespray, no kubeadm
reset and no reboot. It does not reuse manage or unmanage, because their early
exits do not hold for a virtual node -- the execution cluster node carries no SaFE
cluster label, so the "already managed" check fails and control would fall through
to syncClusterStatus, which opens an SSH connection.

Deletion no longer removes the execution cluster node. The provider deregisters it
while releasing the allocation; deleting it here would evict pods whose tasks the
provider has not finished stopping.

shouldSyncMachineStatus returns false for external nodes. Their MachineStatus is
never written, so its zero update time reads as permanently overdue.

Two kubespray guards come with the IsManaged change: an external node reports
managed as soon as it belongs to a cluster, which would otherwise have placed it
in the upgrade inventory for ansible to try to reach.

Co-authored-by: Cursor <cursoragent@cursor.com>

An external workspace does not scale. Its capacity comes from the provider's pool
rather than from spec.Replica, which is deliberately left unset, so the scaling
switch would read the virtual nodes the provider just published as a surplus and
scale down would answer by clearing spec.workspace on them -- removing the very
capacity that was delivered.

The same workspace gets a short requeue. The controller's own backstop is fifteen
minutes, and the events that normally drive a reconcile stop arriving in exactly
the case observation freshness exists to catch: the execution cluster going
unreachable. Without the shorter interval a stale node stays counted as available
for up to a quarter of an hour.

Scale-up candidate selection now excludes external nodes. They satisfy machine
ready and managed while idle, so a native workspace short of a replica would
otherwise bind one and take capacity that is handed out through claims.

Co-authored-by: Cursor <cursoragent@cursor.com>

Records the demand and claim identifiers a workload needs to resume after a
restart or a lost reply, together with the placements the provider approved.

Each identifier is written before the request it belongs to is sent. A request
whose response never arrived can only be reconciled by replaying the same id with
the same body: allocating a fresh one would ask for a second reservation while the
first may already exist, and nothing afterwards would notice the surplus.

Co-authored-by: Cursor <cursoragent@cursor.com>

Ordering, FIFO, priority, dependencies and pause handling are untouched. The
external path attaches at the two points where the existing decision is already
made: when capacity is short, and when it is not.

On a shortage the provider is asked to acquire capacity. Only a genuine shortage
reaches that call -- the earlier checks have already excluded workloads waiting on
a dependency, a start time or a pause, which have no unmet capacity need and would
otherwise grow the pool for work that cannot start.

When capacity looks available locally, the workload still may not leave the queue
until the provider has granted a reservation for this exact workload and dispatch
generation. A plan holds nothing, so losing the devices between planning and
claiming is normal and simply replans on the next pass.

Identifiers are persisted before the request that uses them. A reply lost in
transit can then be reconciled by replaying the same id, where a freshly generated
one would ask for a second reservation while the first may already exist.

Refusals are mapped to distinct waiting reasons. An unprepared image or an
unvalidated profile reported as a capacity shortage would grow the pool for a
problem more nodes cannot solve.

The first release accepts single replica workloads only: a multi-replica workload
needs a stable role and index on every child pod, and the operators creating those
pods do not expose one a reservation can bind to.

Co-authored-by: Cursor <cursoragent@cursor.com>

The dispatcher rechecks the reservation immediately before creating the execution
object. The scheduler already verified it, but time passes while the pods are
built, and a claim revoked or expired inside that window would place a pod on
devices the provider has started reclaiming -- a pod that looks legitimate to
everything downstream.

The pod carries the workload uid, dispatch generation, claim id and revision, unit
key and profile, which is what the provider matches against the reservation after
binding. These are derived from the approved workload and claim; an annotation a
user could write by hand would otherwise amount to an authorisation.

Placement is expressed as required node affinity over the approved virtual nodes
rather than spec.nodeName, so the execution cluster scheduler still performs the
binding and the provider can verify the binding it observes.

Service account token projection is disabled on external pods. The task runs on
provider hardware and speaks a protocol carrying its own identity; a projected
token would place an execution cluster credential inside it for no purpose.

Co-authored-by: Cursor <cursoragent@cursor.com>

Capacity is not returned when a workload ends. The provider has to stop the task
and verify its cleanup first, so the reservation is withdrawn at the terminal
state and the resources stay charged to the workspace until it reports Released.
Revoking is not that confirmation: it only records that the withdrawal was
accepted.

Two places had to agree with that. The scheduler's accounting no longer drops a
finished workload whose claim is still outstanding, because doing so would offer
the next workload capacity the claim transaction is about to refuse -- after this
side had already admitted it. And the release is retried on a timer, since the
provider moving from Revoking to Released produces no event here.

Workspace aggregation stops charging a stale virtual node the flavor's full
resources. For a physical node that accounting is right, the machine is still
there; for a virtual one the allocation behind it may already be gone, and the
same arithmetic would advertise capacity nobody holds.

Co-authored-by: Cursor <cursoragent@cursor.com>

Adds the external_execution section to the chart values and threads it into the
three components that read it: job-manager needs the controller endpoint and mTLS
secret path, resource-manager needs the freshness and resync bounds, and the
webhooks need only the flag that decides admission.

The provider identity is a separate ServiceAccount, rendered only when the feature
is enabled so an untouched deployment grants nothing. Its permissions follow the
ownership split: it may write virtual nodes and their status.external, and read
the workspaces and flavors a node has to be matched against, but nothing that SaFE
itself owns -- no workloads, no clusters, and none of the aggregate fields the
resource manager derives.

Co-authored-by: Cursor <cursoragent@cursor.com>

The freshness predicate is the one piece of new behaviour that changes answer
without any event: only the clock moves. The tests pin it and walk both bounds,
including a far future validUntil paired with a provider that stopped reporting,
which is what a crashed provider looks like from here.

Native predicates are asserted unchanged in the same file, since the external
branch was added inside the functions every physical node also goes through.

On the scheduler side the tests fix the unit expansion, the stability of the
constraints digest the provider recomputes, the refusal of shapes the first
release cannot honour, and the reason mapping -- separating a genuine shortage
from an unprepared image, an unvalidated profile or a transport failure, none of
which more nodes would fix.

Co-authored-by: Cursor <cursoragent@cursor.com>

Three pieces were written but never reached from anywhere. Each was a real gap
rather than an unused helper.

The pod was still built from the image reference the user submitted. The
reservation was granted against the digest the provider resolved at claim time, so
a tag that moved between claim and dispatch would have run content nothing was
admitted for, and the provider would have refused the task after the pod had
already bound.

The configured observation max age was loaded and then ignored. The node
predicates live in the apis package, which cannot read configuration, so every
deployment silently kept the compiled-in default no matter what the chart said.
Both controllers now pass it in after loading config.

Demand withdrawal is called when a workload reaches a terminal state. The demand
would lapse at expires_at on its own, but that window is long enough for the
provider to acquire a node for work that has already finished. It is best effort
and does not hold up the release, which is the part that returns devices.

Also drops the dispatcher's duplicate release path; the scheduler owns that
transition and already retries it until the provider confirms.

Co-authored-by: Cursor <cursoragent@cursor.com>

The local resource comparison decided whether to publish demand or go straight to
plan and claim. That was wrong in both directions, and one of them deadlocked.

What the local view measures is the aggregate of nodes the provider has already
published. It says nothing about what the provider could still acquire, so gating
on it withheld the demand exactly when acquisition was the thing being waited for.
And because a failed plan produced no demand either, a workload whose local view
said "enough" would plan, be refused, queue, and repeat -- with nobody ever asked
to buy anything.

Deduplication was never this side's job: the provider subtracts its ready layout
and in-flight requests before acting on a demand.

So the queue head now states its need unconditionally and then tries for a seat. A
revision is republished only when there is none or the current one is close to
lapsing, because the contract refuses a revision whose body changed and the body
carries its own observation time -- re-sending every pass would either conflict
with the stored revision or make it climb without end. The window is persisted so
a replay reproduces the exact body.

Preemption is withheld at the workload webhook for external workspaces rather than
by where this branch sits. Marking a victim preempted records an intent, not a
release: the devices return only once the provider has stopped the task and
verified cleanup, so the capacity a preemptor was admitted against would not exist
yet. It can be enabled after stop and cleanup close end to end.

An unsupported shape is now a typed error, so a workload that no amount of
capacity could place is rejected rather than left waiting.

Co-authored-by: Cursor <cursoragent@cursor.com>

Restores the local resource comparison on the external path. The capacity the
provider publishes is synced into the workspace aggregate, so this side can tell
whether a workload fits: a virtual node belongs to exactly one workspace, and
every claim against it is visible here.

When it fits, the workload goes straight to plan and claim -- the provider is not
asked to acquire anything. When it does not, the shortage is stated and the
workload waits. A demand still has to exist in the first case, because plan and
claim both reference one and the provider validates its revision, but the renewal
window keeps that to roughly one call per workload every few minutes rather than
one per pass.

A capacity refusal despite local room now restates the need. That combination
means the two sides disagree -- devices held by an unconfirmed cleanup, or an
allocation gone while its node object lingers -- and this side cannot tell which.
Without it the workload would replan forever against a local view nothing
corrects.

Preemption is skipped rather than merely unmarked, so the ordering here does not
depend on the webhook that withholds the annotation.

Co-authored-by: Cursor <cursoragent@cursor.com>

status.external belongs to the capacity provider, which refreshes it on its own
cadence. Replacing the whole status object on every sync carried a stale copy of
that block back with it and made the two writers contend for the same
resourceVersion. Only the five observed fields are patched now.

Co-authored-by: Cursor <cursoragent@cursor.com>

Create, patch, delete and retry now reject a node whose lifecycle belongs to an
external capacity provider. The object is the provider's record of an allocation
it is accounting for; this API cannot hand that allocation back, stop the tasks on
it, or tell the provider its record no longer matches, so a change here would only
leave the two sides disagreeing about what exists.

Batch delete routes through the same guard, and create is checked on the built
object rather than the request so it survives changes to the request shape.

Co-authored-by: Cursor <cursoragent@cursor.com>

Three test files sat apart from their subject. canScheduleWorkload is defined in
scheduler.go, so its cases move into scheduler_test.go and can_schedule_test.go
goes away. The CICD proxy acceptance suite covers pkg/dispatcher and referenced
nothing from its own package, so it moves there too, keeping the integration
build tag that holds it out of a plain go test run.

Also renames the new external node test to match node_types.go, the file whose
predicates it exercises.

Co-authored-by: Cursor <cursoragent@cursor.com>

Seven fixes across the external execution exchange.

The reservation is returned before the finalizer is dropped. Deletion previously
ran the release after the object had already been allowed to disappear, so a
failed release had nothing left to retry from and the Revoking to Released
confirmation was lost with the object -- the provider would hold those devices
with no record on this side that they were owed back.

A new dispatch generation hands back the previous reservation before replacing its
id. Overwriting it stranded the reservation on the provider with nothing naming
it, so a workload that kept failing over leaked one per attempt. A failure there
blocks the new attempt deliberately: waiting is recoverable, a leak is not.

The release path re-reads the state after the withdrawal that rewrote it.
Continuing from the earlier copy patched the withdrawal back out, and the retry
loop then republished one revision under changing bodies forever.

Demand publication records its expiry only once the provider has accepted it, and
a retry resends the identical revision, request id and observation time. Writing
the expiry upfront made a failed publish look like a live demand and suppressed
every retry for the length of the window -- permanently, in the withdrawal case,
where the guard would then skip it forever.

A claim that is definitively gone sends the workload back through admission. The
dispatcher used to recheck a dead reservation every ten seconds with no path to
obtaining a new one, leaving the workload in Pending while it held its share of
the workspace. Failures that might still resolve are still retried in place.

Errors from the external exchange no longer abandon the scheduling pass. A lost
connection or a stale cached object failing its resourceVersion test says nothing
about the workloads behind this one; propagating it stalled admission for the
whole workspace.

The apiserver now applies the configured observation max age and its config map
carries the section, so it no longer judges node freshness by the compiled-in
default while the controllers use the configured one.

Co-authored-by: Cursor <cursoragent@cursor.com>