[RFC] Agent-side harness for higher task goodput on shared inference services #11

Is your feature request related to a problem?

RFC: Agent-side harness for higher task goodput on shared inference services

Status: Proposed. This issue records the design direction and open decisions; implementation proceeds in separately reviewed PRs.

The question is how agentscope-go can let a fixed inference deployment serve more useful agent work while preserving task quality, latency, fairness, and safety. Increasing goroutines or concurrent agents alone is insufficient: every reasoning round, repeated prompt, summary, retry, and speculative child consumes the same constrained inference capacity. Tool waits and human approval can also occupy application resources long after an inference request has finished.

This RFC proposes an incremental harness program: measure complete task cost, reduce unnecessary inference work, govern access to shared backends, and separate logical agent lifetime from scarce execution resources. It builds on the existing agent, runtime, resilience, context, and evaluation packages.

Feedback is requested on the measurement contract, architectural boundaries, and initial implementation scope. The later research tracks remain optional.

1. Scope and terminology

Here, a harness is the agent-side execution environment around the model: context construction, tool interaction, progress control, model admission, budgets, state, recovery, and evaluation.

Report logical sessions, runnable agents, queued model requests, and active inference requests separately. “Supports 10,000 agents” has little meaning without specifying their arrival rate, activity, task mix, quality, and latency.

For planning, model request rate ≈ task arrival rate × mean attempts per task. Input/output tokens per task must include failed attempts where measurable, compression, repairs, verification, and children. A rough model service demand ≈ a × uncached input + b × output + overhead can help explain a workload, but it is not a capacity guarantee: batching, context length, cache state, and prefill/decode interference change the coefficients. Under stable conditions, Little's law relates in-system work to arrival rate and latency; a growing queue is not increased capacity.

Goals

• Increase task goodput per fixed deployment and reduce inference work per successful task without concealing quality regressions.
• Bound queues, local resource use, retry amplification, and the effect of noisy tenants or large agent trees.
• Preserve tool permissions, protocol correctness, cancellation, state ownership, and observable terminal outcomes across supported entry points.
• Provide an embeddable Go library path first, with explicit contracts for hosts that need durable sessions or multiple replicas.

Out of scope

• Implement GPU scheduling, continuous batching, KV-cache management, model training, or a replacement inference server in agentscope-go.
• Promise a universal concurrency multiplier, exact monetary caps without provider support, or exactly-once external side effects.
• Require Kubernetes, Redis, a distributed workflow engine, or a new service for ordinary library users.
• Change stable APIs or defaults as part of publishing this RFC. Cross-provider routing and speculative agents are optional later experiments, not prerequisites for the fixed-deployment objective.

2. Current foundation and gaps

Source baseline: Go 6d4e148, inspected on 2026-09-20. These are source observations, not live-provider benchmark results.

Related work: #8 tracks local-model concerns; #9 is open for the output-limit serialization fix at inspection time. Neither is superseded by this proposal. Merged #4, #5, and #10 provide relevant runtime, backpressure, and tool-history work. Merged does not imply released.

Describe the solution you'd like

3. Architecture: separate session, inference, and tool capacity

flowchart TD
    H[Host: authenticated tenant and session] --> S[Session owner and run policy]
    S --> C[Context assembly and progress control]
    C --> M[Managed model operation]
    M --> Q[Bounded backend admission and attempt budget]
    Q --> P[Provider adapter and inference service]
    P --> M
    M --> C
    C --> T[Permission checks and bounded tool executor]
    T --> C
    C --> W[Approval or external-result wait]
    W --> D[Optional durable continuation store]
    D --> S
    M -.-> E[Usage, traces, task evaluation]
    T -.-> E
    S -.-> E

Use three independent resource controls:

1. Session/run admission: bounded accepted work, one mutable owner per session, and limits on live or resident runs.
2. Inference admission: per-backend locally active requests, request/token rate quotas, and queued estimated work; permits cover the locally observed request/stream lifetime, with uncertain remote occupancy handled separately (§4.3).
3. Tool execution: limits for CPU, processes, network destinations, workspaces, and external tool services, enforced by the appropriate executor/backend.

An agent waiting for a tool, a child, approval, or retry backoff must not hold an inference permit. A parent waiting for children must not hold the only worker resource that those children require. Non-yielding tool execution has its own limit. These rules avoid coupling available model capacity to slow tools and avoid recursive-pool deadlocks.

Reuse runtime, loop, model, and resilience seams. Begin with an internal model-operation executor and experimental runtime policies. A ChatModel decorator may supply a compatibility seam, but a wrapper alone cannot observe hidden transport retries or all final wire transformations. Do not add methods to required stable interfaces simply to implement this RFC.

4. Proposed execution contracts

4.1 One accounting and policy path for managed inference

Every managed inference operation carries trusted tenant/session/run identity, parent run ID, purpose (reasoning, summary, repair, verification, or child work), deadline, destination policy, and budget reference. Identity comes from the host, never from model-generated tool arguments. Trace IDs belong in traces or records, not unbounded metric labels.

The model boundary must cover direct replies, the loop bridge, compression, structured-output fallback, managed subagents, and managed model calls initiated by tools. Inventory these call sites before enabling enforcement. Unwrapped custom models and independently networked tools remain outside this boundary; configuration must identify that scope explicitly.

Separate an operation ID from an attempt ID. Each attempt resolves its destination, validates capabilities, finalizes request content, sizes/reserves it, enters admission, executes, and reconciles once. Summaries and repairs have independent operations charged to the same run tree. They must not appear as free work or duplicate usage already recorded elsewhere.

Existing middleware events remain compatible. A new internal executor should be the source of accounting truth; middleware can project that information for existing consumers. Optional ModelNamer/ContextSizer metadata must survive wrappers where applicable, without inventing metadata for a dynamic fallback destination.

4.2 Admission and fairness

Start with configured limits, bounded queues, and a deterministic scheduling policy. The quota key represents the actual shared backend/deployment and credential quota scope, not merely an agent name or model string. Credentials are never included in labels or logs.

• Bound active requests, queued request count, queued estimated tokens/bytes, and maximum queue wait. Validate oversized requests before queueing. Bound suspended sessions separately.
• Use per-tenant queues with configurable weights and bounded service-class shares (interactive/background). Tenant quotas also aggregate all descendants, so spawning agents cannot manufacture priority.
• Prototype deficit round robin using estimated service cost, with FIFO within each class, age-based promotion, capped accumulated credit, and a maximum dispatchable request size. P0/P1 must define the cost unit, correction/debt for estimation error, and minimum class shares; each queue's credit cap must accommodate its maximum admissible request cost. Large requests must eventually accumulate enough credit; a stream of small arrivals must not starve them. Weights allocate service-cost shares, not equal task-completion counts. Fairness applies at request boundaries; running generations are not safely preemptible by this scheduler.
• Before dispatch, atomically check and acquire the relevant local limits and run reservations; do not hold one scarce permit while waiting indefinitely for another. Queued work holds bounded descriptors, not backend execution slots.
• Cancellation, expiry, dispatch, and shutdown must have a single winner. Rejection reports a structured reason and, when supportable, retry guidance. Local overload must not become an automatic agent retry loop.

For hosted providers, request/token windows follow that provider's documented accounting rather than refunding rate usage as if it were a money reservation. For local servers, a configurable estimate of in-flight token work is a pressure signal, not a claim to reserve real KV memory. Begin conservatively and calibrate with observations.

An in-process scheduler protects only participating calls in that process. Multiple replicas must either receive static portions whose sum fits the deployment budget, or use an optional shared admission coordinator with atomic leases, expiry, and failure policy. P1 uses one process or static allocations; distributed global admission is a later milestone. Unknown external traffic still requires headroom and server-side limits.

4.3 Reservation and attempt semantics

Maintain settled usage, outstanding reservations, and unknown usage separately. A managed run admits a new attempt only when its configured resource envelope allows it. Reserve estimated prompt cost plus an enforced output/reasoning allowance and a documented estimation margin. Reconcile actual usage when available; do not silently turn missing usage into zero or double-count cumulative stream usage. Unknown charges remain conservatively reserved or become explicit debt according to policy.

Children share a parent ledger, with atomic reservations and optional sub-budgets. A reservation at the child is not charged again as another expense at the parent. Limit fan-out, depth, total outstanding tasks, wall time, and total attempts as well as tokens/cost. Reserve a small explicit completion allowance if the application wants a final summary after work stops; it is part of the initial budget, not an escape from it.

Strict token/cost modes require a documented conservative upper bound on billable input after final formatting, an enforceable provider output/reasoning bound, and known prices for monetary policies. An approximate token estimate plus an arbitrary margin does not establish that upper bound. Otherwise reject that strict mode or expose a clearly named best-effort policy. Client-side reservations alone cannot guarantee the final bill after transport ambiguity or a provider that ignores limits. Cancellation requests remote termination but is not evidence that remote computation or billing stopped.

Distinguish locally active requests from potentially active remote work. After an ambiguous timeout/disconnect, release local connection resources but retain bounded unknown-occupancy debt against admission, separately from unknown token/cost charges. Clear that debt on authoritative completion/cancellation acknowledgement or an enforceable server execution deadline. If no such bound exists, a strict remote-occupancy policy stops new admissions when that debt fills its allowance and requires reconciliation. A configurable quarantine timeout may support best-effort operation, but its expiry is not proof of remote completion. State local-only bounds explicitly and use server-side concurrency limits for an enforceable deployment-wide ceiling. Test repeated short client timeouts against a backend that keeps generating; reconnect/retry must not silently erase the uncertainty.

Use one configured attempt budget across agent, transport, structured-output repair, and fallback layers, with each physical request consuming it. Preserve the distinction between transport retry and a new semantic repair operation while charging both. Integrate shared httpx retry behavior instead of adding another retry layer; adapters that cannot expose or disable hidden retries cannot claim strict attempt accounting.

Retry only classified eligible failures, honor Retry-After, use cancellable jittered backoff, and release locally active permits between attempts while retaining any unknown-occupancy debt. A retry re-enters scheduling and consumes request-rate quota. Invalid input, permission denial, local budget rejection, and unsupported capabilities are not transient provider failures. Do not replay a partial visible stream automatically. Fallback must re-check context, modality, tools, output limits, data policy, and the destination's quota before sending.

5. Enhancement tracks and research ideas

A. Context efficiency and cache-aware request construction — high priority

Treat context as a structured working set: stable instructions and tool definitions, task state, recent complete tool rounds, retrieved evidence, and artifact references. Avoid re-sending large raw tool output when a bounded excerpt and an authorized retrieval handle suffice.

• Build a final request-sizing step after middleware, skill expansion, tools, media, and provider formatting. Track estimate provenance and uncertainty. Check input + allowed output/reasoning + margin against the actual destination window; a model card is not evidence of server configuration.
• Preserve tool-call/result pairs, provider reasoning metadata, role boundaries, and media when trimming or summarizing. Required instructions and unresolved calls cannot be casually discarded. If indispensable content cannot fit, return an actionable error rather than silently truncating it.
• Extend existing tool-group/skill mechanisms with task-scoped selection and bounded retrieval. Tool availability is not permission; execution always revalidates the selected tool and arguments. Measure reduced schema tokens against tool-selection errors and extra discovery calls.
• Make tool results structured, bounded, paginated where possible, and explicit about truncation. Keep complete artifacts outside the prompt with retention, version/hash, and per-read authorization. Retrieval and embedding costs/resources are measured even if they use a separate backend.
• Preserve stable prefixes where semantics permit: deterministic tool ordering and schema serialization, stable instruction versions, and volatile run metadata after the reusable prefix. Provider caching remains capability-specific; do not advertise a generic client cache as server KV reuse.
• Compare existing threshold compression with incremental task-state summaries and checkpoint-aligned compaction. Compress when expected future savings justify summary cost and possible cache invalidation. Limit summary attempts and do not recursively summarize a summary request that cannot fit. On failure, use a bounded, protocol-valid policy or stop with an explicit error.

Prefix affinity is an experiment only after fairness works. Tenant/authorization scope, model revision, formatting, tools, sampling-relevant configuration, and mutable data versions matter for any reusable result. Semantic response caching is unsuitable as a default for stateful tasks or tool effects. Even a stable prefix can expose cross-tenant timing information on a shared server; deployment policy must decide whether cross-tenant cache sharing is acceptable.

B. Fewer unproductive reasoning rounds — high priority

Improve the agent's interface to its environment before adding more model calls: clear tool schemas, actionable typed errors, efficient search/read/edit primitives, and bounded tool results. Use deterministic validation for parsable output and known preconditions before asking a model to repair them.

Build on repetition, budget, and watchdog middleware to track progress through explicit task milestones, changed artifacts, validated outputs, or repeated error signatures. A heuristic must not silently declare success. Let applications choose bounded retry, a different tool strategy, user clarification, or a terminal “no progress” outcome. State why a run stopped.

Offer opt-in task policies for maximum reasoning effort/output, iteration/repair limits, tool allowlists, and acceptance checks. Only send settings supported at the provider boundary. Compare concise task-state updates and selective verification with existing prompts on held-out tasks. Spending less on a round is useful only if it does not create more rounds or lower completion quality.

C. Streaming and backpressure — correctness first

Add opt-in provider streaming through the managed executor and then the agent path, preserving the existing ChatResponse contract. Check Error on every chunk, handle cancellation/closure without a final response, and use IsLast as the final assembled response without appending it to accumulated deltas. Account for final usage once, including cache fields.

A stream retains its locally active inference permit until completion or local termination, not just until ChatStream returns a channel; ambiguous termination transfers the remote uncertainty to the admission debt described in §4.3. Configure connection/setup, first-response, idle-progress, and whole-operation deadlines separately; heartbeats are not necessarily semantic progress. No tool side effect starts from incomplete arguments or before permission validation.

Bound buffering by bytes as well as events. For a slow or disconnected subscriber, choose an explicit policy: cancel the run, or detach into an already-configured durable execution path. An in-memory run cannot become durable by losing its client. Persist or terminate when authoritative history cannot be retained; do not silently discard terminal/tool/accounting events. A channel consumer that exits must cancel or hand off ownership, and shutdown must clean up readers, bodies, timers, and permits.

Streaming can improve first-visible-output latency and cancellation responsiveness. Its effect on throughput and memory must be measured independently.

D. Park, resume, and resident-memory control — later, after admission

Define an opt-in run state machine such as queued → preparing → model/tool execution → waiting → runnable → terminal, with waiting reasons for approval, external result, retry deadline, or child completion. These are proposed runtime concepts, not replacements for current public events.

In durable mode, acknowledge accepted input only after its record is committed. Persist continuation state and pending interaction identity before releasing the resident worker. A failed save cannot be reported as a successful durable park. Resume through an authenticated, idempotent input record; reject stale or duplicate ownership epochs and re-check current tool/permission policy.

Preserve existing checkpoint readability and pending-call semantics. Use additive versioned records and explicit migrations for the new mode; old checkpoints do not acquire durability guarantees retroactively. Start with single-process ownership and a storage interface supporting conditional updates. Multi-process ownership needs leases plus fencing tokens, not a mutex in each replica.

There is no general delivery guarantee for tool effects across uncertain failures: an effect may occur zero, one, or multiple times, and retries can duplicate it. Track intent, execution ID, and acknowledged result. Automatically retry only idempotent operations or operations with a backend idempotency key; uncertain non-idempotent effects require reconciliation. Stronger delivery or effect guarantees require an explicit adapter/backend contract. Fencing prevents stale state writes but does not itself fence an external payment or shell process.

Add TTL/retention for parked runs, events, artifacts, and completed child records, with bounded cleanup and explicit expiry outcomes. Run cancellation propagates to descendants; independently persistent background work requires explicit host authorization and a new lifecycle/budget owner.

E. Bounded agent collaboration — later

Prefer one agent with effective tools as the baseline. Delegate when work can be independently specified and checked, using a dependency graph and a compact result contract rather than broadcasting entire histories. Parallelize independent, concurrency-safe tool work under its own limits and preserve deterministic result association.

Give each child fresh mutable state, a scoped toolset, authorized workspace access, a parent budget reference, and cancellation lineage. Share immutable configuration, connection pools, and verified concurrency-safe model clients; do not share mutable histories or unsafely stateful middleware/tools between sessions. Existing worker-agent reuse must not become the basis of a multi-tenant session service.

Evaluate bounded fan-out, early stopping after acceptance, and selective critics against a single-agent baseline at the same total inference budget. Cancel losing branches and charge their consumed/unknown usage. Speculative agents and multi-model routing may improve quality or latency while reducing capacity; retain them only where the measured tradeoff is desirable.

F. Host/runtime efficiency — profile driven

Measure allocations, copying, JSON/formatter overhead, token estimation, event fan-out, mutex contention, GC, and resident session size. Cache immutable schemas and safe token-count fragments with invalidation; avoid reconstructing unchanged prompt components. Respect snapshot ownership when optimizing copies.

Explore bounded session eviction and workspace prewarming only after storage, isolation, cleanup, and lifecycle contracts exist. Keep transport reuse and optional connection limits explicit. Do not introduce object pooling or zero-copy state sharing before profiling demonstrates a material benefit and race tests establish ownership.

6. Compatibility, security, and rollout

Follow STABILITY.md: agent, model, tool, message, permission, and formatter are stable; runtime/loop changes still need migration and behavior documentation. Keep Go 1.25 and /v2 compatibility. No required new dependency in the first slice.

Introduce the managed policy path behind explicit options. Existing constructors and unmanaged calls retain their documented behavior. First offer observation, then enforced admission on selected traffic, then independently selectable context/streaming policies. Observation mode must not issue shadow model/tool requests; any hypothetical queue result is only a simulation. Rollback disables new admissions while draining/reconciling in-flight attempts; it does not forget reservations or orphan durable work.

Publish a conformance matrix for Reply/ReplyStream, loop/session execution, compression/structured output, managed children, and custom models. The matrix must identify hooks, permissions, budgets, stream semantics, and recovery support actually exercised. Unsupported interactive paths continue to fail closed until their protocol supports interaction. Sandbox policy is not a substitute for process/network isolation.

Do not change tenant priority, destinations, tool permissions, or spending limits based solely on untrusted model output. Redact content and credentials by default; distinguish content retention from metric retention. New host endpoints need authorization, request limits, and session ownership checks. Routing to another provider is a data-policy decision as well as a capacity decision.

7. Evaluation: prove capacity gains at an explicit quality level

Extend the existing bench/replay/evalkit tooling instead of creating an unrelated benchmark suite. Add an orchestrator that drives the same versioned task corpus under load, while retaining independent scoring. Existing replay tapes establish protocol/regression behavior; they do not measure live model quality or server scheduling.

Workload matrix

Pin the Go/source revision, task corpus, scoring criteria, model/weight/tokenizer and server versions, hardware/quantization, server context/batching/cache settings, quota, transport configuration, agent policies, and seeds where supported. Hosted runs must identify time window and rate-limit conditions. Do not claim determinism solely from a seed.

Use both closed-loop session/concurrency sweeps and open-loop offered-load tests with scheduled arrival times independent of completion. The latter is needed because concurrency-driven workers slow their own arrivals during overload. Count latency from scheduled arrival, including harness queueing; retain rejected, canceled, timed-out, unscored, and unfinished tasks in the outcome report. Define the observation interval and drain deadline in advance; unfinished tasks cannot disappear from the denominator.

Primary metrics:

• Quality-passing tasks within deadline per second; pass rate across all offered tasks and separately across admitted tasks.
• End-to-end task p50/p95/p99, model queue delay, first-token/first-visible-output latency where meaningful, and cancellation-to-local-cleanup latency.
• Attempts, input/output/cache tokens, compression/repair/child work, and known plus estimated/unknown cost per offered task and per success. Total cost includes unsuccessful work.
• Per-tenant scheduled/observed service-cost shares against configured weights and minimum shares, with completion shares and tail latency reported as outcomes; rejection, expiry, retry, and unknown-usage rates.
• Peak/RSS memory, goroutines, queued bytes/tokens, resident/parked session counts, tool occupancy; backend utilization/KV pressure only when actually observable.

Run paired baseline/candidate trials in randomized order with repetitions and confidence intervals, report cold/warm-cache results separately, and ablate admission, compaction, tool selection, and collaboration separately. Keep the fixed-deployment comparison separate from experiments that change models or hardware. Prefer executable acceptance checks for task quality; any model judge needs calibration, versioning, a held-out corpus, and separate accounting of evaluation cost.

Proposed acceptance gates, to finalize before experiments:

1. Deterministic conformance tests: bounded queues/reservations; cancellation and shutdown races; no cross-session state mix; no duplicate terminal accounting; permission and tool-history contracts; strict-mode rejection when bounds cannot be established. These are correctness gates.
2. Each workload declares Q_min, L95_max, allowed rejection/error rate, and resource ceilings before the run. A candidate must meet them for every required cohort; a higher aggregate cannot hide a harmed minority workload.
3. Use an agreed non-inferiority margin for quality and an agreed improvement threshold for goodput or cost. Report interval estimates, not a single favorable run. Numeric targets require a P0 baseline; this RFC does not invent a throughput multiplier.
4. Admission alone may trade mean latency for predictable tails and bounded failure. Record that outcome honestly; promote optimization defaults only after a repeatable benefit on representative tasks.

8. Delivery plan and community-sized slices

P2a and P2b can proceed independently once their P1 dependencies are satisfied. P3/P4 need not block useful P1/P2 releases. Split every row further into small PRs; do not bundle all execution-path changes into one patch. No owner assignment or release date is implied.

The proposed first P0 slice is a bounded open-loop load driver in bench: explicit arrival times, a limit on running callbacks, and a result record for every planned arrival. It provides load-generation and outcome data; task-quality scoring, model-attempt accounting, and the joined evaluation report remain subsequent work. No runtime scheduling policy changes are needed for this slice.

Other focused P0 contributions include the call-path conformance matrix and a versioned workload manifest. Implementation PRs should link this RFC, describe the delivered subset, and follow AGENTS.md and CONTRIBUTING.md. Provider fixtures, local measurements, and live-service results must remain distinguishable.

Describe alternatives you've considered

Python AgentScope reference

Upstream was inspected at 5ff52f8. These are implementation references, not a claim that Python already implements this entire shared-capacity design.

Additional context

Decisions requested from the community

1. Is fixed-deployment task goodput under declared quality/SLO constraints the right primary objective, with resident-session capacity reported separately?
2. Can we start with P0/P1 and an opt-in managed model boundary, including helper/child calls and explicit unmanaged-call limitations?
3. Which representative task corpus and local/hosted backend configurations should define the first baseline, and what quality/latency margins are acceptable?
4. Is single-process admission plus static per-replica allocation sufficient for the first implementation, leaving shared coordination and durable workflows for later RFCs?
5. Which contract should be designed next after measurement: context/output controls, tenant fairness, or durable interaction waits?