feat(execution): expose SubgraphHeadersBuilder and ExecutionOptions as execution options - #1617

Summary by CodeRabbit

• New Features

  • Added configuration options for customizing subgraph request headers and execution behavior.
  • Added support for distinct per-request headers while safely deduplicating equivalent concurrent subgraph requests.
  • Added an option to disable subgraph request deduplication when each execution must be fetched independently.

• Documentation

  • Clarified deduplication behavior, including how requests are matched and how per-request headers should be provided.

Fixes #1616.

What

Adds two execution options to execution/engine:

// Sets the builder that produces the headers for subgraph requests, and whose
// hash the resolver folds into its request deduplication keys.
func WithSubgraphHeadersBuilder(builder resolve.SubgraphHeadersBuilder) ExecutionOptions

// Sets resolve.ExecutionOptions, e.g. to turn request deduplication off.
func WithExecutionOptions(options resolve.ExecutionOptions) ExecutionOptions

and corrects the documented scope of
resolve.ExecutionOptions.DisableSubgraphRequestDeduplication.

Why

resolve.Context already carries SubgraphHeadersBuilder and ExecutionOptions,
but ExecutionEngine builds its resolve.Context internally and
type ExecutionOptions func(ctx *internalExecutionContext) takes an unexported
struct, so no function of that type can be declared outside the package. Both
fields are therefore unreachable for embedders.

That is a correctness gap rather than a missing convenience. The subgraph
deduplication key is
hash(DataSourceID + renderedInput + SubgraphHeadersBuilder hash). With no
builder the hash is 0, so per-client headers forwarded by any other route — an
http.RoundTripper on the data source client is the only option
execution/engine leaves — are not part of the key. Concurrent operations with
byte-identical subgraph request bodies collapse into one fetch, and the followers
copy the leader's response without ever issuing a request of their own. Every
argument-free operation has an identical body across clients, so concurrent
clients receive each other's data.

The issue has a self-contained reproducer. On an idle machine, eight concurrent
clients produce one upstream fetch and seven responses carrying another client's
data.

With WithSubgraphHeadersBuilder the headers travel through the engine, become
part of the key, and deduplication keeps working for callers that genuinely do
share headers — which is why this is preferable to just switching the feature
off. WithExecutionOptions is there for embedders who would rather disable
deduplication outright.

Tests

New file execution/engine/execution_engine_deduplication_test.go:

• TestWithSubgraphHeadersBuilder/sets the builder on the resolve context
• TestWithSubgraphHeadersBuilder/concurrent identical operations resolve against their own headers —
  8 concurrent executions, 8 distinct header sets, each response matches its own
  headers and the subgraph is hit exactly 8 times
• TestWithSubgraphHeadersBuilder/identical headers are still deduplicated —
  8 concurrent executions sharing one header set collapse into fewer fetches, so
  the option does not disable single flight
• TestWithExecutionOptions/sets the execution options on the resolve context
• TestWithExecutionOptions/disabling subgraph deduplication gives every execution its own fetch

The subgraph in these tests echoes the Authorization header it received, so a
response identifies which caller's headers the fetch was made with, and a fixed
delay keeps the single flight window open for the whole burst.

SubgraphHeadersBuilder had no test coverage before this change.

Verified locally: go test ./engine/ passes, and
go test -race ./engine/ -run 'TestWithSubgraphHeadersBuilder|TestWithExecutionOptions' -count=3
passes with no races.

Compatibility

Additive. Nothing changes for callers that do not pass the new options; both
default to today's behaviour.

Note for implementers

A SubgraphHeadersBuilder must return a copy of its header map from
HeadersForSubgraph. httpclient.makeHTTPRequest assigns the returned value
straight into http.Request.Header and then adds Accept and Content-Type to
it, so a shared map races across concurrent fetches. This is documented on the
new option.

Checklist

• I have discussed my proposed changes in an issue and have received approval to proceed.
• I have followed the coding standards of the project.
• Tests or benchmarks have been added or updated.

The first box is intentionally unchecked: I opened #1616 at the same time as this
PR rather than waiting for approval first, because the change is small and easier
to judge with the code in front of you. Happy to close this and continue in the
issue if you would prefer to settle the approach there.

Open Source AI Manifesto

I have read and verified every line of this change, and the reproducer and tests
back it up.