Skip to content

Go SDK: no-codegen-at-runtime + single-pass codegen#13381

Open
eunomie wants to merge 11 commits into
dagger:mainfrom
eunomie:workspace-go-no-codegen-at-runtime
Open

Go SDK: no-codegen-at-runtime + single-pass codegen#13381
eunomie wants to merge 11 commits into
dagger:mainfrom
eunomie:workspace-go-no-codegen-at-runtime

Conversation

@eunomie

@eunomie eunomie commented Jun 9, 2026

Copy link
Copy Markdown
Member

Original PR: #13210

What this does

Reworks the Go SDK codegen path so that:

  1. A Go module can opt out of codegen at runtime. When
    codegen.automaticGitignore=false in dagger.json, the Go SDK trusts the
    committed dagger.gen.go + internal/dagger files and skips the
    in-container codegen pass at runtime (it errors with an actionable message
    if the committed files are missing). There is no separate opt-in flag:
    opting out of the generated .gitignore already means the module commits
    its generated files, so the runtime can trust them. This makes the
    committed files honest — what is committed is what runs, instead of being
    regenerated at call time and possibly differing from what the user sees in
    git.
  2. The opt-out is gated on engine version compatibility. The generated
    module dispatch is engine-version specific (the empty-parentName typedef
    registration is only emitted by the current codegen). So the skip is only
    honored when the module's pinned engineVersion is at least as new as the
    running engine; on version skew it falls back to runtime codegen, so stale
    or mismatched committed files can never silently run.
  3. generate-module runs once for Go modules, including self-calls.
    Previously self-calls modules ran it twice (the engine built+ran the module
    via asModule just to discover its own types, then ran codegen again). Now
    the Go codegen discovers its own types from the existing packages.Load
    analysis, emits them as introspection JSON, and merges them into the deps
    schema via the engine's new schema(json).merge(...) tool — in a single
    pass. No AST analyzer; packages.Load is retained.
  4. moduleTypes is dropped from the Go SDK. Type discovery happens through
    the generated invoke() dispatcher's empty-parentName arm. Other SDKs
    (Python, TS) keep moduleTypes and the engine-side self-append.

Commits (read in order)

  1. core/modules: add legacyCodegenAtRuntime codegen config field
  2. feat: add schema merge tool for codegen — engine-side
    schema(json).merge(...).contents, exposed over dagql for all SDKs. Kept
    minimal: only the schema / merge / contents the codegen needs (no
    listTypes / hasType / describeType or IntrospectionType graph).
  3. core/sdk/go_sdk: skip codegen at runtime when opted in
  4. cmd/codegen/generator/go: emit empty-parentName module dispatch
  5. cmd/codegen + core: single-pass Go codegen, drop moduleTypes
  6. test: add e2e tests for legacyCodegenAtRuntime config
  7. go: fix Go self-call schema names and cover dependency use (#2)
  8. go: match engine name casing for self-call enum values, args and types
    enum values → SCREAMING_SNAKE, args → lowerCamel, type names/refs → Camel,
    matching what the engine exposes from the same Go source.
  9. docs: regenerate PHP reference for Schema type after rebase
  10. core: key no-runtime-codegen off automaticGitignore, drop legacyCodegenAtRuntime
    — collapses the original separate legacyCodegenAtRuntime flag into the
    existing automaticGitignore=false signal and removes the field, its
    Clone entry and the load-time guardrail.
  11. core: gate no-runtime-codegen on engine version compatibility — only
    honor the opt-out when the module's pinned engineVersion is at least as
    new as the running engine; on skew, fall back to runtime codegen.

Commits 1 and 6 introduce/test an interim legacyCodegenAtRuntime field that
commit 10 then collapses into automaticGitignore; the net surface adds no
new config field.

Status / notes

  • Rebased onto current workspace. Engine builds; core/...,
    cmd/codegen/... unit tests pass; TestSelfCalls passes against a dev engine
    built from this branch.
  • The net new engine API surface is small: only a Schema object exposing
    merge / contents (plus the schema(json) constructor) surfaces in the
    SDKs, so the SDK regen diff is correspondingly small.
  • Design specs + implementation plans were kept local (not in this PR) — happy
    to share separately.

Test plan

  • Self-calls Go module: dagger generate runs codegen once (no asModule
    build/run), produces a compilable dag.MyModule() self binding, self types
    land in internal/dagger/<module>.gen.go.
  • Non-self-calls Go module: unchanged, single pass.
  • Python/TS modules: unaffected (still use moduleTypes).
  • automaticGitignore=false with committed files (and a compatible engine
    version): runtime skips codegen.
  • schemaTools merge — unit (core/schematool_test.go) + integration
    (core/integration/schematool_test.go) cover merge, idempotency, conflict,
    module-constructor reuse, interface/enum, and contents round-trip.
  • No-runtime-codegen e2e (RuntimeCodegenSuite): the missing committed
    generated-files error (TestMissingGeneratedFiles) and the version-skew
    fallback to runtime codegen (TestVersionSkewFallsBackToCodegen).

eunomie and others added 11 commits June 9, 2026 09:57
Introduce codegen.legacyCodegenAtRuntime in dagger.json. When false,
the Go SDK trusts the committed dagger.gen.go + internal/dagger files
and skips the runtime codegen pass. Validated at module load:
false requires automaticGitignore=false (generated files must be
committed). Defaults (nil/true) preserve the legacy behavior.

Signed-off-by: Yves Brissaud <yves@dagger.io>
Add an engine-side schemaTools surface exposed over dagql: load an
introspection schema (dag.schema(json)) and merge a module's
introspection-shaped types into it (merge(moduleTypes, moduleName)),
reading the result back via .contents. The merge stamps
@sourceModuleName and @sourcemap on the merged types so SDK
file-splitters can place them.

Kept minimal — only schema(json).merge(...).contents, which is all the
codegen needs. Provided by the engine so every SDK reasons about
schemas through one implementation. Includes the regenerated SDK
clients and reference docs for the new Schema type.

Signed-off-by: Yves Brissaud <yves@dagger.io>
When dagger.json sets codegen.legacyCodegenAtRuntime=false, Go SDK's
Runtime() skips the in-container codegen pass and builds straight from
the committed dagger.gen.go + internal/dagger files. A baseWithoutCodegen
helper mounts the source as-is (with gitconfig + SSH parity for private
modules) and returns the SSH-unset selectors to bracket the go build.
A precheck errors with an actionable message (self-calls aware) when the
committed files are missing. Codegen() and the default Runtime() path are
unchanged.

Signed-off-by: Yves Brissaud <yves@dagger.io>
Restore the typedef-registration branch generate-module emitted before
moduleTypes was split out. The generated invoke() dispatcher now handles
parentName == "" by returning a *dagger.Module built via
dag.Module().WithObject(...) from baked-in TypeDefCode, so the engine can
materialise the module's typedefs via the Runtime + empty-function-name
path. Adds TypeDefCode() to the parsed object/interface/enum/function/
type-spec types alongside the existing TypeDef() helpers.

Signed-off-by: Yves Brissaud <yves@dagger.io>
Make codegen generate-module run once for Go modules, including
self-calls. The Go codegen emits its own module's types as introspection
JSON (new introspect_emit.go, reusing the packages.Load-parsed types)
and, when --self-calls is set, merges them into the deps schema via the
engine's schemaTools (dag.schema(deps).merge(...)) in a single pass.

Drops the Go SDK's moduleTypes implementation (AsModuleTypes returns
nil,false; ModuleTypes method removed). The engine's runCodegen keeps
its self-append only for SDKs that implement moduleTypes (Python, TS);
Go discovers its types via the empty-parentName dispatcher and merges
codegen-side, so the engine no longer builds+runs the module via
asModule during codegen.

Adds the --self-calls flag, SelfCalls config field, and
experimentalPrivilegedNesting on the codegen exec so the in-container
dag client can call schemaTools.

Signed-off-by: Yves Brissaud <yves@dagger.io>
Cover the Go SDK no-codegen-at-runtime feature end to end:

- the config guardrail rejecting codegen.legacyCodegenAtRuntime=false
  unless codegen.automaticGitignore is also false, enforced during
  module load.
- the missing-generated-files error surfaced by the no-codegen runtime
  path, which points the user at `dagger generate`.

Signed-off-by: Yves Brissaud <yves@dagger.io>
* go: fix self-call field names

Problem:
When a Go module has SELF_CALLS enabled, the Go SDK writes the module schema
that is then used to generate the module's own Go client.

For exported fields, that schema was using the Go name directly. So a field
called Message was exposed as Message.

The engine exposes that same field as message. Because of that mismatch, the
generated Go client could be wrong. In the case we found, it could generate a
field and a method with the same Go name, which does not compile.

Fix:
Emit exported fields with the same name the engine uses. Message now becomes
message.

Test:
Add a Message field to the existing Go self-call fixture and query message from
the integration test. This proves the generated Go code compiles and the field
is available under the expected name.

Signed-off-by: Guillaume de Rouville <guillaume@dagger.io>

* tests: cover self-calls from dependencies

Problem:
The review raised a possible failure where a Go module with SELF_CALLS works
when called directly, but breaks when another module depends on it.

If that happened, the caller could still generate Go code for dag.Dep(), but
the running engine would not expose dep at runtime. The user would only see the
failure when calling the function.

We did not reproduce that exact failure, but this is still a useful area to
cover because it is easy to miss. A module dependency is loaded through a
different path than the module being called directly.

Solution:
Add two small Go fixtures:

- self-calls-as-dep: caller -> dep
- self-calls-transitive: caller -> middle -> dep

Only dep has SELF_CALLS enabled. That keeps the tests focused on the dependency
case from the review.

Tests:
viaDep calls a dependency function that calls itself through dag.Dep().

viaDepContainer returns a container created by that self-call and then reads
stdout from the caller. This proves the returned object is still usable.

viaDepWorker calls back into dag.Dep() from a secondary object. This proves the
case is not limited to methods on the root object.

viaTransitiveDep calls through caller -> middle -> dep. This proves the
self-call still works when the module with SELF_CALLS is a dependency of a
dependency.

Signed-off-by: Guillaume de Rouville <guillaume@dagger.io>

---------

Signed-off-by: Yves Brissaud <yves@dagger.io>
The self-call introspection emitter must reproduce exactly the names the
engine exposes (the TypeDef path passes raw Go names and lets the engine
normalize them). PR #2 fixed exported struct fields. Apply the same to the
remaining names:

- enum values -> ToScreamingSnake (Active -> ACTIVE); previously always
  mismatched, so a self-called enum value sent the wrong wire name.
- function/constructor arguments -> ToLowerCamel (matches the engine; only
  bit non-lowerCamel params such as acronyms).
- object/interface/enum type names and their type references -> ToCamel
  (only bit acronym type names, e.g. HTTPProxy -> HttpProxy).

Add a Color enum to the Go self-calls fixture and a self-call that passes an
enum argument, covering the enum-value casing end to end.

Signed-off-by: Yves Brissaud <yves@dagger.io>
Rebasing onto 1.0-beta hit a conflict in the generated PHP reference
(Binding/Client/Env.html), which was resolved to the 1.0-beta side and
so dropped the Schema type entries added by the schema-merge-tool patch.

Re-run `dagger generate` (engine built from source, so the schema
includes the new Schema type) to restore them: Client.schema()/
loadSchemaFromID and the Binding/Env schema accessors. Also refreshes
the doctum traits.html so php-sdk:api matches.

Signed-off-by: Yves Brissaud <yves@dagger.io>
…enAtRuntime

The Go SDK no-codegen-at-runtime opt-in was a separate
codegen.legacyCodegenAtRuntime flag that was only valid alongside
codegen.automaticGitignore=false, enforced by a load-time guardrail. That
coupling was redundant: opting out of the generated .gitignore already means
the module commits its generated files, so the runtime can trust them instead
of regenerating.

Collapse the two into a single signal. The Go SDK now skips the runtime codegen
pass when codegen.automaticGitignore=false and builds straight from the
committed dagger.gen.go + internal/dagger; unset/true keeps the legacy
runtime-codegen behavior. This also makes the committed files honest: what is
committed is what runs, instead of being regenerated at call time and possibly
differing from what the user sees in git.

- core/modules: drop the LegacyCodegenAtRuntime field, its Clone entry and the
  Validate guardrail; document the dual meaning of AutomaticGitignore.
- core/schema/modulesource: drop the now-empty codegen Validate call.
- core/sdk/go_sdk: key useRuntimeCodegen off AutomaticGitignore and reword the
  missing-generated-files prechecks.
- tests + generated config schema: follow the rename.

Signed-off-by: Yves Brissaud <yves@dagger.io>
useRuntimeCodegen previously skipped the runtime codegen pass for any Go
module with codegen.automaticGitignore=false, trusting the committed
generated files as-is. But the Go SDK's generated module dispatch is
engine-version specific: the empty-parentName typedef registration is only
emitted by the current codegen, so files generated by an older engine fail
to load against a newer one with "unknown object".

The repo's own dev modules pin engineVersion v0.21.4 (the last stable
release) yet run against the in-development engine, so their committed files
don't speak the running engine's module protocol. Keying skip-codegen purely
off automaticGitignore therefore broke every check that loads those modules
(php-sdk:api, ci:bootstrap, the test-split suite, release).

Only honor the opt-out when the module's pinned engineVersion is at least as
new as the running engine; on version skew, fall back to runtime codegen so
the committed files are regenerated to match. This keeps the single-flag
model while making it safe by construction: stale or mismatched committed
files can never silently run.

Signed-off-by: Yves Brissaud <yves@dagger.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants