Go SDK: no-codegen-at-runtime + single-pass codegen#13381
Open
eunomie wants to merge 11 commits into
Open
Conversation
Introduce codegen.legacyCodegenAtRuntime in dagger.json. When false, the Go SDK trusts the committed dagger.gen.go + internal/dagger files and skips the runtime codegen pass. Validated at module load: false requires automaticGitignore=false (generated files must be committed). Defaults (nil/true) preserve the legacy behavior. Signed-off-by: Yves Brissaud <yves@dagger.io>
Add an engine-side schemaTools surface exposed over dagql: load an introspection schema (dag.schema(json)) and merge a module's introspection-shaped types into it (merge(moduleTypes, moduleName)), reading the result back via .contents. The merge stamps @sourceModuleName and @sourcemap on the merged types so SDK file-splitters can place them. Kept minimal — only schema(json).merge(...).contents, which is all the codegen needs. Provided by the engine so every SDK reasons about schemas through one implementation. Includes the regenerated SDK clients and reference docs for the new Schema type. Signed-off-by: Yves Brissaud <yves@dagger.io>
When dagger.json sets codegen.legacyCodegenAtRuntime=false, Go SDK's Runtime() skips the in-container codegen pass and builds straight from the committed dagger.gen.go + internal/dagger files. A baseWithoutCodegen helper mounts the source as-is (with gitconfig + SSH parity for private modules) and returns the SSH-unset selectors to bracket the go build. A precheck errors with an actionable message (self-calls aware) when the committed files are missing. Codegen() and the default Runtime() path are unchanged. Signed-off-by: Yves Brissaud <yves@dagger.io>
Restore the typedef-registration branch generate-module emitted before moduleTypes was split out. The generated invoke() dispatcher now handles parentName == "" by returning a *dagger.Module built via dag.Module().WithObject(...) from baked-in TypeDefCode, so the engine can materialise the module's typedefs via the Runtime + empty-function-name path. Adds TypeDefCode() to the parsed object/interface/enum/function/ type-spec types alongside the existing TypeDef() helpers. Signed-off-by: Yves Brissaud <yves@dagger.io>
Make codegen generate-module run once for Go modules, including self-calls. The Go codegen emits its own module's types as introspection JSON (new introspect_emit.go, reusing the packages.Load-parsed types) and, when --self-calls is set, merges them into the deps schema via the engine's schemaTools (dag.schema(deps).merge(...)) in a single pass. Drops the Go SDK's moduleTypes implementation (AsModuleTypes returns nil,false; ModuleTypes method removed). The engine's runCodegen keeps its self-append only for SDKs that implement moduleTypes (Python, TS); Go discovers its types via the empty-parentName dispatcher and merges codegen-side, so the engine no longer builds+runs the module via asModule during codegen. Adds the --self-calls flag, SelfCalls config field, and experimentalPrivilegedNesting on the codegen exec so the in-container dag client can call schemaTools. Signed-off-by: Yves Brissaud <yves@dagger.io>
Cover the Go SDK no-codegen-at-runtime feature end to end: - the config guardrail rejecting codegen.legacyCodegenAtRuntime=false unless codegen.automaticGitignore is also false, enforced during module load. - the missing-generated-files error surfaced by the no-codegen runtime path, which points the user at `dagger generate`. Signed-off-by: Yves Brissaud <yves@dagger.io>
* go: fix self-call field names Problem: When a Go module has SELF_CALLS enabled, the Go SDK writes the module schema that is then used to generate the module's own Go client. For exported fields, that schema was using the Go name directly. So a field called Message was exposed as Message. The engine exposes that same field as message. Because of that mismatch, the generated Go client could be wrong. In the case we found, it could generate a field and a method with the same Go name, which does not compile. Fix: Emit exported fields with the same name the engine uses. Message now becomes message. Test: Add a Message field to the existing Go self-call fixture and query message from the integration test. This proves the generated Go code compiles and the field is available under the expected name. Signed-off-by: Guillaume de Rouville <guillaume@dagger.io> * tests: cover self-calls from dependencies Problem: The review raised a possible failure where a Go module with SELF_CALLS works when called directly, but breaks when another module depends on it. If that happened, the caller could still generate Go code for dag.Dep(), but the running engine would not expose dep at runtime. The user would only see the failure when calling the function. We did not reproduce that exact failure, but this is still a useful area to cover because it is easy to miss. A module dependency is loaded through a different path than the module being called directly. Solution: Add two small Go fixtures: - self-calls-as-dep: caller -> dep - self-calls-transitive: caller -> middle -> dep Only dep has SELF_CALLS enabled. That keeps the tests focused on the dependency case from the review. Tests: viaDep calls a dependency function that calls itself through dag.Dep(). viaDepContainer returns a container created by that self-call and then reads stdout from the caller. This proves the returned object is still usable. viaDepWorker calls back into dag.Dep() from a secondary object. This proves the case is not limited to methods on the root object. viaTransitiveDep calls through caller -> middle -> dep. This proves the self-call still works when the module with SELF_CALLS is a dependency of a dependency. Signed-off-by: Guillaume de Rouville <guillaume@dagger.io> --------- Signed-off-by: Yves Brissaud <yves@dagger.io>
The self-call introspection emitter must reproduce exactly the names the engine exposes (the TypeDef path passes raw Go names and lets the engine normalize them). PR #2 fixed exported struct fields. Apply the same to the remaining names: - enum values -> ToScreamingSnake (Active -> ACTIVE); previously always mismatched, so a self-called enum value sent the wrong wire name. - function/constructor arguments -> ToLowerCamel (matches the engine; only bit non-lowerCamel params such as acronyms). - object/interface/enum type names and their type references -> ToCamel (only bit acronym type names, e.g. HTTPProxy -> HttpProxy). Add a Color enum to the Go self-calls fixture and a self-call that passes an enum argument, covering the enum-value casing end to end. Signed-off-by: Yves Brissaud <yves@dagger.io>
Rebasing onto 1.0-beta hit a conflict in the generated PHP reference (Binding/Client/Env.html), which was resolved to the 1.0-beta side and so dropped the Schema type entries added by the schema-merge-tool patch. Re-run `dagger generate` (engine built from source, so the schema includes the new Schema type) to restore them: Client.schema()/ loadSchemaFromID and the Binding/Env schema accessors. Also refreshes the doctum traits.html so php-sdk:api matches. Signed-off-by: Yves Brissaud <yves@dagger.io>
…enAtRuntime The Go SDK no-codegen-at-runtime opt-in was a separate codegen.legacyCodegenAtRuntime flag that was only valid alongside codegen.automaticGitignore=false, enforced by a load-time guardrail. That coupling was redundant: opting out of the generated .gitignore already means the module commits its generated files, so the runtime can trust them instead of regenerating. Collapse the two into a single signal. The Go SDK now skips the runtime codegen pass when codegen.automaticGitignore=false and builds straight from the committed dagger.gen.go + internal/dagger; unset/true keeps the legacy runtime-codegen behavior. This also makes the committed files honest: what is committed is what runs, instead of being regenerated at call time and possibly differing from what the user sees in git. - core/modules: drop the LegacyCodegenAtRuntime field, its Clone entry and the Validate guardrail; document the dual meaning of AutomaticGitignore. - core/schema/modulesource: drop the now-empty codegen Validate call. - core/sdk/go_sdk: key useRuntimeCodegen off AutomaticGitignore and reword the missing-generated-files prechecks. - tests + generated config schema: follow the rename. Signed-off-by: Yves Brissaud <yves@dagger.io>
useRuntimeCodegen previously skipped the runtime codegen pass for any Go module with codegen.automaticGitignore=false, trusting the committed generated files as-is. But the Go SDK's generated module dispatch is engine-version specific: the empty-parentName typedef registration is only emitted by the current codegen, so files generated by an older engine fail to load against a newer one with "unknown object". The repo's own dev modules pin engineVersion v0.21.4 (the last stable release) yet run against the in-development engine, so their committed files don't speak the running engine's module protocol. Keying skip-codegen purely off automaticGitignore therefore broke every check that loads those modules (php-sdk:api, ci:bootstrap, the test-split suite, release). Only honor the opt-out when the module's pinned engineVersion is at least as new as the running engine; on version skew, fall back to runtime codegen so the committed files are regenerated to match. This keeps the single-flag model while making it safe by construction: stale or mismatched committed files can never silently run. Signed-off-by: Yves Brissaud <yves@dagger.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Original PR: #13210
What this does
Reworks the Go SDK codegen path so that:
codegen.automaticGitignore=falseindagger.json, the Go SDK trusts thecommitted
dagger.gen.go+internal/daggerfiles and skips thein-container codegen pass at runtime (it errors with an actionable message
if the committed files are missing). There is no separate opt-in flag:
opting out of the generated
.gitignorealready means the module commitsits generated files, so the runtime can trust them. This makes the
committed files honest — what is committed is what runs, instead of being
regenerated at call time and possibly differing from what the user sees in
git.
module dispatch is engine-version specific (the empty-parentName typedef
registration is only emitted by the current codegen). So the skip is only
honored when the module's pinned
engineVersionis at least as new as therunning engine; on version skew it falls back to runtime codegen, so stale
or mismatched committed files can never silently run.
generate-moduleruns once for Go modules, including self-calls.Previously self-calls modules ran it twice (the engine built+ran the module
via
asModulejust to discover its own types, then ran codegen again). Nowthe Go codegen discovers its own types from the existing
packages.Loadanalysis, emits them as introspection JSON, and merges them into the deps
schema via the engine's new
schema(json).merge(...)tool — in a singlepass. No AST analyzer;
packages.Loadis retained.moduleTypesis dropped from the Go SDK. Type discovery happens throughthe generated
invoke()dispatcher's empty-parentName arm. Other SDKs(Python, TS) keep
moduleTypesand the engine-side self-append.Commits (read in order)
core/modules: add legacyCodegenAtRuntime codegen config fieldfeat: add schema merge tool for codegen— engine-sideschema(json).merge(...).contents, exposed over dagql for all SDKs. Keptminimal: only the
schema/merge/contentsthe codegen needs (nolistTypes/hasType/describeTypeorIntrospectionTypegraph).core/sdk/go_sdk: skip codegen at runtime when opted incmd/codegen/generator/go: emit empty-parentName module dispatchcmd/codegen + core: single-pass Go codegen, drop moduleTypestest: add e2e tests for legacyCodegenAtRuntime configgo: fix Go self-call schema names and cover dependency use (#2)go: match engine name casing for self-call enum values, args and types—enum values → SCREAMING_SNAKE, args → lowerCamel, type names/refs → Camel,
matching what the engine exposes from the same Go source.
docs: regenerate PHP reference for Schema type after rebasecore: key no-runtime-codegen off automaticGitignore, drop legacyCodegenAtRuntime— collapses the original separate
legacyCodegenAtRuntimeflag into theexisting
automaticGitignore=falsesignal and removes the field, itsCloneentry and the load-time guardrail.core: gate no-runtime-codegen on engine version compatibility— onlyhonor the opt-out when the module's pinned
engineVersionis at least asnew as the running engine; on skew, fall back to runtime codegen.
Status / notes
workspace. Engine builds;core/...,cmd/codegen/...unit tests pass;TestSelfCallspasses against a dev enginebuilt from this branch.
Schemaobject exposingmerge/contents(plus theschema(json)constructor) surfaces in theSDKs, so the SDK regen diff is correspondingly small.
to share separately.
Test plan
dagger generateruns codegen once (noasModulebuild/run), produces a compilable
dag.MyModule()self binding, self typesland in
internal/dagger/<module>.gen.go.moduleTypes).automaticGitignore=falsewith committed files (and a compatible engineversion): runtime skips codegen.
schemaToolsmerge — unit (core/schematool_test.go) + integration(
core/integration/schematool_test.go) cover merge, idempotency, conflict,module-constructor reuse, interface/enum, and contents round-trip.
RuntimeCodegenSuite): the missing committedgenerated-files error (
TestMissingGeneratedFiles) and the version-skewfallback to runtime codegen (
TestVersionSkewFallsBackToCodegen).