Template step type support: registration and runtime - #262

Pull Request type

• Bugfix
• Feature
• Refactoring (no functional changes, no api changes)
• Build related changes (Please run ./gradlew build --write-locks to refresh dependencies)
• Other (please describe):

Changes in this PR

StepType.TEMPLATE exists in the code, together with the TemplateStep placeholder, TemplateInitiator and the TEMPLATE param sources, but it has no runtime implementation. This PR implements it. A job template can now register a list of steps, and a template step runs that list as one inline workflow instance. The step's outcome is the instance's outcome. This is similar to an Airflow TaskGroup. The step list is stored once in the job template registry instead of being copied into every workflow that uses it.

Nothing changes for existing workflows. A template step only runs after a job template with a step list is registered.

Registering a template

This PR adds a new steps field to JobTemplate.Definition. The existing POST /api/v3/job-templates endpoint registers it:

definition:
  job_type: write_audit_publish
  step_type: template
  version: v3
  params:
    target_table: {type: STRING, value: ""}
    snapshot_id:  {type: STRING, value: ""}
  steps:
    - step:
        id: write
        type: Kubernetes
        transition: {successors: {audit: "true"}}
    - step:
        id: audit
        type: Kubernetes
        transition: {successors: {publish_snapshot: "true"}}
        params:
          table: {type: STRING, expression: "target_table;"}
    - step:
        id: publish_snapshot
        type: Kubernetes
        params:
          snapshot: {type: STRING, expression: "snapshot_id;"}

steps is required when step_type is template and rejected for every other step type. A new @JobTemplateDefinitionConstraint on the definition field of JobTemplate and JobTemplateCreateRequest validates the list on upsert with the same rules as a workflow step list: non-empty, unique step ids, every transition names a step in the list, and the step count limit. These return 400:

step_type: Kubernetes with steps    [job template step definitions] can only be set for the template step type - rejected step type is [KUBERNETES]
step_type: template without steps   [job template step definitions] cannot be null or empty for the template step type
duplicate ids                       [workflow step definitions] contain duplicate step ids
transition to a missing step        [workflow step transition] is invalid for step [write]'s successor step id [audit], which does not exist in steps

The step list is read from the job template that the step names. inherit_from still merges params and tags, but it does not merge steps. Merging step lists across templates needs a defined merge rule first and is out of scope. A registered step list can itself contain a template step, so templates can be nested.

Using a template

A workflow declares a template step and supplies the inputs. This PR adds a sub_type_version field to TemplateStep, so a workflow can pin a template version like a typed step does. Without it, the version comes from the job_template_version workflow param and then default, the same as for typed steps.

template:
  id: publish
  sub_type: write_audit_publish
  sub_type_version: v3
  params:
    target_table: {type: STRING, value: core.trips}
    snapshot_id:  {type: STRING, expression: "return UUID.randomUUID().toString();"}

The template's registered params are merged into the step, the workflow's values are merged over them, and the result is evaluated before the inline instance starts. The evaluated values and all parent workflow params are passed as the inline instance's run params, so every step in the list can read them by name. A later step in the parent workflow reads publish__snapshot_id as it does for any other step.

Runtime

TemplateStepRuntime is structured like the other runtimes that launch child instances. The runtime owns the artifact, the run params it passes down, and the mapping from the child instance status to the step state. A new WorkflowActionHandler.runTemplateInstance method creates the child instance. The step lifecycle is the same as the subworkflow step: start one instance, track it, map its terminal status to a step state, and stop it on terminate. The child instance is created directly from a workflow built out of the registered steps, without run strategy and without a workflow definition lookup, because the step list is registry data and not a registered workflow.

Each parent instance and template step has one inline workflow id, generated by the existing StepHelper.generateInlineWorkflowId with a new maestro_template_ prefix. The child instance id is always 1 and its run id increases by one on each restart. Restarting the parent or the template step loads the previous run of the same child, applies the parent's run policy, and starts the next run. internal_id and workflow_version_id on the child row are the parent's values, the same as foreach and while.

A new TemplateArtifact records the child instance (template_workflow_id, template_instance_id, template_run_id, template_uuid, template_overview) plus the job_type and the template_version resolved at start. A job template version can be updated in place, so the artifact records which version key was used and the child instance's runtime_workflow holds the steps that actually ran.

Terminate works the same as for a subworkflow step: a queued child instance is terminated directly, otherwise the child's actor is woken up and the terminate is retried until the child is in a terminal state.

Reading a param from a step in the template

params.getFromTemplate('<template_step_id>', '<step_id>', '<param>') is added next to getFromSubworkflow and getFromForeach. It checks that the referenced step is a template step, reads the child workflow id and instance id from the artifact, and returns the named param from that step's latest run. It is documented in the SEL function guide.

Inline id recognition and deletion

IdHelper.isInlineWorkflowId now recognizes the template prefix, so getNonInlineParent returns the correct parent for a template child. Both inline deletion stages in MaestroWorkflowDeletionDao get a third id range, so deleting a workflow also deletes its template instances and their step instances. The new range is workflow_id >= prefix AND workflow_id < prefix || '~' on the primary key, the same as the foreach and while ranges.

MaestroWorkflowDao.getInlineWorkflowDefinitionInternal cast the child's initiator to ForeachInitiator. That throws ClassCastException for any non-foreach inline child, including while children today. It now casts to UpstreamInitiator, where getNonInlineParent is declared. The delete-rejection message for an inline id no longer says "foreach".

Ids starting with maestro_ are already rejected by MaestroIdConstraint and MaestroReferenceIdConstraint, so no workflow can be registered under the template prefix. A test asserts this for the template prefix.

Tests

Every new model has round-trip and from-JSON tests with fixtures. The constraint test covers each rejection case. The runtime test covers fresh start, restart with and without a previous artifact, retry when the insert fails, instance step concurrency unavailable, a missing template, every child status mapping, and the terminate paths. The handler test covers fresh and restart runs. JobTemplateManager tests cover loadSteps (found, missing, step type mismatch, no steps) and version resolution from a template step. MaestroParamExtension tests cover getFromTemplate and its failure cases. DAO tests cover the inline definition lookup for a template child, the latest template artifact, and that a template child instance and its step row are deleted with the parent workflow.