Skip to content

Migrate to manifest-based capabilities (Closes CloudTAK#165) - #9

Merged
chriselsen merged 5 commits into
mainfrom
feature/manifest-capabilities
Oct 7, 2026
Merged

chriselsen merged 5 commits into
mainfrom
feature/manifest-capabilities

Conversation

@chriselsen

Copy link
Copy Markdown
Contributor

Summary

Migrates etl-floodhub to CloudTAK's manifest-based capabilities model (v1.1.0). The ETL now ships a capabilities.json manifest that is embedded in the container image as the com.cloudtak.capabilities OCI annotation, so CloudTAK can read the task's compute, permissions and invocations without running it.

Closes TAK-NZ/CloudTAK#165

Issue: TAK-NZ/CloudTAK#165

Changes per issue task

  • capabilities.json: new manifest (Google Flood Hub, memory 1024, timeout 120, required permission feature:submit, incoming schedule invocation defaulting to rate(2 minutes)). The only permission declared is feature:submit, because task.ts only calls this.submit(fc) plus layer-scoped env/ephemeral helpers, and PERMISSIONS in @tak-ps/etl 10.22.2 has feature: ['submit'].
  • Deps / API surface: dependencies updated, @tak-ps/etl range ^10.13.0 (installed 10.22.2), npm audit advisories resolved, EphemeralState type alias adjusted. typescript stays at 6.0.3 because the typescript-eslint peer is <6.1.0.
  • CI OCI-annotation build: both deploy jobs in etl-deploy.yml use docker/setup-buildx-action@v3 and a single docker buildx build with --annotation "com.cloudtak.capabilities=$(jq -c . capabilities.json)".
  • Tests + Task.init + Node 24: 5-test suite (static config, input schema, output schema, no outgoing flow, capabilities.json manifest) run with tsx --test; switched to Task.init(); Node 24 in lint.yml, etl-deploy.yml and engines.node >= 24.
  • CHANGELOG + version bump: 1.1.0 in package.json, package-lock.json and CHANGELOG.
  • Extra: .dockerignore trimmed (adds .agents/) to reduce the docker build context. capabilities.json, task.ts, package*.json and tsconfig.json remain in the context, and the real Dockerfile built successfully.

Deliberate deviation: cloudtak-etl CLI not used

The issue asked to use the cloudtak-etl CLI for building and pushing. We deliberately did not. Its bin/build.ts hardcodes the ECR repository tak-vpc-<Environment>-cloudtak-tasks. TAK-NZ base-infra names the repository <stackname>-etltasks and publishes it through the EcrEtlTasksRepoArn CloudFormation export. Using the CLI would push to a repository that does not exist in our environments, or force changes to base-infra naming.

Instead we use docker buildx build with the com.cloudtak.capabilities OCI annotation and keep the existing CFN-export ECR lookup. The result is the same annotation contract CloudTAK reads, without changing our infrastructure naming.

Verification

Run in a clean worktree with npm ci (all commits individually build green; final HEAD re-checked: lint=0, build=0, tests 5/5):

$ npm ci                      found 0 vulnerabilities   [exit 0]
$ npm run lint                (eslint *.ts test)        [exit 0]
$ npm run build               (tsc --build)             [exit 0]
$ npm test                    tests 5, pass 5, fail 0   [exit 0]
$ npm audit --omit=dev        found 0 vulnerabilities   [exit 0]
$ npm audit                   found 0 vulnerabilities   [exit 0]

capabilities.json validation (checklist section 1 snippet): exit 0, name Google Flood Hub, permissions feature:submit.

{
  "version": "1.0",
  "name": "Google Flood Hub",
  "description": "Ingests river gauge flood status and forecasts, flash flood events and significant flood events from the Google Flood Forecasting API for a configured region and posts them to the map as colour-coded gauge points and impact polygons.",
  "compute": { "memory": 1024, "timeout": 120 },
  "permissions": [
    {
      "resource": "feature:submit",
      "required": true,
      "description": "Submits river gauge points and flash flood / significant event polygons to CloudTAK"
    }
  ],
  "invocations": {
    "incoming": {
      "schedule": {
        "description": "Polls the Google Flood Forecasting API for updated gauge flood status, flash floods and significant events on a schedule.",
        "default": { "enabled": true, "schedule": "rate(2 minutes)" }
      }
    }
  }
}

OCI annotation: Docker 29.x was available locally. Using a throwaway buildx builder and a local registry:2, the exact workflow docker buildx build command was run and imagetools inspect --raw | jq .annotations showed the annotation byte-identical to jq -c . capabilities.json. This was run before .dockerignore was added. The push to real ECR is NOT verified.

Post-merge manual verification

Not done by an agent. Nothing has been deployed.

  • Verify the push to the demo environment succeeds (image lands in the <stackname>-etltasks ECR repo)
  • GET /api/task/raw/etl-floodhub/version/:version returns the capabilities
  • CloudTAK#157: permission-mismatch rejection works for this task
  • CloudTAK#157: update-diff modal shows the expected capabilities diff

- Add capabilities.json (feature:submit only, 1024 MB / 120 s, rate(2 minutes) schedule)
- Build and push with docker buildx in the demo and production deploy jobs,
  embedding the manifest as the com.cloudtak.capabilities OCI annotation
- Keep the existing EcrEtlTasksRepoArn CloudFormation export lookup
- Not adopting the cloudtak-etl CLI: it hardcodes a different ECR repo name

Refs TAK-NZ/CloudTAK#165
- Add tsx and a real test suite (node:test via tsx) covering static config,
  Env and output schemas, and capabilities.json validation
- Replace the no-op test script and lint the test directory
- Use Task.init() for the local and Lambda handler entry points
- Bump Node to 24 in etl-deploy.yml and add engines >= 24
- Document the buildx OCI annotation and npm test in the README

Refs TAK-NZ/CloudTAK#165
- npm update within existing ranges: @tak-ps/etl 10.22.2, eslint 10.12.0,
  typescript-eslint 8.71.1, tsx 4.23.15; npm audit reports 0 vulnerabilities
- typescript stays on 6.0.3: typescript-eslint still caps typescript at <6.1.0
- task.ts: make EphemeralState a type alias instead of an interface so it
  satisfies the Record<string, unknown> parameter of setEphemeral() in
  @tak-ps/etl 10.22 (type-only change, no runtime difference)
- Test: check manifest permissions against the exported PERMISSIONS map, since
  @tak-ps/etl no longer exports isValidPermission from its entry point

Refs TAK-NZ/CloudTAK#165
- Bump version to 1.1.0
- Document the capabilities manifest, OCI annotation build, tests, Node 24,
  dependency updates and the EphemeralState type alias
- Record the deliberate decision not to adopt the cloudtak-etl CLI (it
  hardcodes a different ECR repository name than base-infra's <stackname>-etltasks)

Closes TAK-NZ/CloudTAK#165
Excludes .git, .github, node_modules, dist, test, docs, .agents, .env* and
markdown files. capabilities.json, task.ts, package*.json and tsconfig.json
(everything the Dockerfile build needs) remain in the context; checked with a
COPY . listing of the context.

Refs TAK-NZ/CloudTAK#165
@chriselsen
chriselsen merged commit da03125 into main Oct 7, 2026
1 check passed
@chriselsen
chriselsen deleted the feature/manifest-capabilities branch October 7, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate etl-floodhub to manifest-based capabilities (etl-base / cloudtak-etl)

1 participant