Skip to content

feat!: create the workspace with dlthub init and set up the coding agent - #74

Open
rudolfix wants to merge 10 commits into
develfrom
feat/proxy-dlthub-init
Open

rudolfix wants to merge 10 commits into
develfrom
feat/proxy-dlthub-init

Conversation

@rudolfix

Copy link
Copy Markdown

Overview

dlthub-init no longer ships a workspace of its own. It runs dlthub init from the newest dlt that uv resolves and then installs the dependencies with uv sync. The workspace files, the dependencies and the skills, rules and MCP server of the coding agent all come from dlt, so uvx dlthub-init@latest is current as soon as dlt is, without a dlthub-init release.

Until now the package bundled a minimal workspace (pyproject.toml, uv.lock, .dlt/*, .mcp.json, __deployment__.py) and five Claude-only skills copied from the AI harness. An hourly workflow driven from the agent loop kept that bundle in sync with dlt and the harness, and most of the git history is its commits. The bundle had drifted anyway: the skills no longer matched the harness init toolkit, and agents other than Claude got nothing.

What runs depends on the dlt that uv resolves:

dlt What runs
1.33.0a2 or newer dlthub init --agent. In an existing project it lists the files it changes and the lines it adds and asks before writing; -y accepts. It sets up the detected or named coding agent.
1.27.2 to 1.32.x dlthub init, then dlthub ai init. Nothing is merged: existing files are kept, an existing workspace only gets the agent setup, and --overwrite maps to --force.

Releasing changes too: there is no CHANGELOG.md any more. make prepare-release opens a release pull request, and merging it tags the merge commit and creates the GitHub release with generated notes.

Changes

Every commit is one changelog entry, so this PR should be merged with a merge commit or rebased, not squashed.

CLI (feat!: create the workspace with dlthub init instead of bundling one)

  • Options passed on to dlthub init: --name, --overwrite, --skip, --dependencies, --dry-run, --agent NAME, -y. Options of dlthub-init: DIR (created when missing), --no-sync, --no-telemetry, -v.
  • Breaking: --no-pyproject, --no-gitignore and --merge are removed (merging is the default now), as are the uv installer and the two beaupy prompts.
  • The coding agent is always set up. Inside a Claude Code, Cursor, Codex or Cortex Code session it is detected; outside, dlt picks the agent installed for the user or asks for --agent.
  • Output: dlthub's own output streams live, including its plan, its confirmation and the agent setup. uv sync then runs behind a spinner and its output is replayed when it fails, followed by the next steps. A failed dlthub step prints the step, the exit code and a hint, for example to re-run with -y or --skip.
  • Telemetry: one event per run, sent with a plain HTTP request instead of the posthog package. It uses dlt's anonymous id, honors dlt's opt-out as dlt resolves it, and reports the dlt version and the failed step. --no-telemetry, DLTHUB_INIT_TELEMETRY=0 and DO_NOT_TRACK=1 also turn dlt's telemetry off in the dlthub commands it runs, for that run only.
  • Dependencies are now dlt>=1.27.2, packaging and rich; beaupy and posthog are gone.
  • Deleted: the scaffold, collision handling, prompts, the bundled skills and the scripts that refreshed them. Tests run on pytest.

Workflows and CI

  • ci: remove the hourly workflow that refreshed the bundled workspace: bump-dlthub-init.yml, its changelog script and noop.yml.
  • ci: run on pushes to devel: the push trigger pointed at the stale main.
  • ci: run the integration tests on older dlt that cannot merge: a new older-dlt job runs the end-to-end tests with dlt 1.27.2 (the floor) and 1.32.0 (the newest stable) on Ubuntu and Windows. The existing jobs run everything on the locked dlt on Ubuntu, macOS and Windows.

Releasing (ci: release from a labelled pull request with notes generated by github)

  • make prepare-release [LEVEL=minor|major] runs on your machine, from a clean devel at origin/devel. It checks that the tag and the release branch don't exist and no release pull request is open, shows the commits since the last tag and asks for confirmation. It then bumps with uv version --bump, commits chore: bump version to X.Y.Z on chore/release-vX.Y.Z, pushes and opens a pull request labelled release, which runs CI. It never cleans up, so you stay on the release branch even when a step fails.
  • It runs locally because the org policy doesn't let Actions open pull requests, and a pull request opened with GITHUB_TOKEN would not start CI anyway.
  • Merging that pull request runs release.yml, which tags the merge commit vX.Y.Z and creates the GitHub release with --generate-notes. .github/release.yml keeps the release pull requests out of the notes. make publish still uploads to PyPI by hand.
  • CHANGELOG.md, release-dlthub-init.yml and its changelog script are deleted. The PR template says the title is the line of the release notes.

Development

  • chore: develop and test against dlt 1.33.0a2: [tool.uv] constraint-dependencies locks the dev environment to the first dlt pre-release that merges. It only affects this repo's lock, not users. Drop it once dlt 1.33.0 is released.
  • chore: remove the make targets that pin a workspace to a local platform stack: workspace-env, workspace-local, workspace-dev and scripts/pin_workspace_urls.py were copied from dlthub-start; dlthub-init never talks to a platform stack.
  • chore: add claude rules for tests, docstrings and heresy: .claude/rules and the two skills that point at them. A releasing rule comes with the release commit.
  • docs: describe what dlthub-init runs, its options and the supported dlt versions: README and CONTRIBUTING rewritten.

Try it with the dlt pre-release

uv never picks a pre-release on its own, so pin dlt with --with. Without it you get the newest stable dlt (1.32.0) and the two-step path, which is worth trying too.

From GitHub, without a checkout:

uvx --refresh --from "git+https://github.com/dlt-hub/dlthub-init@feat/proxy-dlthub-init" --with "dlt==1.33.0a2" dlthub-init my-workspace

From a checkout of this branch:

uvx --refresh --from . --with "dlt==1.33.0a2" dlthub-init ../my-workspace

--refresh picks up commits pushed since your last run.

Worth trying:

  • an empty folder, with --agent claude|cursor|codex|cortex and without it from inside an agent session, where the agent is detected
  • an existing project with a pyproject.toml and a .gitignore. dlthub lists what it changes and asks: answering N changes nothing and exits non-zero, -y applies the changes, --skip keeps existing files, and --overwrite overwrites the dependency file, .gitignore and .dlt/.workspace after asking
  • --dry-run, which prints the plan and writes nothing
  • running it again in the same folder

Tests

  • Unit tests cover the dlthub commands for each dlt version and mode, where the global flags and --agent go, the telemetry environment, the failure hints and the uv sync replay.
  • The integration tests run the real dlthub init, the agent setup and uv sync. On dlt 1.33.0a2 they also merge into an existing project with -y; on older dlt they check that existing files are kept. They pass locally on 1.33.0a2, 1.32.0 and 1.27.2.
  • make prepare-release was tested against a scratch clone with a local origin and a stub gh, including the failure paths. The GitHub side of releasing (the label and release.yml) runs for the first time on the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant