Skip to content
BranchesTags

Repository files navigation

JJ-Based Project

Continuously Composed Parallel Development Foundation built with Elixir, Ash Framework, and JJ (Jujutsu) version control. Deploys to K3S via ArgoCD with CloudNativePG.

Overview

This system manages multiple independent change streams (transparent sheets) per developer, continuously composing them in CI every 30 seconds. Immutable snapshots are created from selected streams and promoted through Stage to Production environments.

Core Concepts

  • Stream — An independent line of work identified by a stable JJ change ID. Each user can have many streams simultaneously (e.g., "email validation", "pagination", "refactor"). Streams have states: Active, Composing, Composed, Conflicting, Finished, Abandoned.
  • CI Composition — Every 30 seconds, all Active/Composed streams are overlaid together via JJ multi-parent merge. Validates and compiles only; never deployed. Conflicts detected automatically.
  • Snapshot — An immutable point-in-time merge of selected streams at specific commit versions. Can be production-bound (semantic version like v1.2.7) or experimental (descriptive name like "testing analytics refactor").
  • Environment — Kubernetes namespace: CI (live composition), Stage (snapshot testing), Production (promoted snapshots), Hotfix (isolated emergency fixes).
  • One-Way Sync — JJ is the source of truth. Changes flow JJ to Git (Forgejo) only. No direct pushes to Forgejo allowed.

Stream State Machine

                  +-----------+
                  |  Active   |<---------+
                  +-----+-----+          |
                        |                |
                  begin_composing   reactivate (on edit)
                        |                |
                  +-----v-----+    +-----+-----+
                  | Composing  |    |  Finished  |
                  +-----+-----+    +-----+-----+
                        |                ^
                  mark_composed          |
                        |             finish
                  +-----v-----+          |
              +-->|  Composed  |----+----+
              |   +-----+-----+    |
              |         |          |
         resolve    mark_conflicting
              |         |          |
              |   +-----v------+   |
              +---|Conflicting |   |
                  +-----+------+   |
                        |          |
                     abandon    abandon
                        |          |
                  +-----v-----+    |
                  | Abandoned  |<--+
                  +-----------+

Tech Stack

Layer Technology
Language Elixir 1.19+ / Erlang/OTP 28
Framework Phoenix 1.8, Ash 3.x
Database PostgreSQL via CloudNativePG
API JSON:API (AshJsonApi) with OpenAPI/Swagger
VCS JJ (Jujutsu) with Git backend
Container Docker, multi-stage build
Orchestration K3S via K3D (local), Traefik ingress
GitOps ArgoCD watching Forgejo
Git Host Forgejo (self-hosted, on K3S)

Prerequisites

brew install jj kubectl helm argocd k3d
brew install --cask docker    # needs sudo
mix archive.install hex igniter_new
mix archive.install hex phx_new

Local Development

1. Start PostgreSQL

docker compose up -d

Or use your own PostgreSQL with config in config/dev.exs.

2. Setup and Run

mix setup
mix ash.setup               # creates DB, runs migrations
mix run priv/repo/seeds.exs  # seed test users and environments
mix phx.server               # start at http://localhost:4000

3. Verify

curl http://localhost:4000/health/ready
curl http://localhost:4000/api/json/users -H "Accept: application/vnd.api+json"
curl http://localhost:4000/api/ci/status

Swagger UI: http://localhost:4000/api/json/swaggerui

K3S Deployment (Local)

Quick Start

Ensure Docker Desktop is running, then:

cd /path/to/jj_based_project
./scripts/setup-k3s.sh

This creates a K3D cluster with:

  • Local Docker registry on port 5050
  • CloudNativePG operator
  • ArgoCD
  • 4 namespaces (ci, stage, production, hotfix)
  • App deployed to production with CloudNativePG PostgreSQL

Manual Step-by-Step

# 1. Create registry + cluster
k3d registry create registry.localhost --port 5050
k3d cluster create jj-dev \
  --port "8080:80@loadbalancer" \
  --port "8443:443@loadbalancer" \
  --registry-use k3d-registry.localhost:5050

# 2. Create namespaces
kubectl apply -f k8s/base/namespace.yaml

# 3. Install operators
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm install cnpg cnpg/cloudnative-pg -n cnpg-system --create-namespace --wait

helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n argocd --create-namespace \
  --set server.service.type=ClusterIP --wait

# 4. Build and push image
docker build -t k3d-registry.localhost:5050/jj-based-project:latest .
docker push k3d-registry.localhost:5050/jj-based-project:latest

# 5. Deploy PostgreSQL
kubectl apply -f k8s/cnpg/cluster.yaml -n production
# Wait for: kubectl get cluster jj-app-db -n production

# 6. Deploy app
kubectl create serviceaccount jj-app -n production
kubectl apply -f k8s/base/secret.yaml -n production
kubectl apply -f k8s/base/rbac.yaml
kubectl apply -f k8s/base/deployment.yaml -n production
kubectl apply -f k8s/base/service.yaml -n production
kubectl apply -f k8s/base/ingress.yaml -n production

# 7. Run migrations
kubectl exec -n production deployment/jj-app -- /app/bin/migrate

Access Points

Service URL Notes
App http://localhost:8080 Via Traefik ingress
JSON API http://localhost:8080/api/json/ JSON:API endpoints
Swagger UI http://localhost:8080/api/json/swaggerui Interactive API docs
Health http://localhost:8080/health/ready Liveness + DB check
CI Status http://localhost:8080/api/ci/status Reconciliation worker
ArgoCD kubectl port-forward svc/argocd-server -n argocd 9090:443 https://localhost:9090
Forgejo kubectl port-forward svc/forgejo-http -n forgejo 3000:3000 http://localhost:3000

API Reference

All JSON:API endpoints are at /api/json/. Full OpenAPI spec at /api/json/open_api.

Users

# List users
GET /api/json/users

# Create user
POST /api/json/users
{"data":{"type":"user","attributes":{"name":"Alice","email":"alice@example.com","role":"tech_lead"}}}

# Get user
GET /api/json/users/:id

Roles: developer, tech_lead, admin

Streams

# List all streams
GET /api/json/streams

# Create stream
POST /api/json/streams
{"data":{"type":"stream","attributes":{
  "description":"Driver email validation",
  "user_id":"<uuid>",
  "change_id":"<jj-change-id>",
  "parent_ref":"main"
}}}

# Update description
PATCH /api/json/streams/:id
{"data":{"type":"stream","attributes":{"description":"Updated description"}}}

# Abandon stream
PATCH /api/json/streams/:id  (action: abandon)

# Finish stream
PATCH /api/json/streams/:id  (action: finish)

Stream states: active, composing, composed, conflicting, finished, abandoned

Commits

# List commits
GET /api/json/commits

# Get commit
GET /api/json/commits/:id

Divergent Paths

# List divergent paths
GET /api/json/divergent_paths

# Get divergent path
GET /api/json/divergent_paths/:id

Snapshots

# List snapshots
GET /api/json/snapshots

# Create snapshot
POST /api/json/snapshots
{"data":{"type":"snapshot","attributes":{
  "name":"v1.2.7",
  "snapshot_type":"production_bound",
  "base_version":"v1.2.6"
}}}

# Deploy to stage
PATCH /api/json/snapshots/:id  (action: deploy_to_stage)

# Promote to production (production_bound only)
PATCH /api/json/snapshots/:id  (action: promote_to_production)

# Retry failed infrastructure deployment
PATCH /api/json/snapshots/:id  (action: retry_deployment)

# Discard snapshot
PATCH /api/json/snapshots/:id  (action: discard)

# Invalidate (stale after hotfix)
PATCH /api/json/snapshots/:id  (action: invalidate)

# Mark rolled back
PATCH /api/json/snapshots/:id  (action: mark_rolled_back)

Snapshot types: production_bound, experimental

Snapshot statuses: created, testing_in_stage, deployed_to_production, failed_infra, failed_code, rolled_back, invalidated, discarded

Environments

# List environments
GET /api/json/environments

# Get environment
GET /api/json/environments/:id

CI Status

# Get reconciliation worker status
GET /api/ci/status

# Trigger manual reconciliation
POST /api/ci/trigger

Project Structure

lib/jj_based_project/
  accounts/
    user.ex                    # Ash resource: User
  accounts.ex                  # Domain: Accounts

  vcs/
    stream.ex                  # Ash resource: Stream (state machine)
    commit.ex                  # Ash resource: Commit
    divergent_path.ex          # Ash resource: DivergentPath
    reconciliation_worker.ex   # GenServer: 30s CI composition loop
  vcs.ex                       # Domain: VCS

  deployments/
    snapshot.ex                # Ash resource: Snapshot
    snapshot_stream.ex         # Ash resource: SnapshotStream (join)
    environment.ex             # Ash resource: Environment
    deployer.ex                # K8S deployment pipeline
  deployments.ex               # Domain: Deployments

  jj/
    cli.ex                     # JJ CLI wrapper (System.cmd)
    k8s_client.ex              # K8S API client (Req-based)

  repo.ex                      # AshPostgres.Repo
  application.ex               # OTP supervision tree
  release.ex                   # Release tasks (migrate)

lib/jj_based_project_web/
  router.ex                    # Routes: health, API, JSON:API
  ash_json_api_router.ex       # AshJsonApi router (3 domains)
  controllers/
    health_controller.ex       # /health/live, /health/ready
    ci_status_controller.ex    # /api/ci/status, /api/ci/trigger

k8s/
  base/                        # Deployment, Service, Ingress, Secrets, RBAC
  overlays/
    production/                # Kustomize overlay
    stage/
    hotfix/
  cnpg/cluster.yaml            # CloudNativePG PostgreSQL cluster
  forgejo/values.yaml          # Forgejo Helm chart values
  argocd/application.yaml      # ArgoCD Application definitions

scripts/
  setup-k3s.sh                 # Full local K3S setup script

JJ CLI Wrapper

The JjBasedProject.Jj.Cli module wraps JJ commands as Elixir functions:

Function JJ Command Purpose
init_repo/1 jj git init Initialize JJ repo with Git backend
create_change/2 jj new <parent> Create new stream/change
describe/3 jj describe -m Set change description
compose/2 jj new rev1 rev2 ... Overlay multiple streams (multi-parent merge)
check_conflicts/1 jj log -T conflict_description Detect conflicts in composition
rebase/3 jj rebase -s <src> -d <dst> Foundation updates
log/2 jj log --no-graph -T Structured change history
git_push/2 jj git push One-way push to Forgejo
create_bookmark/3 jj bookmark create Named pointer for deployment
abandon/2 jj abandon Abandon a change
undo/1 jj undo Undo last operation
write_file/3 (File.write) Simulate changes in working copy

All functions return {:ok, result} or {:error, %{exit_code, output, args}}.

Deployment Pipeline Flow

Developer creates stream
        |
        v
Stream Active in CI -----> 30s reconciliation cycle
        |                         |
        |                    Compose all Active streams
        |                    via JJ multi-parent merge
        |                         |
        |                    Check conflicts
        |                    /          \
        |               Clean         Conflicts
        |                |                |
        |          Mark Composed    Mark Conflicting
        |
        v
Tech lead creates Stage Snapshot
  (selects streams + commit versions)
        |
        v
Snapshot deployed to Stage namespace
  (K8S Deployment + CNPG PostgreSQL)
        |
        v
Health checks pass? ----No----> Failed (Infra: retry, Code: discard)
        |
       Yes
        |
        v
Team tests in Stage
        |
        v
Promote to Production (production_bound only)
        |
        v
Health checks + smoke tests
        |
        v
Contributing streams -> Finished
All CI streams rebased onto new foundation
Stale Stage snapshots -> Invalidated

Hotfix Flow

Production emergency
        |
        v
Create hotfix stream from Production tag
  (isolated from CI)
        |
        v
Fix applied + tested in Hotfix namespace
        |
        v
Fast-track: Stage -> Production
        |
        v
Foundation update for all CI streams
Stale Stage snapshots invalidated

Environment Variables

Variable Required Default Description
DATABASE_URL prod - PostgreSQL connection string
SECRET_KEY_BASE prod - Phoenix secret (generate with mix phx.gen.secret)
PHX_HOST prod example.com Hostname for URL generation
PHX_SERVER prod - Set to true to start server
PORT no 4000 HTTP port
POOL_SIZE no 10 Database connection pool size

Useful Commands

# Development
mix phx.server                    # Start dev server
mix ash.codegen <name>            # Generate migration after resource changes
mix ash_postgres.migrate          # Run migrations
mix run priv/repo/seeds.exs       # Seed data

# K3S
kubectl get pods -n production    # Check production pods
kubectl logs -f -l app=jj-app -n production   # Tail app logs
kubectl exec -n production deployment/jj-app -- /app/bin/migrate  # Run migrations
kubectl exec -n production deployment/jj-app -- /app/bin/jj_based_project remote  # IEx remote shell

# Docker
docker build -t k3d-registry.localhost:5050/jj-based-project:latest .
docker push k3d-registry.localhost:5050/jj-based-project:latest
kubectl rollout restart deployment/jj-app -n production  # Deploy new image

# Cluster management
k3d cluster start jj-dev         # Start stopped cluster
k3d cluster stop jj-dev          # Stop cluster
k3d cluster delete jj-dev        # Delete cluster entirely

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages