Skip to content
qaraxPublic

Latest commit

 

History

585 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

qarax

qarax is a management platform for virtual machines running on Cloud Hypervisor and Firecracker.

Architecture

qarax consists of two main components:

  • qarax (control plane) -- Axum REST API server managing VM and host lifecycle, backed by PostgreSQL
  • qarax-node (data plane) -- gRPC service running on hypervisor hosts, managing VM execution via Cloud Hypervisor and Firecracker

Supporting crates:

  • cli -- Command-line client for the qarax API
  • qarax-init -- Minimal PID 1 init process for OCI-booted VMs
  • common -- Shared logging and telemetry helpers

Communication flow: control plane -> gRPC (proto/node.proto) -> qarax-node -> selected hypervisor backend.

Building

make build       # build all packages and generate OpenAPI spec
make test        # run tests (auto-starts Postgres via Docker)
make lint        # cargo clippy --workspace -- -D warnings
make ci-checks   # run every CI check (make test-deps first)

Running locally

make run-local           # compose mode: qarax + qarax-node + Postgres in Docker Compose
make run-local VM=1      # VM mode: the node is the appliance image as a bootc microVM
make stop-local          # stop the stack

Requires Docker, Docker Compose, KVM (/dev/kvm), and a Rust toolchain.

The API is served under /v1 (e.g. http://localhost:8000/v1/vms), with Swagger UI at http://localhost:8000/swagger-ui. Lists return one page, {"items": [...], "next_cursor": ...}: pass limit (1 to 1000, default 100) and the previous page's next_cursor as cursor for the next.

Installing a release

Releases are published as static binaries (GitHub releases) and images (ghcr.io/qarax/qarax for the control plane, ghcr.io/qarax/qarax-vmm-host for nodes). docs/INSTALL.md sets one up on real hosts, and CHANGELOG.md lists what each release changes and what an upgrade needs.

More documentation

See docs/DEVELOPMENT.md for running the node as a bootc microVM (make run-local VM=1).

See docs/NETWORK_ISOLATION.md for VPC-style subnet grouping, cross-host VXLAN-backed private routing, VM security groups, and the intended CLI workflow.

See docs/PLACEMENT_POLICIES.md for host reservation classes, placement labels, VM placement policies, scheduler ordering, and CLI/API examples.

CLI quickstart

See the CLI README for full usage. Quick version:

cargo build -p cli --release

qarax configure --server http://localhost:8000
qarax host list
qarax vm list

Host provisioning

qarax uses bootc (bootable containers) to deploy hypervisor hosts. The appliance image includes qarax-node, Cloud Hypervisor, Firecracker, and all dependencies.

# Register the host
qarax host add --name node-01 --address 10.0.0.42 --user root

# Build and push the appliance
make appliance-build
make appliance-push

# Deploy and initialize
qarax host deploy node-01 --image ghcr.io/yourorg/qarax-vmm-host:latest --ssh-key ~/.ssh/id_ed25519
# Install the node's certificates (below), then:
qarax host init node-01

See the qarax-node README for runtime dependencies and configuration.

Node certificates

qarax-node's gRPC API only serves clients that present a certificate from its CA. Each node needs a server certificate for the address its host is registered under (--address), and the control plane needs a client certificate. scripts/gen-node-tls.sh makes both; run it again with more nodes to issue theirs from the same CA:

scripts/gen-node-tls.sh ./node-tls 10.0.0.42 node-02.example.com

# Control plane: node_tls_dir (default /etc/qarax/node-tls, env NODE_TLS_DIR)
install -d -m 700 /etc/qarax/node-tls
cp node-tls/control-plane/* /etc/qarax/node-tls/

# Each host: qarax-node --tls-dir (default /etc/qarax-node/tls)
ssh root@10.0.0.42 'install -d -m 700 /etc/qarax-node/tls'
scp node-tls/nodes/10.0.0.42/* root@10.0.0.42:/etc/qarax-node/tls/

Keep node-tls/ca-key.pem off the hosts. Both sides read their files at startup: the control plane refuses to start without them, and qarax-node exits until they are there (its systemd unit restarts it). Restart both after replacing certificates. The local stack (make run-local) and the e2e tests generate their own.

Host credentials

For new hosts, keep SSH passwords outside Qarax and register an external credential reference:

# Resolve from the qarax process environment.
export NODE_01_SSH_PASSWORD='...'
qarax host add --name node-01 --address 192.0.2.10 --user root \
  --credential-ref env://NODE_01_SSH_PASSWORD

# Resolve from a secret mounted into the qarax container.
qarax host add --name node-02 --address 192.0.2.11 --user root \
  --credential-ref file:///run/secrets/node-02-ssh-password

Environment variables and mounted files are read only when a host deploy or upgrade starts. The reference is stored in PostgreSQL, but neither references nor resolved secrets are returned by the API or written to logs. File references must use absolute paths visible inside every control-plane replica. Environment variables must likewise be configured consistently on every replica.

Native secret-manager API providers are not yet included. Vault Agent, Secrets Store CSI Driver, and similar sidecars can expose secrets as mounted files for file:// references. Direct vault://, AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault references require a future provider implementation.

Qarax does not store host passwords. A migration drops the former hosts.password column, so operators upgrading from an older release must register credential references before the next deploy or upgrade. A deploy request may still supply an operation-scoped password or private-key path; those values are used for that operation and are never persisted.

VM boot configuration

Default boot artifacts are configured per environment in configuration/ (base.yaml, local.yaml, production.yaml), selected by the APP_ENVIRONMENT env var (default: local):

vm_defaults:
  kernel: "/var/lib/qarax/images/vmlinux"
  initramfs: "/var/lib/qarax/images/initramfs.gz"
  cmdline: "console=ttyS0 console=hvc0 root=/dev/vda1"

Demos

Working demo setups in demos/:

Demo Description
oci/ OverlayBD lazy-pull disk workflow
boot-source/ Direct kernel + initramfs boot
hooks/ Lifecycle webhooks
etcd-cluster/ 3-node etcd cluster on VMs
k8s-cluster/ 3-node kubeadm Kubernetes cluster
gpu-passthrough/ VFIO GPU passthrough
firecracker/ Firecracker backend lifecycle demo
hyperconverged/ Single-VM control plane + node
sandbox/ Ephemeral auto-reaping VMs with prewarmed pool claims
sse-events/ Server-Sent Events stream

Releases

Packages

Used by

Contributors

Languages