jerbob.me

Management tasks on agentless containers

2021-05-12T00:00:00+00:00

Running one-off agentless commands.

Earlier this year, I was tasked with dealing with a Django deployment that had gone awry.

This particular project was running in an ECS container, and the underlying issue was caused by a logical error in our entrypoint script that was failing to run migrations before starting the WSGI server. This meant that we were only notified of the problem once the required fields were accessed.

Since this issue with our entrypoint wasn’t readily apparent, the immediate stop-gap here would be to run migrations manually, but there’s no easy way to do this if your containers are running on EC2-backed ECS with the SSM Agent disabled, or if you’re backed by Fargate.

Thankfully, our database and cache were accessible behind an SSH bastion, meaning it was possible to spin up a local instance of the project hooked up directly to the live database. I hacked together this docker-compose.yml to allow us to run this one-off management command:

version: "3.7"

x-bastion-service: &bastion-service
  image: kroniak/ssh-client
  volumes:
    - "${SSH_AUTH_SOCK}:${SSH_AUTH_SOCK}"
    - "~/.ssh/id_rsa:/root/.ssh/id_rsa"
    - "~/.ssh/id_rsa.pub:/root/.ssh/id_rsa.pub"
  environment:
    - SSH_AUTH_SOCK
  entrypoint: "ssh -vp 22 -N user@bastion.host -o StrictHostKeyChecking=no -L"

x-live-environment: &live-environment
  environment:
    - DEPLOY_ENV=live
  links:
    - "postgres:postgres.production.host"

services:
  web:
    <<: *live-environment
    build: .
  depends_on:
    - db

  db:
    <<: *bastion-service
    command: "0.0.0.0:5432:postgres.production.host:5432"

If you’re not familiar with docker-compose syntax, this essentially defines a new service, db, that is actually just a SSH client masquerading as a database. It’s simply forwarding requests to the container’s port 5432 through the bastion and directly to postgres.production.host:5432.
We then just add a link to proxy any requests intended for the production host to our fake database.

Needless to say, this is a terrible idea to run against production, and you shouldn’t be using this for an established production environment. In an ideal scenario, your entrypoints would be written and tested well before you run into issues with migrations. These were not ideal circumstances, but this compose happened to save us a few hours of downtime.

It was a fun little challenge to solve, and hopefully this helps anyone who might be in a similar position. If you do end up needing something like this, the only things worth keeping in mind are:

Ensuring that initialisation actions are idempotent.
If you do have non-idempotent tasks running on initialisation, disable them.
Ensuring that your container is otherwise identical to production.
Don't enable any debug settings, or point your local instance to a development configuration.

Keep this in mind, and you should be able to execute your management command as per:

docker-compose exec web python src/manage.py migrate

Moving from pulseaudio to pipewire

2021-03-13T00:00:00+00:00

Some personal notes on pipewire.

I’ve seen a lot of talk about pipewire recently, particularly given the proposal to route all audio to pipewire for Fedora 34, so I figured I’d try it out. Since switching audio stacks would usually be a pretty daunting task, I’ve documented my process below. As it turns out, switching to pipewire from pulseaudio is incredibly simple, so this is more of a personal reference than anything.

Installation

Installing pipewire is just a case of using your package manager:

yay -S pipewire

If you’d like to use pipewire as a replacement for pulseaudio, or to run JACK software, install the relevant plugins to replace those sound systems:

yay -S pipewire-pulse pipewire-jack

Their systemd services should already be enabled, but it can’t hurt to make sure:

systemctl --user enable --now pipewire-media-session pipewire-pulse pipewire

This should be all that’s required. It may be worth rebooting to ensure that the pulseaudio/JACK daemons aren’t still running, but that’s it! You’ll still be able to use software that depends on pulse/JACK as if you had those servers running.

Practical uses

If you have both pipewire-pulse and pipewire-jack installed, your pulseaudio sinks will appear in JACK software natively. This offers some promising prospects with using software for either system interchangeably. For example, you could use Catia, the JACK transport patchbay, to visualise your pulseaudio sinks and route audio more intuitively than you might be able to with pactl.

An obligatory uses.tech page

2020-11-30T00:00:00+00:00

A collection of hardware & software I use.

Shell

My shell is Zsh, with the zinit plugin manager. I also source oh-my-zsh later on in my ~/.zshrc, for the few QoL things I can’t find in other plugins. You can see the list of plugins I use in my dotfiles.

Terminal Emulator & Font

When I’m actually working, I just use uxterm with Ypnose’s excellent envypn font. It’s a tiny bitmap font that gives me plenty of breathing room for vertical and horizontal code splits, even on the tiny laptop monitors that I use most often.

When I’m not particularly in need of screen real estate, I’m prone to font-hopping every other day. My current choice is Relative Mono 11 Pitch at around 20px.

System

My operating system of choice is currently Arch, and my window manager is bspwm. I have no swap, instead opting for earlyoom to handle OOMs for me. I try and keep my system clean and simple, running only what I need.

Editor

My preferred editor is Neovim, the hyperextensible Vim-based text editor. I use it alongside coc.nvim for VSCode-esque intellisense. Auto-imports and static typechecking are provided by coc-pyright, a coc.nvim port of Microsoft’s pyright. This workflow gives me some of the smartness of a fully-fledged IDE, while staying as close to the terminal experience as possible.

Utilities

Since I primarily write (and work) in Python, these will all be Python libraries and/or utilities:

poetry

I use poetry to set up any new generic project. It’s a great dependency resolver and project manager that hides a lot of the complexity involved in publishing your own packages.
django-stubs

If you’re not familiar with type-checked python, I highly recommend watching this PyCon talk. django-stubs is a package containing type stubs for your type checker, allowing it to more accurately resolve types for things that use Django’s internal metaclass magicks. Invaluable for anyone interested in type-hinting their Django projects.

Peripherals

I use a mechanical keyboard - the “Glorious GMMK Pro” - with lubed panda switches. For audio, I have a pair of ATH-M50xs, with an Antlion ModMic Uni stuck on for calls.