Beth Anderson

Managing Multiple Git(Hub) Profiles

Mon, 09 Jun 2025 00:00:00 +0000

A little Git(Hub) hack that I use to manage config for a couple of GitHub accounts (work plus personal if I’m doing a 10% or hackday thing)

Instead of changing one ~/.gitconfig file, I have a simple one (plus also my generic aliases in there so they’re available for all profiles), with:

[includeIf "gitdir:~/home/"]
path = ~/home/.gitconfig

[includeIf "gitdir:~/work/"]
path = ~/work/.gitconfig

Then in the ~/work/.gitconfig file I have:

	email = work.email@example.com
	name = Beth Anderson

[github]
	user = "githubusername"

…and a different one in ~/home/.gitconfig.

Then if I’m in the ~/work/... directory it automatically uses my work user, and home for home…

Z Shell Performance

Fri, 06 Jan 2023 00:00:00 +0000

My zsh is taking so long to launch now, like up to 40 seconds, and I wanted to find out why.

So I did some Googling and found out that adding zmodload zsh/zprof to the top of my .zshrc file meant that I could run zprof (after reloading the shell). This yielded:

num	calls	time					self	name
1)	1	3067.88	3067.88	74.90%	1344.09	1344.09	32.82%	nvm_auto
2)	2	909.13	454.57	22.20%	909.13	454.57	22.20%	jenv
3)	2	1723.79	861.90	42.09%	852.65	426.32	20.82%	nvm
4)	1	779.97	779.97	19.04%	710.71	710.71	17.35%	nvm_ensure_version_installed
5)	1	90.30	90.30	2.20%	72.26	72.26	1.76%	nvm_die_on_prefix
6)	1	69.26	69.26	1.69%	69.26	69.26	1.69%	nvm_is_version_installed
7)	1	47.94	47.94	1.17%	47.94	47.94	1.17%	handle_completion_insecurities
8)	1	18.56	18.56	0.45%	18.56	18.56	0.45%	compinit
9)	2	17.09	8.54	0.42%	17.09	8.54	0.42%	nvm_grep
10)	2	12.73	6.36	0.31%	12.73	6.36	0.31%	grep-flag-available
11)	2	11.29	5.65	0.28%	11.29	5.65	0.28%	down-line-or-beginning-search
12)	1	4.79	4.79	0.12%	4.77	4.77	0.12%	powerlevel9k_vcs_init
13)	1	4.67	4.67	0.11%	4.67	4.67	0.11%	termColors
14)	2	2.91	1.45	0.07%	2.91	1.45	0.07%	env_default
15)	15	2.05	0.14	0.05%	2.05	0.14	0.05%	up-line-or-beginning-search

…which showed that nvm is taking a long time to load. I don’t use it a lot but I do want it sometimes so instead I’m using https://github.com/lukechilds/zsh-nvm#lazy-loading which lazy-loads nvm with some additions in the .zshrc file:

export NVM_LAZY_LOAD=true
export NVM_COMPLETION=true
plugins=(zsh-nvm [plus your others])

It’s gone from around 40 seconds to around 4 seconds!

However rbenv and jenv are also slow! I couldn’t find a reasonably easy way to improve that but I am using evalcache which will cache the init commands, saving some time. Installed with:

git clone https://github.com/mroth/evalcache ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/evalcache

…and in .zshrc:

plugins=(evalcache [plus your others])

Then instead of eval "$(jenv init -)" in your .zshrc use _evalcache jenv init -

Sadly that’s one less excuse though…

What We Learned

Mon, 15 Aug 2022 00:00:00 +0000

What We Learned

Be a facilitator and accelerator not a gatekeeper. Find ways to include people.
Companies are often hard to communicate across so things get reinvented.
Introducing a new system adds complexity. Only removing an old system reduces complexity.
Don’t try to solve every problem. Do one thing well. (Don’t ignore the big problems though).
Don’t just build tooling; model processes.
Tools are temporary, ideas are forever…(well a bit longer than tools anyway).
Add value without adding complexity.
Teams depend on your stuff. You should too!
Don’t build everything straight away. Get something out there and embrace feedback.
Work with people. Go and speak to them and include their ideas.
The real problem you’re solving is not technical, it’s people.
Ensure senior management understand the problem and support you.

Maze Generator in Go

Tue, 14 Jun 2022 00:00:00 +0000

Named after the Overlook hotel in The Shining, Goverlook generates a maze

Implemented with a stack, this approach is one of the simplest ways to generate a maze using a computer. Consider the space for a maze being a large grid of cells (like a large chess board), each cell starting with four walls. Starting from a random cell, the computer then selects a random neighbouring cell that has not yet been visited. The computer removes the wall between the two cells and marks the new cell as visited, and adds it to the stack to facilitate backtracking. The computer continues this process, with a cell that has no unvisited neighbours being considered a dead-end. When at a dead-end it backtracks through the path until it reaches a cell with an unvisited neighbour, continuing the path generation by visiting this new, unvisited cell (creating a new junction). This process continues until every cell has been visited, causing the computer to backtrack all the way back to the beginning cell. We can be sure every cell is visited.

As given above this algorithm involves deep recursion which may cause stack overflow issues on some computer architectures. The algorithm can be rearranged into a loop by storing backtracking information in the maze itself. This also provides a quick way to display a solution, by starting at any given point and backtracking to the beginning.

Horizontal Passage Bias Mazes generated with a depth-first search have a low branching factor and contain many long corridors, because the algorithm explores as far as possible along each branch before backtracking.

Using a depth-first maze-generation algorithm with a following recursive routine:

Given a current cell as a parameter
Mark the current cell as visited
While the current cell has any unvisited neighbour cells
- Choose one of the unvisited neighbours
- Remove the wall between the current cell and the chosen cell
- Invoke the routine recursively for a chosen cell

JSON Format

Mazes are represented as a collection of collections of “cells”. Each cell has a northRoute and westRoute boolean.

{
	"cells": [
		[{
			"northRoute": false,
			"westRoute": false
		}, {
			"northRoute": false,
			"westRoute": false
		}],
		[{
			"northRoute": false,
			"westRoute": true
		}, {
			"northRoute": true,
			"westRoute": true
		}]
	]
}

Code

Maze Generation: github.com/betandr/goverlook Boilerplate (you complete the algorithm): github.com/betandr/gomazegen-boilerplate

Mille French Vocabulary - Privacy Policy

Mon, 31 Jan 2022 00:00:00 +0000

Hi, this is to confirm that none of your private data is stored or captured at all. Mille runs on your device and stores your progress locally.

No data is transmitted or captured from your device. It’s just an app that helps you learn French. I wrote it for myself and thought it might be useful.

Beth

Building Go with Containers

Thu, 03 Jun 2021 00:00:00 +0000

Building Go with containers.

Build for a particular target

Such as make PLATFORM=darwin/amd64:

FROM --platform=${BUILDPLATFORM} golang:1.16.4-alpine3.13 AS base
WORKDIR /src
ENV CGO_ENABLED=0
COPY go.* .
RUN go mod download
COPY . .

FROM base as build
ARG TARGETOS
ARG TARGETARCH
RUN GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -o /out/server .

FROM base as unit-test
RUN go test -v .

FROM scratch AS bin-unix
COPY --from=build /out/server /

FROM bin-unix AS bin-linux
FROM bin-unix AS bin-darwin

FROM scratch AS bin-windows
COPY --from=build /out/server /server.exe

FROM bin-${TARGETOS} AS bin

Repo: github.com/betandr/congo

Using Multi-stage builds to build a container without Go development dependencies

# syntax=docker/dockerfile:1
FROM golang:1.16 AS builder
WORKDIR /server
COPY . .
COPY ./server .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app ./server

FROM alpine:latest
WORKDIR /root/
COPY --from=builder /server/app .
CMD ["./app"]  

Repo: github.com/betandr/congo

Managing OS X Credentials for Multiple GitHub Repos

Thu, 11 Jul 2019 00:00:00 +0000

By default git stores credentials globally but it can be configured to cache credentials on a per-repo basis.

On OS X a tool called osxkeychain is used to store your GitHub credentials. This tool stores your credentials in the OS X Keychain which you can see by running the Keychain Access app in your OS X Applications. To ensure this helper is working, run:

git config --global credential.helper osxkeychain

To authenticate myself with GitHub I prefer to use Personal Access Tokens as they’re a more granular and easily-repudiated credential, so you can visit github.com/settings/tokens and generate a token. Usually I have repo scope only as I just need to manage repos.

Keep a note of this token now as you won’t be able to see it again once you leave the GitHub page. It should be kept somewhere safe, such as a password manager like gopass. I have a little script that gets my access token and copies it into the clipboard buffer (although that might not be the most secure approach) ready for me to paste when my credentials are requested.

At this stage you can use your username and this access token when git asks for your credentials, however these credentials will be stored for all repos from GitHub. To use these credentials on a per-repo basis, run:

git config --global credential.usehttppath true

This will mean that git can cache different credentials for github.com/person/example and github.com/person/another rather than just one set of credentials for github.com. Note that you should use Clone with HTTPS, such as git clone https://github.com/person/example.git not Clone with SSH such as git clone git@github.com:person/example.git because using SSH just use an SSH key (if you have one set up) rather than the access token.

When git needs credentials, it will prompt you as:

Cloning into 'example'...
Username for 'https://github.com/person/example.git': YOUR_USERNAME
Password for 'https://betandr@github.com/person/example.git': YOUR_ACCESS_TOKEN

At this stage you can create another access token in a different GitHub account and then use this different when working with another repo on the same system.

You will be able to see these different credentials in your OS X Keychain by running Keychain Access and searching for github. You should see a list of credentials for GitHub which, when clicked on, look like:

These credentials can then be deleted from your Keychain if you no longer require them.

Kubernetes Ingress with TLS on GKE

Wed, 26 Jun 2019 00:00:00 +0000

Using an Ingress Controller we’re going to secure traffic to our Kubernetes cluster with TLS and a Let’s Encrypt certificate.

To follow these instructions you will need:

A Kubernetes cluster you can manage; recommended 3 nodes.
A domain name that you can create DNS A records for
Although this tutorial was tested with GKE 1.13.6-gke.13 the same approach can be taken on other types of Kubernetes clusters.

Most Kubernetes deployment examples show you how to deploy a container and then expose the container to public traffic via the service, although sometimes you might like to have more control over ingress to your cluster services.

Kubernetes Ingress was added in Kubernetes v1.1 and exposes HTTP and HTTPS routes from outside the cluster to services running within the cluster. Traffic routing is controlled by rules defined on the ingress resource which we’ll see later.

To run kubectl you’ll need to have access to your cluster. For GKE the command gcloud container clusters get-credentials will do this. To ensure you’re actually using the correct cluster, you can run kubectl config current-context

If you have a few clusters you can list all of the contexts using kubectl config get-contexts and then set a different context using kubectl config set-context $NAME where $NAME is one of the entries in the NAME column when running ...get-contexts. It’s always good to double check which cluster you’re going to make changes to.

Setup

We’re going to use Helm to install tooling such as Cert Manager. The Helm documentation covers installation. We need to set up a Service Account and Role Bindings for Kubernetes’ Role-Based Access Control.

Create a file called tiller-rbac-config.yaml containing:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
--- 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata: 
  name: tiller
roleRef: 
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects: 
  - 
    kind: ServiceAccount
    name: tiller
    namespace: kube-system

Create these resources by running:

kubectl create -f tiller-rbac-config.yaml

Then initialise the tiller service account by running:

helm init --service-account tiller

Service

Next up we need to schedule some work to the cluster so for this we’re going to use a Hello, World app.

Create a file called deployment.yaml containing:

--- 
apiVersion: apps/v1
kind: Deployment
metadata: 
  labels: 
    app: hello
  name: hello
  namespace: default
spec: 
  replicas: 1
  selector: 
    matchLabels: 
      app: hello
  template: 
    metadata: 
      labels: 
        app: hello
    spec: 
      containers: 
        - 
          image: "betandr/gollo-world:latest"
          livenessProbe: 
            httpGet: 
              path: /
              port: 8888
          name: hello
          ports: 
            - 
              containerPort: 8888
          readinessProbe: 
            httpGet: 
              path: /
              port: 8888

Then schedule this by running:

kubectl create -f deployment.yaml

You can check on the deployment’s progress using:

kubectl get deployment hello

This should show your deployment is available, such as:

NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
hello     1         1         1            1           37s

Create a service for this workload by creating a file called service.yaml containing:

--- 
apiVersion: v1
kind: Service
metadata: 
  name: hello
  namespace: default
spec: 
  ports: 
    - 
      name: http
      port: 80
      protocol: TCP
      targetPort: 8888
  selector: 
    app: hello
  type: NodePort

Note that while the service does run on port 80, this service does not have public ingress into the cluster. The hello service is only accessible from within the server. Create this service using:

kubectl create -f service.yaml

You can check on the service’s progress using:

kubectl get service hello

Your service should be assigned an internal IP address, but not a public IP address as we don’t wish this service to have public ingress. Instead we’re going to route all of our traffic to this service via an ingress controller.

Ingress

We should now have a service running on port 80 serving traffic within the cluster to our container running in a pod on port 8888. Now we can create our Ingress Controller.

Because we wish to use HTTPS for our service we need to use a certificate from Let’s Encrypt. Cert Manager will obtain this certificate for us but initially we will need to create the Ingress Controller without the references to the certificates. This is because you need to create the ingress first to obtain the IP address which is used when creating the DNS A-record (we will see this later); all before we can get the certificate.

Create a file called ingress.yaml containing:

--- 
apiVersion: extensions/v1beta1
kind: Ingress
metadata: 
  name: demo-ingress
spec: 
  rules: 
    - 
      host: YOUR_DOMAIN_NAME_HERE
      http: 
        paths: 
          - 
            backend: 
              serviceName: hello
              servicePort: 80

Replace YOUR_DOMAIN_NAME_HERE with your own domain name, in a format such as example.com This manifest sets up a rule which matches requests from your host (as an HTTP host header field) to a particular service—in this instance the hello service running on port 80.

At this stage we are currently not using TLS. We want to initially create this ingress without TLS until we have cert-manager running and a demo-tls secret available. Otherwise we’ll have no ingress at this stage. It’s a little bit horse and cart but we’ll add TLS later.

To create the ingress controller run:

kubectl create -f ingress.yaml

Obtain the external IP address of the ingress:

kubectl get ing demo-ingress

It may take some time to assign an external IP—as GCP is provisioning you a load-balancer—but when one is assigned, visit your domain management console and create an A record with the new IP. This will mean that a DNS lookup of your domain name will yield the IP address of your ingress controller so requests to your domain name will be forwarded to your cluster’s Ingress Controller.

This all takes some time; to get an external IP, to propagate the DNS A record, and then for everything to start responding. Visiting your domain will likely return a Google 404 for a while as it takes some time to provision the load-balancers and is normal.

While this is happening you can watch cluster events using:

kubectl get events --sort-by=.metadata.creationTimestamp

…and to check the progress of your DNS record you can use dig.

When, eventually, everything is working, you should see a response of hello, world! when requesting http://YOUR_DOMAIN_NAME/. Note we use http at the moment. Next we’ll set up https

Install and Configure Certificate Manager

We will use Jetstack’s cert-manager to manage certificates, providing us with TLS.

Install cert-manager using the command:

helm install stable/cert-manager \
  --name cert-manager \
  --namespace kube-system \
  --version v0.5.2

Next we’re going to create a ClusterIssuer which will provide the ability to obtain certificates from a central authority; Letsencrypt in our case. A cluster-wide issuer like this may not be the correct option for a multi-tenanted cluster but for our use it’s fine. Something to keep in mind at least. YMMV.

Create a file called issuer.yaml with the contents:

--- 
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata: 
  name: letsencrypt-issuer
spec: 
  acme: 
    email: ADD_YOUR_EMAIL@ADDRESS.com
    http01: {}
    privateKeySecretRef: 
      name: letsencrypt
    server: "https://acme-v02.api.letsencrypt.org/directory"

Create the issuer by running:

kubectl apply -f issuer.yaml

To watch the status of the issuer, run:

kubectl describe ClusterIssuer letsencrypt-issuer

…which will show you the issuer’s progress in obtaining a certificate.

Next create the cluster’s Certificate which will hold the certificate that Let’s Encrypt has issued us.

Create a file called certificate.yaml containing:

--- 
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata: 
  name: demo-certificate
  namespace: default
spec: 
  acme: 
    config: 
      - 
        domains: 
          - YOUR_DOMAIN_NAME_HERE
        http01: 
          ingress: demo-ingress
  commonName: YOUR_DOMAIN_NAME_HERE
  dnsNames: 
    - YOUR_DOMAIN_NAME_HERE
  issuerRef: 
    kind: ClusterIssuer
    name: letsencrypt-issuer
  secretName: demo-tls

Create the certificate resource with:

kubectl create -f certificate.yaml

To check the progress of your certificate, run:

kubectl describe certificate demo-certificate

You may see some messages such as http-01 self check failed for domain or ValidateError but these are likely to be transitory and things should begin to settle down when you enable TLS through the Ingress Controller.

Previously we created the Ingress resource with some lines commented out. Now we should have our certificate issued so can enable TLS via the ingress controller.

Edit ingress.yaml to contain the following:

--- 
apiVersion: extensions/v1beta1
kind: Ingress
metadata: 
  annotations: 
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
    certmanager.k8s.io/cluster-issuer: letsencrypt-issuer
  name: demo-ingress
spec: 
  rules: 
    - 
      host: faworth.co.uk
      http: 
        paths: 
          - 
            backend: 
              serviceName: hello
              servicePort: 80
  tls: 
    - 
      secretName: demo-tls

Then update the ingress controller using (note apply rather than create ):

kubectl apply -f ingress.yaml

Now running:

kubectl get ing demo-ingress

…you should see 80, 443 in the PORTS column as we now have TLS over port 433 enabled.

After some time your domain should start responding using TLS: https://YOUR_DOMAIN_NAME_HERE/ (note https rather than http now). However this could take quite a while. At first you may see nothing, then 404s from Google, then eventually your site should be responding.

Now we have our service (hello, world) accessible via TLS with a Let’s Encrypt certificate!

Metal Band Name AI Twitter Bot

Wed, 08 Aug 2018 00:00:00 +0000

Metal Band Bot is a Twitter bot which uses a Recurrent Neural Network to think up a brand new metal band name at two minutes to midnight every night.

Metal Band Bot (twitter.com/MetalBandBot) uses the textgenrnn library to train an RNN model from a data file of band names.

textgen.train_from_file('data/bands.txt', num_epochs=1)
textgen.save('metal_bands.hdf5')

This model is then used to generate a new, unique name.

textgen = textgenrnn('weights/bands.hdf5')
band_name = textgen.generate(1, return_as_list=True, temperature=0.8)[0]

Code

Metal Band Bot: github.com/betandr/mbrnn

Delivering with GCP at the BBC

Mon, 23 Jul 2018 00:00:00 +0000

There probably is a software engineer’s equivalent of the Rifleman’s Creed and it’s probably something like…

This is my infrastructure stack.

There are many like it, but this is mine.

So, this is ours… but first, some context.

Having been at the BBC for 9 years I’ve seen a few major changes in infrastructure. When I started, our process was largely to edit HTML and Javascript locally and FTP the changes up to the live server. The live servers were the source control and to make a change a Web Developer would copy the live site and update it, then send it back again. I’m not sure if I’m actually allowed to fully recount one of the stories of that time but suffice to say the Wayback Machine is a good method to recover lost HTML…

At this time we also built Rails apps and hosted these on our own on-prem infrastructure and had a great Rails team. Incidentally, it was a Rails App that powered my first BBC project; Visual Radio.

The next big shift-change was to a platform called Forge. This supported Java services, implemented usually in Spring, and PHP web applications, implemented with frameworks such as Zend. On the journey to a mature software organisation this was a time that many found restrictive but it supported the organisation in defining process and control over how we deliver features to our audience.

Shortly after we had finished migrating all of our apps to Forge, we started building new apps in the cloud with Amazon Web Services. To support this new platform we developed a system called Cosmos which acts as a delivery shim between the engineering and delivery teams at the BBC and the cloud provider. This provides a mechanism by which we can safely and, with relatively low friction, release to our cloud services provider and the audience. Cosmos also provides secure VMs, rollbacks, and a host of other benefits.

Let’s skip to the present and, 10months ago, I joined a fledgling team at the BBC called Datalab, a team tasked with bringing together what we know about our content into one place and using machine learning to enrich it. Not only did we have our own roadmap of products to deliver but—as we were a greenfield project without any legacy—we would also move into Google Cloud Platform (GCP). We arrived at GCP with no infrastructure and no legacy so set about defining how we’d use this new platform.

‘Data visualisation’ during the 1951 general election.

Previously, to build an API, we’d typically employ an architectural pattern of some application code (such as Scala), implemented with a framework (such as Akka), running in an application server (such as Spray or Akka HTTP), packaged as a binary (such as an RPM package), deployed onto a snapshot machine image (such as CentOS 7), deployed as a VM (such as an EC2 instance), running behind a load-balancer (such as an Elastic Load Balancer) [see fig 1.]. Usually we’d have a set number of these instances running and scale out under increased load. We’d follow the maxim of scale out fast, scale in slowly to benefit from the overhead and cost of spinning up new instances. It takes in the order of large values of seconds to small values of minutes to scale out.

Fig. 1 — API stack

To deploy a new feature we’d check the feature into our Git repository, then the build pipeline would be triggered. This would build a new VM for us, with our application binary baked in. We could then deploy to our pre-production environments using a rolling deployment which replaced the VMs with a new instance one by one. After running our smoke tests we’d promote this to production, again using a rolling deployment. If the smoke tests failed we’d rollback by pushing an older, known to be working, version of our application.

Moving into GCP allowed us the opportunity to re-think our stack and the first thing on our list was to benefit from containers. Let’s deftly skip over why containers are a good idea but they were a great fit for our project; a system built using microservices.

We also made the decision to move to Google Kubernetes Engine (GKE) as our container orchestrator. This means our containers need only be declared as workloads and GKE will make this request a reality. We can benefit from autoscaling in our Kubernetes (K8s) cluster and benefit from another optimisation; swapping HTTP internal traffic to RPC. Rather than our components being deployed in a VM and communicating via HTTP our pattern now is to have external HTTP(S) ingress and internal RPC communications. We implemented this with gRPC.

The choice of language in the implementation of any component is important for an organisation and we decided to use Python as our house language. This is mostly because our team comprises of a number of different roles such as data scientist, data engineer, and software engineer but we all have Python in common—and we think it’s important that everybody can work on all parts of the system. Our containers run Flask/Green Unicorn for HTTP and gRPC Protocol Buffers for RPC. Compared to a full VM, we can scale out in seconds—especially due to each container having a lightweight parent based on Alpine Linux and the actual container layer is very small.

We now have containers and a cluster to put them on [see Fig. 2], but we needed to think about how we delivered features to production. One of our goals is the potential to release a feature on the first morning you join the team. This means there should be no complex processes, no restricted access, no arcane knowledge, and no hierarchy.

Fig. 2 — Container Application Structure

Firstly we merge features we wish to release into our Git repository. This triggers our Continuous Integration (CI) pipeline (in fact any commit to any branch does). To run our CI we use Drone and define a pipeline for each component. With Drone we can run our automated testing and validate the containers we’re building. Our repos are then in a constant state of healthy or broken (which inhibits a release).

To release that feature we tag it, which triggers Google Container Builder to build our image and push it to our Container Registry.

At this point our Continuous Delivery (CD) tooling takes over, triggered by the presence of a new version of a container. To run our CD we use Spinnaker which runs our delivery pipelines. These pipelines define how a microservice should be deployed into the cluster. Typically we deploy automatically into the cluster to our stage environment (more on that later) and then send a message to our team’s Slack channel to let us know a deployment is ready to be promoted. Our smoke tests are then run and the feature promoted to production.

These deployments run as a green/blue deployment (or red/black in Spinnaker/Netflix parlance). So instead of performing a rolling update of replacing node by node, we spin up a mirror deployment on our cluster and switch the service from the old deployment to the new using traffic splitting. This means that the previous deployment is still running and we can slowly bring a new feature into service and run canary tests while this is happening. If we experience any issues we can simply send the traffic back to the old deployment without any issues. We leave the previous deployment running, but disabled, until we’re confident the new deployment is operating correctly. Google also recently announced the availability of Kayenta on the platform which provides automated canary analysis tooling.

We generally run two environments; stage and production. However, because we run everything on a K8s cluster these environments can be as similar to each other as we need them to be; we don’t have a pre-production environment which is markedly different to production. These environments are deployed as versions of a Kubernetes service. So each container, say foo-service, is deployed as a Kubernetes service foo-service-stage and foo-service-production. This approach allows us to deploy interim environments too, for experiments etc. So foo-service-experiment-bar environments are possible too.

Our microservices communicate via RPC but to provide HTTP ingress we use a Kubernetes Ingress Controller which accepts HTTP, terminates the TLS connection, and forwards to our service within the cluster. This is implemented with a Google Cloud Load Balancer and provides secured ingress to our cluster.

We’re not replacing our VM strategy but adding more deployment options to our enterprise. While this post started with a story of how far we’ve come in the last few years, I’m sure that if we check back in a few months things will have continued to change and move forward at an ever-increasing rate at a velocity that’s well supported by our infrastructure and tooling.

Originally published at How we deliver with GCP at the BBC on the BBC Design + Engineering Blog at Medium.