Anton Zalialdinov, expert software engineer blog

Configure Shadowsocks Client on Ubuntu

Wed, 04 Sep 2024 00:00:00 GMT

Configure Shadowsocks Client on Ubuntu

Shadowsocks is a popular option for building secure tunnels with masked traffic. It's simple and lightweight. Unlike other similar tools like WireGuard or OpenVPN, Shadowsocks lacks a well-organized development process. This means there is no centralized documentation, multiple versions exist, and client configuration can be tricky.

Server

Setting up the server is quite simple. Below is a sample Docker Compose file. You’ll need to install Docker to use it.

version: "2.1"
services:
  shadowsocks:
    image: shadowsocks/shadowsocks-libev:v3.3.5
    container_name: shadowsocks
    ports:
      - 8388:8388
      - 8388:8388/udp
    environment:
      - METHOD=chacha20-ietf-poly1305
      - PASSWORD=your_password
    restart: always

Client

The Shadowsocks client works as a local proxy, meaning you configure and connect it to your remote server, but locally it is exposed as an HTTP or SOCKS5 proxy.

Windows: There is a nice GUI client that allows you to set a global proxy with a single click. I found it to be almost as good as the WireGuard client. However, some applications ignore system proxy configurations, so their traffic may bypass the tunnel.
Android: The client experience is also smooth—one client, and all traffic is wrapped.
Ubuntu: While there are plenty of clients, I couldn't find one that works like WireGuard. You’ll need to configure each app individually, and in some cases, it may be impossible.
macOS/iOS: I haven’t checked yet =)

Configuring Shadowsocks to Wrap All Traffic on Ubuntu

A proper solution for Ubuntu is to use Shadowsocks in combination with a tool like tun2socks, which streams your traffic from a network adapter to the local Shadowsocks proxy. Let’s configure this combination.

First, install the following tools:

I downloaded the binaries from the release pages and moved them to /usr/bin.

Assuming you have a server running as configured above, we will use the sslocal binary. Your client command will look like this:

sslocal -s "shadowsocks_server:shadowsocks_port" -k "your_password" -m "chacha20-ietf-poly1305" --local-addr 127.0.0.1:1080 -U

The output should look something like this:

Now, you can test the client with curl:

curl --proxy "127.0.0.1:1080" "https://ifconfig.me"

It should return the IP address of your remote Shadowsocks server.

Next, you need to create a local network adapter and route all traffic through it. Then, use tun2socks to direct traffic from that adapter to the Shadowsocks tunnel.

Create the network adapter with the following commands:

ip tuntap add mode tun tun0
ip addr add 10.0.0.2/24 dev tun0
ip link set dev tun0 up

Run tun2socks with:

tun2socks -device tun0 -proxy socks5://127.0.0.1:1080

You can check if it works with:

curl --interface tun0 http://ifconfig.me

The output should be your Shadowsocks server's IP address.

Proper tun2socks output looks like this:

Now, let's route your traffic through the tun0 network interface and exclude traffic going to the Shadowsocks server. Otherwise, it will create a loop.

First, obtain your gateway address and outer interface name:

# gateway IP address
ip route | grep default | awk '{print $3}'
# outer interface
ip route | grep default | awk '{print $5}'

Next, configure the traffic routing:

ip route add "shadowsocks_server_ip" via "gateway_ip" dev "outer_interface"
ip rule add to "shadowsocks_server_ip" lookup main priority 1000

The last command might cause issues. If something goes wrong, you may end up without internet access. Here are commands to route traffic to the tunnel and to revert it if needed:

To route traffic to the tunnel:

ip route add default dev tun0 table 100

To revert the routing:

ip route delete default dev tun0 table 100

If everything works fine, you can open a browser and visit iplocation.net. It should show the location of your Shadowsocks server. You can also test it with:

curl "https://ifconfig.me"

WireGuard to Shadowsocks: Ready Docker Setup

Debugging Keycloak Extension

Tue, 03 Sep 2024 00:00:00 GMT

Debugging Keycloak Extension

It’s hard to understand how your code will behave just by looking at it. To do it properly, you’d have to run it in your head. This becomes especially challenging with complex codebases like Keycloak. Instead of manually analyzing the source code or reading through the documentation, it makes sense to spend some time setting up a debugger and walking through the execution path.

Tools and Codebase

In this example, I’m going to use IntelliJ IDEA as my Java IDE and Docker Desktop for running Keycloak. Here are the tools you will need:

Download and install the tools, and clone the repositories to your local environment. Follow the official installation guides. You will also need to build Keycloak by following the official instructions here. In this guide, I will be using Keycloak 25.0.0, so before building, check out the correct version with the command:
git checkout tags/25.0.0.

Keycloak runs tests during the build process, but you can skip them with the following command:
./mvnw -pl quarkus/deployment,quarkus/dist -am -DskipTests clean install.

We need the local Keycloak build because it will allow us to navigate through not only the extension sources but also the Keycloak code itself.

The Keycloak build took 2:49 minutes in my case ;-)

Configuration and Running

Open IntelliJ and load the project folder. Since this project uses Maven as the build tool, IntelliJ will rely on .pom files. On the IntelliJ welcome screen, click "Open" and choose keycloak-external-claim-mapper.

You need to add a debug profile:

Click Edit Configuration as shown in the screenshot above.
Click "Add," and choose Remote JVM Debug.
Set Use module classpath to external-claim-mapper.

Now, let’s run Keycloak with the sample extension. Open the keycloak-external-claim-mapper folder and run the following terminal command:
docker compose up --build.
You should see logs from all three containers: Keycloak, Postgres, and MockServer. The following logs indicate that Keycloak is running in debug mode:

The following configuration manages debug mode in Keycloak, and you can find it in the Dockerfile:

ENV DEBUG='true'        # Debug mode on/off
ENV DEBUG_SUSPEND='n'   # Wait for debugger connection at startup; useful when stepping through startup code
ENV DEBUG_PORT='*:5005' # Debugger port; this should be mapped to localhost

Debugging with Breakpoints

To start debugging, you need to attach the debugger, set breakpoints, and trigger code execution. Let’s attach IntelliJ to the Keycloak instance running in Docker. To do that, hit the "Attach" button in the top-right corner or press the Shift+F9 shortcut.

To set a breakpoint, find and open ExternalClaimMapper.java, click on line 141, and press the Ctrl+F8 shortcut.

To trigger code execution, mappers are executed during token generation, so you need to request a user token from the endpoint. Below is the curl command:

curl 'http://localhost:8080/realms/dev/protocol/openid-connect/token'  \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'client_id=account' \
  --data-urlencode 'grant_type=password' \
  --data-urlencode 'username=test' \
  --data-urlencode 'password=test'

If everything is set up correctly, you should be able to step through your extension code with all variables available in the call stack. There’s also an evaluation tool, allowing you to test certain functions without needing to rebuild.

You can debug not only the extension but also Keycloak itself. It’s not always easy to find where the HTTP requests enter the codebase. To do this, open the Keycloak source code and search for @POST or @GET. Among the results, one relevant file is TokenEndpoint.java. If you set a breakpoint here, you can walk through all steps of the Keycloak token generation process, which is very useful when setting up features like token exchange.

How to Add External Data to Keycloak Token

Get Database Create Script from Prisma Schema

Mon, 02 Sep 2024 00:00:00 GMT

Get a Create Script from Prisma Schema

TL;DR: Use this package: prisma-sql-gen

Prisma provides a way to create a database from your schema using the following command:
npx prisma migrate dev.
However, this method has the following limitations:

It uses migrations and performs checks, which makes it slow when the application has a large migration history.
It requires a shell command to run, complicating app startup logic.

In my case, the migration folder contained approximately 100 migrations, with an additional 3-5 migrations being added monthly. With this amount, creating the database from scratch took nearly 10 minutes, which was a significant portion of the pipeline execution time. Prisma is the tool of choice for Node.js when dealing with databases, similar to Entity Framework for .NET or NHibernate for Java.

Now, imagine every team developing backend applications in Node.js having such a migration folder. What a waste of computational resources! Another concern is the use of npx. In modern Kubernetes environments, most developers equip Node.js apps with shell scripts, which will require npx and the Prisma CLI to be installed in the image. This increases the attack surface of the app and makes the image heavier.

Ideally, migration checks and database initialization should be handled in the app's entry point. Fortunately, Prisma exposes its internal package used by the npx CLI:
@prisma/migrate.

Internally, it uses JSON-RPC to communicate with the core Prisma tool, written in Rust:
prisma-engines. You can also extract the raw create script using the CLI. The command that prints a large SQL script to the console is:
npx prisma migrate diff --from-empty --to-schema-datasource prisma/schema.prisma --script

All we need to do is find the correct command and intercept the JSON response from the Rust core. After exploring the migrate package:
prisma/migrate, I created the following snippet:

import { SchemaEngine } from '@prisma/migrate'
import { resolve } from 'path'

// Inherit the internal engine class
class GeneratorEngine extends SchemaEngine {
  public createScript: string
  constructor() {
    super({ projectDir: process.cwd(), schemaPath: '' })
    this.createScript = ''
    // Assign a custom handler to the JSON RPC response handler. Prisma's original handler is more complex, but we only need it for a single command.
    this['handleResponse'] = (response: string) => {
      try {
        this.createScript = JSON.parse(response).params.content
      } catch (error) {
        throw `Invalid response: ${error}, response ${response}`
      } finally {
        this.stop()
      }
    }
  }
}

// Wrap with an empty try-catch block, because the engine works with dummy handlers
export const schemaToScript = async (schemaPath: string = 'prisma/schema.prisma') => {
  const engine = new GeneratorEngine()
  try {
    await engine.migrateDiff({
      script: true,
      from: { tag: 'empty' },
      to: { tag: 'schemaDatamodel', schema: resolve(schemaPath) },
    })
  } catch (error) {}

  return engine.createScript
}

How to Add External Data to Keycloak Token

Sun, 01 Sep 2024 00:00:00 GMT

How to Add External Data to User JWT in Keycloak

If you have Keycloak integrated into your application, you’ve probably faced a situation where you need some data from a remote endpoint in the user token, like custom roles. This guide will show you how to include external data in a user JWT using Keycloak.

Let’s assume you have a remote data source and you want this data included in the user token. In this guide, I will mock an external API using Mockserver. My example will contain two services: Keycloak and Mockserver (external API).

Keycloak has a set of embedded tools to connect external user accounts and data providers. However, these tools don’t always fit external APIs and require a lot of OpenID protocol details to be implemented. The simplest way to get custom data into a user token is to use a protocol mapper. Keycloak has a set of built-in mappers, but there are no mappers that can call external endpoints.

In this guide, I’m going to use a custom protocol mapper with that functionality implemented.

Steps Overview

Set up the environment with Keycloak, Mockserver, and PostgreSQL.
Configure the external claim mapper in Keycloak.
Test the configuration and view the results in the token.

Test Environment Setup

Let’s prepare the environment for testing the mapper. To get Keycloak with the keycloak-external-claim-mapper added, I’m going to create a Dockerfile with the following content:

FROM alpine:3.20 as build

ARG VERSION=0.0.2

RUN \
  wget https://github.com/zloom/keycloak-external-claim-mapper/releases/download/${VERSION}/external.claim.mapper-${VERSION}.tar.gz;\
  mkdir -p /providers;\
  tar -C /providers -zxvf external.claim.mapper-${VERSION}.tar.gz;

FROM quay.io/keycloak/keycloak:25.0.0 as keycloak

COPY --from=build /build /opt/keycloak/providers

We need to spin up the external API mock, Keycloak, and a database, so let’s use Docker Compose. Below is the content for docker-compose.yaml:

version: "3"

services:
  postgres:
    container_name: postgres
    image: postgres:14.13
    environment:
      POSTGRES_DB: 'keycloak'
      POSTGRES_USER: 'keycloak'
      POSTGRES_PASSWORD: 'keycloak'
    ports:
      - 5432:5432
    volumes:
      - ./postgres/data:/var/lib/postgresql/data

  keycloak:
    container_name: keycloak
    build: 
      context: .
      dockerfile: Dockerfile
    environment:
      KEYCLOAK_ADMIN: 'admin'
      KEYCLOAK_ADMIN_PASSWORD: 'admin'
      KC_DB: 'postgres'
      KC_DB_URL: 'jdbc:postgresql://postgres:5432/keycloak'
      KC_DB_USERNAME: 'keycloak'
      KC_DB_PASSWORD: 'keycloak'
    command: start-dev
    ports:
      - 8080:8080
    depends_on: 
      - postgres   

  mockserver:
    container_name: mockserver
    image: mockserver/mockserver:5.15.0
    environment:
      MOCKSERVER_INITIALIZATION_JSON_PATH: /mockserver/initializer.json
      SERVER_PORT: 8081
    ports:
      - 8081:8081
    volumes:
      - ./initializer.json:/mockserver/initializer.json

Mockserver requires configuration, so I will create initializer.json with the following mock configuration:

[
  {
    "httpRequest": {
      "path": "/userprofile"
    },
    "httpResponse": {
      "statusCode": 200,
      "headers": {
        "content-type": [
          "application/json"
        ]
      },
      "body": {
        "roles": {
          "values": [
            "role1",
            "role2",
            "role3"
          ]
        }
      }
    }
  }
]

After creating all these files, you should have a ready folder with three files: Dockerfile, docker-compose.yaml, and initializer.json. Switch to this folder and run docker compose up -d wait when all containers started and open http://http://localhost:8080/ in browser you should see keycloak login window.

Keycloak configuration

Protocol mappers are grouped with client scopes. In fact, a Keycloak client scope is a set of JWT mappers. A single scope can be assigned to multiple clients. Also, each client has its own scope called ${client-name}-dedicated.

Go to http://localhost:8080, and log in with the username and password admin.
Open Clients from the left menu.
Open the account client, and in the Capability config section, check the Direct access grants box. Click the save button.
Switch to the Clients Scopes tab, open account-dedicated, and in the Mappers tab, click the Configure a new mapper button.
Select External claim mapper and set the following values:
- Name: test
- Remote url: http://mockserver:8081/userprofile
- Request headers: Key Content-Type, Value: application/json
- Token Claim Name: test
- Claim JSON Type: JSON
Click the save button

Testing the Mapper

You can test that the mapper sends a request to the remote endpoint using the following curl statement:

curl 'http://localhost:8080/realms/master/protocol/openid-connect/token'  \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'client_id=account' \
  --data-urlencode 'grant_type=password' \
  --data-urlencode 'username=admin' \
  --data-urlencode 'password=admin'

This request should give you a valid token with the data from the remote endpoint:

{
   "access_token" : "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJUZGMxRW1mb2c2cm4yYlZ5RGw0al81VjFKMUMtUHBKWWNKVEJISWF2WndNIn0.eyJleHAiOjE3MjkyMDI4NDIsImlhdCI6MTcyOTIwMjc4MiwianRpIjoiZTQ5OTRkNjctMzU0Yi00MzY1LThlYzEtMTQxYzgzZDgzZDBlIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwL3JlYWxtcy9tYXN0ZXIiLCJzdWIiOiI3OWRhZjUxZi0yNjI0LTQxNGEtOWYxMC1hZDA0NDI2ZTk5NWUiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJhY2NvdW50Iiwic2lkIjoiNjdjYzg4MjQtNmI1NC00ZTkwLTliNzctOTFlYWQ5YzVlYmNiIiwiYWNyIjoiMSIsInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6ImVtYWlsIHByb2ZpbGUiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsInRlc3QiOnsicm9sZXMiOnsidmFsdWVzIjpbInJvbGUxIiwicm9sZTIiLCJyb2xlMyJdfX0sInByZWZlcnJlZF91c2VybmFtZSI6ImFkbWluIn0.Jd-ThcB_QTuXoSaqsLg-Km0wA4XeLOIpni0cCmqJitV43L6hVs2RCDtwucWpXrWf4SeY9P3_YbCzx3vhHNL904cUQDXnhg0lZufhdzUUSnF5vSsjVuOLW9odyAi44dmRhBKI3FQFUGxv2JAoB1iISGCzmpdhFgzOHmmjbgrOuL5pcvQHG6cLNCfkappounDwccPtyTaJmOrEb3yDJzof0gjr7TbfXBcpQUCZn6OifXkzWwbpyKEDjL5eEWQd-0sZFexuFhOQX5d5uOJL0Wx5G8GCBhPsbQipyk5GCf_YaEo353pZ_nAYyvN75mmUMOJILpHUK2Ex55JgCBmi1N7SFg",
   "expires_in" : 60,
   "not-before-policy" : 0,
   "refresh_expires_in" : 1800,
   "refresh_token" : "eyJhbGciOiJIUzUxMiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJmZmFkYjdiMy00Nzg3LTRhYjAtYWI2ZS0yMmExZDM0YjBmNTIifQ.eyJleHAiOjE3MjkyMDQ1ODIsImlhdCI6MTcyOTIwMjc4MiwianRpIjoiOTZkOWRiNGMtOGZkOC00MDI1LTgyNjItZDNmYmIzODg0YTQxIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwL3JlYWxtcy9tYXN0ZXIiLCJhdWQiOiJodHRwOi8vbG9jYWxob3N0OjgwODAvcmVhbG1zL21hc3RlciIsInN1YiI6Ijc5ZGFmNTFmLTI2MjQtNDE0YS05ZjEwLWFkMDQ0MjZlOTk1ZSIsInR5cCI6IlJlZnJlc2giLCJhenAiOiJhY2NvdW50Iiwic2lkIjoiNjdjYzg4MjQtNmI1NC00ZTkwLTliNzctOTFlYWQ5YzVlYmNiIiwic2NvcGUiOiJiYXNpYyB3ZWItb3JpZ2lucyBlbWFpbCBwcm9maWxlIHJvbGVzIGFjciJ9.BQJPApuQ_-krRMBPP8vuwDyD1L9BJCxrhP4-BujYGbT4bWu8NRLAEJssYdg1e7yk6ajJxXeP6-pbk2tpYvgOCQ",
   "scope" : "email profile",
   "session_state" : "67cc8824-6b54-4e90-9b77-91ead9c5ebcb",
   "token_type" : "Bearer"
}

Decoded token holds additinal claims:

You can see keycloak request logs in mockserver container:

Debugging Keycloak Extension

Jaeger Docker Setup: Production-Ready Tracing Server

Mon, 23 Jun 2025 00:00:00 GMT

Jaeger Docker Setup: Production-Ready Tracing Server

TL;DR https://github.com/zloom/telemetry

On one of my recents projects I needed a simple and reliable way to handle telemetry in a legacy .NET project.
No Kubernetes. No big team. Just me — and a minimal server.

Most mature teams run observability stacks in Kubernetes or pay for SaaS services like New Relic or Datadog.
But if your project is small, Kubernetes might be overkill, and SaaS might not fit your budget or privacy requirements.

I had worked with Jaeger before — it’s a solid tracing solution, especially when paired with the OpenTelemetry standard, which is now supported almost everywhere.

Jaeger does provide a sample Docker setup — but in newer versions it's an all-in-one container with in-memory storage, which isn’t production-ready.
Older versions (like 1.63) include SPM (Service Performance Monitoring), but they still use all-in-one mode.

To make it production-worthy, you need to combine Jaeger with the OpenTelemetry Collector, which takes some work.
And on top of that, you have to solve CORS, SSL, and proxy issues when collecting telemetry from web apps — and don’t forget about Cassandra metrics.

This setup solves all of that.
It’s designed to be robust, minimal, and easy to run — even on a modest VPS.

Server overview

Each service individual files located in corresponing subfolders. Once you run it you will have components state for example cassandra data files wich may be huge this will appear in data subfolder with service name (data/cassandra for exampe). The only config you need to update to raise server is is caddy DOMAIN

...
caddy:
    <<: *logging_default
    container_name: telemetry_caddy
    build:
      context: ./caddy
      dockerfile: Dockerfile
    environment:
      DOMAIN: localhost
...

I dont reccomend to remove x-logging_default mixin, i reduced verbocity in cassandra but components may still generate huge amount of logs so your server may be oveflowed with docker logs. This have to match with your server public endpoint something like my_telemetry.io' then https://my_telemetry.io/v1/traces would be your OTEL_EXPORTER_OTLP_ENDPOINT for exporting telemetry from apps. Once you assigned dns properly you can start server with docker compose up -d caddy will obtain letsencrypt sertificate automatically, if you run it with localhost DOMAIN it will generate selfsigned sertificate. There is also compose.dev.yam you can run this setup with docker compose -f compose.yaml -f compose.dev.yaml up -d then trace generator will be added to your setup, this usefull if you want to test setup. Server paths are configured as following domain/jaeger-query is jaeger builtin trace explorer and SPM. domain/prometheus is prometheus. I not reccomend to change service version as services are tested againts each other, something may stop working 💥

Server components overview

Let me explain the role of each component.

Caddy

A lightweight web server — handles SSL, CORS, and exposes both public and internal endpoints. Configuration is in caddy/Caddyfile.

Originally I tried using Nginx, but it's not a great fit for Docker setups. Certbot integration for SSL is too complex, and Nginx doesn’t resolve service names properly inside Docker. You can still use it, but I recommend keeping it outside the container network.

Jaeger Collector

Jaeger's trace collector. It accepts only filtered traces from otel-collector and writes them to Cassandra.

Ideally, the otel-collector should write traces directly, but the public image otel/opentelemetry-collector-contrib doesn’t support Cassandra. To eliminate the Jaeger collector, you’d need to build a custom Otel Collector image — so there’s room for improvement here.

Jaeger Query

The main Jaeger UI — includes both the trace viewer and SPM (Service Performance Monitoring) panel. It depends on Cassandra and Prometheus.

If your project already uses Grafana, it might make sense to replace this with Grafana dashboards.

Otel Collector

The telemetry collector. It receives distributed traces, handles tail-based sampling, and generates trace metrics. It forwards traces via gRPC to the Jaeger Collector and exposes metrics for Prometheus.

In this setup, I use tail sampling, meaning the decision to keep or drop a trace is made server-side. Exporters just send everything.

Tail sampling is heavily promoted by the OpenTelemetry team. Jaeger developers still support head and remote sampling, but these require a two-way connection and real-time configuration sync, which makes the whole system more fragile. Also, many exporters don’t support this because it’s not part of the official spec.

That’s why I prefer sticking to the standard. Simple and reliable beats complex and fragile — unless you have very specific requirements. Public standards also help avoid vendor lock-in.

Sampling is the main way to control server load, along with disk size and retention period. In this setup, the rules are:

Always keep ERROR traces
Keep 90% of all other traces
Wait 5 seconds (decision_wait) before deciding

If you're adding telemetry to your services for the first time — don’t enable sampling immediately. Record all traces first to check trace quality. For example, you might not record errors properly yet — and that would break your sampling logic. Improve trace data first, then add sampling.

Prometheus

A time-series database with a nice UI. In this setup, it’s used to store SPM data and monitor Cassandra disk usage — which is critical to keep the server running 24/7.

You need to understand how much telemetry your server can store based on two key metrics: disk size and retention window. This setup handling stably 10k rps with 500 GB of disk and 2-day retention.

Other important metrics:

Otel Collector sampling metrics (otelcol_processor_tail_sampling*)
These help you understand how many traces were dropped and how the samplers behave.

Cassandra metrics (cassandra_*)
Monitor disk usage and storage health.

Cassandra

Main storage for trace data. I included a metrics exporter and a custom logging config to reduce log volume — default Cassandra logging is very noisy.

Cassandra is often used in Jaeger setups because it supports TTL (time-to-live) natively, which makes trace expiration simple and reliable.

Cassandra Schema

A one-time init container. It runs only when the database is empty and initializes the schema for Jaeger.

The default TTL value is set via the docker-compose file to 172800 seconds = 2 days.

CI/CD and Development Recommendations

The deployment pipeline is very simple — I use GitHub Actions and deploy directly to a single VPS.

name: deploy

on:
  push:
    branches:
      - main
  workflow_dispatch:

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: your_env
    steps:
      - uses: actions/checkout@v4

      - name: Create .env file from secret
        run: echo "${{ secrets.ENV_FILE }}" > src/.env

      - name: Upload infra configs
        uses: appleboy/scp-action@v0.1.4
        with:
          host: ${{ vars.HOST }}
          username: ${{ secrets.USER }}
          password: ${{ secrets.PASSWORD }}
          port: 22
          source: "./*"
          target: /telemetry

      - name: Restart infra
        uses: appleboy/ssh-action@v1.0.3
        with:
          host: ${{ vars.HOST }}
          username: ${{ secrets.USER }}
          password: ${{ secrets.PASSWORD }}
          script: |
            docker compose -f /telemetry/compose.yaml up -d --build

Not exactly kubectl apply, but it’s fast and works 😎

⚠️ Service updates:
This pipeline might not fully update some services if they rely on config files mounted as volumes. To avoid this, I recommend creating custom Dockerfiles (e.g. for Caddy) and embedding configuration files > inside the image itself. That way, any config change will trigger a rebuild and full restart of the affected service.

Security note

This server does not include authentication by default. While telemetry collection endpoints (like /v1/traces) are often open, service dashboards like Jaeger and Prometheus should be protected.

The simplest way is to add Basic Auth via Caddy. I didn’t include it here to avoid storing password hashes in the public repo — but it’s literally a one-line change.

In production, you should integrate this setup with your centralized authentication system (e.g., SSO or LDAP).

Good luck with your first traces!

Mastering Telemetry in a Legacy .NET Project

Mastering Telemetry in a Legacy .NET Project

Sat, 14 Jun 2025 00:00:00 GMT

Mastering Telemetry in a Legacy .NET Project

TL;DR Jump to practical examples ➜

A few words about me: in my work I try not just to develop features, but to solve underlying problems that slow down development as a whole.

One day, a former colleague reached out and offered me a chance to join his company as a team lead. I had always wanted to test myself in this role, so I gladly accepted.

To get familiar with the real issues, I first joined the support team and was asked to find ways to reduce costs.

A few words about the project:

a large, old system
heavily based on Microsoft technologies
the core process: data aggregation and ETL

What I discovered

Every day started with incident analysis.
The team was very efficient at it: a few long-time specialists knew exactly which parts of the system caused trouble most often.

Developers mainly relied on logs and server metrics.
If servers stopped or used too many resources, they would catch it quickly and dig through logs.

For experienced members, this was usually enough — they knew the system well.
But new developers often got lost, and even experts still spent time investigating and writing up root causes.

In short: it worked, but it cost time and money.
As a fresh pair of eyes, my first thought was clear:
✅ We need proper telemetry.

Good telemetry would help new developers:

see how the system is structured,
understand what components exist and how they interact,
and quickly see where problems come from.

With that in mind, I started designing a practical solution.

My starting point and choice of tools

I had worked with telemetry before. On one large project, I used New Relic — it was powerful, but for this company it wasn’t a good fit: they prefer to rely on their own infrastructure and hardware.

They do have a Kubernetes cluster, but few specialists to maintain it, and it’s not yet fully used in production.

On another project, I had used Jaeger and liked it. It was easy enough to run and works well with the OpenTelemetry standard.
From my experience, sticking to open standards is always the safest choice. It’s better to use two components from different vendors that follow the same standard than to get locked into a single vendor’s ecosystem — I learned this the hard way with OpenID and Auth0.

So for this project, I decided:

Not to spend time setting up Kubernetes.
Instead, run Jaeger in Docker on a Linux server.
Use the OpenTelemetry SDK in the .NET applications for integration.

This way, the setup stays simple, works with existing infrastructure, and remains flexible for future changes.

Architecture and Iterative Improvements

When designing serious systems, it is always better to start on paper. Even if mistakes are made, it is much easier (and cheaper) to fix a sketch than to rework a live system. Another critical point is documentation. On one of my projects, we made it a standard practice to write a short RFC for every substantial change. For my initial proof of concept, I skipped the docs, but as soon as it became clear that the task was complex, I wrote documentation to formalize my thoughts.

In my experience, the best practice is to keep documentation as close to the code as possible. This works especially well in a monorepo. Documentation should be clear enough that any new developer encountering the system can understand its purpose and flow without long handover meetings — this directly reduces IT business costs.

My typical RFC skeleton looks like this:

Problem description
Proposed solution
Alternatives considered
Open questions

Depending on the task, I might add extra sections — for example, a snapshot of the current state if I’m planning to improve an existing component.

Equally important is not just to describe what was built, but to explain why it was built that way. This helps others trust the decisions and extend them safely in the future.

Real-world telemetry setup

Returning to the telemetry topic — the final version of the architecture looked quite simple but was built iteratively with these principles in mind. One significant change was made early on, but the end result was robust: it could handle up to 160 GB of telemetry data daily and store it for 2 days, running entirely on a single server with 4 GB RAM, 500 GB disk, and 4 CPU cores.

The main challenge turned out to be disk usage. As more services integrated with the collector, the volume grew fast — so keeping storage under control became the key constraint.

Example RFC

I won’t include the full real RFC here 😉 — you can find good examples in many open-source repositories:
Keycloak RFC example
Jaeger RFC issue
Vitess / Postgres-style design RFC

The goal is not just the format, but the discipline: sketch first, write an RFC second, then implement and refine. Of course, this documentation should be reviewed by other experts on your team — just like real implementations.

Development approaches

One of my core principles is developer convenience.
For me, this means everything in the repository must run with minimal effort. Docker helps a lot with this: of course, a production server can never look exactly like a developer’s laptop, but you can get it close enough — and make deployment dead simple.

I followed this principle from day one. Even if the rest of the system needs manual tweaks or tribal knowledge, your own corner of the codebase should be green, clean, and predictable.

In practice, this means:

If your service is basically a docker-compose.yml, then docker compose up must Just Work™.
This saves time: instead of changing things directly on the production server, I could test everything locally first, then push updates with confidence.
Even without formal Infrastructure as Code, you can apply the same discipline manually — it pays off quickly.

Make it easy for others to help

Everything needed to build and test your component should be ready in the repository.
This way, you can easily ask teammates for help, and they won’t waste hours just trying to run your code.

Unfortunately, many legacy projects fail at this basic rule.

Problem investigation: real examples

On this project, I had help from one senior developer and a junior DevOps engineer. Together, we tackled a few interesting challenges — let me share how.

1) The dependency problem

In one of the past projects with telemetry, an architect had used the OpenTelemetry SDK — but for some reason, he just copied its source code directly into the project instead of installing it as a proper dependency.
At first, I couldn’t understand why, but once I started using this SDK myself, the problem became clear: the SDK depends on very recent .NET packages, which often conflict with older packages, like Entity Framework 6.

Luckily, in our case, only a mock library for EF was causing a conflict.
Once I reproduced the issue and showed it to the other developer, he quickly found a solution.
Resolving complex dependency conflicts is always tricky — sometimes the best approach is to build your dependencies manually, picking only the pieces that won’t break your project.

2) The storage problem

In this setup, we used Cassandra because it supports TTL out of the box — metrics older than two days should auto-delete. In theory, this should have solved the problem. But in practice, the server kept stopping.

Our DevOps tried to clean up the storage manually and tune Cassandra to handle it better, but after a while, he gave up — and we all drifted into guess-shooting again.
When you catch yourself doing that, it usually means you don’t have enough information yet and need better tools.

So I added proper metrics to Cassandra.
Once I had them, it became obvious that the server always stopped right before the TTL would clear the data.
I paused data collection, checked how much space was truly needed for stable operation, and once we added enough disk space, the issue disappeared.

3) Trace fragmentation

Another subtle but important challenge was trace fragmentation.
In theory, all services should pass along the same trace context so that you can see the full story of a request, from the API gateway down to the database.

In reality, this works well when services share common ASP.NET startup code — but in legacy projects, there are always exceptions and customizations.
In my case, I had a mix of Windows Services and a few old Web Services.

For Windows Services, I needed to implement proper span creation for scheduled jobs. I used the .NET Activity API — it works well and avoids spreading OpenTelemetry SDK dependencies all over the codebase.

For the legacy web service, it was enough to bring its startup flow closer to a typical ASP.NET configuration — once that was done, context propagation started working out of the box.

4) CORS handling

A monitoring service should be able to start and collect telemetry even if the main application services are down.
To make this possible, I exposed a dedicated endpoint for client-side telemetry collection.

However, browsers enforce strict security rules that block requests to hosts other than the web app itself — the classic CORS issue.

Originally, the project had a proxy controller in the backend to work around this.
I suspect the developer who added it simply ran into CORS problems and solved it that way.

In my final setup, I handled CORS directly at the Nginx level.
Think of Nginx here as an ingress controller in Kubernetes terms — it’s a good place to terminate CORS and handle routing.
My local Docker setup made it easy to experiment with CORS configurations before deploying to production.

5) Aggregated telemetry statistics

Once you have tracing working, you quickly realize that just traces are not enough — you also need a clear monitoring panel for high-level stats like RPS, error rates, and so on.

For this, I used Jaeger’s built-in tool called SPM.
However, it turned out to be quite tricky to configure correctly: basically, you need to run and combine two collectors to make it work as intended.

In my experience, this complexity exists partly because Jaeger was not originally designed with the full OpenTelemetry standard in mind. From what I’ve seen, the project seems to be split between the original Jaeger Authors and the broader Linux Foundation governance — but don’t take this too seriously, it’s just my personal guess. 😄

Despite this, Jaeger remains a very good and reliable tool — and once set up, SPM provides helpful aggregated stats alongside detailed traces.

6) Authorization

Luckily, I had access to a corporate authentication server, so the infrastructure team handled most of the effort for me.
However, it’s important to know that Jaeger does not have built-in authentication out of the box.

If you need something simple, it’s usually best to add basic authentication on top of Nginx.
I did exactly that when my setup wasn’t yet integrated with the company’s infrastructure — it worked well enough to keep the telemetry endpoint protected until a full single sign-on was ready.

So these were the main corner problems I faced. I hope sharing them will help you set up telemetry in your own projects with fewer surprises.

Key takeaway

Whether it’s local dev setup or production tuning — clarity and simplicity always pay off.
A clean local environment, clear documentation, and good observability reduce surprises and make teams faster and more confident.

Jaeger Docker Setup: Production-Ready Tracing Server

WireGuard to Shadowsocks: Ready Docker Setup

Sun, 15 Jun 2025 00:00:00 GMT

WireGuard to Shadowsocks: Ready Docker Setup

TL;DR https://github.com/zloom/ss-wg-tunnel

WireGuard is a popular VPN protocol integrated everywhere. It is part of the Linux kernel, it is dead simple by itself, and open source — as every security tool should be. It works perfectly: you can run a server with a single Docker command. Previously, I used it as my VPN tool with this simple compose:

services:
  wireguard:
    image: lscr.io/linuxserver/wireguard:latest
    container_name: wireguard
    cap_add:
      - NET_ADMIN
      - SYS_MODULE # optional
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Etc/UTC
      - PEERS=10
    volumes:
      - /path/to/wireguard/config:/config
    ports:
      - 51820:51820/udp
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
    restart: unless-stopped

You only need a remote server with Linux and Docker, then copy this file to the server and run with docker compose up. Clients are available for every platform — they work perfectly on mobile and desktop. All your traffic goes as expected.

It has only one big disadvantage: it uses UDP and has a unique fingerprint, so your provider can easily block your traffic. A year or two ago it stopped working on my home internet.

What is the alternative?
There is one well-known tool — Shadowsocks. Just like WireGuard, you can run a server pretty easily:

services:
  shadowsocks:
    image: shadowsocks/shadowsocks-libev:v3.3.5
    container_name: shadowsocks
    ports:
      - 8388:8388
      - 8388:8388/udp
    environment:
      - METHOD=chacha20-ietf-poly1305
      - PASSWORD=your_password
    restart: always

It works well, but the problem is the clients. It’s not as convenient: there is an Android client and it works, but apps and browsers ignore it. On PC, you can run a local client which acts as a local proxy, so you need to configure your apps to use it. For example, I use Firefox with it — I added this in network settings:

With that, you can use Firefox with VPN — it works fine, even DNS leaks are prevented. Unfortunately, not every app respects the system proxy settings. In Windows it’s a bit better; in Linux the situation is worse.

Shadowsocks also works fine on Android TV, so you can watch whatever you want with just Shadowsocks. An important point: inside a protected perimeter, WireGuard traffic is not blocked because it is de facto an industry standard.

So I came to an obvious solution: combine them.

The complex part here is the Server B content — a combination of WireGuard and Shadowsocks. In total, it took me about a month to put it into a ready-to-use compose file 😅.

Setup

Here is the full setup in Docker Compose: https://github.com/zloom/ss-wg-tunnel

Clone and run with:

docker compose up

Main limitations:
This works reliably only on Docker Desktop. The main challenge was setting up a multihop network between containers — quite tricky work. The key difference is that Docker Engine provides less network isolation than Docker Desktop: it uses the host network stack, so configuring routes inside containers becomes complicated. On Desktop, Docker runs in a fully isolated VM, so there’s no interference with the host network.

In this setup, all tools except WireGuard are written in Golang — this is intentional. I plan to combine everything into a single Golang app later. I originally wanted to use wireguard-go too, but ran into routing issues. Switching to the kernel version of WireGuard solved the problem.

When running your server with this setup, you might face issues with Docker Desktop autostart. I worked around this by adding a custom daemon. It’s not polished yet, so I haven’t included it in the repo.

Besides Docker, you also need a server: a VPS in your country or your own PC at home. I initially planned to run it on a Raspberry Pi because it’s compact, but when I found it works only on Docker Desktop, I repurposed my old PC and put it in the garage (its coolers are noisy!). If you run it at home, you’ll also need a static IP and a router that supports port forwarding. The setup is not too hard. In the end, I got a stable VPN server that works without manual intervention, even after reboots, and has a nice dashboard!

I hope this setup will help you build your own site-to-site VPN!

Configure Shadowsocks Client on Ubuntu

Anton Zalialdinov, expert software engineer blog

Configure Shadowsocks Client on Ubuntu

Configure Shadowsocks Client on Ubuntu

Server

Client

Configuring Shadowsocks to Wrap All Traffic on Ubuntu

Related Posts

Debugging Keycloak Extension

Debugging Keycloak Extension

Tools and Codebase

Configuration and Running

Debugging with Breakpoints

Related Posts

Get Database Create Script from Prisma Schema

Get a Create Script from Prisma Schema

How to Add External Data to Keycloak Token

How to Add External Data to User JWT in Keycloak

Steps Overview

Test Environment Setup

Keycloak configuration

Testing the Mapper

Related Posts

Jaeger Docker Setup: Production-Ready Tracing Server

Jaeger Docker Setup: Production-Ready Tracing Server

Server overview

Server components overview

Caddy

Jaeger Collector

Jaeger Query

Otel Collector

Prometheus

Cassandra

Cassandra Schema

CI/CD and Development Recommendations

Security note

Related Posts

Mastering Telemetry in a Legacy .NET Project

Mastering Telemetry in a Legacy .NET Project

What I discovered

My starting point and choice of tools

Architecture and Iterative Improvements

Real-world telemetry setup

Example RFC

Development approaches

Make it easy for others to help

Problem investigation: real examples

Key takeaway

Related Posts

WireGuard to Shadowsocks: Ready Docker Setup

WireGuard to Shadowsocks: Ready Docker Setup

Setup

Related Posts