Hi all,

We’re back for another issue. I’m still trying my time to make this news letter become a weekly again :(.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• View the latest HubSpot Developer Platform updates in Spring Spotlight

  Spring Spotlight 2026 is live and we’ve rounded up the top updates for developers here. Featuring:
  
  

  • AI-Assisted Development: Developer MCP Server lets you build apps faster with AI coding tools like Cursor, Claude Code, and Codex.
  

  • Modern API Infrastructure: Date-based versioned APIs and versioned docs for predictable releases.
  

  • MCP: Unlock HubSpot data via the HubSpot MCP Server (remote) to connect any compatible MCP client and create custom workflows and integrations.
  

  • Projects 2026.03: Includes serverless functions, UI extension support, and migration from 2025.1 and 2023.2.
  

• Understanding Wi-Fi 4/5/6/6E/7/8 (802.11 n/ac/ax/be/bn)

  decode and understand all the thing behind these cryptic number.

• How an HTTP header caused time.gov to skew from UTC

  In the United States, the National Institute of Standards and Technology (NIST) maintains the official U.S. time reference. On a recent project I needed a trustworthy clock and time.gov was a convenient option. To validate that the provided reference was accurate, I opened time.gov in two browser windows side-by-side, but found that the provided clock offset estimates disagreed by a margin larger than I could tolerate. When I compared to another source, an NTP client, I found even more disagreement.

• Floating point from scratch: Hard Mode

  In famous problem. 0.1 + 0.2 is equal 0.30000000000000004. Best way to to understand is to re-implement them from the first principle, the hard way.

• Virtual Memory: A Deep Dive into Page Tables, TLBs, and Linux Internals

  From page faults to NUMA topology: how the Linux kernel manages memory, and what that means for the performance of data-intensive systems

• Let's Build a Regex Engine

  Ever wondered how regex works under the hood? How does it understand an incantation like “<\/?[\w\s]*>|<.+[\W]>” and magically produces a desired result? This series is going to describe exactly how it works and how to implement a feature-rich regex engine.

• Native Secure Enclave backed ssh keys on MacOS

  that MacOS Tahoe can generate and use secure-enclave backed SSH keys! This replaces projects like secretive

• Making your own programming language is easier than you think

  Most programmers dream of their own perfect programming language.

• My queries to monitor autovacuum

  Without vacuum, deadtuple (dead row) are accumulate in Postgres, the engine needs to scan them and exclude them, vacuum suppose to delete them. Here are many tip and tricks to monitor that process

• PGKeeper: Building the bouncer we needed for Postgres

  Figma outgrow pgbouncer and share experience writing their own Postgres connection pooling.

• NPM Security best practices

  cheatsheet covers several npm security best practices and productivity tips, useful for JavaScript and Node.js developers. This list was originally based on the 10 npm security best practices from the Snyk blog.

• High Performance Git

  a book written for engineers who need Git to stay fast as repositories, histories, and teams get larger: build and CI engineers, monorepo owners, devprod teams, and anyone tasked with figuring out Git beyond the usual four commands.

Code to read

• ywawky

  MacOS Web Server written entirely in ARM64 assembly

• honker

  SQLite extension + bindings for Postgres NOTIFY/LISTEN semantics with durable queues, streams, pub/sub, and scheduler

• boring

  A simple command line SSH tunnel manager that just works.

• backpressure

  Prioritized semaphore/rate-limiter/adaptive throttle for load management.

Tools

• plow

  A high-performance HTTP benchmarking tool that includes a real-time web UI and terminal display

• pg_column_tetris

  A PostgreSQL extension that can enforce optimal column alignment to minimize row padding waste.

• pgwrl

  Cloud-native continuous backup for PostgreSQL - WAL/base-backup streaming, compression, encryption, retention, and monitoring in a single binary.

• leandex

  a pure SQL / PL/pgSQL control plane for conservative autonomous reindexing, in the same extension-avoidance spirit as pg_ash and PgQue. It installs into a separate control database, talks to target databases through postgres_fdw user mappings and dblink, and rebuilds only with reindex index concurrently. No schema is installed into target databases. No C extension, no shared_preload_libraries, no sidecar worker, no restart.

• hyperframes

  HyperFrames lets AI agents compose videos by writing HTML, CSS & JS.

• pg-trickle

  Have u ever has to write some cronjob to refresh materialized view? this tool handle all of that, natively inside Postgres

• remotion

  Create real MP4 videos with React. Parametrize content, render server-side and build applications.

• trailtool

  aggregates CloudTrail logs to simplify analysis for AI agents

• redai

  A terminal workbench for AI-driven vulnerability discovery and live validation.

Hi all,

Trivy is a security scanner to find vulnerabilities, misconfigurations, secrets and very widely used. Imagine, Trivy got under attack itself. Attackers compromised Trivy GitHub Actions by force-updating tags to deliver malware, exposing CI/CD secrets across affected pipelines.

Then last week, two versions of the litellm Python package on PyPI were found to contain malicious code.

All of that just mean we need to be extra triple careful when dealing with any external dependenies. Any external dep is a liability. In this issue we had included 2 tools that hopefully will allow us to run those app in sandbox instead.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Autonoma — AI Agents That Find Your Bugs Before Your Users Do

  Let’s be honest: your test suite is probably lying to you. You’ve got green checkmarks in CI, 80% coverage, and your last deploy still broke the checkout flow on mobile.

  The problem isn’t that you’re bad at testing. It’s that test maintenance is a full-time job nobody wants, and the coverage you think you have is mostly happy paths you already tested manually. Meanwhile, your users are the ones discovering the real bugs.

  Autonoma takes a different approach. Instead of making you write and maintain test scripts, it runs AI agents that navigate your web and mobile apps like real users: clicking through forms, testing edge cases, handling state transitions, all in parallel across hundreds of browsers and devices.

  The best thing? If after 30 days a bug reaches production, we will refund you the money.

• My DIY FPGA board can run Quake II (part 4)

  An incredible story of someone dived into electronics design with almost zero prior experience. Their goal was ambitious: build an FPGA-based device capable of booting Linux, connecting to a screen and keyboard, and allowing me to write and compile code directly on the machine itself.

• EXPLAIN's Other Superpowers

  Most people who work with PostgreSQL eventually learn two commands for query tuning: EXPLAIN and EXPLAIN ANALYZE. In this post we’ll take a look at a few of those lesser-known options.

• PostgreSQL and Huge Pages: Boosting Database Performance the Right Way

  When you run PostgreSQL on a high-performance server with a lot of RAM, one of the most overlooked performance tunings is Huge Pages. Understanding and configuring Huge Pages correctly can significantly improve performance, especially for databases with large shared memory requirements.

• PostgreSQL WAL replication and Checkpoints: why they might lag and how we fixed it

  Anyone who operate Postgres with high write will need to handle and optimize WAL and AutoVacuum. We can defer these job to AI, but it’s better to understand all the knowledge.

• Ctrl-C in psql gives me the heebie-jeebies

  Whatever the reason, if you’re a psql command-line user, Ctrl-C is in your muscle memory. So now you’re looking at the words Cancel request sent, followed shortly after by the not-really-an-error message ERROR: cancelling statement due to user request. But what’s going on behind the scenes?

• Automating RDS Postgres to Aurora Postgres Migration

  In 2024, the Online Data Stores team at Netflix conducted a comprehensive review of the relational database technologies used across the company, and decide to standardize on Amazon Aurora PostgreSQL as the primary relational database offering for Netflix teams.

• Rewriting our Rust WASM Parser in TypeScript

  Switching from Rust to TypeScript make the code faster. Not because Rust is slow, or TypeScript is fast. But simply the code to serialize/unserialize the object back and forth when passing between both world outweight the performance gain.

• Two React Design Choices Developers Don’t Like—But Can’t Avoid

  Developers have never been shy about disliking certain React APIs. They feel awkward, restrictive, or just plain counterintuitive. But the reality is that the two most complained‑about design choices in React weren’t arbitrary at all — they were early signs of deeper constraints that every UI model eventually runs into

• A Complete Guide to Bookmarklets

  Imagine the ability to inject the JS code, and run it on any webpage whenever you click a bookmark, and it acts on the urrent page? That’s what Bookmarklet is

Code to read

• diff

  A difference algorithm module for Go

• defuddle

  Get the main content of any page as Markdown.

• nokolexbor

  drop-in replacement for Nokogiri. It’s 4.7x faster at parsing HTML and up to 1352x faster at CSS selectors.

Tools

• agent-safehouse

  Sandbox your local AI agents so they can read/write only what they need

• edgejs

  Running Node apps inside a WebAssembly Sandbox

• gnata

  Pure-Go implementation of JSONata 2.x for high-throughput streaming evaluation. And the story of re-impelment it in Go with AI saves the compnay $500k USD per year

• secretlint

  Pluggable linting tool to prevent committing credential.

• pgtui

  A Postgres TUI client that utilizes your terminal text editor for inserts & updates

• slowql

  SQL static analyzer for performance, security, compliance and cost. 272 rules. Completely offline. Works in CI pipelines.

• pg_stat_ch

  A PostgreSQL extension that captures per-query execution telemetry and exports it to ClickHouse in real-time. Unlike pg_stat_statements which aggregates statistics in PostgreSQL, pg_stat_ch exports raw events to ClickHouse where aggregation happens via ClickHouse’s powerful analytical engine.

• go-lsp

  Go LSP helper library support 3.17 of the LSP specification

Hi all,

Merry Christmas everyone. It’s the end of years, take sometime off and enjoy the season.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• What Actually Moved the Needle for Engineering Leaders in 2025 — With Salesforce, Snyk, Port & Qodo

  What’s actually working for engineering teams — and what’s not? Hear from Benjamin Stice (Salesforce VP Engineering), Clinton Herget (Snyk Field CTO), Yonatan Boguslavsky (Port CTO), and Dedy Kredo (Qodo Co-Founder) in this candid panel discussion. Three fast, honest segments: What we’re leaving behind: practices, tools, or org habits that no longer serve the team (and how we’re unwinding them without chaos). What we’re doubling down on: the investments that produced real outcomes in quality, delivery, and developer experience, and how we’re scaling them. What we’re betting on next: the experiments and measured risks we’ll run in 2026 to meet reliability, security, and growth goals. No platitudes. Just real lessons from leaders who’ve steered teams through a year of AI transformation.

• How when AWS was down, we were not

  One of the most massive AWS incidents transpired on October 20th. The long story short is that the DNS for DynamoDB was impacted for us-east-1, which created a health event for the entire region. It’s the worst incident we’ve seen in a decade. authress shares with us how they maintain a high SLI to match our SLA reliability commitment even when the infrastructure and services we use don’t.

• How to Think About GPUs

  Part 12 of “How to Scale Your Model”, an free ebook that aim to demystify the science of scaling language models: how TPUs (and GPUs) work and how they communicate with each other, how LLMs run on real hardware, and how to parallelize your models during training and inference so they run efficiently at massive scale.

• How Memory Maps (mmap) Deliver 25x Faster File Access in Go

  Memory maps are a modern Unix mechanism where you can take a file and make it part of the virtual memory. In Unix context, modern means that it was introduced in the 1980s or later. You have a file, containing data, you mmap it and you’ll get a pointer to where this resides. Now, instead of seeking and reading, you just read from this pointer, adjusting the offset to get to the right data.

• Bloom filters

  The original motivation for the creation of Bloom filters is efficient set membership, using a probabilistic approach to significantly reduce the time and space required to reject items that are not members in a certain set.

• What the heck is AEAD again

  It stands for “Authenticated Encryption with Associated Data” but if you are struggling to remember what is it. This post sumarize nicely.

• sql joins

  There are many SQL JOINs guides and tutorials, but this one takes a different approach. We try to avoid misleading wording and imagery, and we structure the material in a different way. The goal of this article is to clarify your mental model

• Postgres Scan Types in EXPLAIN Plans

  The secret to unlocking performance gains often lies not just in what you ask in a query, but in how Postgres finds the answer. The Postgres EXPLAIN system is great for understanding how data is being queried. One of secretes to reading EXPLAIN plans is understanding the type of scan done to retrieve the data. The scan type can be the difference between a lightning-fast response or a slow quey

• 21 Lessons from 14 Years at Google

  Addy Osmani is a legendary engineer. In his own words: “When I joined Google ~14 years ago, I thought the job was about writing great code. I was partly right. But the longer I’ve stayed, the more I’ve realized that the engineers who thrive aren’t necessarily the best programmers - they’re the ones who’ve figured out how to navigate everything around the code: the people, the politics, the alignment, the ambiguity.”

• The Anatomy of a Web Performance Report

  WebPerformance Report was designed around that principle. Although its underlying data comes from automated performance tests, the report itself is intentionally high-level. It is structured to help anyone, developers, designers, marketers, and executives, understand what happened, why it matters, and where the opportunities lie. This article walks through the anatomy of a Web Performance Report, explaining each section, why it exists, and what decisions it enables.

• Did you know your browser has two accessibility trees?

  To implement accessibility, Chrome has this pipeline DOM mutation -> AXEvent -> Accessibility tree -> OS accessibility API -> Screen reader. But here’s the strange part: Chrome doesn’t have one accessibility tree, it has two. This article traces the full lifecycle of an accessibility event. From the moment the DOM changes, through Blink’s internal systems, through its internal accessibility tree, and finally back out into the BrowserAccessibility tree that screen readers query.

Code to read

• fizzy

  37signals release this kanban style app. A good opportunity to dive into the source and see how DHH write user facing code.

• Heat.js

  A lightweight JavaScript library that generates customizable heat maps, charts, and statistics to visualize date-based activity and trends.

• tinyauth

  The simplest way to protect your apps with a login screen.

• tiny-diffusion

  A character-level language diffusion model trained on Tiny Shakespeare

• uvm32

  Minimalist, dependency-free virtual machine sandbox for microcontrollers and other resource-constrained devices. Single C file, no dynamic memory allocations, asynchronous design, pure C99

Tools

• rustnet

  A cross-platform network monitoring terminal UI tool built with Rust

• driverjs

  Lightweight JavaScript library for product tours, highlights, and contextual help to guide users through your product

• RealtimeVoiceChat

  This project lets you chat with a Large Language Model (LLM) using just your voice, receiving spoken responses in near real-time. Think of it as your own digital conversation partner.

• Nano-PDF

  Edit PDF files with Nano Banana

• mkslides

  Use mkslides to easily turn markdown files into beautiful slides using the power of Reveal.js!

• dbdock

  Enterprise-grade PostgreSQL backup and restore. Beautiful CLI with real-time progress tracking.

• termboard

  powerful management tool for PostgreSQL. You can use it to monitor, optimize or configure multiple PostgreSQL instances.

• pg_clickhouse

  Interfaces to query ClickHouse databases from PostgreSQL

• tiny-rdm

  (Tiny Redis Desktop Manager) - A modern, colorful, super lightweight Redis GUI client for Mac, Windows, and Linux.

Hi all,

Sometime when my though wandering the world, I just wonder how amazing TCP was design. The world all connect together because of it. Sometime you disconnect your VPN, then re-connect, and your SSH connection just resume. How amazing resilient it is. Today we discover that by learning about TCP protocol.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Build automation tools for AI agents. Earn passive income and win your share of $1M in prizes

  Developers earned $563K in September alone building automation tools on Apify. Now there’s $1M more in prizes through January 31. Build web scrapers, API wrappers, MCP servers for AI agents. Earn up to $2K per tool based on monthly active users, compete for $30K/$20K/$10K grand prizes, plus $2K weekly spotlight prizes. Ongoing marketplace revenue continues after the challenge ends.

• The Internet is Cool. Thank you, TCP

  The internet is incredible. It’s nearly impossible to keep people away from. But it can also be unreliable: packets drop, links congest, bits mangle, and data corrupts. Oh, it’s dangerous out there! (I’m writing this in Kramer’s tone)

• Building a Simple Search Engine That Actually Works

  Sometimes you just want something that: Works with your existing database, Doesn’t require external services, Is easy to understand and debug. That’s what the OP built. A search engine that uses your existing database, respects your current architecture, and gives you full control over how it works.

• Faster Index I/O with NVMe SSDs

  Marginalia is an experimental search engine. The Marginalia Search index has been partially rewritten to perform much better, using new data structures designed to make better use of modern hardware. This post will cover the new design, and will also touch upon some of the unexpected and unintuitive performance characteristics of NVMe SSDs when it comes to read sizes.

• Myths Programmers Believe about CPU Caches

  As a computer engineer who has spent half a decade working with caches at Intel and Sun, I’ve learnt a thing or two about cache-coherency. This was one of the hardest concepts to learn back in college – but once you’ve truly understood it, it gives you a great appreciation for system design principles.

• Building blobd: single-machine object store with sub-millisecond reads and 15 GB/s uploads

  I decided to experiment with writing an object store from scratch as a fun learning exercise, and to see how much I could get out of NVMe disks I had on my bare metal machines. Specifically, I wanted to really optimize for random reads and small objects — serving user content where the lower the latency, the better. This would not focus on other features; S3 is bottomless, distributed, and managed for example.

• Postgres Internals Hiding in Plain Sight

  Postgres has an awesome amount of data collected in its own internal tables. Postgres hackers know all about this - but software developers and folks working with day to day Postgres tasks often miss out the good stuff. Today I want to walk through some of the most important Postgres internal data catalog details. What they are, what is in them, and how they might help you understand more about what is happening inside your database.

• Listen to Database Changes through the Postgres WAL

  This could be a cheap way to have a Kafka-like consumer out of the box by consuming your Postgres WAL log. WAL log is like a stream of change happen to your database (update/create/delete) and you can decode them to JSON to see the change and act on them

• The Inner Workings of JavaScript Source Maps

  A deep dive into how JavaScript source maps work under the hood, with examples showing how all the pieces fit together.

• Importing vs fetching JSON

  Beside the usualy way of await fetch to load JSON API, we now can also use import directly. We had this in NodeJS with build tool already but now it’s natively available in the browser too. Lets see when we should use what.

• Why NaN !== NaN in JavaScript (and the IEEE 754 story behind it)

Code to read

• pgFirstAid

  Easy-to-deploy, open source PostgreSQL function that provides a prioritized list of actions to improve database stability and performance.

• top_secret

  Filter sensitive information from free text before sending it to external services or APIs, such as chatbots and LLMs.

• pg_easy_replicate

  Easily setup logical replication and switchover to new database with minimal downtime

• justgage

  a handy JavaScript plugin for generating and animating nice & clean dashboard gauges. It is based on Raphaël library for vector drawing

• navcat

  javascript navigation mesh construction and querying library for 3D floor-based navigation

• stormy

  Minimal neofetch-like weather CLI. Just fun code to read.

• portal

  Public Open Relay To Access Localhost

• nano-vllm

  A lightweight vLLM implementation built from scratch.

• zensical

  A modern static site generator by the creators of Material for MkDocs

Tools

• chandra

  OCR model that handles complex tables, forms, handwriting with full layout.

• perspective

  Perspective is an interactive analytics and data visualization component, which is especially well-suited for large and/or streaming datasets. Use it to create user-configurable reports, dashboards, notebooks and applications, then deploy stand-alone in the browser, or in concert with Python and/or Jupyterlab.

• pg_statviz

  A minimalist extension and utility pair for time series analysis and visualization of PostgreSQL internal statistics.

• check_postgres

  Nagios check_postgres plugin for checking status of PostgreSQL databases. You probably cannot use this as it’s because it’s for Nagios but the script can adopt to any monitoring system.

• sloggo

  Minimal RFC 5424 syslog collector and viewer based on DuckDB. Runs as a single, resource-friendly process

• slim-select

  Advanced select dropdown without any dependencies

• caswaf

  HTTP & OAuth Gateway and Web Application Firewall (WAF) based on ModSecurity

• stylus

  Lightweight status page for home infrastructure

Hi all,

It’s already autumn and we will continue to get more issue moving forward. Hope everyone have a great haloween ahead. I also shared a story about someone almost got hack by being trick into running code for an interview. Remind so stay safe these day, never run untrusted code on your machine.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Build your next on HubSpot with the flexibility of an all-new Developer Platform

  The HubSpot Developer Platform gives you the tools to build, extend, and scale with confidence. Create AI-ready apps, integrations, and workflows faster with a unified platform designed to grow alongside your business.

• How to create an OS from scratch

  May sound overwhelm but it’s very approachable with step by step building from previous foundation. Eg to write the initial boot loader you just need: loop: jmp loop

  times 510-($-$$) db 0 dw 0xaa55

  A very detail setup so you can prepare emulator to start learning these.

• Caching

  Every time you use a computer, caches work to ensure your experience is fast. Everything a computer does from executing an instruction on the CPU, to requesting your X.com feed, to loading this very webpage, relies heavily on caching. You are about to enjoy a guided, interactive tour of caching: the most elegant, powerful, and pervasive innovation in computing.

• Big O

  A visual introduction to big O notation.

• Consistent hashing

  This post is an introduction to consistent hashing, an algorithm for designing a hash table such that only a small portion of keys has to be recomputed when the table’s size changes.

• Understanding PostgreSQL Tablespaces

  By default all of the Postgres data stay in the same directory, under the same path and therefore the same volume. This limit our scale, many cloud provider has a max volume size, and they all share same IO. What can we do to put the database at different location instead?

• How I Almost Got Hacked By A 'Job Interview

  Be careful out there with sophisicated phishing attempt nowadays. Bottom line is never ever run untrusted code directly on your machine.

• Find where a specific object was allocated in JavaScript with DevTools

  Maybe you’re debugging a network request, but the stack trace has been laundered away by a batching mechanism. Maybe you have an enormous React codebase with kilometers of prop drilling, and you can’t be bothered to hit Go To References all morning. In any case, sometimes you just wish you could see where that specific object in the debugger was created.

• WireGuard topologies for self-hosting at home

• How to stop Linux threads cleanly

  Once you get into the business of starting threads, you’re probably also in the business of stopping them. However the former is much easier than the latter. With “stopping” I mean stopping the thread while giving it a chance to run some cleanup operations before fully terminating. Or in other words, we want to terminate a thread while ensuring that memory is freed, locks are released, logs are flushed, and so on

• pqr.sql: Generate QR Codes with Pure SQL in PostgreSQL

Code to read

• rack-attack

  Rack middleware for blocking & throttling

• sift

  A lightweight terminal UI for displaying Go tests

• oto

  ♪ A low-level library to play sound on multiple platforms ♪

Tools

• pgsync

  keeps PostgreSQL/MySQL/MariaDB as your source of truth and publishes denormalized documents to Elasticsearch/OpenSearch—continuously and transactionally.

• pgcalendar

  A PostgreSQL extension that provides infinite calendar functionality for recurring schedules with exceptions

• relay

  Fast & free browser trusted HTTPS certificates

• builder

  Visual Development for React, Vue, Svelte, Qwik, and more

Hi all,

We came back with other issue. I had got a new job with lot of duties last few month but things are cooling down so we’re going back :).

My friend, Joel Dare has also start writing a news letter, you may consider to subscribe there. He’s also creator of neat which as its name, a very neat css framework.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Transform Your Career: Build, Learn, and Network at AWS re:Invent 2025 | Las Vegas.

  Transform your skills at AWS re:Invent 2025. Over five dynamic days, you’ll dive into technical training, explore the latest technologies, and build valuable peer connections.Take home practical solutions that deliver instant value to your work and advance your career.
  
  

  Join us in Las Vegas from December 1-5 to:
  
  

  ● Access insider knowledge through interactive labs, technical sessions, and workshop
  

  ● Accelerate your learning through direct mentorship from AWS experts
  

  ● Build your network and connect with peers who share your technical interests through AWS Community Programs at the Developer Pavilion
  

  ● Experience live music, amazing food, and more at re:Play, the ultimate tech celebration
  

  ● Attend in person and receive a 50% discount voucher for any AWS Certification exam
  

  ● Team up and save - Get one free pass for every 10 passes purchased*
  
  

  Our 2025 event catalog is now available! From experimenting with new solutions to interactive workshops and hands-on training, we’ve got something for everyone at AWS re:Invent.

• Inside the Apollo "8-Ball" FDAI (Flight Director / Attitude Indicator)"

  The Apollo “8-Ball” FDAI (Flight Director/Attitude Indicator) was a crucial instrument in Apollo spacecraft, used to display the vehicle’s orientation. The article details its complex internal workings, which include a multi-axis rotating ball mechanism driven by three motors and a servo loop. The specific unit examined was later modified for use in a Space Shuttle simulator, with changes made to its components and markings.

• Branch prediction: Why CPUs can’t wait?

  There are some latency numbers that every programmer should know, according to Jeff Dean, and one of them is branch misprediction, which costs around 5ns in 2012, and the latency remains roughly the same as the time of writing this post. So what is branch prediction, what happens when it’s mispredicted, and why is it costly?

• How to Slow Down a Program? And Why it Can Be Useful.

  Making programs slower can be useful to find race conditions, to simulate speedups, and to assess how accurate profilers are. To detect race conditions, we may want to use an approach similar to fuzzing. Instead of exploring a program’s implementation by varying its input, we can explore different instruction interleavings, thread or event schedules, by slowing down program parts to change timings. This approach allows us to identify concurrency bugs and is used by CHESS, WAFFLE, and NACD.

• UUIDv7 Comes to PostgreSQL 18

  UIDv7 is a highly important and long-awaited feature that addresses a major pain point for developers who use UUIDs as primary keys in databases. Unlike older versions like UUIDv4, which are randomly generated and lead to poor performance and index bloat in B-tree indexes, UUIDv7 incorporates a Unix Epoch timestamp. This makes it time-ordered, ensuring that new UUIDs are inserted sequentially, which significantly improves performance for write-heavy workloads. The article emphasizes that this quiet but impactful addition “brings the best of both worlds” by providing global uniqueness while also being sortable and performance-friendly.

• Three Algorithms for YSH Syntax Highlighting

  I always like syntax highling algorithm. It’s something we take for granted and didn’t ask how it can be done. Andy walked us through how he do it for his own shell language.

• Rendering Crispy Text On The GPU

  The webpage describes a new method for GPU text rendering that uses vector data of glyphs instead of pre-baked textures, addressing issues like poor quality with certain fonts and large texture sizes. The approach involves loading raw Bézier curves, sending them to the GPU, and rasterizing them at runtime with a temporal accumulation technique for high-quality anti-aliasing. This method also allows for customized subpixel anti-aliasing to eliminate color fringing on different monitor layouts.

• CRDT: Text Buffer

  Collaboratively editing strings of text is a common desire in peer-to-peer applications. For example, a note-taking app might represent each document as a single collaboratively-edited string of text.

  The algorithm presented here is one way to do this. It comes from a family of algorithms called CRDTs, which I will not describe here. It’s similar to the approaches taken by popular collaborative text editing libraries such as Yjs and Automerge. Other articles have already been written about these similar approaches (see the references section below), but this article also has a nice interactive visualization of what goes on under the hood.

• P-Hacking in Startups

  Speed kills rigor. In startups, the pressure to ship fast pushes teams to report anything that looks like an improvement. That’s how p-hacking happens. This piece breaks down three common cases—and how to avoid them

• How Compiler Explorer Works in 2025

  explains how Compiler Explorer works, a popular online tool for compiling code and viewing assembly output. The site handles millions of compilations annually, using Google’s nsjail tool for security to isolate compilation processes. To manage a massive 4 terabyte collection of over 4,700 compiler versions and reduce latency, it uses squashfs images mounted over NFS. The article concludes by discussing the site’s future plans, including the addition of an AI explanation tool and user accounts.

• strace tips for better debugging

  Have you ever scratch your head and see why your program seems hang? with very low CPU utilization? When you do strace it just show a bunch of FUTEX. In this short article we just learn some useful options and flags of strace to debug these easiser

• Git Notes: git's coolest, most unloved­ feature

  the short of it is: they’re cool for appending notes from automated systems (like ticket or build systems) but not really for having interactive conversations with other developers (at least not yet)

Code to read

• miniDiffusion

  A reimplementation of Stable Diffusion 3.5 in pure PyTorch. The Hacker News thread also had a lot of userful and great comment.

• bake

  a Makefile formatter and linter. It only took 50 years!

• naturally

  Natural (“version number”) sorting with support for: name, legal doc numbering,

• filterql

  A tiny query language for filtering structured data

Tools

• peaks.js

  JavaScript UI component for interacting with audio waveforms

• manim

  Animation engine for explanatory math videos

• LinkLiar

  Link-Layer MAC spoofing GUI for macOS

• hurl

  run and test HTTP requests with plain text.

• pgactive

  Active-active Replication Extension for PostgreSQL (pgactive)

• pgrwl

  Cloud-Native PostgreSQL WAL receiver. Stream, compress, encrypt, upload, retain and monitor your WAL archive.

• s3mini

  Tiny S3 client. Edge computing ready. No-dep. In Typescript. Works with @cloudflare @minio @Backblaze @digitalocean @garagehq

• gonzo

  A powerful, real-time log analysis terminal UI inspired by k9s. Analyze log streams with beautiful charts, AI-powered insights, and advanced filtering - all from your terminal

Hi all,

I recently becomes a heavy LLM user in coding, It helps me write a lot of helper and tedious script which will take me days or even weeks now I can do in ¹⁄₁₀ of time. It is great when you know how to do something but dont’ want to sit down and write it. Especially on UI. So I plan to share more about LLM/AI to help all of us.

On other note, Mozilla is shutting down Pocket on July 8, 2025. In spirit of self-hosted, I had switch to linkding. I wrote this simple script to import Pocket archive to linkding. The script was written by Gemini btw.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Your data, built your way. Twilio Segment. For data you can depend on.

  Customer Data You Can Depend On
  We know you run your business on data, so you better be able to depend on it.

  
  Twilio Segment was purpose-built so that you don’t have to worry about your data. Forget the data chaos, dissolve the silos between teams and tools, and bring your data together with ease.

  
  So that you can spend more time innovating and less time integrating.

  
  Talk to sales and get a guided tour of Twilio Segment tailored to your business.

• What is Entropy?

  People say many things about entropy: entropy increases with time, entropy is disorder, entropy increases with energy, entropy determines the arrow of time, etc.. But I have no idea what entropy is, and from what I find, neither do most other people. This is the introduction I wish I had when first told about entropy, so hopefully you find it helpful. My goal is that by the end of this long post we will have a rigorous and intuitive understanding of those statements, and in particular, why the universe looks different when moving forward through time versus when traveling backward through time.

• How DCI Movie Delivery and Encryption works

  The Cinema Industry is using its own standards for creating and distributing movies in a secure way. The DCI (Digital Cinema Initiatives) specification defines everything from file formats and encryption to the projection systems itself.

  The specification itself is publicly available but relies on various IEEE (Institute of Electrical and Electronics Engineers) and SMPTE (Society of Motion Picture and Television Engineers) standards, which have to be purchased.

• OpenAI: Scaling PostgreSQL to the Next Level

  At scale of OpenAI, I imagine they might shard PostgreSQL. But they utilize an unsharded architecture with one writer and multiple readers, demonstrating that PostgreSQL can scale gracefully under massive read loads.

• Talking To Your Mailserver Is Not as Hard as You Think!

  Email is very well distributed where you cannot get lock down by a provider. As long asyou own domain, you can move email anywhere. In email world, generally SMTP is for sending email and IMAP is for receiving mail. In this article, we focus on IMAPv4.

• An illustrated guide to Amazon VPCs

  In this section, I talk about why VPCs were invented and how they work. This is critical to understand because almost everything you do in AWS will happen inside of VPC. If you don’t understand VPCs, it will be difficult to understand any of the other networking concepts.

• Writing your own CUPS printer driver in 100 lines of Python

  Printing on unix-like operating systems like Linux or macOS is usually done through the CUPS daemon. CUPS is responsible for handling printer configuration, scheduling print jobs and actually talking to the printer over the wire. In this article we discuss a printer system for ticket using FGL. And they implement a CUPS driver to take the rastered pixel data and ouput FGL.

• Understanding logical replication in Postgres

  A deeper look at how PostgreSQL logical replication works, from WAL internals and logical decoding to streaming protocols and replica lag. If you’ve ever wondered what’s happening under the hood, this post connects the dots.

• Field Notes From Shipping Real Code With Claude

  Think of this post as your field guide to a new way of building software. By the time you finish reading, you’ll understand not just the how but the why behind AI-assisted development that actually works.

• How Reladiff Works

  diffing dataset is very useful in data engineering. incremental syncing, detect drift etc. how can we create a tool that could efficiently diff large SQL datasets across different databases and servers. This article build such as a tool with: Divide, Hash, And Conquer

• Building an AI Server on a Budget

  Instead of renting the GPU we can pay up-front cost to build a server with GPU. This post can serve as a simple foundation into this process with sourcing parts and assbler thing together.

• Too Many Open Files

  Every backend dev will hit file limit at some point and we run something like ulimit -n 65000 to fix it. But how we debug it, how we find out how many file it openning? and what is these file descriptor.

• X-Ray Defence: Hidden Resource, Sudden Lifeline

• Taking out the Trash: Garbage Collection of Object Storage at Massive Scale

• A Trip Down Memory Lane: How We Resolved a Memory Leak When pprof Failed Us

  Warpstream control plane is written in Go, which has excellent built-in support for debugging application memory issues with pprof. We’ve used pprof hundreds of times in the past to debug performance issues, and usually memory leaks are particularly easy to spot. But in this case, this isn’t something that pprof could help with.

• Crafting Effective Prompts for Agentic AI Systems: Patterns and Practices

  Curated collection of system prompts for top AI tools. Perfect for AI agent builders and prompt engineers. Incuding: ChatGPT, Claude, Perplexity, Manus, Claude-Code, Loveable, v0, Grok, same new, windsurf, notion, and MetaAI.

• The Book of Secret Knowledge

  A collection of inspiring lists, manuals, cheatsheets, blogs, hacks, one-liners, cli/web tools and more.

Code to read

• Your data, built your way. Twilio Segment. For data you can depend on.

  Customer Data You Can Depend On
  We know you run your business on data, so you better be able to depend on it.

  
  Twilio Segment was purpose-built so that you don’t have to worry about your data. Forget the data chaos, dissolve the silos between teams and tools, and bring your data together with ease.

  
  So that you can spend more time innovating and less time integrating.

  
  Talk to sales and get a guided tour of Twilio Segment tailored to your business.

• fast-mcp

  Use Ruby DSL to let AI Model call function inside your Ruby app

• rails-mcp-server

  A Ruby implementation of a Model Context Protocol (MCP) server for Rails projects. This server allows LLMs (Large Language Models) to interact with Rails projects

• mcp-go

  A Go implementation of the Model Context Protocol (MCP), enabling seamless integration between LLM applications and external data sources and tools.

• chafa-go

  Go bindings for the Chafa terminal graphics library. Render high quality images on the terminal without CGO or external dependencies.

• typed-ffmpeg

  offers a modern, Pythonic interface to FFmpeg, providing extensive support for complex filters with detailed typing and documentation.

• mini-photo-editor

  Online webgl photo editor with effects, filters and cropping

Tools

• beachpatrol

  Browser’s End-user Automation CLI Hub. Potentialize All Tasks Regarding Online Life.

• glowui

  442 open source icons in 2 styles, made for interfaces. MIT license

• Why upgrade PostgreSQL?

  Build by the fame depesz, it tells you what you got between 2 PG version.

• PostgreSQL Conf: Parameters Documentation

  A simple list of all PostgreSQL parameter.

• AirAP

  airplay to an ios device, use your iphone as an airplay receiver

• markitdown

  Python tool for converting files and office documents to Markdown.

• contextgem

  Effortless LLM extraction from documents

• Turn docstrings into LLM-functions

• postgres-mcp

  Postgres MCP Pro is an open source Model Context Protocol (MCP) server built to support you and your AI agents throughout the entire development process—from initial coding, through testing and deployment, and to production tuning and maintenance.

• pg_auto_failover

  Postgres extension and service for automated failover and high-availability

• revive

  🔥 ~6x faster, stricter, configurable, extensible, and beautiful drop-in replacement for golint

• pg_exporter

  Advanced PostgreSQL & Pgbouncer Metrics Exporter for Prometheus

Hi all,

So this week, Github CodeQL leaked their token; the token was only publicly exposed for about 2 seconds, but that’s enough. Then, we had Malware found on npm infecting local packages with a reverse shell. And an atop heap issue. The world truly fell apart last week, no joke. That’s why it’s helpful to be paranoid whenever we run any piece of software on our CI, on our dependencies—always vet them yourself.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• 30 years of PostgreSQL buffer manager locking design evolution

  An in-depth exploration of the evolution of PostgreSQL’s buffer manager locking mechanisms over three decades, highlighting key design decisions and their impact on performance and concurrency.

• How to Write Blog Posts that Developers Read

  A guide offering practical advice on crafting technical blog posts that effectively engage and inform developer audiences, emphasizing clarity, structure, and relevance.

• Identity Tokens Explained: Best Practices for Better Access Control

  An overview of identity tokens, detailing their role in authentication and authorization processes, and providing best practices for implementing secure and efficient access control mechanisms.

• Blasting Past Webp: An analysis of the NSO BLASTPASS iMessage exploit

  A technical analysis by Google’s Project Zero team on the NSO Group’s BLASTPASS exploit, which targeted Apple’s iMessage using malicious WebP images to achieve zero-click remote code execution.

• Visual-Focused Algorithms Cheat Sheet

  A concise reference guide summarizing key algorithms related to visual data processing, offering quick insights into their applications and implementations.

• Train & Weather Tracker with Raspberry Pi & E-Ink

  If you have some old Raspberry Pi hardware lying around, this could be a great hobby project.

• MySQL transactions per second vs fsyncs per second

  An examination of the relationship between MySQL’s transaction throughput and the filesystem’s synchronization operations, discussing how fsync frequency impacts database performance.

• Better Shell History Search

  An article discussing techniques to enhance shell history search functionality, including the integration of tools like fzf for more efficient command retrieval and improved productivity.

• Life Altering PostgreSQL Patterns

  A discussion on transformative PostgreSQL design patterns, such as using UUID primary keys, implementing timestamp fields, enforcing strict foreign key constraints, utilizing schemas for organization, and employing enum tables for data integrity.

• A Practical Guide to Taming Postgres Isolation Anomalies

  A comprehensive guide addressing PostgreSQL’s transaction isolation anomalies, providing practical examples and strategies to understand and mitigate issues like dirty reads, lost updates, and phantom reads.

• Cloudflare open-sourcing OpenPubkey SSH (OPKSSH): integrating single sign-on with SSH

  Single sign-on for SSH is a dream. No longer dealing with keys, editing ~/authorized_keys files, or modifying LDAP or PAM configurations. Easy to revoke and offboard developers.

• MCP (Model Context Protocol): Simply explained in 5 minutes

  An introduction to the Model Context Protocol (MCP), explaining its purpose, functionality, and how it can streamline development processes by providing contextual model information.

Code to read

• hann

  A fast approximate nearest neighbor search library for Go

• Physix

  A simple Physics engine in GoLang

• shezem-rs

  A Rust implementation of a fast audio fingerprinting system inspired by Shazam, for audio recognition and identification. It focuses on speed, efficiency and simplicity

• pgledger

  A ledger implementation in PostgreSQL

Tools

• gaia

  A tool by AMD to run LLM Agents on Ryzen AI PCs in Minutes

• stable-virtual-camera

  Generative View Synthesis with Diffusion Models

• star-vector

  a foundation model for SVG generation that transforms vectorization into a code generation task. Using a vision-language modeling architecture, StarVector processes both visual and textual inputs to produce high-quality SVG code with remarkable precision.

• gopass

  The slightly more awesome standard unix password manager for teams

• lakekeeper

  an Apache-Licensed, secure, fast and easy to use Apache Iceberg REST Catalog written in Rust. Apache Iceberg is a project that allow us to run SQL query on a set of file(CSV, Parquet, JSON) directly from storage such as S3.

• goavatar

  This package provides a simple way to generate unique, symmetric identicons based on an input string (e.g., an email address or username). It uses an MD5 hash to create a deterministic pattern and color scheme, then mirrors the design for a visually appealing avatar.

Hi all,

Last week, NextJS has a new security vulnerability, CVE-2025-29927 that allow by pass middleware auth checking by setting a header to trick it into thinking this is an internal request and skip the auth middleware check. If you’re hosted on Vercel there is no action to take, but if you run the vercel server in your infra, you should update.

Another one for k8s, if you’re running ingress-nginx, you should update this patch. The attack require the ability to have a pod already running inside K8S network, such as sharing namespace with other users, or exposing admission controller webhook to internet. So if your cluster is dedicated to you, and not doing stuff like host network or expose admission controller service, you can buy sometime for the upgrade.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• When You Deleted /lib on Linux While Still Connected via SSH

  A classic tale of Linux horror. You’re SSH’d into a machine, you make one wrong move, and suddenly your system is broken. This post walks through how to recover from this self-inflicted nightmare.

• Think About SQL MERGE in Terms of a RIGHT JOIN

  RIGHT JOIN is an esoteric feature in the SQL language, and hardly ever seen in the real world, because almost every RIGHT JOIN can just be expressed as an equivalent LEFT JOIN. There is, however, one place in the SQL language where RIGHT JOIN is surprisingly ubiquitous and today we learn about it.

• Zero-Knowledge Proofs

  An approachable introduction to the magic of zero-knowledge proofs—how they let you prove something without revealing the details. Perfect for anyone curious about cryptography without diving into hardcore math.

• Understanding Solar Energy

  This one’s a bit outside the usual programming realm, but trust me—it’s fascinating. A deep dive into how solar panels actually work, the engineering behind them, and why understanding electricity at this level is surprisingly useful.

• A Synchronization Engine for Everyone

  Syncing data is one of those things that sounds easy until you try to build it. This post walks through a synchronization engine design that’s both simple and powerful.

• Leaking Passwords (and more!) on macOS

  A deep dive into how macOS can unintentionally leak sensitive data like passwords, thanks to clipboard behavior and debugging tools. A must-read for anyone who cares about security.

• A Deep Dive into the Inline Background Overlap Problem

  Ever tried putting text over an image and ended up with weird background overlaps? This post explains why that happens and how to fix it. If you do anything with CSS, you’ll want to bookmark this one.

• BPF: Or How I Stopped Worrying and Loved the Kernel

  BPF is like a superpower for peeking into Linux. Whether you care about performance monitoring, security, or just love cool system hacks, this is a fantastic read.

• A History of Null Pointer Dereferences on macOS

  A nostalgic yet technical look at how null pointer dereferences have plagued macOS over the years. Great for those who love debugging deep OS issues.

• Improving Recommendation Systems & Search in the Age of LLMs

  AI-powered recommendations are everywhere, but how do they really work? This post explores how LLMs are changing search and recommendation engines for the better.

• Ruby Ractors and Lock-Free Data Structures

  If you’re a Rubyist curious about parallelism, this is for you. A practical look at Ruby’s Ractors and how they enable truly parallel execution without locks.

Code to read

• retroboy

  A Game Boy emulator written in Rust.

• rhino

  A Rack web server written in Ruby from scratch.

• konva

  an HTML5 Canvas JavaScript framework that extends the 2d context by enabling canvas interactivity for desktop and mobile applications.

• landrun

  Need to run untrusted code safely? This sandboxing tool might be your new best friend.

• Physix-go

  A physics engine in Go that makes simulating motion, collisions, and forces as simple as possible.

• fastplotlib

  A blazing-fast plotting library for massive datasets, using modern GPU rendering to keep things smooth.

• zimdjson

  Parsing gigabytes of JSON per second. Zig port of simdjson with fundamental features.

Tools

• playwright-mcp

  A Model Context Protocol (MCP) server that provides browser automation capabilities using Playwright. This server enables LLMs to interact with web pages through structured accessibility snapshots, bypassing the need for screenshots or visually-tuned models.

• debugbar

  Powerful devtools for Ruby on Rails. Inspired by the Laravel Debugbar.

• xata agent

  an open source agent that monitors your database, finds root causes of issues, and suggests fixes and improvements. It’s like having a new SRE hire in your team, one with extensive experience in Postgres.

• pgFormatter

  Free Online version of pgFormatter a PostgreSQL SQL syntax beautifier (no line limit here up to 100000 characters). This SQL formatter/beautifier supports keywords from SQL-92, SQL-99, SQL-2003, SQL-2008, SQL-2011 and PostgreSQL specifics keywords. May

• atlas

  A database schema management tool that works like Terraform—diff, apply, and keep your schema in check.

• typespec

  Define your API once in a simple tsp file, then generate OpenAPI, gRPC, client code, and docs automatically.

• ferronweb

  A fast, memory-safe web server powered by Rust. Similar to Caddy or Nginx but written in Rust.

Hi all,

This week the github action tj-actions/changed-files are compromise and steeling credential of github action env. I had always found the concept of trusting randome github action is scary. Especially github make the action appear under the global namespace /marketplace/actions make it seems trust worth. Had github adopt a <user-org>/name probably more people will aware. I recommend for any non official github action, fork it to your own org or account and review source code manually.

Second thing, if you has been using WHOIS to find out domain owner or registra info, that is going to change. ICANN is Launching RDAP; Sunsetting WHOIS and a CLI to replace whois too.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Developer AI for real software engineering work

  Lots of coding AIs have cool X demos on greenfield apps. But the day-to-day of a pro software engineer working on a team looks…a little different from vibe coding. Enter Augment Code. The first developer AI built for teams and large codebases, Augment works on codebases of millions of lines of code and thousands of files, bringing full codebase context to every keystroke. Customers like Datastax, Observe, Kong, and Lemonade trust Augment because it’s fast, high quality, and secure. Even better - you don’t have to switch your IDE - Augment works in VS Code, JetBrains, and even Vim. Augment is free to try and never trains on code without consent. Start building for free today.

• IO devices and latency

  Non-volatile storage is a cornerstone of modern computer systems. Every modern photo, email, bank balance, medical record, and other critical pieces of data are kept on digital storage devices, often replicated many times over for added durability. we’re going to cover the history, functionality, and performance of non-volatile storage devices over the history of computing, all using fun and interactive visual elements.

• Zen and the Art of Microcode Hacking

  Google release the full details of EntrySign, the AMD Zen microcode signature validation vulnerability which they initially disclosed on Feb 2025. This vulnerability allows an adversary with local administrator privileges (ring 0 from outside a VM) to load malicious microcode patches. We have demonstrated the ability to craft arbitrary malicious microcode patches on Zen 1 through Zen 4 CPUs

• Going down the rabbit hole of Git's new bundle-uri

  Exploring an unpopular git bundle-uri that can help speed up Git by pre-populate git local object cache from https instead of fetching from git server.

• Dive Into WebGPU — Part 1

  WebGPU is a modern graphics API designed to provide high-performance graphics and computation capabilities across different platforms, including web browsers, desktops, and mobile devices. It is intended to be a successor to the WebGL API, offering more advanced features, better performance, and greater flexibility for developers

• CSS Relative Colors

  CSS is evolving realy fast. This is a useful feature used to cover by CSS pre-processor like SCSS or LESS.

• What Makes Code Hard To Read: Visual Patterns of Complexity

  The question everyone have an answer for. The OP share with us an interesting concept: visual patterns of code, specifically the ones that make their brain hurt!

• Paxos made visual in FizzBee

  Any distributed system you will hear the term Raft or Paxos. Some expert consider Paxos is super simple, some say it’s the most complicated. In this post, the OP modeling Paxos with FizzBee, a design specification language and model checker to specify distributed systems at a much higher level of abstraction than a programming language for system analysis and design. So we will learn 2 things in this post.

• Sorting Algorithms with CUDA!

  Building on a previous post on sorting algorithms, I implemented the same algorithms using CUDA to explore performance improvements through parallel computing. The goal is to see how we can leverage the power of parallel computing to speed up our sorting algorithms. I went for a NVIDIA recruiting event some days ago, that was a great event and it motivated me to try to rewrite the sorting algorithms using CUDA.

• PostgreSQL: 1 trillion rows in Citus

  “PostgreSQL scales” - we have all heard this phrase over and over again. However, the question is: What does this actually mean? Does it mean 1 million rows? Or maybe even 1 billion rows? So, on a rainy weekend, I decided to do a little experiment to figure out if it is possible to squeeze 1 trillion rows (= 1000 billion rows) into my local personal computer

• Getting started with Compilers

  As an engineer we’re all curious how a compilers to made. The bad news is there is no single resource to grasp all of that. The good news are all the resource widely available for us to get started.

• Azure’s Weakest Link? How API Connections Spill Secrets

  Binary Security found the undocumented APIs for Azure API Connections. In this post we examine the inner workings of the Connections allowing us to escalate privileges and read secrets in backend resources for services ranging from Key Vaults, Storage Blobs, Defender ATP, to Enterprise Jira and SalesForce servers.

Code to read

• ruby_llm

  A delightful Ruby way to work with AI. No configuration madness, no complex callbacks, no handler hell – just beautiful, expressive Ruby code.

• filter

  a powerful, web-based image editor built with React and TypeScript. It provides a modern, intuitive interface for quick image edits and filters, optimized for both desktop and mobile devices.

• ⌘K: Fast, composable, unstyled command menu for React.

  If you ever need a short-cut like a spotlight search for your app. You would want this

• ghw

  Go HardWare discovery/inspection library

• shelgon

  a robust Rust framework for building interactive REPL (Read-Eval-Print Loop) applications and custom shells. It provides a flexible, type-safe foundation with built-in terminal UI capabilities using ratatui.

• octotools

  An agentic framework with extensible tools for complex reasoning

• Reticulum

  The cryptography-based networking stack for building unstoppable networks with LoRa, Packet Radio, WiFi and everything in between.

Tools

• PRevent

  Prevent merging of malicious code in pull requests

• agents-json

  The agents.json Specification is an open specification that formally describes contracts for API and agent interactions, built on top of the OpenAPI standar

• shorts_maker

  a Python package designed to facilitate the creation of engaging short videos or social media clips. It leverages a variety of external services and libraries to streamline the process of generating, processing, and uploading short content.

• MotifyX

  Elevate your designs with our curated collection of modern background patterns. Preview, customize, and implement with just a few clicks.

• smart-turn

  an open source, community-driven, native audio turn detection model. Hugging Face model is available

• dicedb

  an open-source, fast, reactive, in-memory database optimized for modern hardware.

• pgdog

  Horizontal scaling for PostgreSQL.

• nubmq

  A blazing-fast KV store written in pure Golang without any dependencies with native pub-sub support, engineered for high-frequency, contention-heavy workloads

• olmocr

  Toolkit for linearizing PDFs for LLM datasets/training

Hi all,

In the wave of ByBit exchange being hack for 1.6billion, and the hack is very sophisciated, exploit developer access key to change an s3 bucket. The attack start by having the developer run some untrusted docker compose with privileged: true I’ll include a few more tools in security, vulnerable scan, and supply chain attack.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Developer AI for real software engineering work

  Lots of coding AIs have cool X demos on greenfield apps. But the day-to-day of a pro software engineer working on a team looks…a little different from vibe coding. Enter Augment Code. The first developer AI built for teams and large codebases, Augment works on codebases of millions of lines of code and thousands of files, bringing full codebase context to every keystroke. Customers like Datastax, Observe, Kong, and Lemonade trust Augment because it’s fast, high quality, and secure. Even better - you don’t have to switch your IDE - Augment works in VS Code, JetBrains, and even Vim. Augment is free to try and never trains on code without consent. Start building for free today.

• Tracking You from a Thousand Miles Away! Turning a Bluetooth Device into an Apple AirTag Without Root Privileges

  Imagine someone could turn your laptop, smartphone, or even your gaming console into a tracking device without your knowledge. Our research team discovered a way this can happen through Apple’s Find My network The Find My network uses over a billion Apple devices worldwide. We found a security problem that lets hackers use this system to track almost any device with Bluetooth capabilities - not just Apple products. We call this attack “nRootTag.”

• The Complete Guide to Cloud-Native Ransomware Protection in Amazon S3 and KMS

  In this post, we’ll cover how to prevent ransomware and provide resources & code for 11 different ransomware prevention use cases to ensure preventative controls are in place to prevent against types of ransomware attacks targeting AWS S3.

• TLS/SSL Certificates - For The Rest Of Us

  The internet is insecure by default. Attackers can intercept and modify traffic, so we need a way to secure communication and verify the server’s identity. Encryption ensures data privacy, but without verification, hackers could impersonate a legitimate website, intercepting traffic while still using encryption. This is why TLS/SSL combines encryption with authentication.

• Kafka Transactions Explained (Twice!)

  When consuming Kafka in a consumer group, the offset each consumer has bit read the message and ack it is recorded by Kafka. When they crashed or restart, they can resume from the commited ack position. But what happen if a record has been processed by consumer but failed to send the ack, the app will consume the record again on restarting. How can we somehow do 2 thing: commit the offset and the result of operation somehow in a single atomic operation. Enter Kafka transactions allow.

• CSS Meets Voxel Art: Building a Rendering Engine with Stacked Grids

  We usually don’t include this tutorial style link but this one is really interesting to see. We will learn how transforms, perspective, and stacked grids can create a fully addressable 3D space and push the boundaries of what’s possible with pure CSS

• Crossy Road with React Three Fiber

  In this tutorial, we learn how to create a simplified clone of the mobile game Crossy Road with React Three Fiber. The goal of the game is to move a character through an endless path of static and moving obstacles. We have to go around the trees and avoid getting hit by a car or a truck. We start with the basic setup: setting up the scene, camera, and lighting. We learn how to draw the player and the map’s trees, cars, and trucks. We cover how to animate the vehicles and add event handlers to move the player through the map. Finally, we add logic to detect collisions between the player and the cars.

• Representing graphs in Postgresql

  A very command problem when we need to re-present a relationship in databsae such as family, friend. Can come in very handy to write a single SQL query to find relationship instead of building them from the app level

• XOR explanation

  XOR is every where. We use them in Linux permission: umask, chmod. We use them in Cryptography. We will learn about their hardware implementation, about their usage and all kind of trick to use them efficiently.

• Writing an LLM from scratch, part 8 -- trainable self-attention

  Write up when following “Build a large language model from scratch” book.

• How I Automated My Podcast Transcript Production With Local AI

  Den has been running podcast for half a decade. He use WhistpeX, run it locally and build a workflow to transcibe his podcast to text. He shared with us the setup. I think local ai is really useful and worth to explorer the self hosted path. It is much easiser to self hosted than we think even

• Tips to debug hanging Go program

  Go specific but come in very handy when working with Go app.

• Why fastDOOM is fast

  FastDOOM was a port of DOOM but it is 30% faster. On some complicated map, it’s even 48% faster. We will dive into the secret sauce of the technique that is used to optimized it. very low level thing like use the right assembly instruction to make code run in fewer instructions.

• Performance optimization, and how to do it wrong

  I recently tried to optimize convolutions using SIMD instructions, but what I thought would be a simple task ended up taking me days, with issue after issue popping up one after another. Some of them make sense in hindsight, but others were utterly baffling. While the specific examples are for direct convolution, these considerations apply to pretty much any code with a hot loop.

Code to read

• orama

  A complete search engine and RAG pipeline in your browser, server or edge network with support for full-text, vector, and hybrid search in less than 2kb.

• pdfslick

  a library which enables viewing of and interaction with PDF documents in React and SolidJS apps. It’s build on top of Mozilla’s PDF.js, and utilises Zustand to provide a reactive store for the loaded documents.

• zap: blazingly fast backends in zig

  I recently pickup Zig and I think this is going to replace Rust and C for me when I needed them. It has all the power of Rust while less restricted, and have power of C while being more friendly than C.

• rayhunter

  A tool to detect cell site simulators on an Orbic mobile hotspot

• phony

  E164 international phone number normalizing, splitting, formatting.

• gore

  Yet another Go REPL that works nicely. Featured with line editing, code completion, and more.

• PRevent

  Prevent merging of malicious code in pull requests

Tools

• Peerdb: PostGres Replication Tool

  If you ever need to keep postgres in-sync with another datawarehouse, or even another postgres, this is the way to go.

• mlx-audio

  A text-to-speech (TTS) and Speech-to-Speech (STS) library built on Apple’s MLX framework, providing efficient speech synthesis on Apple Silicon

• sqldef

  Idempotent schema management for MySQL, PostgreSQL, and more

• rogue

  An intelligent web vulnerability scanner agent powered by Large Language Models

• pphack

  The Most Advanced Client-Side Prototype Pollution Scanner

• FerretDB

  A truly Open Source MongoDB alternative

• kunai

  powerful tool designed to bring actionable insights for tasks such as security monitoring and threat hunting on Linux systems. Think of it as the Linux counterpart to Sysmon on Windows, tailored for comprehensive and precise event monitoring.

• Postgres query plan visualization tools

Hi all,

Welcome to second issue of BetterDev. I’m lagging behind a few issues but catching up. There are many things happens with LLM/AI recently. I encourage to give these tools a try, on your own hardware.

As a reader of this newsletter, you probably has the itch to build some app all the time. My friend at lapa.ninja has been curating almost 7000 landing pages, all hand pick. So you can get the inspiration next time when you want to build a landing page or a dashboard for your app.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• How Core Git Developers Configure Git

  In this post, I’ll go through some of the perhaps obscure Git config settings that I have personally globally enabled and go into them to explain what they do and why they should probably be the default settings.

• Hacking chromecast

  Do you want to be able to discover chromecast device on your network and broadcast arbitrary media to it? Sure, there is many tools to do it. But what is the protocol behind it. This is a 4 part series dive into that. We will learn the basic, the protocol, the casting server and gotcha.

• Minecraft from scratch

  A Minecraft clone from scratch with only modern OpenGL. No game engines or frameworks. Using Go so it’s very h igh level and easy to follow.

• What would happen if we didn't use TCP or UDP?

  Switches, bridges, routers, load balancers, firewalls—these network boxes keep the internet running. Routing, blocking, mirroring, duplicating and deduplicating traffic in ways most people never think about. Without them, this document wouldn’t have reached you

• Build your own SQLite, Part 5: Evaluating queries

  We had been featuring this series in previos issue. In this part we will learn how to evaluate a SQL query. If you are interesting of understanding deep into your stack, these article provide great approach into learning and rebuilding them.

• Performance Speed Limits

  Sometimes you just want to know how fast your code can go, without benchmarking it. Sometimes you have benchmarked it and want to know how close you are to the maximum speed. Often you just need to know what the current limiting factor is, to guide your optimization decisions.

• Scaling with PostgreSQL without boiling the ocean

  Scaling PostgreSQL successfully doesn’t always require a full team of DBAs and experts. The beauty of PostgreSQL is that solutions often lie within the database itself - by rethinking your data access patterns from first principles, you can solve many business problems at scale.

• Running Pong in 240 Browser Tabs

  What do you do with your unclosed browser tabs? I find that they take up a lot of screen space. So this week I figured out how to run pong inside mine.

• Video streaming using http 206 partial content

  Implementing video streaming might be a pain for some cases, especially when we talk about memory management alongside scalability of the application. One common way is using HTTP Live Streaming Protocol where a single video could be broken down into smaller chunks and sent through multiple HTTP requests. But then we need to transcode video ahead of time. Can we stream MP4 file? has it playable asap instead of downloading whole video? Turn out we can thank to the modern browser support of byte “Rang: byte” header.

• Multi-Hop WireGuard

  WireGuard is a very powerful VPN which support multiple topology and routing. Can come in very handy when you want to perform complex routing, especially in environment where you want to control egress. Give this a read to learn about these setups: Site Gateway as a Spoke, Internet Gateway as a Spoke, Chain of Hubs, Hub is Also a Site Gateway.

• The power of interning: making a time series database 2000x smaller in Rust

  interning is a technique of storing only one copy of each unique piece of data in memory.Very common use in string so we don’t need to store the string multiple time. Imagine big JSON document with repeated field. In this post we look into a time series database and how interning help with different data type.

• 3,200% CPU Utilization

  A tale into debugging a Java program to see why it use all 32 cores.

Code to read

• micro_events.py

  Micro event loop library to teach the basic concepts of python coroutines and how event loop libraries might be implemented

• opendbc

  Python API for your car. Control the gas, brake, steering, and more. Read the speed, steering angle, and more.

• authie

  Improve user session security in Ruby on Rails applications with database session storage

• bsky-parser

  Bluesky text parser that outputs parsed text with rich text facets

• human-regex

  Human-friendly regular expression builder with English-like syntax.

• re2js

  the JavaScript port of RE2, a regular expression engine that provides linear time matching

• nativewebp

  This is a native WebP encoder written entirely in Go, with no dependencies on libwebp or other external libraries. Designed for performance and efficiency, this encoder generates smaller files than the standard Go PNG encoder and is approximately 50% faster in execution

Tools

• cartography

  a Python tool that consolidates infrastructure assets and the relationships between them in an intuitive graph view powered by a Neo4j database.

• gochromecast

  A simple tool to stream your media to android TV or chromecast device.

• Waveform Renderer

  A lightweight and customizable TypeScript library for rendering audio waveforms on HTML canvas. Create beautiful, interactive audio visualizations with ease.

• rails-diff

  Compare Rails-generated files with the ones in your repository

• Cap.So

  Open source Loom alternative. Beautiful, shareable screen recordings.

• stagehand

  An AI web browsing framework focused on simplicity and extensibility.

• BemiDB

  Postgres read replica optimized for analytics. It consists of a single binary that seamlessly connects to a Postgres database, replicates the data in a compressed columnar format, and allows you to run complex queries using its Postgres-compatible analytical query engine.

• pgwatch

  PostgreSQL metrics monitor/dashboard

Hi all,

Happy new year. Welcome to the first issue of 2025. I’m trying to become more regular this year. Looking forward to a new year and hope everyone continue to build cool thing. For me, I was unable to launch my another SaaS last year. I did overhault my cloudcost https://ec2.shop and much more happy with its state now.

This week our sponsor is WarpStream, they have build an amazing Kafka alternative while 100% compatible with Kafka and way more cheaper. If you are running Kafka in Kubernetes, or looking to replace Redis with Kafka, like Honeybadger team did, then definetely give WarpStream a try.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Pixel Federation Swaps Amazon MSK for WarpStream, Saves 83% on Apache Kafka Costs

  The video game company Pixel Federation uses Kafka as the message bus to power its real-time data streaming infrastructure. However, they were leveraging Amazon MSK and their bill was growing faster than their data volumes and they had to use complex networking like VPC peering. By switching to WarpStream, they saved 83% and ditched VPC peering and manual partition rebalancing

• Operating System in 1,000 Lines

  build a small operating system from scratch, step by step. You might get intimidated when you hear OS or kernel development, the basic functions of an OS (especially the kernel) are surprisingly simple. Even Linux, which is often cited as a huge open-source software, was only 8,413 lines in version 0.01. Today’s Linux kernel is overwhelmingly large, but it started with a tiny codebase, just like your hobby project. We’ll implement basic context switching, paging, user mode, a command-line shell, a disk device driver, and file read/write operations in C. Sounds like a lot, however, it’s only 1,000 lines of code!

• A 2-ply minimax chess engine in 84,688 regular expressions

  So without further ado, I present to you … Regex Chess: sequence of 84,688 regular expressions that, when executed in order, will play a (valid; not entirely terrible) move given a chess board as input.

• Mistakes engineers make in large established codebases

  Working in large established codebases is one of the hardest things to learn as a software engineer. You can’t practice it beforehand (no, open source does not give you the same experience). Personal projects can never teach you how to do it, because they’re necessarily small and from-scratch.

• Seconds Since the Epoch

  People, myself included, like to say that POSIX time, also known as Unix time, is the number of seconds since the Unix epoch, which was 1970-01-01 at 00:00:00. This is not true. Or rather, it isn’t true in the sense most people think. For example, it is presently 2024-12-25 at 18:51:26 UTC. The POSIX time is 1735152686. It has been 1735152713 seconds since the POSIX epoch. The POSIX time number is twenty-seven seconds lower.

• How NAT traversal works

  How 2 devices under a NAT can connect directly to each other without a server beind in the middle to relay traffic?

• Linux Routing Fundamentals

  Routing table define what what happens when an IP packet is sent from or through a Linux box. But the simplify is which destination a package should send to? how do your computer know that a LAN ip need to send to othe laptop, and the rest go out to internet?

• Writing a simple pool allocator in C

  When you need memory for your program, you can malloc. Similarly to malloc, a pool allocator allows the user to allocate memory at run time. The pool allocator, however, is much faster than malloc1, at the cost of having a fixed pool size. It allows the user to allocate and free memory blocks (referred to as chunks, from now on) in O(1) constant time. This implementation also uses very little memory: when creating the pool, a very small Pool structure is allocated, along with the pool itself. Free chunks are used to store information, so the memory impact is minimal.

• Behind the scenes with Stream Live, Cloudflare’s live streaming service

  Stream Live accepts audio/video content from broadcasters and makes that content available to viewers around the world in real time through the Cloudflare network. This post look into how they build broadcast ingestion with RTMPS, SRT, WHIP and Spectrum. The data is converted and store in their Durable Object, then serve to viewer through HLS or DASH protocol.

• B-Trees: More Than I Thought I'd Want to Know

  B-Trees were essentially “better” Binary Search Trees, with some hand-waving done that they had improved performance when used in database applications. But it’s more than just that, today we will look into disk constraint, slotted pages to see how B-Tree node fit naturally into a page - a chunk of data that the harddrive read all in the same time.

• SQL NULLs are Weird

• Breaking Up with Long Tasks or: how I learned to group loops and wield the yield

  Arrays are in every web developer’s toolbox, and there are a dozen ways to iterate over them. Choose wrong, though, and all of that processing time will happen synchronously in one long, blocking task. The thing is, the most natural ways are the wrong ways.

Code to read

• rumale

  a machine learning library in Ruby. Rumale provides machine learning algorithms with interfaces similar to Scikit-Learn in Python.

• go-haystack

  Track personal Bluetooth devices via Apple’s “Find My” network using OpenHaystack and Macless-Haystack with tools written in Go/TinyGo. No Apple hardware required!

• postal-mime

  Email parser for browser and serverless environments

• MegaParse

  File Parser optimised for LLM Ingestion with no loss 🧠 Parse PDFs, Docx, PPTx in a format that is ideal for LLMs.

• @smoores/epub

  A Node.js library for inspecting, modifying, and creating EPUB 3 publications.

Tools

• ggh

  Recall your SSH sessions (also search your SSH config file)

• termo

  a simple terminal emulator that can be used to create a terminal-like interface on your website

• wasm-vips

  libvips for the browser and Node.js, compiled to WebAssembly with Emscripten.

• recur

  Retry a command with exponential backoff and jitter (+ Starlark expressions)

• vanir

  a source code-based static analysis tool that automatically identifies the list of missing security patches in the target system. By default, Vanir pulls up-to-date CVEs from Open Source Vulnerabilities (OSV) together with their corresponding signatures so that users can transparently scan missing patches for an up-to-date list of CVEs.

• OpenGFW

  OpenGFW is a flexible, easy-to-use, open source implementation of GFW (Great Firewall of China) on Linux

• Pig: The Postgres Extension Wizard

  Ever wished installing or upgrading PostgreSQL extensions didn’t feel like digging through outdated readmes, cryptic configure scripts, or random GitHub forks & patches? The painful truth is that Postgres’s richness of extension often comes at the cost of complicated setups—especially if you’re juggling multiple distros or CPU architectures. Enter Pig, a Go-based package manager built to tame Postgres and its ecosystem of 340+ extensions in one fell swoop.

• pgspot

  Spot vulnerabilities in postgres SQL scripts

Hi all,

It’s chrismast time. Time for led lighting project. This year I came across this WLED project, a fast and feature-rich implementation of an ESP8266/ESP32 webserver to control NeoPixel (WS2812B, WS2811, SK6812) LEDs or also SPI based chipsets like the WS2801 and APA102! They had all the resouces from hardwares to softwares, Web UI,mobile app to control your leds.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Stop worrying about your to-do list.

  Zapier connects the apps you use every day, so you can focus on what matters most. Start working more efficiently - Create your free account today.

• Compromising OpenWrt Supply Chain via Truncated SHA-256 Collision and Command Injection

  BetterDev usually doesn’t include these kind of vulnerability report. However, this one is very interesting because it happen through using a cache key as a truncate hash of user input data, lead to collision. And caching is common particularlly in web dev, picking right cache key is important to avoid cache poisoning.

• Applied offensive security with Rust

  it’s a from-theory-to-practice guide and you may enjoy it if you are a developer and want to learn security

• Breaking the 4Chan CAPTCHA

  This project was entered into as a learning experience, to enhance my knowledge of machine learning, as well as TensorFlow specifically. At the end, I wanted to have a trained machine learning model that runs in the browser to reliably (at least 80% accuracy, >90% preferred) solve the 4Chan CAPTCHA. These goals were achieved - let’s talk about how I got there!

• Demystifying git submodules

  Throughout my career, I have found git submodules to be a pain. Because I did not understand them, I kept getting myself into frustrating situations. So, I finally sat down and learned how git tracks submodules. Turns out, it’s not complex at all. It’s just different from how git tracks regular files. It’s just one more thing you have to learn.

• Diving into eBPF: Building a Process Tracer from Scratch

  Today i want to share a story about how i ended up writing a simple process tracer for linux. Using eBPF in go to fix a github actions which i actually didn’t need. We will go over each piece and hopefully you will learn something form it.

• Writing down (and searching through) every UUID

  Most of time you see UUID re-present as text based, but it’s actually a 128 bit integer. The OP makes everyuuid.com to display both the number and text based of every single UUID. They share the challenge of rendering, ordering an searching.

• PostgreSQL for everything

  How to reduce complexity and move faster? Just Postgres for everything.

• What you need to know about SQLite

  A compilation of lessons about what the SQLite database engine can and cannot do. It’s written for Rails, but you can apply all the lessons to use SQLite generally.

• Deletes are difficult

  We tend to optimize db for insert and select, and overlook delete. At first glance, the DELETE command seems straightforward. But more ofthen than not on production the delete break down. We’ll uncover why DELETE operations demand careful consideration and explore how to handle them effectively

• The Case for Shared Storage

  If you operate Kafka, you know how important the storage layer is. In K8S, when a pod went down, and come back up, we need o make sure the storage is mounted to the same node, lead to slowness of pod because it needs to detach and re-attach volume. WarpStream is essentially serverless Kafka. They aim to solve this storage layer. In this post, they share with us why they choose to use a shared storage system.

• Using Transformers.js for AI in the Browser

  A practical journey to use transformersjs.js to perform ML task directly in browser.

• How To Improve Interaction-to-Next-Paint : React

Code to read

• go-taglib

  This project is a Go library for reading and writing audio metadata tags. It provides a portable solution with no external dependencies required, thanks to an embedded WASM binary. The most interesting part is WASM. So there is no CGO invoke at all even though taglib is a C++ codebase. A very interesting approach

• music-metadata

  Stream and file based music metadata parser for node. Supporting a wide range of audio and tag formats.

• pg_query

  Ruby extension to parse, deparse and normalize SQL queries using the PostgreSQL query parser

• halloy

  IRC application written in Rust

• htmy

  Async, pure-Python rendering engine.

Tools

• quill

  Simple mac binary signing and notarization from any platform (replacing the codesign utility for simple use cases).

• studio

  A lightweight Database GUI in your browser. It supports connecting to Postgres, MySQL, and SQLite.

• fastmcp

  Model Context Protocol (MCP) servers are a new, standardized way to provide context and tools to your LLMs, and FastMCP makes building MCP servers simple and intuitive. Create tools, expose resources, and define prompts with clean, Pythonic code:

• tonic

  A rust implementation of gRPC, a high performance, open source, general RPC framework that puts mobile and HTTP/2 first.

• kaluma

  A tiny and efficient JavaScript runtime for RP2040 (Raspberry Pi Pico).

• bunbuster

  Ridiculously fast web & TCP fuzzer designed for brute-forcing directories, subdomains, and files on web servers.

Hi all,

Welcome to thanksgiving issue of BetterDev. Hope everyone had a safe and warm thanksgiving. It’s getting so cold these days. If you are in warzone such as Ukraina or Gaza, I really hope thing will become better.

AI field has been evolve quite fast this month with a lot of amazing tooling. I added many of them in self-hosted section to help you try them out.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Memory: The Forgotten History

  This article explores the historical development of memory in computing, from the early innovations in storage to the evolution of modern memory technologies. It reflects on the forgotten milestones in memory history and their impact on current computing.

• Why did Windows 95 setup use three operating systems?

  why Windows 95 setup goes through three operating systems: MS-DOS, Windows 3.1, and then Windows 95. Why not go from MS-DOS straight to Windows 95?

• A Day in the Life: The Global BGP Table

  Border Gateway Protocol (BGP) is the routing protocol for the Internet. Much like the post office processing mail, BGP picks the most efficient routes for delivering Internet traffic. In this post, we’re exploring the intra-day shenanigans with an eye to finding some of the ridiculous things that go on out.

• against /tmp

  Why using /tmp to store file is a usually a bad idea.

• How we prevent conflicts in authoritative DNS configuration using formal verification

  Learn how CloudFlare use formal verification to mathematically prove properties about DNS addressing behavior, even when different systems (owned by different teams) at Cloudflare have contradictory views on which IP addresses should be returned.

• How DRAM Changed the World

  This blog post from Micron explores the transformative role of DRAM (Dynamic Random Access Memory) in the evolution of technology. It highlights DRAM’s impact on everything from personal computing to cloud infrastructures and its continued importance in the development of modern devices.

• Best Practices for Bulk Optimization of Queries in PostgreSQL

  After reading this article, you’ll learn about the techniques we use to analyze SQL query performance when dealing with millions of queries per day and monitoring hundreds of PostgreSQL servers.

• Loading the World! OpenStreetMap Import In Under 4 Hours

  The OpenStreetMap (OSM) database builds almost 750GB of location data from a single file download. OSM notoriously takes a full day to run. A fresh open street map load involves both a massive write process and large index builds. It is a great performance stress-test bulk load for any Postgres system. I use it to stress the latest PostgreSQL versions and state-of-the-art hardware. The stress test validates new tuning tricks and identifies performance regressions.

• Build your own personal SIRI with LLAMA-3 like a PRO!

  In this easy-to-follow tutorial, you will learn how to build your own voice assistant Siri with the LLAMA-3 AI Model.

• SRE deep dive into Linux Page Cache

  In this series of articles, I would like to talk about Linux Page Cache. I believe that the following knowledge of the theory and tools is essential and crucial for every SRE. This understanding can help both in usual and routine everyday DevOps-like tasks and in emergency debugging and firefighting. Page Cache is often left unattended, and its better understanding leads to the following:

• Tiny GraphRAG (Part 1)

  We’re going to build a tiny 1000 line implementation of a GraphRAG algorithm originally invented by Microsoft. I consistently hear people talk about this algorithm at meetups, but it appears there are several orders of magnitude of people talking about it than actually using it or implementing it. Likely because the reference implementation is enormous and rather complex. So let’s break it down and see if there’s any merit to the hype around this approach.

• Boosting Postgres INSERT Performance by 2x With UNNEST

• Why TCP needs 3 way handshakes

Code to read

• nokolexbor

  High-performance HTML5 parser for Ruby based on Lexbor, with support for both CSS selectors and XPath. A drop-in replacement for Nokogiri.

• mittsu

  Mittsu is a 3D Graphics Library for Ruby, based heavily on Three.js

• rill

  Go toolkit for clean, composable, channel-based concurrency

• jsep

  a simple expression parser written in JavaScript. It can parse JavaScript expressions but not operations. The difference between expressions and operations is akin to the difference between a cell in an Excel spreadsheet vs. a proper JavaScript program.

• micromark

  small, safe, and great commonmark (optionally gfm) compliant markdown parser

• datamapplot

  Creating beautiful plots of data maps

• pglite-fusion

  Embed an SQLite database in your PostgreSQL table. AKA multitenancy has been solved.

Tools

• PoWA

  performance tool compatible with all PostgreSQL versions (down to 9.4) allowing to collect, aggregate and purge statistics gathered from multiple PostgreSQL instances from various Stats Extensions.

• superfile

  Pretty fancy and modern terminal file manager

• LTESniffer

  An Open-source LTE Downlink/Uplink Eavesdropper

• memsparkline

  Track the RAM usage of a process and its descendants in real time

• ovault: Secure Vault for OAuth Credentials

  Ovault is an open-source tool designed to securely store OAuth credentials for applications. It offers encrypted storage, helping developers manage sensitive tokens without compromising security.

• skyvern

  Automate browser-based workflows with LLMs and Computer Vision

• lexical

  An extensible text editor framework that does things differently. At somepoint you would definetely need a JS text editor, having another option is great.

Hi all,

Welcome to another issue of BetterDev! This week I come across Colmi, a smart ring where you can write your own software to interact with it. It’s also have a $12.51 deal on AliExpress so very affordable to toy around with hardware.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• WarpStream Makes Apache Kafka Simpler and More Affordable

  WarpStream is a drop-in replacement for Apache Kafka that has no interzone networking fees, no disks to manage and requires zero cross-account IAM access, so raw data never leaves your environment. You’ll never again have to do things like partition or broker rebalancing, deal with snapshot replication issues or worry about over-provisioning, as auto-scaling is automatic and you’re always right-sized. Join customers that have saved over 80% by replacing self-hosted Kafka and MSK with WarpStream. Sign up for a free WarpStream account and get $400 in credits that never expire.

• Should We Chat, Too? Security Analysis of WeChat’s MMTLS Encryption Protocol

  The first public analysis of the security and privacy properties of MMTLS, the main network protocol used by WeChat, an app with over one billion monthly active users. While they were unable to develop an attack to completely defeat WeChat’s encryption, the implementation is inconsistent with the level of cryptography you would expect in an app used by a billion users, such as its use of deterministic IVs and lack of forward secrecy.

• Open source python client to read your data from the Colmi R02 family of Smart Rings

  Colmi is a cheap (as in $20) “smart ring” / fitness wearable that includes the following sensors: Accelerometer, sleep tracking, gestures, heart rate and blood oxygen. The coolest thing is you can write your own client to interact with it through bluetooth.

• solar-powered and self-hosted website

  How cool it’s to setup a website run on solar powered at home? Follow this journey.

• Upgrading Uber’s MySQL Fleet to version 8.0

  Uber upgraded their databae from 5.7 to 8.0. If you had use Uber app, you can use the app is no joke. Routing driver, provide real time upgrade etc. A very complicated app. The strategy that they used to upgraded it is worth a read for us. One important point is not being able to rollback once a v8.0 node is promoted to primary. There is risk and they careful testing to accept that risk.

• Sensible SQLite defaults

  SQLite got a lot of attention recently. If you ever try to use it for some high load you most likely disappointed at its performance. In this post we will look at a few sensible default to help that.

• PostgreSQL Streaming Replication (WAL); What It Is And How To Configure One

• Dealing with trigger recursion in PostgreSQL

  Many a beginner falls into the trap of trigger recursion at some point. Usually, the solution is to avoid recursion at all. But for some use cases, you may have to handle trigger recursion. This article tells you what you need to know about the topic. If you were ever troubled by the error message “stack depth limit exceeded”, here is the solution.

• Schema changes and the Postgres lock queue

  There are two classes of breakage that can occur when applying database migrations: Migrations that make incompatible changes to the schema, breaking client applications, Migrations that lock a database object for an unacceptable amount of time, causing the application to become unavailable as reads and writes start to fail. Today we’re going to talk about the second type of breakage: how long running queries together with DDL statements can lock out reads and writes from a table, causing application downtime.

• Understanding DNS resolution on Linux and Kubernetes

  Resolve DNS on k8s is a bit messy. For convenience there is a few way to hit a servie with just a name, a name and namespace or the cluster domain. What is the rule there?

• Fixing a DNS leak in OpenVPN setup

  After Turkey banned Discord, I had to jump through some hoops, fix my VPN, and learn a bit about how DNS works. Today I’m here to share what I have learned while trying to… you know. Find a way to use Discord again. Surprisingly, this ban ended up being a positive experience for me.

• Chat with your PDF using Pinata,OpenAI and Streamlit

  In this tutorial, we’ll build a simple chat interface that allows users to upload a PDF, retrieve its content using OpenAI’s API, and display the responses in a chat-like interface using Streamlit. W

Code to read

• denko

  Electronics programming in Ruby

• ratatui

  A Rust crate for cooking up terminal user interfaces (TUIs)

• go-blueprint

  allows users to spin up a quick Go project using a popular framework

Tools

• Keyguard

  With new of Bitwarden moving to close source, this is an alternative client for the Bitwarden® platform, created to provide the best user experience possible.

• openvmm

  a modular, cross-platform, general-purpose Virtual Machine Monitor (VMM), written in Rust.

• trippy

  Trippy combines the functionality of traceroute and ping and is designed to assist with the analysis of networking issues.

• pgroonga

  a PostgreSQL extension to use Groonga as index. PGroonga makes PostgreSQL fast full text search platform for all languages!

• pg_parquet

  If you ever want a way to export and load parquet file like how BigQuery does it, this finally happens for PostgreSQL.

• julep

  A new DSL and server for AI agents and multi-step tasks

Hi all,

Welcome to another issue of BetterDev! I’ve been exploring LLMs more and, while they’re not perfect or likely to replace programming jobs, they’re great for pattern recognition and repetitive tasks. I see LLMs becoming as common as email, with most people using providers and a few hosting their own. That’s why I’m sharing more LLM content, focusing on first principles to help programmers understand and build them from scratch.

Don’t worry, this newsletter isn’t turning into AI “hype.” I simply want to share useful tools and insights with our audience

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Seamlessly Integrate Product Usage Insights with Your Customer Data

  Unlock unparalleled insights with the improved Amplitude app for HubSpot. This integration empowers you to seamlessly sync user data and behavior insights, ensuring real-time analytics at your fingertips. Elevate your customer engagement with advanced segmentation, allowing marketers to identify product qualified leads. Try the app and start optimizing your strategies today

• LLM from scratch with Pytorch

  Have you ever feel overwhelm with all the news about LLM? what is these 70B 4B or 3.5 4.0? Or when you go and check out the code of some project, it’s super short and simple, where is the logic? In this link, we go back to that root to understand how to build an LLM by using Pytorch.

• Started a guide to writing FUSE filesystems in Python

  Have you ever feel like iCloud drive, Google Drive or Dropbox app is magic? What power them? It’s pretty much FUSE behind the scene. Can we turn random data available through RPC into a file system?

• End-to-End Encrypted Cloud Storage in the Wild: A Broken Ecosystem

  Cloud storage is ubiquitous: Google Drive, Dropbox, and OneDrive are household names. However, these services do not provide end-to-end encryption (E2EE), meaning that the provider has access to the data stored on their servers. The promise of end-to-end encrypted cloud storage is that users can have the best of both worlds, keeping control of their data using cryptographic techniques, while still benefiting from low-cost storage solutions.

• Crafting qr codes

  QRCode is everywhere nowsaday. However, they doesn’t need to be just square black and white box. How can we generate QRcode that looks artsy while still readable to the decoder?

• From bare metal to a 70B model: infrastructure set-up and scripts

  With a small team and in a few monthes, imblue trained a 70B parameter model from scratch on their own infrastructure that outperformed zero-shot GPT-4o on reasoning-related tasks. They’re sharing an end-to-end guide for setting up the required infrastructure: from bringing up the initial cluster and installing the OS, to automatically recovering from errors encountered during training.

• Optimizing Postgres table layout for maximum efficiency

  what if I told you that simply reordering your columns could reduce the size of your tables and indexes by 20%? This isn’t some obscure database trick — it’s a direct result of how Postgres aligns data on disk.

• How ngrok build their data platform

  This article is a look at how ngrok, a platform to enable you expose local service on your laptop to internet, built their data platform. Given their team size of 1 engineer, this maybe more useful and relevant to us more than scale of big corp(where there are hundred engineers building a data platform).

• Confusing or misunderstood topics in systems programming

  Threads and the general concept of concurrency are among the most widely misunderstood and confusing topics for newer developers. This series is meant to help clarify some of these concepts and provide a better understanding of how they work.

• Why GOV.UK’s Exit this Page component doesn’t use the Escape key

• The Four Horsemen of network communication

• Revisiting the DOS memory models

• Pushy to the Limit: Evolving Netflix’s WebSocket proxy for the future

  a very high level discussion of how Netflix improve and evolve their websocket infrastructure to perform many kind of tasks. There isn’t much technical discussion on this post, but more about concept and system design.

Code to read

• neighbor

  Nearest neighbor search for Rails.

• qrframe

  code-based qr code generator.

• exiftool-vendored.js

  Fast, cross-platform Node.js access to ExifTool.

• fuqr

  an awesome qr code generator in Rust. Go beyond the normal QR Code, it can generate animation QR code as well. By the same author of above qrframe package.

• lm.rs

  Minimal LLM inference in Rust

• run-llama-locally

  Running Llama locally with minimal dependencies

• rose

  A music manager with a virtual filesystem.

• dito

  an advanced Layer 7 reverse proxy server written in Go

Tools

• player.style

  Video and audio player themes built with Media Chrome, for every web player and every web app framework.

• duckstation

  is an simulator/emulator of the Sony PlayStation™ console, focusing on playability, speed, and long-term maintainability. The goal is to be as accurate as possible while maintaining performance suitable for low-end devices. “Hack” options are discouraged, the default configuration should support all playable games with only some of the enhancements having compatibility issues.

• auto-southwest-check-in

  A Python script that automatically checks in to your Southwest flight 24 hours beforehand.

• DeskPad

  Certain workflows require sharing the entire screen (usually due to switching through multiple applications), but if the presenter has a much larger display than the audience it can be hard to see what is happening. DeskPad creates a virtual display that is mirrored within its application window so that you can create a dedicated, easily shareable workspace.

• qrbtf

  AI & parametric QR code generator. View the actualy deployment on https://qrbtf.com

• sequin

  Add streaming capabilities to Postgres.

• gitbutler

  A Git GUI branch management tool. Support Linux/Mac/Window. Build with Tauri/Rust/TypeScript so it won’t be as slow as Electron. If you’re a visual person, give it a try.

• rsql

  Command line interface for DuckDB, LibSQL, MariaDB, MySQL, PostgreSQL, Redshift, Snowflake, SQLite3 and SQL Server

Hi all,

Welcome to another issue of BetterDev. This week PostgreSQL 17 is released. It has a lot of amazing feature. Time to update and look over the release note. If you are a Postgres user, consider subscribe to Postgres Weekly as well

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Not too many days left for DevSecCon 2024

  Developing AI Trust, hosted by Snyk on October 8-9. With 20+ sessions and 5 themed tracks, DevSecCon 2024 is packed with DevSecOps lessons and hands-on experiences from industry trailblazers. For all our developers, here’s one that you just can’t miss: Leonardo Zanivan from Okta, who will be discussing how to Secure Node.js Applications from Supply Chain Attacks. Register today!

• Compiling to Assembly from Scratch

  Have you been trying to learn how compilers and programming languages work? Then come along! Let’s make a compiler that goes all the way from source to assembly from scratch—no shortcuts This book will teach you enough compiler theory and assembly programming to get going. It uses a subset of TypeScript that reads like pseudocode and targets ARM 32-bit instruction set.

• A Reintroduction to Programming

  Imagion composing a BMP image by hand, or writing a executable file manually? What kind of knowledge need to do that? Mastery of computer programming unlocks power, flexibility, speed, and debugging prowess across all of your work. We will learn in depth what you use every day, by exploring memory, instructions, syscalls, functions, structure and cognition.

• Using YouTube to steal your files

  In my security research I often come across weird quirks and behaviours that aren’t particularly useful beyond a neat party trick. It’s always a good idea to keep track of them though, perhaps one day they’ll be just the missing piece you need.

• Building an image search engine on Postgres

  Have you ever want a search to find similar image? In this blog we’ll build a basic image search engine using Postgres. We’ll use a pre-trained model to generate embeddings for images and text, then store those embeddings in Postgres. The pgvector extension will enable us to conduct similarity searches on these embeddings using both images and raw-text as queries.

• Production RAG with a Postgres Vector Store and Open-Source Models

  Part 6 of an AI Engineering open-source models tutorial series. We focus on RAG on this article. A RAG pipeline is what allows your AI model(s) to leverage the knowledge of your private/corporate data in its inference. It consists of the following stages: loading, indexing, storing, querying and evaluation.

• Don't Sleep on AbortController

  In JavaScript, more often that not you may want to cancel a request? Imagine an auto-completed, when user type more, you may want to cancel the previous request, there is no point to get them. AbortController is a global class in JavaScript that you can use to abort these fetch request. And also anything else, not just http request.

• Hierarchical data types

  Parent-child relationship happens a lot in context of web dev. Even harder when it’s nested. Example, nested comment. In this shortarticle we look at the ltree Postgres extension to store and retreive that data type.

• Monitor your local and real-user Core Web Vitals performance in DevTools

  This post is part of an ongoing series about Chrome’s efforts to improve the DevTools Performance panel. Learn about these tools to help analyze performance of our front-end app. In this post we’ll take a closer look at each of the new features: Real-time local Core Web Vitals performance, Real-user experience data, Recommendations to configure your local environment, Information to help you reproduce issues.

• How Discord Reduced Websocket Traffic by 40%

  Websocket bandwidth is especially chatty for Discord because they need to broadcast the message to all connection client. Also, decreasing bandwidth usage would lead to a more responsive experience.

• Typescript Monorepo with NPM workspace

• The Ultimate Guide to PostgreSQL Data Change Tracking

Code to read

• pagoda

  Rapid, easy full-stack web development starter kit in Go

• golang-set

  A simple, battle-tested and generic set type for the Go language. Trusted by Docker, 1Password, Ethereum and Hashicorp.

• ruby-pg

  Ruby PostgreSQL database performance insights. Locks, index usage, buffer cache hit ratios, vacuum stats and more. Also, if you’re a Rails dev, checkout the rails-pg-extras

• harper

  The Grammar Checker for Developers. Can be think of your own Grammarly replacement.

• weather_landscape

  Visualizing Weather Forecasts Through Landscape Imagery. Traditional weather stations often display sensor readings as raw numerical data. Navigating these dashboards can be overwhelming and stressful, as it requires significant effort to locate, interpret, and visualize specific parameters effectively.

Tools

• go-jet

  Type safe SQL builder with code generation and automatic query result data mapping

• revive

  🔥 ~6x faster, stricter, configurable, extensible, and beautiful drop-in replacement for golint. Use by a lot of other Go project to lint their Go code.

• portr

  Open source ngrok alternative designed for teams. Tunnel http, tcp or websocket connections.

• dalibo

  Visualizing and understanding PostgreSQL EXPLAIN plans made easy.

• bytebase

  Have you ever feel tedious and risky when you acquire prod db credential, to run some kind of data modification query? Or feel wrong when giving someone access to a postgres db user and have to follow up when revoke/rotate password? Bytebase might solve these pain points. A single place to track change, query db, manage access. Bill itself World’s most advanced database DevOps and CI/CD for Developer, DBA and Platform Engineering teams.

• container-desktop

  Podman desktop companion

• schedule-x

  an event calendar focused on covering the needs of modern web apps: responsive design, internationalization, and extensibility. You can ship it with default settings in a matter of minutes. Surely come in handy when you need some sort of calendar-ish displaying for your app

Hi all,

Welcome to another issue of BetterDev. This week we will learn about some crypto, a topic many time we are taugh to just use a library instead of writing our own. Therefor, it’s great to dive deep into the concept.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• HubSpot's Smart CRM offers a comprehensive, customizable company record, allowing marketing, sales, and service teams to have a single, unified view of the customer

  However, most GTM teams still struggle to understand how customers are using their products. Now, teams can combine the power of HubSpot and Amplitude’s customer behavior data to accelerate growth.

  The new and improved Amplitude app for HubSpot allows you to uncover your most qualified leads using product usage insights. Now, marketers can use key product signals for personalized lead nurturing, pass qualified leads to Sales, and work with Service to re-ignite at-risk customers.

  Try the improved app today to enrich your HubSpot customer data with Amplitude’s product usage data

• Cryptography 101 with Alfred Menezes

  Alfred Menezes, a professor in the Department of Combinatorics and Optimization, Faculty of Mathematics, at the University of Waterloo in Canada. His research field is cryptography. He has made quite a few of his lecture available for free, both of slide, video, exercises and handout.

• What is entropy?

  Engineer no doubt encounters the term entropy quite a bit when it come to randomess. But what is it? how to define it? This is the draft to a book about that exact topic. And here is the introduction post by author.

• Introduction to WebAssembly

  WebAssembly, or WASM for short, is a new technology for running portable programs in a safe and efficient manner primarily aimed at the web platform. Similarly to ASM.js, WASM aims at a low level-of abstraction suitable as an intermediate representation of a higher-level program — i.e. WebAssembly code is intended to be generated by compilers rather than being written by humans. T

• OpenTelemetry Tracing in 200 lines of code

  Developers tend to treat tracing as deep magic, and OpenTelemetry is no exception. OpenTelemetry may be even more mysterious given how many concepts your are exposed to even with beginning examples. They are likely a lot simpler than you expect! Once you peel back the layers, I find a useful mental model of tracing looks like “fancy logging” combined with “context propagation” a.k.a “passing some IDs around”. The examples are JS code but the concept can be applied to your favorite language SDK given OpenTelemetry popularity.

• Visual guide to SSH tunneling and port

  If you always confuse with ssh remote port, local port forwarding this visualization might help.

• Improving rendering performance with CSS content-visibility

  On Mastodon, an opensource platform that is similar to Twitter. People can define custom emoji. The op library call emoji-picker-element is used to render the picker. It’s choke with 19k emoji. How can we improve performance here to render a bunch of emojis as fast as possible.

• Optimising your Database for Analytics

  Your database is configured for the needs of your day-to-day application activity, but what if you need to run complex analytics queries against your application data? Let’s look at how you can optimise your database for an analytics workload without compromising the performance of your application.

• Detecting the use of "curl | bash" server side

  Installing software by piping from curl to bash is obviously a bad idea and a knowledgable user will most likely check the content first. So wouldn’t it be great if a malicious payload would only render when piped to bash? A few people have tried this before by checking for the curl user agent which is by no means fail safe - the user may simply curl the url on the commandline revealing your malicious code. Luckily the behaviour of curl (and wget) changes subtely when piped into bash. This allows an attacker to present two different versions of their script depending on the context :)

• Cryptography in Go: AES encryption

  Lets learn how to encryption and decrypt with AES. You will learn stuff like AES-CBC or AWS-GCM. It uses Go as the language but the idea can be applied to your favorite langugage.

• Why MySQL Replication Is Fast

  Replication being slow—replication lag—is a common complaint, but MySQL replication is actually really fast. Let’s run a controlled experiment and peek inside the Performance Schema and binary logs to see why.

• Building LLM-powered applications in Go

  In other words, LLM-powered applications are a lot like other modern cloud-native applications: they require excellent support for REST and RPC protocols, concurrency and performance. These just so happen to be the areas where Go excels, making it a fantastic language for writing LLM-powered applications. This blog post works through an example of using Go for a simple LLM-powered application. It starts by describing the problem the demo application is solving, and proceeds by presenting several variants of the application that all accomplish the same task, but use different packages to implement it. All the code for the demos of this post is available online.

Code to read

• fair

  a Go library designed to ensure fairness in the resource-constrained environments. It helps distribute the limited resources (e.g., database/blob storage throughput, job execution resources etc.) evenly across multiple clients during the time of shortage, preventing over-allocation and starvation based on client behavior.

• maud

  Compile-time HTML templates for Rust

• nextgen

  Generate your next Rails app interactively! This template includes production-ready recommendations for testing, security, developer productivity, and modern frontends. Plus optional Vite support! ⚡️

• iou

  a Ruby gem for working with the io_uring API. IOU provides a simple and idiomatic API for working with io_uring.

• LLaMA-Omni

  a low-latency and high-quality end-to-end speech interaction model built upon Llama-3.1-8B-Instruct, aiming to achieve speech capabilities at the GPT-4o level.

• WordLlama: Things you can do with the token embeddings of an LLM

  WordLlama is a fast, lightweight NLP toolkit that handles tasks like fuzzy-deduplication, similarity and ranking with minimal inference-time dependencies and optimized for CPU hardware.

Tools

• MiniCPM

  An edge-side LLM that surpasses GPT-3.5-Turbo.

• The Ultimate Web Reconnaissance & Vulnerability Scanner 🚀

  your ultimate web application reconnaissance suite, designed to supercharge the recon process for security pros, pentesters, and bug bounty hunters. It is go-to web application reconnaissance suite that’s designed to simplify and streamline the reconnaissance process for all the needs of security professionals, penetration testers, and bug bounty hunters

• scramble

  an open-source Chrome extension that leverages AI to enhance your writing directly in your browser. It’s designed to be a more customizable and privacy-respecting alternative to Grammarly.

• rainfrog

  a database management tui for postgres

• maelstrom

  a suite of tools for running tests in isolated micro-containers locally on your machine or distributed across arbitrarily large clusters. Maelstrom currently has test runners for Rust, Go, and Python, with more on the way.

• There

  A native menubar app to track friends, teammates or city time zones on macOS.

• sops

  Simple and flexible tool for managing secrets. Used to be managed by Mozilla but become a community project now.

• skyvern

  Automate Browser-based workflows using LLMs and Computer Vision

• spann3r

  3D Reconstruction with Spatial Memory

Hi all,

Welcome to another issue of BetterDev. This week we will learn about making SWIFT payment, imagine you don’t have to rely on Stripe, and just be able to make payment right from your bank through SWIFT Protocol.

If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Snyk is thrilled to announce DevSecCon 2024, Developing AI Trust Oct 8-9, a FREE virtual summit designed for DevOps, developer and security pros of all levels

  Hear from John Hammond & Daniel Miessler on some of the critical strategies and prescriptive DevSecOps approach needed to build and maintain trust in the age of AI-powered development. Save your spot

• How to Send a SWIFT Wire From Scratch

  SWIFT is the most widely used international payment method in the world. In this tutorial we will be going over how to send a SWIFT wire programatically to your bank, what information you might need about your recipient to allow the payment to clear successfully, and the software you might need to do so. The following example can be implemented in code using iso20022.js.

• We Spent $20 To Achieve RCE And Accidentally Became The Admins Of .MOBI

  A tale of how not renewing domain allow one to register and control an endpoint, lead to RCE. Another reminder that if you ever has a domain in use, then stopping using it, don’t let it expired. Also, yet another story to stick with traditional TLD such as .com, .net etc

• B-trees and database indexes

  B-trees are used by many modern DBMSs. Learn how they work, how databases use them, and how your choice of primary key can affect index performance.

• SLO: Elastic vs Datadog vs Grafana

  A very hand-on comparison of SLO functionality on 3 platforms. Really good and detail write up if you’re evaluating these observability platfor.

• The Undeniable Utility Of CSS :has

  CSS is getting better and better recently with ton of of highly-requested CSS features delivered. Today, we will looks into the :has pseduo class and share some of the most interesting real-world use cases I’ve found so far, along with some truly mindblowing experiments.

• Better-performing “25519” elliptic-curve cryptography

  In recent years, you may notice we started to use SSH key with ED25519 instead of RSA. You will notice the key length is much shorter, but more secure. Another Two cryptographic algorithms that have become increasingly popular are x25519. AWS has optimized the x25519 and Ed25519 cryptographic algorithms in their AWS LibCrypto library, improving both performance and security and they share the benchmark and how we can use it here.

• The Many SHAs of a GitHub Pull Request

  If you ever want to get the SHA of the Github commit in a Github action. You will quickly realize depend on pull request or branch push, the data path to get the SHA is different. In this we will demystify it

• Creating a Git commit: The Hard Way

  To create git commit, we use Git high-level commands (also known as Porcelain commands) like git add, and git commit. However, there is another group of Git commands, known as Plumbing commands, that handle the low-level operations. In this blog post, we want to create a Git commit using these low-level operations, and not the git commit command.

• Bitten by Unicode

  There are character that looks the same. But depending on where one type the message, it may be a different unicode code of the underlying data. So today we will learn the tail of hyphen and hyphen-minus.

• Shrinking Big PostgreSQL tables: Copy-Swap-Drop

• Dynamically estimating and scaling Postgres’ working set size

Code to read

• tetris-sql

  Using SQL’s Turing Completeness to Build Tetris

• acts_as_recursive_tree

  Make use of recursive queries in Rails when using Postgresql or SQLite to retrieve tree-based data.

• annotaterb

  Adds comments summarizing the model schema or routes in your. Really good codebase to learn how to parse and insert code into the existing files

• LivePortrait

  Efficient Portrait Animation with Stitching and Retargeting Control

• live-pprof

  Instead of cluttering up your computer with Docker, Prometheus, Grafana or even K8S just to monitor a Go app’s heap size, use this package to Monitor a Go app’s performance. Note that, it’s build mainly for local dev

• fast_float

  Fast and exact implementation of the C++ from_chars functions for number types: 4x to 10x faster than strtod, part of GCC 12, Chromium, Redis and WebKit/Safari. C++ is the language I never wrote, but I do plan to read to understand how float parse from characters works.

Tools

• pixijs

  to provide a fast, lightweight 2D library that works across all devices. The PixiJS renderer allows everyone to enjoy the power of hardware acceleration without prior knowledge of WebGL. Also, it’s fast. Really fast.

• css-triggers

• pgbackweb

  🐘 Effortless PostgreSQL backups with a user-friendly web interface! 🌐💾

• tmail-flutter

  A multi-platform (Flutter) application for reading your emails, with your favorite devices, using the JMAP protocol!

• svgedit

  Powerful SVG-Editor for your browser

• shoutrrr

  A notifications library and a CLI to send notification to multiple services such as discord, slack, telegram and more.

Hi all,

Now, let’s dive into the newsletter. If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• A collaborative IPv6 book

  The intention is a practical introduction to IPv6 for technical people, kept up to date by active practitioners.

• How SSH Secures Your Connection

  If you’ve ever remotely connected to any UNIX-like server, you have likely used SSH, short for “Secure Shell”. SSH provides, as the name implies, secure shell access to remote machines and is used pretty much everywhere. But what exactly does “secure” mean here, and how is this security provided by the protocol? This post will take a look at SSH’s security features and how they protect against example attacks.

• Extreme Pi Boot Optimization

  🚀 3.5 sec to Linux userspace code

• The Design & Implementation of the CPython Virtual Machine

  For every bytecode compiled language, the most interesting part of its implementation is its virtual machine. Even if you are not a compiler engineer, learning about such internal implementation can give you new performance tricks and insights that you may be able to use in other places of your job.

• The web's clipboard, and how it stores data of different types

  Have you ever notice when you copy the html text, sometime the format is kepted as well, such as table and font style? How does that kind of data is serialized and pass between the applications.

• Programming ZKPs: From Zero to Hero

  Imagine you want to prove that you have a key to a room? You don’t need to show me the key. But if I look through the window, and see you’re inside. I know you have the key. How can we apply that kind of process into programming to prove it?

• Dealing with rejection (in distributed systems)

  Backpressure is one of the most important practical details that every good distributed system has to get right if it’s going to stand a chance at survival in production. Without a good backpressuring system, a small increase in load or an errant client can easily knock over the entire system and leave it stuck in a death spiral from which it will never recover without manual intervention — usually by shutting off all the clients.

• Easy to Overlook PostgreSQL Performance Issues in Rails Apps

  It says Rails app in the title but it can be applied to app like Django or any kind of ORM and it abstracts you away from thinking and optimize your db. Example,

• Tuning PostgreSQL parameters in Amazon RDS

  Tailor towards AWS RDS but it’s good and generic guide to cover many PostgreSQL tuning parameters

Code to read

• netlink

  package provides a simple netlink library for go. Netlink is the interface a user-space program in linux uses to communicate with the kernel. It can be used to add and remove interfaces, set ip addresses and routes, and configure ipsec.

• victor

  a lightweight, zero-dependencies Ruby library that lets you build SVG images using Ruby code

• allgood

  ✅ Quickly set up a health check page for your Rails app

• LitServe

  is an easy-to-use, flexible serving engine for AI models built on FastAPI. Features like batching, streaming, and GPU autoscaling eliminate the need to rebuild a FastAPI server per model.

Tools

• Calendar.js

  A javascript drag & drop event calendar, that is fully responsive and compatible with all modern browsers. These kind of project you may not need it right now but come in handy when you need it at a later time

• perspective

  A data visualization and analytics component, especially well-suited for large and/or streaming datasets.

• pgmonitor-extension

  This extension provides a means to collect metrics within a PostgreSQL database to be used by an external collection source (Prometheus exporter, Icinga/Nagios scraper, etc). Read the announcing post from CrunchyData

• pgstream

  an open source Change-Data-Capture(CDC) command-line tool and library that offers Postgres replication support with DDL changes to any provided output. Once you have a stram of change happen to your data, it can solve many use case. Example, implement webhooks

• WatchYourLAN

  Lightweight network IP scanner. Can be used to notify about new hosts and monitor host online/offline history

Hi all,

This week, I’ve already started to feel the first signs of autumn in the air where I live. I hope everyone can savor these last few days of summer and gear up for the fall season.

As engineers, we’re often advised not to reinvent the wheel—use Redis for caching, ElasticSearch for searching, and so on. But let’s challenge that mindset. If you’re curious, dive in and experiment. Build your own search engine, write your own database. That’s the spirit behind this week’s curated links. I hope you find them inspiring.

Now, let’s dive into the newsletter. If you enjoy BetterDev, please spread the word by sharing it with your friends. And if you’d like to support my work, buying me a coffee would be much appreciated.

• Realtime mouse pointer

  In collabortion app you will usually see the cursor of all the user moving around. How would we implement that? At scale. Canvas engineering share with us here. Some good find such as using a Binary serialization instead of JSON to reduce CPU load on backend

• Postgres as a search engine

  Build a retrieval system with semantic, full-text, and fuzzy search in Postgres to be used as a backbone in RAG pipelines. We will do all: full text search, semantic search, fuzzy search with Postgres

• What’s Really Going On in Machine Learning? Some Minimal Model

  It’s surprising how little is known about the foundations of machine learning. Yes, from an engineering point of view, an immense amount has been figured out about how to build neural nets that do all kinds of impressive and sometimes almost magical things. But at a fundamental level we still don’t really know why neural nets “work”—and we don’t have any kind of “scientific big picture” of what’s going on inside them.

• Linux Pipes are Slow

  Some programs use a particular system call “vmsplice” to move data faster through a pipe. When not using vmsplice, Linux pipes are slower than what I would have expected. Since we cannot always use it, lets see exactly why that was, and whether it could be improved.

• Continuous reinvention: A brief history of block storage at AWS

  Marc Olson has been part of the team shaping Elastic Block Store (EBS) for over a decade. In this post, Marc provides a fascinating insider’s perspective on the journey of EBS. He shares hard-won lessons in areas such as queueing theory, the importance of comprehensive instrumentation, and the value of incrementalism versus radical changes. Most importantly, he emphasizes how constraints can often breed creative solutions. It’s an insightful look at how one of AWS’s foundational services has evolved to meet the needs of our customers (and the pace at which they’re innovating).

• Ethernet History Deepdive – Why Do We Have Different Frame Types?

  Take us into the history of Ethernet to see why things happened the way they did. It will also significantly enhance your network knowledge.

• Build your own SQLite, Part 1: Listing tables

  As developers, we use databases all the time. But how do they work? In this series, we’ll try to answer that question by building our own SQLite-compatible database from scratch.Source code examples will be provided in Rust, but you are encouraged to follow along using your language of choice, as we won’t be relying on many language-specific features or libraries.

• Advanced Monitoring with AI and Prometheus: Detecting and Mitigating Memory Leaks

  Detect memory by just looking at growth rate of memory is simple. What if we train a model of leak pattern, and apply it to detect it sooner and more efficent?

• How Postgres stores data on disk – this one's a page turner

  databases are basically magic, and SQL is the arcane tongue that allows you to channel that magic. In fact, it’s easy to think of databases like a black box where you make sure your tables are indexed sensibly and your queries aren’t doing anything silly, and the rest just happens. But at the end of day, they are just software, that works with hardware, and use some great tricks or abstraction. So we will dive into that

• How Postgres stores oversized values – let's raise a TOAST

Code to read

• transformers-ruby

  State-of-the-art transformers for Ruby

• lo

  A Lodash-style Go library based on Go 1.18+ Generics (map, filter, contains, find…)

• pie

  Enjoy a slice! A utility library for dealing with slices and maps that focuses on type safety and performance.

• ruroco

  a tool that lets you execute commands on a server by sending UDP packets. The commands are configured on the server side, so the client does not define what is going to be executed, it only picks from existing commands.

• sqlite-vec

  A vector search SQLite extension that runs anywhere!

• ai-toolkit

  Various AI scripts. Mostly Stable Diffusion stuff.

• chroma.js

  JavaScript library for all kinds of color manipulations. Working with color is really fun because it invole a lot of matrix manipulation and some cool tip.

• tinykeys

  A tiny (~650 B) & modern library for keybindings

Tools

• go-vcr

  Record and replay your HTTP interactions for fast, deterministic and accurate tests

• vega

  a visualization grammar, a declarative language for creating, saving, and sharing interactive visualization designs. With Vega, you can describe the visual appearance and interactive behavior of a visualization in a JSON format, and generate web-based views using Canvas or SVG.

• Milkdown

  A plugin driven framework to build WYSIWYG Markdown editor

• anyquery

  Query anything (JSON, CSV, GitHub, Notion, Airtable, etc.) with SQL

• dozzle

  Real-time logging and monitoring for Docker in the browser

• pg_duckdb: DuckDB + Postgres extension

  DuckDB-powered Postgres for high performance apps & analytics. Check out the introduction post

• gonb

  A Modern Go Kernel for Jupyter Notebooks

Hi all,

Another issue for another week. Trying to get back to my schedule. If you like BetterDev, please help spead word out by refer to your friends. Buy me a coffee would be great too.

This week we had a big security issue with Github. If you had archive the whole directory, the Github token may leak into the artifact. Read more here: https://unit42.paloaltonetworks.com/github-repo-artifacts-leak-tokens/

I also found this amazing tool call that allow you to auto apply job to linkedin. Try it out to increase your chance. The job market is tough these day.

• Audit your app for free today with Alerty JavaScript SDK

  The Alerty JavaScript SDK takes 7 minutes to set up and covers your app and database with all the APM and error-tracking features you’d expect, all at a fraction of the big monitoring tools cost. But it doesn’t stop there. Alerty Agents use your frontend telemetry data to give you tailored guidance and feedback to debug issues and optimize your app’s performance. Audit your app for free today.

• What every SRE should know about GNU/Linux resolvers and Dual-Stack applications

  deep dive into the GNU/Linux local facilities used to convert a domain name or hostname into IP addresses, specifically in the context of dual-stack applications. Although it may seem simple at first glance, the entire process involving stub resolvers is filled with complexities and subtle nuances

• blocking=render: Why would you do that?!

  WebKit have recently announced their intent to implement the blocking=render attribute for script and style elements, bringing them in line with support already available in Blink and generally positive sentiment in Firefox. The blocking=render attribute allows developers to explicitly mark a resource as render blocking, but… why on earth would you want to do that?!

• Sort, sweep, and prune: Collision detection algorithms

  The problem of collision detection is pretty common in video game programming. It’s a prerequisite to the implementation of certain game mechanics or simulations. Here I’ll cover several related approaches, starting with the simplest and building up to the sweep-and-prune algorithm. I won’t cover other approaches, such as space partitioning or spatial tree subdivision.

• Common Causes of Memory Leaks in JavaScript

  Memory leaks are a silent threat that gradually degrades performance, leads to crashes, and increases operational costs. Unlike obvious bugs, memory leaks are often subtle and difficult to spot until they start causing serious problems.

• Compiling a Custom Browser to Bypass Anti-Bot Measures

  In this blog post, I will be documenting the journey veritas and I took to extract the AES keys and browser flags/fingerprint from the Supreme anti-bot system. This work was done using the ticket.js anti-bot from March, 2021, and is being published now that Supreme has migrated away from the ticket anti-bot system in favour of using Shopify. Extracting these keys allows for a complete bypass of the anti-bot system.

• A quick beginner’s guide to animation

  Not a really programming article but I enjoy and learn something from it when trying to do some animation

• Syntax Highlighting in Hand-Coded Websites

  could it be possible to build syntax highlighting directly into a font, skipping JavaScript altogether? Could I somehow leverage OpenType features, by creating colored glyphs with the COLR table, and identifying and substituting code syntax with contextual alternates?

• What you should know about Linux memory overcommit in PostgreSQL

  Useful as memory overcommit may be for other applications, it is bad news for your PostgreSQL database. With this article I am trying to raise the awareness of this problem. I’ll give you hints for the proper kernel configuration and discuss the implications for a containerized setup

Code to read

• whois

  whois implements in Ruby

• go-sqlbuilder

  Ever want to see how a query builder is built? This could be a good read.

• templ

  A language for writing HTML user interfaces in Go.

• wireguard-rs

  Implementation of wireguard in Rust

• pgai

  Bring AI models closer to your PostgreSQL data

Tools

• tldfinder: A streamlined tool for discovering TLDs, associated domains, and related domain names.

  Read the companion post Hacking Beyond .com — Enumerating Private TLDs

• NetAlertX

  WIFI / LAN intruder detector. Scans for devices connected to your network and alerts you if new and unknown devices are found.

• AIGoat

  AI-Goat is a deliberately vulnerable AI infrastructure hosted on AWS, designed to simulate the OWASP Machine Learning Security Top 10 risks (OWASP ML Top 10). You can deploy this into your own infra and practice exploting. The app deployed with terraform, in a standalone VPC so you can quickly spin up and destroy everything without leaving something insecure running in your infra. Read the comprehensive introduction post

• super-linter

  Combination of multiple linters to run as a GitHub Action or standalone. One linter to rule them all.

• agentUniverse

  a multi-agent framework based on large language models. agentUniverse provides you with the flexible and easily extensible capability to build single agents.

• wpscan

  WPScan WordPress security scanne

• SmoothMQ

  An improved drop-in replacement for SQS

Hi all,

We come back with a new issue this week. If you like BetterDev, please help spead word out by refer to your friends. Buy me a coffee would be great too.

• Audit your app for free today with Alerty JavaScript SDK

  The Alerty JavaScript SDK takes 7 minutes to set up and covers your app and database with all the APM and error-tracking features you’d expect, all at a fraction of the big monitoring tools cost. But it doesn’t stop there. Alerty Agents use your frontend telemetry data to give you tailored guidance and feedback to debug issues and optimize your app’s performance. Audit your app for free today.

• Wi is Fi: Understanding Wi-Fi 4/5/6/6E/7 (802.11 n/ac/ax/be)

  Make your own educated Wi-Fi upgrade decisions. An extensive guide with everything and parameter about Wi-Fi protocol.

• The biggest-ever global outage: lessons for software engineers

  Unless you were under a rock, you likely heard about the CrowdStrike / Windows outage that took down critical services like airlines, banks, supermarkets, police departments, hospitals, TV channels, and more, around the world. Gergely covers recap, root cause, fix, and learning in this post.

• A Deeper Dive of kube-scheduler

  Getting a proper understanding of kube-scheduler is difficult because there are no online resources explaining how it actually works. The scheduling framework README provides a good outline but isn’t comprehensive. The scheduling framework is also a somewhat leaky abstraction so you really do need to know how the scheduler works internally if you want to write a non-trivial plugin.

• WebGPU Unleashed: A Practical Tutorial

  your gateway to the dynamic world of graphics programming. This free book will teach you graphics programming in JavaScript using WebGPU, drawing from my own learning journey.

• Introduction to Linux interfaces for virtual networking

  Linux has rich virtual networking capabilities that are used as basis for hosting VMs and containers, as well as cloud environments. In this post, I will give a brief introduction to all commonly used virtual network interface types. There is no code analysis, only a brief introduction to the interfaces and their usage on Linux. Anyone with a network background might be interested in this blog post. A list of interfaces can be obtained using the command ip link help.

• How I Created 175 Fonts Using Rust

  How hard it is to create font? Can we generate, build them dynamically like writing code? Yes, this page document the process on how to do just that by putting together many tools. You will also learn a fair bit amount about font structure too.

• Patterns for Memory Efficient DOM Manipulation with Modern Vanilla JavaScript

  best practices to avoid excess memory usage when managing updating the DOM to make your apps blazingly fast™️.

• Speeding up index creation in PostgreSQL

  Without indexes, there are no such things as efficient search operations, no unique constraints, and no primary keys. The main question now is: what happens during index creation, and how can we speed up the process? Creating indexes on billions of rows is quite costly, because data has to be sorted and turned into an efficient search structure.

• How to Run Mistral Models Locally - A Complete Guide

  In the rapidly evolving landscape of artificial intelligence, Mistral AI has emerged as a beacon of innovation, charting new territories in the realm of large language models (LLMs). With the introduction of its groundbreaking models, Mistral AI not only advances the frontier of machine learning but also democratizes access to cutting-edge technology. This guide aims to elucidate the intricacies of Mistral AI’s offerings and provide a comprehensive roadmap for harnessing their capabilities locally.

• Writing a system call tracer using eBPF.

  One of feature that CrowdStrike falcon provide is listen to system call to detect suspicious activity, such as when someone read /etc/password. How do they do that? In this we learn a way to do so throuhg eBPF to track these syscall.

• The fastest way to copy data between Postgres tables

Code to read

• editor

  Source code editor in pure Go.

• Deep Live Cam

  real time face swap and one-click video deepfake with only a single image (uncensored)

• pg_statviz

  a minimalist extension and utility pair for time series analysis and visualization of PostgreSQL internal statistics.

• canvg

  SVG parser and renderer on Canvas. It takes the URL to the SVG file or the text of the SVG file, parses it in JavaScript and renders the result on Canvas. It also can be used to rasterize SVG images.

• peerfetch

  Peer-to-peer HTTP over WebRTC.

• fontations

  a number of foundational crates for reading and manipulating OpenType font files. It is motivated by a desire to have more robust and performant open tools for a variety of font engineering and production tasks.

Tools

• sniffnet

  Comfortably monitor your Internet traffic. Native, not electron, cross platform. Extensive featurset. Strongly suggest to give it a try, blow my mind with 17K stars on github and a vast amount of featurset.

• greenmask

  a powerful open-source utility that is designed for logical database backup dumping, obfuscation, and restoration. It offers extensive functionality for backup, anonymization, and data masking

• floating-ui

  A JavaScript library to position floating elements and create interactions for them.

• whirl

  CSS loading animations with minimal effort!

• svgviewer

  a tool to view, edit svg in browser and export image in JSX, PNG or data uri.

• rapidscan

  The Multi-Tool Web Vulnerability Scanner

• Nettacker

  Automated Penetration Testing Framework - Open-Source Vulnerability Scanner - Vulnerability Management

• approf

  pprof is a tool for visualization and analysis of profiling data. Usually your language instrument tool generate pprof file, then you use a web tool to view it, or export to SVG. This is a native macOS app for pprof file.

• torchchat

  a small codebase showcasing the ability to run large language models (LLMs) seamlessly. With torchchat, you can run LLMs using Python, within your own (C/C++) application (desktop or server) and on iOS and Android.

Hi all,

We come back with a new issue this week. If you like BetterDev, please help spead word out by refer to your friends. Buy me a coffee would be great too.

• Hacking Millions of Modems (and Investigating Who Hacked My Modem)

  What would you do if one day you’re realized your modem was being hack and MiTM your request? In this amazing and crazy stories, a Cox modem was hacked. The bug was also really interesting, if repeated a same request multiple time, we will eventually can by pass the authentication.

• I am a puts debuggerer

  printf vs debugger is a holy war in programming, and has cause endless internet fight or fame. Today, we won’t go into that. We will simply go through a list of common scenartio and see how we can use printf to debug the situation. This is a Ruby article, but if you like the approach, they can easily ported to your own language. Methodlogy is what we will focus here.

• What is PID 0?

  pid 1 is the first process the kernel start. If you user docker and want to run multiple process, you definetely had deal with the pid 1 before. but what about pid 0 ? what is that?

• Architectures for Central Server Collaboration

  If you need to do an editor, a web diagram drawing tool, how would you handle the real time aspect? This post records some though on How to architect a real-time collaborative app when you do have a central server.

• Detecting a PS2 Emulator: When 1*X does not equal X

• How We Improved the Performance of a Userspace TCP Stack in Go by 5X

  Throughput - the rate at which you can send or receive data across a network Throughput is usually quoted in bits per second, and on modern internet networks, these range from a few millions of bits per second (Mb/s) to a few billions of bits per second (Gb/s). If you had work with a remote connection such as SSH through a VPN tunnel you will notice the differencies of responsiveness when workign with a code editor. Coder is self-hosted cloud development environments, and they build their own TCP stacks, and share with us experience and their deep dive to increase perf 5x.

• Cracking the Scheduling Code in Hay Day

  Hay Day is a game where you planted and sold fruit tree. Tree takes time to grow. Trade need to be matched. How can we put together these thing in a game?

• Promises From The Ground Up

  In order to understand Promises, we need a surprisingly deep understanding of how JavaScript works and what its limitations are. Without that context, Promises won’t really make much sense.

• why do i have a slow commit in PostgreSQL?

  Normally, COMMIT is a very fast statement in PostgreSQL, so that is worth investigating. In this article, I will explore the possible reasons for a slow COMMIT and discuss what you can do about it.

• Attacking NIST SP 800-108

  NIST 800 are guidelines, recommendations, technical specifications, and annual reports of NIST’s cybersecurity activities. 800-108 focus on recommendation for Key Derivation Using Pseudorandom Functions. In this post, we will look into a novel security issue with this standard

• Encryption At Rest: Whose Threat Model Is It Anyway?

  To quote to OP, “Unsurprisingly, I’ve heard a few infosec thought leader types insist that encryption-at-rest is security theater over the years. I disagree with this assessment in the absolute terms, but there is a nugget of truth in that assertion.”. To which I agree.

• Custom PostgreSQL extensions with Rust

  I used to be able to pick-up Rust in a single week and become productive very fast. Rust has open us the ability to write low level component in such a high level language. This article exporers pgrx, which simplifies the creation of custom PostgreSQL extensions to bring more logic closer to your database.

Code to read

• Take an image. How can we generate an ascii-text that can resemble the image in terminal, with color. This projects show that kind of algorithm to do it. Read the introduction post here too

• tinypool

  A minimal and tiny Node.js Worker Thread Pool implementation (38KB)

• rill

  Go concurrency with channel transformations, type safety, batching and error handling. No boilerplate

• go-datastructures

  A collection of useful, performant, and threadsafe Go datastructures.

• blind_index

  Securely search encrypted database fields

• yolov10

  Real-Time End-to-End Object Detection

• pyreft: Representation Finetuning for Language Models

  a representation fine-tuning (ReFT) library that supports adapting internal language model representations via trainable interventions

Tools

• tbls

  a CI-Friendly tool for document a database, written in Go. Document a database automatically in GFM format. Output database schema in many formats

• pgCompare

  a straightforward utility crafted to simplify the data comparison process, providing a robust solution for comparing data across various database platforms.

• katex

  The fastest math typesetting library for the web.

• gh-dash

  A GitHub CLI extension to display a dashboard with pull requests and issues by filters you care about.

• oapi-codegen

  Generate Go client and server boilerplate from OpenAPI 3 specifications

• PKI.js

  a pure JavaScript library implementing the formats that are used in PKI applications (signing, encryption, certificate requests, OCSP and TSP requests/responses). It is built on WebCrypto (Web Cryptography API) and requires no plug-ins.

• pg_analytica

  Postgres extension that speeds up analytics queries by upto 90%

• motion-canvas

  A TypeScript library that uses generators to program animations and an editor to preview that animation Check the introduction page

• ragflow

  an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.

Hi all,

We come back with a new issue this week. If you like BetterDev, please help spead word out by refer to your friends. Buy me a coffee would be great too.

I’m a bit busy this week so we don’t have a lot of links but we will resume next week :-).

• Uber Migrates 1T Records from DynamoDB to LedgerStore to Save $6M Annually

  Nothing as good as when company throw out the number. Pair it with another one detail on engineering aspect of How LedgerStore supports trillions of indexes

• The Worst Website In The Entire World

  You would have a good laugh after this. Please go ahead and read it. I won’t spoil anything here.

• How Ahrefs Saved US$400M in 3 Years by NOT Going to the Cloud

  Title say it’s all.

• Designing a Scalable OCPP server

  OCPP, short for Open Charge Point Protocol, is like the universal language of EV charging. It sets the standard for how charging stations talk to central systems, enabling functions like monitoring, management, billing, and even integration with third-party systems. This open-source protocol isn’t just a buzzword — it’s backed by a robust community of vendors and manufacturers, making it a widely adopted and trusted solution.

• Llama 3 implemented in pure NumPy

  Understand the exact structure with working implementation of the Llama 3 model. This is over my head, but definetely something I want to dip my toe in. These Llama model is amazing.

• Building a SaaS App? You should probably stick to a single subdomain

  When building a SaaS app, you will face the question: “should we use app.domain.com” or “company-name.domain.com” ? in this article, shortcut.com shared their approach. For me, personally, I prefer to use a single subdomain because of the same login session can be shared to join other team.

• Visualizing algorithms for rate limiting

  rate limiting is like measureing speed? how can you measure speed at a given time? you need to measure volume, divide by time, but what define a time window? In this post, we’ll explore the three most common algorithms to answer each of these questions: Fixed windows, sliding windows and toke buckets

• That time PostgreSQL said "no thanks, I don't need your index"

  An interesting tale of victory after PostgreSQL rejected our attempts to optimize a query. We had a query we were observing that was growing in latency. We analyzed it and determined the best course of action was to add an index. PostgreSQL disagreed, and outrightly refused to employ the index. Following is the story of the negotiation that took place to push past the blatant rejection of Postgres.

• PostgreSQL Stories: From slow query to fast—via stats

  You might have heard “it’s important to know your data” when working with databases. But what does that look like in practice? This post is one answer: an example of how knowing our data helped us debug a slow query.

• Making a Postgres query 1,000 times faster

  Mattermost query Postgres data then index into ElasticSearch for search, but they discovered a slow query. The walk us through the investigation process and run EXPLAIN to debug the slowness.

• Advanced psql scripting with coproc

  In this post, let’s see how to control psql from outside with the coproc functionality in bash.

Tools

• pareas

  GPU-accelerated compiler

• pg_lakehouse

  an extension that transforms Postgres into an analytical query engine over object stores like S3 and table formats like Delta Lake. Queries are pushed down to Apache DataFusion, which delivers excellent analytical performance. Combinations of the following object stores, table formats, and file formats are supported.

• pg_repack

  Reorganize tables in PostgreSQL databases with minimal locks

Hi all,

We come back with a new issue this week. If you like BetterDev, please help spead word out by refer to your friends. Buy me a coffee would be great too.

Many link about LLM this week, especially focus on fundamental knowledge and using them locally on your own hardware, instead of just a wrapper of OpenAI.

• Turning AirPods into a Fitness Tracker to Fight Cancer

  A fun side project for a great cause featuring Core Motion, SwiftUI, a little help from AI, and a pair of AirPods to count 100 push-ups a day.

• How LLMs Work, Explained Without Math

  A basic question I think a lot of people have about the GenAI revolution is where does the apparent intelligence these models have come from. In this article, I’m going to attempt to explain in simple terms and without using advanced math how generative text models work, to help you think about them as computer algorithms and not as magic.

• A simple guide to local LLM fine-tuning on a Mac with MLX

  walk through the process of fine-tuning step-by-step. It won’t cost you a penny because we’re going to do it all on your own hardware using Apple’s MLX framework.

• t’s always TCP_NODELAY. Every damn time

  The first thing I check when debugging latency issues in distributed systems is whether TCP_NODELAY is enabled. And it’s not just me. Every distributed system builder I know has lost hours to latency issues quickly fixed by enabling this simple socket option, suggesting that the default behavior is wrong, and perhaps that the whole concept is outmoded.

• Time based css animation

  Using time for animation is very common in shader programs and various other places. CSS can not start a timer like JavaScript does, but nowadays it’s possible to define a custom variable with the CSS Houdini API to track time in milliseconds. The value of animation is calculate based on this custom variable. It’s a different approach to perform animation compare to keyframes. This goes beyond my ability to understand, I’m not super good at CSS animation. But still a good read to dabble, I always like to perform animation with pure css.

• How not to change PostgreSQL column type

  One of the surprises that comes with developing applications and operating a database cluster behind them is the discrepancy between practice and theory, development environment and the production. A perfect example of such a mismatch is changing a column type. Super easy to change stuff in dev, but I dare you to randomly go in a prod db and change data type

• ECDSA: Elliptic Curve Signatures

  When you generate SSH key like RSA or ECDSA and peform encryption and signature verification. What is the math behind them? I always though they are complicated, but they are way more straighforward when you dabble into its math.

• Postgres Bloat Minimization

  In database, there is always a trade off. You trade storage tooling gain cpu time. trade fragment for speed. all of that become bloated. How can we minimize these?

• 13 Tips to Improve PostgreSQL Insert Performance

• An Applied Introduction to eBPF with Go

  discusses how to develop eBPF programs with Go, with practical example at Edgedelata. A bit Go specific but learning about eBPF will boost your system programming knowledge.

• A 100x speedup with unsafe Python

  We’re going to speed up some numpy code by 100x using “unsafe Python.” Which is not quite the same as unsafe Rust, but it’s a bit similar, and I’m not sure what else to call it… Python specific but please read-on, I’m sure the same concept can be apply to any language that has a C-binding to native C library.

Code to read

• gepub

  a generic EPUB library for Ruby : supports EPUB 3

• llama_parse

  an API created by LlamaIndex to efficiently parse and represent files for efficient retrieval and context augmentation using LlamaIndex frameworks.

• squawk

  A linter for Postgres, focus on migration.

• superfile

  Pretty fancy and modern terminal file manager. Even though I cannot see myself use this tool, I absolutely want to dig into its source code and see how it’s done. One thing I always wanted to understand about the event/gui loop in these GUI/TUI app.

• ntcharts

  Terminal Charts for the Golang BubbleTea framework and your TUI

• bash_tls

  a minimal TLS implement that is well enough to make a simple HTTPS request to most web servers. Very nice to understand what happen at the byte level in a TLS session

• tine

  a tiny text editor.

Tools

• pintorajs

  An extensible text-to-diagrams library that works in both browser and node.js. Similar to tool like MermaidJS to draw diagram with text.

• secret-llama

  Fully private LLM chatbot that runs entirely with a browser with no server needed. Supports Mistral and LLama 3

• asitop

  Perf monitoring CLI tool for Apple Silicon

• mactop

  a terminal-based monitoring tool “top” designed to display real-time metrics for Apple Silicon chips. It provides a simple and efficient way to monitor CPU and GPU usage, E-Cores and P-Cores, power consumption, and other system metrics directly from your terminal!

• sioyek

  a PDF viewer with a focus on textbooks and research papers

• hugeicons

  A collection features over 3,800 stroke React icons, available at no charge and crafted to meet a diverse array of design requirements.

• pgmoneta

  a backup / restore solution for PostgreSQL.

• pg_tier

  A Postgres extension to tier data to external storage such as AWS S3.

• pgslot-notify-bot

  not monitoring pgslot closely and if a consumer die, you may wake up with a full disk of pgslot WAL. This small tool monitors their growth and alert via slack

• pgdsat

  PostgreSQL Database Security Assessment Tool

• gpt-home

  ChatGPT at home! Basically a better Google Nest Hub or Amazon Alexa home assistant. Built on the Raspberry Pi using the OpenAI API. A very cool hardware project with DIY style

• llama_index

  a data framework for your LLM applications

Hi all,

We come back with a new issue this week. If you like BetterDev, please help spead word out by refer to your friends. Buy me a coffee would be great too.

• Build an 8-bit computer from scratch

  An 8-bit microprocessor is a type of microprocessor that processes data and executes instructions using 8-bit binary numbers. In simple terms, it means that the processor can handle data in chunks that are 8 bits wide. This tutorial walks through building a fully programmable 8-bit computer from simple logic gates on breadboards. Included are parts lists, sources, and schematics to make it easier to follow along or even build your own!

• A Beginner's Guide to the ESP8266

  ESP8266 is just a chip name that many companies has designed their own board for it. The most interesting thing is they have Wifi so you can do a lot of IoT with it. Many of them cost just $5 for a board. Also, checkout the ESPHome project for similar thing as well.

• Reverse engineering a software crack

  An amazing crack that doesn’t patch the original binaries. How does it works? by using

• HTML attributes vs DOM properties

  Given a dom node object, what is the different between div.getAttribute(‘foo’) and div.foo? sometime they are same, some time they are different. why so?

• How can we make pg_dump and pg_restore 5 times faster?

  In this blog, we’ll discuss an idea called “Parallel Snapshotting”. This idea could be integrated into Postgres upstream in the future to make pg_dump and pg_restore parallelizable at a single table level.

• Exploring the Enigma of Database Vacuuming

  Before we discuss what VACUUM does and its implication. How data is actually stored on disk. What happens when a tuple is inserted, updated, or deleted? Understanding this will help us understand what VACUUM does, why it’s needed, and its implications.

• What Is Amazon Resource Name (ARN)?

  In this article, you will understand ARNs and their significance within the Amazon Web Services (AWS) ecosystem. You’ll get to dissect the structure of an ARN and how you can retrieve them. By the end of this article, identifying and deciphering ARNs will be a breeze for you.

• Build, notarize, and sign Golang binaries for MacOS with GitHub Actions

  Even it target at Go, this articule is useful for anyone want to sign binary so it can run on MacOS Catalina and later without gotting the unidentified warning.

Code to read

• NAND

  is a Turing equivalent 16-bit computer made entirely from a clock and NAND gates emulated on the web. NAND features its own CPU, machine code language, assembly language, assembler, virtual machine language, virtual machine translator, programming language, compiler, IDE, and user interface. NAND is based on the Jack-VM-Hack platform specified in the Nand to Tetris course and its associated book. You absolutely have to checkout and see how this project is done

• autotuner

  a tool designed to help you tune the garbage collector of your Rails app. Autotuner integrates into Rack as a middleware and collects data from the garbage collector between requests. It will then intelligently provide suggestions to tune the garbage collector for faster bootup, warmup, and response times. Read tehe companion post too

• OpenVoice

  Instant voice cloning by MyShell.

Tools

• paradedb

  an Elasticsearch alternative built on Postgres. We’re modernizing the features of Elasticsearch’s product suite, starting with real-time search and analytics.

• pgddl

  Have you ever want to generatea nice “CREATE TABLE” command from an existing tbale? or “CREATE USER” for an existing user? This tool help you generate that.

• makefile-graph

  Turn your makefile into a graph

• dify

  an open-source LLM app development platform. Dify’s intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.

• unovis

• esphome

  ESPHome is a system to control your microcontrollers by simple yet powerful configuration files and control them remotely through Home Automation systems.

• svgviewer

  View, edit, and optimize SVGs.

• svg-path-editor

  Online editor to create and manipulate SVG paths

Hi all,

We come back with a new issue this week. If you like BetterDev, please help spead word out by refer to your friends. Buy me a coffee would be great too.

This week I come across awesome-secure-defaults, a collection of libs to help us secure our app by setting right HTTP header, crypto libs, CSRF,…Browswing these kind of libs usually will help identify gap in your knowledge, because there is term or concept you might not know. So def give us a read. Now onward to our issue.

• Building a GPS Receiver, Part 1: Hearing Whispers

  I know zero thing about GPS receiver. And I may not build this after reading this article, but my knowledge indeed improve on this topic. I understand more about the technology that we use day to day

• How two processes can communicate in one device using IPC

  Simple way we can use shared file, you can use a client <-> server, but that involve you are need to setup port, sending and serializing data. What if you want your processes to be independent, to communicate, and yet not distribute some arcane shell scripts along with them? It turns out that other IPC mechanisms allow you to do all these. This a series so make you check all the part.

• Building a weather data warehouse: Loading a trillion rows of weather data into TimescaleDB

  Challenge of building a data warehouse. How to load a massive amount of data quickly,

• DNS tcpdump by example

  analyze traffic of DNS is a easier and fun practice because it has no encryption, unlike HTTPS or SSH. So you can actually look at the payload. In this article we will walk through th exercise to analyze this traffic of querying a DNS end to end, what happen, where do they get their data. Knowing this knowledge will put you on the top 10% of software programmer.

• Slack Journey Migrating to AWS IMDSv2

  Have you ever query 169.254.169.254 you deal with AWS IDMS service. Anyone can query that from an EC2 instance. In 2019, they release IMDSv2, which require a token so it’s more secure. Slack running 60,000 AWS instance. How can they roll this out safely across the fleet of services on all these 60,000 server to use IDMSv2

• Profile Node.js performance with the Performance panel

  The JS Profiler is going away in Chrome 124, so we would need to familiar ourself with the newer approach.

• Calculus made easy

  With all the trend about LLM, maybe time to dust up our math skill in high school.

• SQLite on Rails: The how and why of optimal performance

• Rebuilding Tables online using pg_repack in PostgreSQL

  Any database engine, including PostgreSQL, requires some routine maintenance to ensure optimal performance. Especially when there are massive delete or purge operations to delete rows from a Table, the fragmentation builds up within a table. Reclaim this fragmented space from disk requires a table to be rebuilt

• An unfair advantage: multi-tenant queues in Postgres

Code to read

• hyperformula

  an open-source headless spreadsheet for business web apps. It comes with over 400 formulas, CRUD operations, undo-redo, clipboard support, and sorting. Built in TypeScript, supported by the Handsontable Team

• gemfast

  a fast and secure rubygems server written in Go.

• validates_email_format_of

  Validate e-mail addreses against RFC 2822 and RFC 3696 with this Ruby on Rails plugin and gem.

• Open Sora Plan

  This project aim to reproduce Sora (Open AI T2V model), they wish the open source community contribute to this project.

• trdsql

  CLI tool that can execute SQL queries on CSV, LTSV, JSON, YAML and TBLN. Can output to various formats.

Tools

• wedges

  An ever-expanding, open-source React UI library built with the Wedges Design System, Radix primitives, and Tailwind CSS.

• go mysql server

  A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

• litellm

  Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)

• ragas

  Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines

• pgxman

  package manager for PostgreSQL extensions, along with a repository of packages. It integrates with native build systems for installation.

Hi all,

We come back with a new issue this week. If you like BetterDev, please help spead word out by refer to your friends. Buy Me a coffee would be great too.

This issue is my favourite issue with a lot of pracitcal, action-able item which close to our day to day work. I hope you like them.

• How to write a code formatter

  We use code formatter all day long. But what is the process of implement such as a tool?

• A search engine in 80 lines of Python

  In this post I will walk you through the journey of buliding a search engine from scratch using Python. As usual, all the code I’ve written can be found on my GitHub (microsearch repo). This implementation doesn’t pretend to be a production-ready search engine, just a usable toy example showing how a search engine works under the hood.

• The Simple Beauty of XOR Floating Point Compression

  I recently implemented a small program to visualize the inner workings of a scheme that compresses floating point timeseries by XORing subsequent values. The resulting visualizations are quite neat and made it much easier for me to understand this beautiful algorithm than any of the explanations that I had previously encountered.

• A simple query with a big problem

  Render.com staging pipeline got blocked due to what looked like a simple query

  SELECT e.id FROM events e JOIN postgres_dbs db ON (e.data ->> ‘serviceId’) = db.database_id LIMIT 1;

  Can you spot what’s wrong? Let’s read

• Setting up PostgreSQL for running integration tests

  I’ll explain how to set up PostgreSQL for tests and discuss some common pitfalls to avoid.

• Object structure in JavaScript engines

  Love it or hate it. We will stick with JavaScript for quite a long time. Understanding these objects and its memory lay out is a must to improve your performance optimization skill

• I accidentally built a meme search engine

  Or: how to learn about clip/siglip and vector encoding images

• How macOS opens a file in the correct app

  One of feature of Mac is it can detect the right app to open a file, even without a file extension. How did it works and how it evolve

• Can You Grok It – Hacking together my own dev tunnel service

  Lets learn how to re-create a service similar to ngrok by re-using existing software such as nginx and openssh. There is also a great repository for these tunneling softwares

• How do machine grok data

  By apparently overtraining them, researchers have seen neural networks discover novel solutions to problem

Code to read

• crystal ruby

  a gem that allows you to write Crystal code, inlined in Ruby. All you need is a modern crystal compiler installed on your system.

• great-tables

  Easily generate information-rich, publication-quality tables from Python. Read how it’s built

• devpod

  Like Codespaces but open-source, client-only and unopinionated: Works with any IDE and lets you use any cloud, kubernetes or just localhost docker. Pretty big so hard to grok all the code but will give an idea on how to approach these kind of system

• PythonMonkey

  A Mozilla SpiderMonkey JavaScript engine embedded into the Python VM, using the Python engine to provide the JS host environment.

Tools

• madge

  is a developer tool for generating a visual graph of your module dependencies, finding circular dependencies, and giving you other useful info.

• hysteria

  a powerful, lightning fast and censorship resistant proxy.

• rotz

  Fully cross platform dotfile manager and dev environment bootstrapper written in Rust.

• pg_tde

  This is an experimental encrypted access method for PostgreSQL 16. Read introduction post

• awesome-postgres

  A curated list of awesome PostgreSQL software, libraries, tools and resources,

• redka

  Redis re-implemented with SQLite

• postgres index advisor

  A PostgreSQL extension for recommending indexes to improve query performance.

Hi all,

We come back with a new issue this week. If you like BetterDev, please help spead word out by refer to your friends. Buy Me a coffee would be great too.

This week, I was helping a friend doing some network setup, and I realize there is many thing that I don’t understand so I share a link to learn about linux networking. It’s heavily in C, but I guess there is no way around to truly gasp these concept.

• What even is a JSON number?

  Not a question people generally ponder, and seems fairly straightforward. It’s a number, obviously! But the question turns out to be somewhat hard to answer, and for API designers especially, the answer is really important to know!

• Building My First Homelab Server Rack

  Michael share with us how he built his first homelab server rack, a genuine post without affiliate links so we know he wasn’t bias.

• Using Multiple GitHub Accounts

  If you share laptop for home and work, or doing consultant for clients, you probably know the hassle of using different github account. This short guide help you configure just that, to make your life easiser by having git automatically load the right account based on directory.

• Everything I know about SSDs

  Solid State Devices using NAND Flash, how they differ from Hard Drives, and how they affect file deletion and recovery

• pid-controller explainer

  A PID controller is a way to solve problems with the following formulation:

  You can change some input to the system, called the process variable You have a sensor which monitors something about the system You want the sensor measurement to be close to some target value, called the set point

  The PID controller is a good way to decide what the input to the system should be without knowing anything about the internal workings of the system, except that the change in output is roughly proportional to the input.

• Postgres Locks Explorer

  lock is a very powerful primitive. We use lock all the time in thread programming with mutex. Postgres offer locks to work as across server, so application can request lock and use the central Postgres server as a coordinator. This small website is like a flash card to review all different locks in Postgres.

• Taming PostgreSQL WAL File Growth

  Have you ever wake up with an alert and look into Postgres datadir you notice pg_wal is growing massively in size? Read on to learn about why this happens, whether you should be concerned, and what you can do about it.

• Optimizing SQLite for servers

  SQLite default config is optimized for embeded use case, so most people trying it will encounter poor performances and the dreaded SQLITE_BUSY error. However, you can configure SQLite to reach ~8,300 writes / s and ~168,000 read / s concurrently, with 0 errors, on a ~40€ / m commodity virtual server with 4 vCPUs

• Linux networking

  This is somewhat a very low level. But recently I found I had not enough knowledge to understand networking at a very fundamental level. What exactly happen when we make a request, how the tcp connection is open, how the package is sent. how it picks which ethernet interface to send to? So I decide to google around and found this. I plan to studying these. I think understand just 50% of this is already put you on top 10% of developers.

Code to read

• excelize

  Go language library for reading and writing Microsoft Excel™ (XLAM / XLSM / XLSX / XLTM / XLTX) spreadsheets

• vernier

  Next-generation Ruby 3.2.1+ sampling profiler. Tracks multiple threads, GVL activity, GC pauses, idle time, and more.

• rpgp

  Pure rust implementation of OpenPGP

• facesjs

  A JavaScript library for generating vector-based cartoon faces

Tools

• pg_basebackup

  Since Postgres 12, it has built-in support to take a base backup of a PostgreSQL cluster. If you have been glueing together pg_dump, pg_restore, etc you will love this tool too. Pair it with an optinal compression setting to get the best compression rate.

• doltgresql

  Imagine a Postgres with git style log, diff, branch, and merge functionality. This is just that. The company behind it originaly released a MySQL flavored, but now they are taming Postgres.

• automq

  a cloud-native, serverless reinvented Kafka that is easily scalable, manage-less and cost-effective. It uses Object storage(s3, gcs) as the main storage. Using EBS for durable cache layer to accelerate write. It’s 100% compatible with Apache Kafka greater than 0.9.x and not lose any good features of it, but cheaper and better.

• drawdb

  robust and user-friendly database entity relationship (DBER) editor right in your browser. Build diagrams with a few clicks, export sql scripts, customize your editor, and more without creating an account.

• glimmer

  a DSL framework to build GUI app in Ruby. It contains a DSL engine to express he GUI, and a data binding lib to sync GUI with model attributes bidirectionally.

Hi all,

We come back to my second week :). If you like BetterDev, please help spead word out by refer to your friends. Buy Me a coffee would be great too

In previous email, I said about a new self hosted section. Due to a bug the sefl hosted doesn’t appear in email, you can go back to re-read the previous issue for awesome tool that you can run on your own hardware.

Onward to our news issue today.

First thing, there is a suppy chain attack on xz, it has assigned CVE-2024-3094, If you had install xz 5.6.x, that version is affected. If it’s a server, better to destroy and rebuild a new server with xz 5.4.

If you’re using mac, you can check which xz version you had installed

brew info xz

If it return 5.6, perform an brew upgrade xz and it will rollback to 5.4.

Continue on this story, no one know the real identity of the attacker so far. He has spend 2 years contribute code in a legitimate way. 2 years of works. Had he implement the backdoor correctly and not causing slowness, probably no one has notice. Someone shared a similar story on mastodon.

Another dance in open source community is the Redis license change, rest assured that if you’re just using Redis as an end-user theree is nothign change, you will be continue to use it. On top of that, Linux foundation has already launch Valkey , which is a fork of Redis right before the license change. In Redis codebase, top changeset contributions by employe is Tencent, not even Redis Lab. Alibaba comes in third place, then Huawei. So I think there will be definetely a path to the official Redis, similar to the IO.js story back in the old day, where everyone switch to the community fork. So no action require as an end-user, in the meantime, exploring the new ValKey project.

• Garbage Collection for Systems Programmers

  Let’s talk about one of the most performance-sensitive programs you run every day: your operating system. Since every speedup gives you more computer to compute with, an OS is never fast enough, so you can always find kernel and driver developers optimizing the bejesus out of their code.

• Radios, how do they work?

  A brief introduction to antennas, superheterodyne receivers, and signal modulation schemes.

• JavaScript Visualized: Promise Execution

  Due to its easi of use, and its forgiven model, promise remain an unclear? should I use return Promise<Value> or, return Promise.Resolve(value). all of that. Let look deeper to understand their execution model.

• Hacking our PostgreSQL driver to get more out of Amazon RDS performance insights

  In the process of improving Knock performance, Knock engineers found ourselves in the weeds hacking our Postgres driver, and we wanted to share some fun learnings we’ve gathered so far.

• Running a Raspberry Pi with a read-only root filesystem

  Many applications that run on Raspberry Pis and similar single-board computers — for example, environmental data loggers that report to a central database server — don’t really need to store any state locally on the Pi’s SD card. This means you can run the Pi with a read-only root filesystem, which will dramatically increase the SD card’s lifetime. Also good discussion on Hacker News for alternative method

• How does Linux nat a ping

  Pretty much any home computers are behind a NAT, mean multiple computer share a same public ip. In the TCP/UDP packet, there is a port, the pair ip:port can be used to track which host and send the packet back. But ping operate with ICMP, and ICMP packet doesnt have a port field. So how does NAT work for ICMP packets?

• Attacker Techniques: Gesture Jacking

  When a website ask you to hold down a key, what can happen? What is User Gesture? How to defend.

• File Monitoring with eBPF and Tetragon

  Anytime you start to implement HIPAA, SOC, NIST PCE-DSS, the file integrity monitoring come up? who changed this file? when? what is changed? In this articule, we look into that with Tetragon. Not just for security, say you have a config file, such as an env file that is source before application run, tracking changes to this file can help spot and solve problem in an outage too.

• finding memory leaks in Postgres C Code

  What is the process of this? what tooling, what setup, what is a step by step guide in layman term.

• Digital wallets and the “only Apple Pay does this” mythology

  Apple Pay is great, but I think there is some misunderstanding out there about the details of how it works.

• Memory leaks in Go

  A useful article to go over commont leaks in Go, method to find them.

Code to read

• charcoal

  Faster utf8.Valid using multi-byte processing without SIMD. Up to 50% faster than the standard library. Can be a useful tool to deal with email parser for example where we want to filter out non utf8 character

• weron

  Overlay networks based on WebRTC

• node-sql-parser

  Parse simple SQL statements into an abstract syntax tree (AST) with the visited tableList and convert it back to SQL

• node-telnet-client

  A simple telnet client for Node.js

• lorax

  Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

• pgvecto

  Scalable, Low-latency and Hybrid-enabled Vector Search in Postgres. Revolutionize Vector Search, not Database.

• noticed

  a gem that allows your application to send notifications of varying types, over various mediums, to various recipients. Be it a Slack notification to your own team when some internal event occurs or a notification to your user, sent as a text message, email, and real-time UI element in the browser, Noticed supports all of the above (at the same time)!

Tools

• pglite

  Lightweight Postgres packaged as WASM into a TypeScript library for the browser, Node.js, Bun and Deno. Use case is you may run an interactive embeded postgres in browser for a playground postgres application.

• tetragon

  owerful real-time, eBPF-based Security Observability and Runtime Enforcement such as monitoring file changes, block operation to file etc.

• monogon

  A pure Go minimal operating system. Based on Linux and K8S. It runs on a fleet of bare metal or cloud machines and provides users with a hardened, production ready Kubernetes, without the overhead of traditional Linux distributions or configuration management systems.

• monolith

  CLI tool for saving complete web pages as a single HTML file

• retina

  eBPF distributed networking observability tool for Kubernetes.

• Atlassian Pragmatic drag and drop

  a performance focused drag and drop library that can be used to power any drag and drop experience on any tech stack. Pragmatic drag and drop makes working with the browsers powerful and flexible built in drag and drop functionality safe and easy.

Hi everyone,

I’ve returned this week and hope to make this newsletter a more regular occurrence. I apologize if my absence and lack of updates have disappointed any of you.

Since we last caught up, I’ve made several updates. I’ve polished my side project, ec2.shop, which I believe is the best tool for quickly viewing all on-demand, spot, and reserve prices on a single page. Give it a try and let me know what you think.

One significant change in my perspective since last year is my strong belief in the importance of self-hosting more nowadays, rather than relying on free tiers or cloud providers.

I’ve witnessed cases where individuals have lost years of photos after being locked out of their Google accounts, with no means of contacting a real person to resolve the issue. Moving forward, I’ll be adding a self-hosted section so we can started to move into the direction of owning our own content. It’s also the reason my newsletter platform is home-grown, self-made.

Becuase it’s been a while since I last sent out this newsletter, if you nolonger enjoy the content, you can click unsubscribe at the bottom of this email.

• Fifty Things you can do with a Software Defined Radio

  As a software engineer, I like to tinker around hardware projects, physical thing that you can touch, and show to our faimily. Radio is fastinating because there are many valueable information being pass around. You can listen to airplane communication, track aircraft via ADS-B, you will write some code to decode the signal and make sense of them. You can even throw a front-end to render them on a map for example.

• HTTP/2 and HTTP/3 explained

  Understand better how HTTP works in each version.

• GPT in 500 lines of SQL

  It just proves that if you want something done right, you have to do it yourself. Encouraged by this optimistic forecast, today we will implement a large language model in SQL.

• Stylized image binning algorithm

  Given a color image? How can we convert it to black and white, pixel-art-like images? The cool thing about these pixel-art image is they can be easily embed into a README.md and render great. We will learn the binning algorithm, implement purely just with JavaScript.

• Optimizing Javascript for fun and for profit

  JavaScript is quick and easy to write. The language is very forgiable so can easily write code that isn’t optimize. In this post we will learn a few way to optimize JS.

• How Figma’s databases team lived to tell the scale

  I’m honst everytime I get into Figma, I’m frustrated and think how everyone using them? The UI is so clunky, I always have to make sure I don’t accidently move something. But anyway, they operate at massive scale and today they share the story they migrate and shard their database.

• The Day Soft Deletes Caused Chaos

  Soft delete is the practice of not deleting the row physically but create a flag that we delete a row. It’s very simple to implement because deleting data isn’t easy. Such as deleting a user lead to remove many relevant data, relevant foreign key. Data can also refer to external data, such as user avatar store on S3. because of that, many use soft delete. it does add complexity for operation though, and what can go wrong, we will learn today.

• Linux Crisis Tools

  When you have an outage caused by a performance issue, you don’t want to lose precious time just to install the tools needed to diagnose it. Here is a list of “crisis tools” Bredan recommend installing on your Linux servers by default. I would suggest to add 4 mores: nmap, nc, netstat and strace

• One Billion Row Challenge - From 95s to 1.96s

  The One Billion Row Challenge is to write a Java program for retrieving temperature measurement values from a text file and calculating the min, mean, and max temperature per weather station. There’s just one caveat: the file has 1,000,000,000 rows! It has been expand and adopt by many languages or eco system. This one show an optimization to bring up almost 20x performance. There one another similar challenge of moving a billion postgres rows on 100 budget

Code to read

• rotary

  do you remember years ago you can dial a number on the phone, they play some music. We replicate this setup here with old hardware, open source project like Asterisk to do that. If you have a kid def you would want to look into it. Not an end to end soluition but I show a full setup, how thing tie together, which component you need.

• pgwire

  Have you ever wonder what is the process of writing a database library ? This project shows that, implement PostgreSQL wire protocol in Rust

• maxtext

  a high performance, arbitrarily scalable, open-source, simple, easily forkable, well-tested, batteries included LLM written in pure Python/Jax and targeting Google Cloud TPUs. MaxText typically achieves 55% to 60% model-flop utilization and scales from single host to very large clusters while staying simple and “optimization-free” thanks to the power of Jax and the XLA compiler.

• debugbar

  Powerful devtools for Ruby on Rails. Inspired by the Laravel Debugbar.

• fgprof

  fgprof is a sampling Go profiler that allows you to analyze On-CPU as well as Off-CPU (e.g. I/O) time together. I’m always interested in how to implement this CPU profiling tool. The project is less than 1000 LOC so you can grasp the idea behind quickly

Tools

• pgzx

  There are already tool to help write Postgres extension with Rust such as pgrx. Now we can do with Zig. After the last 2 years working with Rust, I feel like it’s such as great language that is truely innovative compare with other, but there is still room for a low level language that isn’t too strict such as Zig. Using zig to write a postgess extension feel great.

• atrament

  A small JS library for beautiful drawing and handwriting on the HTML Canvas. An use case is you can use to take user signature and have it smoothen out. Check the demo as well.

• grover

  A Ruby gem to transform HTML into PDFs, PNGs or JPEGs using Google Puppeteer/Chromium. Useful when generating invoice

• cargosvg: Elegant, simple interactive maps

  If you want to generate some kind of activity maps this is a good tool. Example in my email forward app I had a world map on homepage which can probably re-impelemnt with this tool

• goqite

  a persistent message queue Go library built on SQLite and inspired by AWS SQS (but much simpler).

• janai

  An open source alternative to ChatGPT that runs on the desktop.

• ruby-advisory-db

  A database of vulnerable Ruby Gems. useful to not miss a vulnerable if you are using Ruby

• pgcopydb

  Copy a Postgres database to a target Postgres server (pg_dump | pg_restore on steroids)

• EchoVault

  highly configurable, distributed, in-memory data store and cache implemented in Go. Think of it like a Redis alternative but speak the same protocol, therefore we can reuse the same client code, use the same our favorite Redis client in our language. With the recent Redis license change, it’s time to look for alternative.

This week I come across a tool call LocalAI. I always curious how to replicate OpenAI, have them run locally and this LocalAI is the answer, bill itself as free, Open Source OpenAI alternative. Self-hosted, community-driven and local-first. Drop-in replacement for OpenAI running on consumer-grade hardware. No GPU required. Runs ggml, gguf, GPTQ, onnx, TF compatible models: llama, llama2, rwkv, whisper, vicuna, koala, cerebras, falcon, dolly, starcoder. It defs live up to its hype with 12k+ stars on Github. Super easy to get started.

• Using AWK and R to parse 25tb

  Knowing tools like awk, sed helps immensively when need to process text based log file fast. AWK in itself is a whole programing language. Lets follow this journy of bringing the query time and cost down from 8 minutes and $20 to a tenth of a second and less than a penny,

• How Discord Stores Billions of Messages — Big Surprises in System Design

  The biggest supprise is they don’t use cache.

• What is in that .git directory?

  Well, I think most of you reading this blog use git more or less on a daily basis, but have you ever looked into what is in the .git folder that git creates? Let’s explore it together and understand what is going on in there.

• An Interactive Intro to CRDTs

  Have you heard about CRDTs and wondered what they are? Maybe you’ve looked into them a bit, but ran into a wall of academic papers and math jargon?

• Profiling Python and Ruby using eBPF

  Dive into the internals of profiling interpreted Python and Ruby code using eBPF

• BPFAgent: eBPF for Monitoring at DoorDash

  Metrics, logs, and traces provide vital information. But these signals almost entirely rely on application-level instrumentation, which can leave gaps or conflicting semantics across different systems. DoorDash decided to seek potential solutions that could provide a more complete and unified picture of their networking topology

• Scaling GraphQL with Postgres - Lessons learned from our database timeout issues

  Cycle app has a lot of issue about time out db connection. They workaround by run health check more often to auto restart. Wh

• Five Great Features of the PostgreSQL Partition Manager

  pg_partman is a tool to partition your table into either time based or number based. In this article they discussed 5 important features to help managing large tables in PostgresSQL.

• Fine-Tuning Mistral7B on Python Code With A Single GPU!

• Flui Simulation

  A step by step notion note of Kynd, explain how to design and implement a fluid simulation. Very detail at every step, together with codepen to run and follow. You will learn a lot follow this

• Speeding up the JavaScript ecosystem - The barrel file debacle

  Part 7 of this series about JS performance. In this, Marvinhh discussed why the import-re-export in a index.js, while convenience for consumer cause perfomrnace degrade.

Code to read

• mods

  LLM based AI is really good at interpreting the output of commands and returning the results in CLI friendly text formats like Markdown.

• paradedb

  PostgreSQL for Search

• Viselect

  A high performance and lightweight library to add a visual way of selecting elements, just like on your Desktop. Zero dependencies, super small. Support for major frameworks!

• ziggy-pydust

  A toolkit for building Python extensiions in Zig

• faststream

  simplifies the process of writing producers and consumers for message queues, handling all the parsing, networking and documentation generation automatically.

• httpx

  An HTTP client library for ruby that support HTTP/2

• nokolexbor

  Nokolexbor is a drop-in replacement for Nokogiri. It’s 5.2x faster at parsing HTML and up to 997x faster at CSS selectors. Another similar one is Oga but it has been lack of active development

Tools

• evidence

  an open source, code-based alternative to drag-and-drop business intelligence tools. You can write markdown, and generate embed chart(through query your warehouse), like traditional BI tools but now can also embed them into markdown.

• audiomotion

  A high-resolution real-time audio spectrum analyzer JavaScript module with no dependencies

• logster

  Log viewer UI and framework for rack

• Chrome-GPT

  An experimental AutoGPT agent that interacts with Chrome

• TRAKTOR

  Set up true multimaster replication clusters with vanilla PostgreSQL servers.

• pg_cirrus

  Set up 3-Node highly available PostgreSQL cluster

• obligator

  Simple and opinionated OpenID Connect server designed for self-hoster

• Caddy Security

  Auth Portal and Authorize Plugins for Caddy v2

• payloadcms

  The best way to build a modern backend + admin UI. No black magic, all TypeScript, and fully open-source, Payload is both an app framework and a headless CMS.

• pgmetrics

  Collect and display information and stats from a running PostgreSQL server. A must say to have, very easy to install with a single binary. A similar tools but tailor toward Ruby/Rails is PgHero

Last week, the famous Insomnia rugpull us and require an account to use the software. In process, many has lost their project data when the migration happen from local to cloud. There is a new uprising star emerge, let give Bruno a try.

This week, I came across a very interesting tool to fork the db, a feature I think every engineer has been dream about. Still in beta but give pg_branch a try. It’s written in Rust so the code base is more approachable compare with C/C++.

• Writing a Debugger From Scratch - DbgRs Part 5 - Breakpoints

  There is an infamous saying: One cannot understand thing until they made it. In this series, Tim shared how to approach writing a debugger. This part is about breakpoints but make sure to check other parts too.

• Hardening macOS

  There are many options that is convenience but it isn’t secure enough. Such as disk encryption is not enable, lock screen isn’t auto activate. This is a guide to enhance your security, especially useful when connecting to public wife for example.

• Understanding Postgres IOPS: Why They Matter Even When Everything Fits in Cache

  Disk IOPS (Input/Output Operations Per Second) is a key metric for measuring the performance of any disk system. It represents the number of read and write operations that can be performed per second. For PostgreSQL, which relies heavily on disk access, understanding and optimizing disk IOPS is crucial for achieving optimal performance.

• From Chaos to Recovery: How We Restored Our AWS Microservice After Accidental Deletion at Dolap

  Dolap accidently remove their comment service infra. They click delete on “Cloudformation” stack, which is a process that cannot be cancel. Panic. The key thing help them is “deletion protection” on RDS. While normal stateless component will be removed, the core place where data remains is safe. So do yourself a favor and enable “deletion protection” on all of your stateful resource today.

• Your Cache Headers Could Probably be More Aggressive

  It’s common for modern hosts to cache static assets in a flexible, but not most optimal way. Let’s explore why that is and what we can do to push cache performance (for some assets) even further.

• Kind Engineering: How To Engineer Kindness

  How to be kind and provide constructive feedback instead of being a brilliant jerk?

• Storage challenges in the evolution of database architecture

  AWS RDS has a physical storage size limit of 128TiB for each RDS cluster. Postman has a db of 95TB and growing at 2TB per month. So they plan to prune the old data. However, just drop a table won’t reclaim the space. They needs to rebuild the table, but it’s a costly process. This is real world engineering, read to see how they become creative and overcome the challenge.

• Intercepting and modifying Linux system calls with ptrace

  How software fails is interesting. But real-world errors can be infrequent to manifest. Fault injection is a formal-sounding term that just means: trying to explicitly trigger errors in the hopes of discovering bad logic, typically during automated tests.

• Navigating personal information with care

  This article is a Rails solution by 37signals fork themselve. But many principal can be applied to anyone. Every engineer would need to debug issue, and have a need at poke into some kind of console app to access to the sensitive data. How do we log those access, get approval to run those. how they glue these parts together

• Upgrading frontend dependencies with confidence

  A very practicle advice you can do today to handle front-end upgrade. It leads us through a process of setting up github action, run and take screenshot of your app. Compare them visually. of course, we will need some tips to hide flaky element along the line. Such as when first load page there might be no pop-up, but waiting 5 second and a pop-up may appear. Or having animation that appear based on time.

Code to read

• bitset

  bitset is a mapping between non-negative integers and boolean values. It should be more efficient than map[uint] bool.

• sonnet

  High performance JSON library in Go. Fully compatible with the Go standard library, Both the encoder and the decoder passed all tests from it.

• atone

  A VecDeque (and Vec) variant that spreads resize load across pushes.

• n64.js

  an n64 emulator in JavaScript

• mistral-sr

  Reference implementation of Mistral AI 7B v0.1 model.

• terminal7

  a terminal multiplexer re-designed for remote servers and hi-res touch screens. A reincaranation of tmux and screen, Terminal7 is a hybrid app that works best on the iPad.

• prosopite

  Rails N+1 queries auto-detection with zero false positives / false negatives.

• gemstash

  A RubyGems.org cache and private gem server

• mimic.css

  an css animations library because everyone else is doing it.

Tools

• flameshow

  a terminal Flamegraph viewer.

• iredis

  A Terminal Client for Redis with AutoCompletion and Syntax Highlighting.

• release-please

  generate release PRs based on the conventionalcommits.org spec

• gitify

  GitHub notifications on your menu bar. Available on macOS, Windows & Linux

• small chrome extension to monitor (and optionally block) other extensions’ network calls

• ebitengine

  open source game engine for the Go programming language. Ebitengine’s simple API allows you to quickly and easily develop 2D games that can be deployed across multiple platforms, including Nintendo.

• instant.dev

  Build APIs with JavaScript and Postgres, fast.

• ollama

  Get up and running with large language models, locally.

• sqlcoder

  a 15B parameter model that is fine-tuned on a base StarCoder model. It slightly outperforms gpt-3.5-turbo for natural language to SQL generation tasks on the sql-eval framework, and outperforms popular open-source models. It also significantly outperforms text-davinci-003, a model that’s more than 10 times its size.

• hyperdx

  helps engineers figure out why production is broken faster by centralizing and correlating logs, metrics, traces, exceptions and session replays in one place. An open source and developer-friendly alternative to Datadog and New Relic.

• argo-cd

  a declarative, GitOps continuous delivery tool for Kubernetes.

• woodpecker

  a simple CI engine with great extensibility.

Hi everyone, This week, many resorts in Las Vegas got hack and have to shutdown their system. That means many operation is handled by paper and pen. The hack appears to be executed by BlackCat/ALPHV ransomware group and you can read about their statement and analysis here. A reminder for all of us to be more caution with social engineering hacks.

• What's the toughest bug you ever found and fixed

  A jpeg parser, running on a surveillance camera, which crashed every time the company’s CEO came into the room. Read to know why :-).

• In a git repository, where do your files live?

  We know that it’s in your .git directory, but where exactly in there are all the versions of your old files?

• Network health overview with mtr, ss, lsof and iperf3

  Sometimes our Internet connection seems weird, and we don’t understand why. We know something is wrong, but we cannot really pinpoint the issue. And other times, there is weird file access that goes undetected simply because we don’t know what to look for. Logs won’t be helpful in those case because they might be logged. Lets’ look at the tool that help us identity and analyze network traffic.

• How Microsoft does Quality Assurance (QA)

  Microsoft has played an outsized role in the development and importance of quality assurance across the industry. Microsoft was the first major company to come up with a specialized testing role which went well beyond manual testing.

• Get All That Network Activity Under Control with Priority Hints

  Open up the browser’s network tab and you’ll see a lot of activity. Assets are being downloaded, information’s being submitted, events are being logged, and more. How can we priority more important request, and dedicated more browser(end-user) bandwidth to these.

• All the ways to capture changes in Postgres

  How do we track changes to a database and react to it? Such as sending an email when a new customer are added to the table out of bound?

• SQL join flavors

  There is more to SQL joins than you might think. Let’s explore them a bit. I confess that I didn’t know about lateral join

• Active active Postgres 16

  While I’m a big fan of Postgres, I have used MongoDB extensively too and their replicaset is amazing. All the failover and done automatically, with built-in client auto-retry and auto discover help the whole setup. With Postgres, we still have a lot of work to do to get there, but thing are improving in Postgres 16.

• Securing Your Node.js Apps by Analyzing Real-World Command Injection Examples

  No matter your stack is, at some point, NodeJS will enter the table. Knowing how to operate them securely is super useful.

• Scaling Kafka to Support PayPal’s Data Growth

  To handle the tremendous growth of PayPal’s streaming data since its introduction, Kafka needed to scale seamlessly while ensuring high availability, fault tolerance, and optimal performance. In this blog post, we will provide a high-level overview of Kafka and discuss the steps taken to achieve high performance at scale while managing operational overhead, and our key learnings and takeaways.

Code to read

• chrono

  a pure JS lib to turn arbitraty date string to right datetime such as “2 day ago”, “week from now”, “this friday”…etc

• ruby-clock

  A ruby job scheduler which runs jobs each in their own thread in a persistent process.

• pdfplumber

  Plumb a PDF for detailed information about each text character, rectangle, and line. Plus: Table extraction and visual debugging

• Maroto

  In every engineer career, at some point, you would need to generate a PDF. This lib allow you to generate PDF as if working with bootstrap, a row has column, a column has component. Support many component for bar code, data matrix, custom style.

• re2

  Ruby bindings to re2, a “fast, safe, thread-friendly alternative to backtracking regular expression engines like those used in PCRE, Perl, and Python”.

• lychee

  Fast, async, stream-based link checker written in Rust. Finds broken URLs and mail addresses inside Markdown, HTML, reStructuredText, websites and more!

Tools

• lux

  Fast and simple video download library and CLI tool written in Go. Support many services like youtube, bilibili, tiktok, Tumblr, vimeo, instagram etc

• plain

  a Rails engine that serves as an Artificial Intelligence (AI) assistant for your Rails project.

• fx

  Terminal JSON viewer & processor

• pzip

  A blazing fast concurrent zip archiver and extractor.

• peerdb

  Postgres-first data-movement platform that makes moving data in and out of Postgres fast and simple. It enables you to sync, transform and query data across your stores using simple SQL commands. We implement multiple Postgres native and infrastructural optimizations for 10x faster data-movement in and out of PostgreSQL.

• vale

  an open-source, command-line tool that brings your editorial style guide to life. It runs entirely offline, no cloud. single binary. Think of it like a tool that you run to enforce a certain kind of rule and language for your document. It supports markup language well.

• electric

  Local-first sync layer for web and mobile apps. Build reactive, realtime, local-first apps directly on Postgres. Build your app with its standalone db, locally on device storage, sync to a Postgres db on the background for persistent across device, conflict free.

Happy tuesday everyone. We’re back for other issue. Lot of exiciting article. Hope you like this issue. This issue is a bit short and lack the summary but hope you still like it.

• Running my own LLM

  Instead of just using OpenAI and consume, we can look into how to run and play with a real LLM on your own computer. That will help us learn much faster, especially if we’re dipping our toe into AI/ML. Also checkout the petals tool at the tool section in this issue.

• Optimizing LLMs From a Dataset Perspective

  This article focuses on improving the modeling performance of LLMs by finetuning them using carefully curated datasets. Specifically, this article highlights strategies that involve modifying, utilizing, or manipulating the datasets for instruction-based finetuning rather than altering the model architecture or training algorithms. This article will also explain how you can prepare your own datasets to finetune open-source LLMs.

• How They Bypass YouTube Video Download Throttling

  This sounds so easy. Just get youtube raw video and download it. But turning out they throttling it, how do those download script by pass it?

• Session management security: Best practices for protecting user sessions

• Apple bleee. Everyone knows What Happens on Your iPhone

• How does UTF-8 turn “😂” into “F09F9882”?

• How fast are Linux pipes anyway?

• JPEG and EXIF Data Manipulation in Javascript

• Exploring the design space of binary search trees

• Fun with PostgreSQL Puzzles: Recursive Functions with Animations

• What actually happens when you COPY in Postgres

• Optimizing Postgres's Autovacuum for High-Churn Tables

Code to read

• pgJQ

  Use jq in Postgres

• Magnus

  Ruby bindings for Rust. Write Ruby extension gems in Rust, or call Ruby from Rust.

• adornable

  Python has a super cool feature where you add @name on top of a function and name is call, wrap your function inside. Now we can do similar to that in Ruby, implement entirely as a library at Ruby level. no new keyword, just pure Ruby syntax.

• LLaMA2-Accessory

  An Open-source Toolkit for LLM Development 🚀

• libbpf-rs

  Minimal and opinionated eBPF tooling for the Rust ecosystem. Read the introduction post to play around

• bore

  a simple CLI tool for making tunnels to localhost

Tools

• Run large language models at home, BitTorrent‑style

• outlines

  Generative Model Programming

• chDB

  an embedded OLAP SQL Engine powered by ClickHouse

• llama-gpt

  A self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device.

• serge

  is a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!

• Llama2 Embeddings FastAPI Service

• netmaker

  WireGuard® automation, quick way to setup VPN with admin ui to manage user.

• netbird

  Another wireguard based VPN automation with SSO/MFA. Wireguard is very performance and easy to setup, but there isn’t an easy solution to manage user out of the box. Therefore we need tool like netmaker or netbird

Hi all,

I’m really sorry for sending out no issue in last few months. Works get in but I’m finally manage to spend time again on this newsletter. I missed it so much and cannot wait to share exciting link with everyone once again.

• Putting the “You” in CPU

  Curious exactly what happens when you run a program on your computer? Read this article to learn how multiprocessing works, what system calls really are, how computers manage memory with hardware interrupts, and how Linux loads executables. The site is also available on github.

• Llama from scratch (or how to implement a paper without crying)

  I want to provide some tips from my experience implementing a paper. I’m going to cover implementing a dramatically scaled-down version of Llama for training TinyShakespeare. This post is heavily inspired by Karpathy’s Makemore series, which I highly recommend.

• Zero-Downtime PostgreSQL Cutovers

  Even when using Multi-AZ on RDS, there is still a chance of downtime where the DNS switch to the standby in other AZ and our app need to switch together. There is a brief moment the app is still connecting to the old primary instance. In this, Instacart shared how they achieved that.

• Optimizing speed on eBay.com

  For every 100 milliseconds improvement in search page loading time, eBay saw a 0.5% increase in “Add to Cart” count. Through the adoption of Performance Budgets, and a focus on key user-centric performance metrics, eBay was able to make significant improvements to site speed.

• Migrating Netflix to GraphQL Safely

  In 2022, a major change was made to Netflix’s iOS and Android applications. They migrated Netflix’s mobile apps to GraphQL with zero downtime, which involved a total overhaul from the client to the API layer. lets dive into to see how they do this safely for 100s of millions of customers without disruption.

• After four years of SMR storage, here's what we love—and what comes next

  Dropbox for sure store tons of data. In 2018, they adopt high-density SMR (shingled magnetic recording) technology for storage drives. SMR technology has reduced the power requirements of storage servers, lowered the cost of each exabyte, and enabled more efficient server rack designs. Today, we will learn about all of that.

• 10 Postgres tips for beginners

  Getting started with PostgreSQL can be both exciting and challenging. It’s a whole eco system with so many toolings, features. In this post, Postgre.ai team discuss 10 tips to for beginner to get started.

• Understanding database Indexes in PostgreSQL

• Why macOS anti-malware scans can behave oddly

  macOS Catalina and later include an anti-malware scanning service, XProtect Remediator (XPR). Many didn’t know about it.

• Lamport Clocks: Determining the Order of Events in Distributed Systems

  Distributed systems are behind many services that we use every day. A significant challenge of said systems is to understand and manage the order of events across these independent machines. How can we know which event happened before another? What caused what?”

• Resilient LL Parsing Tutorial

  In this tutorial, I will explain a particular approach to parsing, which gracefully handles syntax errors and is thus suitable for language servers, which, by their nature, have to handle incomplete and invalid code. Explaining the problem and the solution requires somewhat less than a trivial worked example, and I want to share a couple of tricks not directly related to resilience, so the tutorial builds a full, self-contained parser, instead of explaining abstractly just the resilience.

Code to read

• sqledge

  Replicate postgres to SQLite on the edge.

• risor

  Fast and flexible scripting for Go developers and DevOps.

• learndb-py

  implementing a relational database management system (RDBMS) (sqlite clone) from scratch.

• mayu

  a live-streaming server-side component-based VDOM rendering framework written in Ruby

• luxon

  A library for working with dates and times in JS by Moment.

• svg-drawing

  A canvas for easy sketching and replay, exportable in SVG format.

• oto

  ♪ A low-level library to play sound on multiple platforms ♪

Tools

• fuite

  a CLI tool for finding memory leaks in web apps. Simply run and point it at your web app

• pqmq

  A lightweight message queue. Like AWS SQS and RSMQ but on Postgres. Read the introduction post by its creaator, tembo

• mailcatcher

  If you ever want to preview email on your app, mailcatcher is perfect for that. The project emerged 12 years ago and I though it’s was abandone but turning out still good going.

• HyperUI

  Free Open Source Tailwind CSS Components

• nuclei

  Fast and customizable vulnerability scanner based on simple YAML based DSL.

• toaster

  Pure CSS 3D Editor

• Easing functions

  Easing functions specify the rate of change of a parameter over time. Very useful when doing animation. Widely use in CSS animation. This site is like a cheatcode to help you pick the right one.

• maily

  Open source editor for creating beautiful, pre-designed, mobile-ready emails

These continue to be big weeks in AI world. I share some LLM tools. Enjoy.

• Cryptographic Best Practices

  Putting cryptographic primitives together is a lot like putting a jigsaw puzzle together, where all the pieces are cut exactly the same way, but there is only one correct solution. Thankfully, there are some projects out there that are working hard to make sure developers are getting it right.

• A Journey through Color Space with FFmpeg

  For those who want to understand color spaces, transform videos from one color space to another, or read how I almost went crazy working out why my Canva-generated videos looked slightly off, color-wise

• Semantic search in iMessage, iMessage wrapped, and ai conversations

  iMessage is slow and clunky when searching message. At the end, it’s just text store in a SQLIte database. If we can reverse engineer it, we can yield better search result

• I Made My Blog Solar-Powered, Then Things Escalated

  A very detail write up on how to generate enough power for the OP usage on his equipment. Fun experiment and I think it feeling great when knowing the blog is powered by Solar energy :)

• Load Balancing

  In this post we’re going to focus on the ways that a single load balancer might distribute HTTP requests to a set of servers. We’ll start from the bottom and work our way up to modern load balancing algorithms.

• Systems design 2: What we hope we know

  Someone asked if I could write about the rise of AI and Large Language Models (LLMs) and what I think that means for the future of people, technology, society, and so on. Although that’s a fun topic, it left me with two problems: I know approximately nothing about AI, and predicting the future is hard even for people who know what they’re talking about. Instead, I’ll tell you a bunch of things I do know that are somehow related to the topic, and then you can predict the future yourself.

• Tracing Notifications at Slack

  Slack is a very complicated system with complicated rule. Mute channel, when to send notification, where(email or device), did user see this notification in email already? Debugging notification is complicated at Slack scale, so they build a system to trace the notification across the system.

• Improving Fault Tolerance with RPC Fallbacks in DoorDash’s Microservices

  Challenge of Microservice is they call each others, one client request turns into many internal RPC call, and amplify the failure rate when making these requests due to nature of network. Retrying is often use, but if the service is completeley failed, how can we skip it immediately. This is DoorDash strategy for that

• Setting Up PostgreSQL Failover and Failback, the Right Way!

  help beginners understand and set up server replication in PostgreSQL using failover and failback. Postgres evolve fast and process to setup Postgres replication is way simpler than in Postgres 10 or 12 for example.

• Time Ranges without Overlapping

  Storing time ranges in a database is easy, but you must also prevent overlapping rows for some applications. Accomplishing this requirement can be challenging because of race conditions.

• Progress Reporting In PostgreSQL

  A tip to monitor the progress of certain operations in PostgreSQL? Yes, we can monitor the progress of these operations in PostgreSQL: index creation, re-index, vacuum, and more

Code to read

• sandpack

  A component toolkit for creating live-running code editing experiences, using the power of CodeSandbox.

• PL/Rust 1.0: Write Postgres Functions in Rust

  One of a neat way to extend Postgres is define your own function using SQL. But needless to say SQL has its limit, so there are PL/Python where you can define function with Python code. Now that go futher, you can define function in Rust, which compile to native machine code, not interprer through Postgres engine, so you will get great performance

• pgwire

  PostgreSQL wire protocol implemented as a rust library.

• credcheck

  A PostgreSQL extension provides few general credential checks, which will be evaluated during the user creation, during the password change and user renaming. By using this extension, we can define a set of rules like password policy or password expiration

• mrsk

  A tool by DDH to deploy web app anywhere. Similar to capistrano but with a strong focus on docker deployment and coordination with load balancer.

• web-llm

  This project brings language model chats directly onto web browsers. Everything runs inside the browser with no server support and accelerated with WebGPU. We can bring a lot of fun opportunities to build AI assistants for everyone and enable privacy while enjoying GPU acceleration.

• tokenizer

  Pure Go implementation of OpenAI’s tiktoken tokenizer. Also checkout OpenAI’s Go API client

Tools

• tabby

  Self-hosted AI coding assistant

• timoni

  a package manager for Kubernetes, powered by CUE and inspired by Helm.

• wayback

  A self-hosted archiving service integrated with Internet Archive, archive.today, IPFS and beyond.

• guardllama

  The Ultimate Open-Source VPN, Ad-blocker and DNS resolver 🦙🛡️

• list.js

  The perfect library for adding search, sort, filters and flexibility to tables, lists and various HTML elements. Built to be invisible and work on existing HTML.

• easeprobe

  A simple, standalone, and lightweight tool that can do health/status checking, written in Go.

• hydra

  A modern, open source replacement for enterprise data warehouses

• supavisor

  A cloud-native, multi-tenant Postgres connection pooler

• pgpool-II

  a tool that can help you manage and optimize the performance of your PostgreSQL database cluster. Enable you to do Load balancing, Connection pooling, Query caching and more.

• Iconify

  makes it easy to avoid vendor lock-in. You can use many open source icon sets with a large choice of open source icon components. Thousands of high quality icons from 100+ icon sets, all validated, cleaned up, optimised and always up to date

• Is your Postgres ready for production?

  your database ready for whatever may come on launch day or even 2 months in? Here’s a handy checklist to make sure you’re not caught flat footed.

• How to read MySQL EXPLAINs

  When you prepend the EXPLAIN keyword to the beginning of a query, it explains how the database executes that query and the estimated costs. By leveraging this internal MySQL tool, you can observe the following: The ID of the query — The column always contains a number, which identifies the SELECT to which the row belongs. The SELECT_TYPE — If you are running a SELECT, MySQL divides SELECT queries into simple and primary (complex) types, as described in the table below.

• Storing OAuth tokens

  I recently start working on a new SaaS app and I evaluate where to store auth token. I has always store them in cookie with HTTPOnly flag. Here, Dan walk us through the storage layer of an auth token. Dan is also author of letter to a news developer which you should subscribe if you haven’t

• SimulatedRides: How Lyft uses load testing to ensure reliable service during peak events

  Lyft explains why record and playback are not suitable for them and lead them to develop a custom performance testing framework that worked in production.

• Clocks and Causality - Ordering Events in Distributed Systems

  In distributed systems, logical clocks play a key role in the ordering of system events. What are the various logical clock designs, and how do they help with event ordering? This article answers these questions.

• The Secret Life of fsync

  We have heard that commodity ssd/hdd don’t provider durability such as a call to fsync won’t flush data reliably and can suffering data loss in case of power failure. there is nothing the OS can do about it. enterprise grade drives had this capability with battery/capacitor. Lets find out those claims.

• Navigating the unpredictability of everything

  As a programmer, we always have to deal with unpredictability. unknow requirements ahead of time, unknow of scale of our product. So what can we do to navigate this landscape?

• The Ultimate Guide to Image Optimisation

  Image optimisation is often the first step in a person’s web performance and page speed journey. It’s a relatively easy fix with potentially high rewards. Optimising a single image isn’t complicated or time-consuming, and it could equate to site speed gains, better UX, and noticeably improved Core Web Vitals.

• Anyone Can Play Guitar Series

  This is a really fun experiment. Sonic Pi is software to make sounds and music driven by code. Sonic Pi comes with an IDE of sorts. But you can also control it from a CLI to make sound. Pair it with musical notes parser and now we can play song.

• Hello Dolly: Democratizing the magic of ChatGPT with open models

  We don’t usually list product announcement but I think this is worth to mention. We all know and love ChatGPT from OpenAI. Since then, Meta released LlaMA together with the weight. Stanford built the Alpaca model, which was based on LLaMA, but tuned on a small dataset of 50,000 human-like questions and answers that, surprisingly, made it exhibit ChatGPT-like interactivity. Databricks’ show that, dolly-v1-6b, a large language model (blog post) trained on the Databricks machine learning platform, demonstrates that a two-years-old open source model can do. It’s good experiment to play around with these models.

Code to read

• thumbhash

  A very compact representation of a placeholder for an image. Store it inline with your data and show it while the real image is loading for a smoother loading experience. It’s similar to BlurHash but with a few advanage

• ruby-openai

  OpenAI API Ruby Client, Generate text with ChatGPT, transcribe and translate audio with Whisper, or create images with DALL·E…

• grover

  A Ruby gem to transform HTML into PDFs, PNGs or JPEGs using Google Puppeteer/Chromium. I used it to generate invoice for my app.

• python-openai

  The OpenAI Python library provides convenient access to the OpenAI API from applications written in the Python language.

• otpauth

  one time password generation for JS eco system: browser, node, deno and bun

• go-nbd

  Pure Go Network Block Device(NBD) server and client library. NBD is a network protocol that can be used to forward a block device (typically a hard disk or partition) from one machine to a second machine. It is similar to how EC2 EBS works.

• pg_gpt

  Postgres <> ChatGPT using PGX Rust to write PG extension. If you always want to write Postgres extension but don’t know where to start, and don’t want to write C, this is a very small repository that build on top of PGX, you can see how easy it is to write Postgres plugin with PGX in Rust, Everything is abstracted out. I picked this example because it involve dealing with external HTTP API, which is common in web dev, and also because it’s GPT :)

• icsp

  Command-line iCalendar (.ics) to CSV utility. Pure bash, no dependencies.

Tools

• git-fame

  A command-line tool that helps you summarize and pretty-print collaborators based on contributions

• serge

  A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.

• ccurl

  when dealing with third part API, we may usually use the copy as CURL of chrome to export a CURL command to run. Paste cookie in terminal is tedious. Turning out we can get the cookie value through chrome remote debuggin protocol. This small script show a way to do so.

• gotraceui

  a tool for visualizing and analyzing Go execution traces. It is meant to be a faster, more accessible, and more powerful alternative to go tool trace

• pgcat

  PostgreSQL pooler with sharding, load balancing and failover support.

• pgwarehouse

  a tool to sync Postgres database to a Snowflake, ClickHouse, or DuckDB warehouse. You can do a full sync once initially setup, then incremental later on.

• Scaling Kubernetes to 7,500 nodes

  At this point no one doesn’t know OpenAI and their ChatGPT. They run a massive K8S cluster with 7,500 nodes. Needless to say that come with many challenges they share with us here.

• Things DBs Don't Do - But Should

  Writing or talking about things databases don’t do may sound a bit silly. There are obviously many things databases don’t do. Example, automatic backup, version control of row…

• Tinkering with Unix domain sockets

  A most normal way for client <-> server communication is to open a port, and listen to traffic to that port. To secure sometine we use localhost, but there is also other way call Unix socket. The most famous one use it is docker with its /var/run/docker.sock which sometime we can mount into a container and interact with docker on host. In this post, we explore this socket concept.

• Generate diagrams programmatically

  Drawing Diagram by hand can be very tedious. To connect relationship we have to constantly move thing around to make room and space. D2 is a modern text-to-diagram language. It is built with extensibility in mind. This post show an example of how we can use SQL to generate the diagram of table relationship.

• Inside logical replication in PostgreSQL: How it work

  Logical replication uses a publish and subscribe model with one or more subscribers subscribing to one or more publications on a publisher node. Subscribers pull data from the publications they subscribe to and may subsequently re-publish data to allow cascading replication or more complex configurations.

• The Wonders of Postgres Logical Decoding Messages

  Did you know there’s a function in Postgres that lets you write data which you can’t query? A function that lets you persist data in all kinds and shapes but which will never show up in any table? Let me tell you about pg_logical_emit_message()! It’s a Postgres function that allows you to write messages to the write-ahead log (WAL) of the database.

• Introduction to Postgres Backups

  There’s a variety of ways to back up your data and this post aims to explain the basic tools involved in backups and what options you have, from just getting started to more sophisticated production systems.

• The tar archive format, its extensions, and why GNU tar extracts in quadratic time

  Can you imagine being annoy by a very very slow untar and think you can beat it by writing your own untar? The OP did just that, lead us through the wolrd if tar format, why it is slow and a few tip we can do to speed up.

• Beautiful and mind-bending effects with WebGL Render Targets

• How We Keep Our Government Apps Running With High Reliability: A Peek at Our Incident Management Strategy

  Get to know Nadinastiti, our Technical Program Manager, and Estu Fardani, our Cloud Platform Engineer at GovTech Edu, as they shed light on the best practices for incident management in GovTech Edu’s engineering teams

• awesome-crdt

  A collection of awesome CRDT resources. Useful when you want to learn about collaborative editing

Code to read

• basaran

  an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models.

• tragic-method

  A collection of script depicting the strange quirks of programming languages.

• noseyparker

  a command-line program that finds secrets and sensitive information in textual data and Git history.

• Minisearch

  Tiny and powerful JavaScript full-text search engine for browser and Node

• transformers.js

  Run 🤗 Transformers in your browser

• go-ansi-parser

  Have you ever see code like \u001b[1;31;40mHello World\033[0m in your log because color code get send to text log? How can we parse them?

• revive

  ~6x faster, stricter, configurable, extensible, and beautiful drop-in replacement for golint

• coltrane

  🎹🎸A music theory library with a command-line interface. Super cool as a guitar beginner to look up chord I must say

Tools

• chainloop

  an open source software supply chain control plane, a single source of truth for artifacts plus a declarative attestation process. SecOps teams can declaratively state the attestation and artifacts expectations for their organization’s CI/CD workflows, while also resting assured that latest standards and best practices are put in place.

• libgsqlite

  A SQLite extension which loads a Google Sheet as a virtual table.

• steampipe

  Use SQL to instantly query your cloud services (AWS, Azure, GCP and more). Open source CLI. No DB required

• Tremor Components

  Ready, Beautiful UI Components for building analytical interfaces. Feed data and they show nice chart.

• jsoncrack

  Have you ever want to dig through a huge JSON object? filter out a certain array. Nested it etc. jq is great if you know the shape of object ahead of time. But if you’re looking for a needle in a huge JSON, a visual tool might be better

• modern font stacks

  Use system font avoid many gotcha such as layout shift, flashing. istant render because the font is pre-installed. But which font stack offer the best on all the OS, how does they looks like. Use this site to choose

• readyset

  a lightweight SQL caching engine written in Rust that helps developers enhance the performance and scalability of existing applications.

• pg_timetable

  Schedule job in Postgrs by using Postgres table itself. Using a cron you always have to deal with lock to ensure there are no 2 instances run at same time. Putting this in the db itself make operating simpler. You can schedule SQL query, system command, or built-in function. Refresh materialize view is a perfect match.

• pgcat

  PostgreSQL pooler with sharding, load balancing and failover support. There is a post by instacart on adopting it

• greptimedb

  An open-source, cloud-native, distributed time-series database with PromQL/SQL/Python supported.

• How discord stores trillions of messages

  Rewrite in Rust, migrate to ScylaDB from cassandra. But the most important thing to me is the request coalescing. If two user read the same row, there should only one query to the database.

• How Cloudflare runs Prometheus at scale

  They run 916 Prometheus instances with a total of around 4.9 billion time series. CloudFlare sure knows how to scale Prometheus

• Demystifying bitwise operations, a gentle C tutorial

  writing code that operates on bits can be tedious, less readable. higher level languages usually abstract them out but it’s a fundamental useful to know and understand these concept. You don’t need to know C to follow these

• Stack Machines and Where To Find Them

  A very long post explain recursion, how function works, invent automatic memory, arn a bit of Polish (notation), and discover the power of stack machines. To conclude, we’ll try writing a Rust macro that parses and executes Forth code at compile time.

• How not to design a UI library

  It hard to design a UI library because UI is async by default. You have two approach, declarative or imperative. On top of that, add event handle, where to put it to avoid blocking main threaad. Separation of logic(rendering) and behaviour. Suddently it brings every concept into it

• Common Beginner Mistakes with React

  Josh taugh React and noticed that there were a handful of things that kept catching students off guard. In this tutorial, we’re going to explore 9 of the most common gotchas

• Get smarter every day

  Every day Refind picks 5 links from around the web that make you smarter, tailored to your interests. Refind is a must-read newsletter loved by 100,000+ curious minds. Sign up for free today

• Using LLaMA with M1 Mac

  There are many open source implementation of large language models. But we cannot experiment locally. Thanks to Georgi Gerganov and his llama.cpp project it is possible to run Meta’s LLaMA on a single computer without a dedicated GPU.

• Store Trees As Materialized Paths

  Currently used approaches for trees in databases like adjacency lists or the nested set model are too complex. Storing the full path with every row simplifies tree querying and management.

• How we built our multi-platform design system at Booking.com

  Building a design system that works is a challenge at any scale. Building a design system for 150+ product teams, used by 200+ designers and 800+ developers, and serving 4 different platforms? That’s a challenge requiring lots of special considerations.

Code to read

• textual

  a TUI (Text User Interface) framework for Python inspired by modern web development.

• potatis

  🥔 MOS-6502 and NES emulator in Rust (SDL/WebAssembly/Android/Embedded/Cloud)

• finder

  Given an element, it will generate the shortest CSS selector to identify that element uniquely. This is useful say you’re building a tool to design UI and it works with element/block. User click an element to focus it, this can be used to identify those elements

• chrono

  Given random arbitrary string like “today”, “last friday 2am” it can generate a date object. Really cool to see how it’s implemented.

• Rux

  A jsx-inspired way to render view components in Ruby. With this and another library call phlex I think we have more viable method to render view in Rails. I was never a fan out string rendering in traditional Ruby app.

• valgo

  a type-safe, expressive, and extensible validator library for Golang use generic.

Tools

• pyroscope

  Continuous Profiling Platform. Debug performance issues down to a single line of code. Very nice webui and supprot many languages to ingest data. Has a nice demo you can check

• pgrok

  Poor man’s ngrok - a multi-tenant HTTP reverse tunnel solution through SSH remote port forwarding

• zrok

  ngrok equivalent built base on OpenZiti which create a zero trust overlay network to join your nodes together

• pg_graphql

  GraphQL support for PostgreSQL

• pigsty

  A tool to create a Postgres deployment with all the tooling around it like RDS

• mathesar

  a straightforward open source tool that provides a spreadsheet-like interface to a PostgreSQL database.

• dalai

  Dead simple way to run LLaMA on your computer.

• ultralytics

  a cutting-edge, state-of-the-art (SOTA) model that builds upon the success of previous YOLO versions and introduces new features and improvements to further boost performance and flexibility.

• nebullvm

  Plug and play modules to optimize the performances of your AI systems

• hyperswitch

  an Open Source Financial Switch to make payments Fast, Reliable and Affordable. It lets you connect with multiple payment processors and route traffic effortlessly, all with a single API integration.

• Scaling Etsy Payments with Vitess: Part 1 – The Data Model

  Etsy Payment system reached the point where vertical scale won’t work anymore. They used biggest instance on GCP. They migrated this to Vitess. They have to change schema and update data model and roll out in 2 phases and share with us how they do in this article

• Get smarter every day

  Every day Refind picks 5 links from around the web that make you smarter, tailored to your interests. Refind is a must-read newsletter loved by 100,000+ curious minds. Sign up for free today

• What Every Programmer Should Know About Memory

  As CPU cores become both faster and more numerous, the limiting factor for most programs is memory access. Hardware designers have come up with ever more sophisticated memory handling and acceleration techniques–such as CPU caches–but these cannot work optimally without some help from the programmer. Unfortunately, neither the structure nor the cost of using the memory subsystem of a computer or the caches on CPUs is well understood by most programmers. This paper explains the structure of memory subsystems in use on modern commodity hardware, illustrating why CPU caches were developed, how they work, and what programs should do to achieve optimal performance by utilizing them.

• DataDog Exactly-Once Ingestion and Multi-Tenancy at Scale

  DataDog handle massive amount of data so they build a system to redesign how they ingest and store log call Husky. Husky’s architecture decouples storage and compute so that they can be scaled independently. In this post we learn how Husky organized its data and establish consensus.

• Building a Minimalistic Virtual Machine

  we could build a simple Virtual Machine (VM) with no dynamic linking and a small set of minimalistic APIs that remain stable over time, it would make it a lot easier to build software without worrying about becoming randomly broken by changing APIs or dependencies. The author build such a VM call UVM and share with us here.

• Consistent Hashing Algorithm

  Consistent hashing is a technique to associate a resource with a set a nodes evenly, but also support adding new nodes into system without reallocating resource around.

• PostgreSQL Basics: Object Ownership and Default Privileges

  discussed how roles (users and groups) are created and managed in PostgreSQL Depending on your background with permissions, particularly in other database products, some of those nuances in how permissions work may have been surprising.

• Animate a Camera Fly-through on Scroll Using Theatre.js and React Three Fiber

  I always wonder how those walk through, fly through or 3d object model are being build. This is an end to end tutorial to show you the entire process of implement JS, CSS code and build the animation

• Use Maps More and Objects Less

  In JavaScript, its very easy to create object with no extra syntax: just do a {foo: 1}. But turning out there is another better and fast way: const o = new Map();o.set('foo', 1). Map is optimize for adding/removing key frequently

• ProjectDiscovery' Best Kept Secrets Tools

  A walk-through of multiple tools that can be very useful when audit and analyse a network system. This is the step call analyze footprint of any system to find relevant components. By running these tools on your own network can yield interesting result sometimes.

• Gossip Glomers

  A very good way to learn and build reliable distributed system. The famous Kyle of Jepsen team up with fly.io to build a challenge, where you implement the system to communicate with a pre-build server follow a protocol. Your program needs to communicate the right way to pass the challenge.

• Get One Free Month of Skillshare Explore thousands of hands-on creative classes with one inspiring membership.

  Skillshare is a platform that offers thousands of classes in a variety of subjects. From creative arts to business and technology, there’s something for everyone. Expert instructors guide students every step of the way and a community of passionate learners provides support and inspiration. Whether you’re looking to learn something new or enhance your existing skills, Skillshare has something for you. Explore the platform today and start your learning journey.

Code to read

• NoteSH

  fully functional sticky notes App in your Terminal!

• opossum

  Rudimentary web browser written in Golang

• TabFS

  A chrome extension that mount your browser tabs as a file system and you simply interact with part of it like the title, close tab etc. Read the companion post here

• An authentication system generator for Rails applications.

  generate a pre-built authentication system into a rails application (web or api-only) that follows both security and rails best practices. By generating code into the user’s application instead of using a library, the user has complete freedom to modify the authentication system so it works best with their app.

• DOMPurify

  a DOM-only, super-fast, uber-tolerant XSS sanitizer for HTML, MathML and SVG. When scraping site or need to display a certain user input such as email or html in browser, this come in handy.

• PGMQ: ostgres Message Queue

  A project that build a queue on Postgres, but expose extra SQL function where you call in to queue/enqueue job. Its API is similar to AWS SQS. I love this kind of code base, I got to learn Rust, learn how to write extension with pgx, a tool to write Postgres extension in Rust, and can also evaluate to use this tool for my lightweight queue in the future

Tools

• soft-serve

  A tasty, self-hostable Git server for the command line. You SSH into it and browse it through a TUI (terminal based user interface). You can even click on it too, in a terminal just like a normal remote SSH session.

• pgmanage

  Web tool for database management

• sqlite_fdw

  SQLite Foreign Data Wrapper for PostgreSQL. This means that PostgreSQL now can access and join data with another SQLite database.

• pgpq

  tream Arrow file data into Postgres. Tools like BigQuery, RedShift already support Arrow format due to their compression, easiser to parse and fast.

• pgweb

  Cross-platform client for PostgreSQL databases

• karafka-web

  a webui to manage kafka jobs, consumers, routing, cluster info

• keep

  a simple CLI tool that contains everything you need to start creating Alerts. builtin providers (e.g. sentry/datadog or slack/pagerduty), 100% open sourced.

• FlexGen

  a high-throughput generation engine for running large language models with limited GPU memory (e.g., a 16GB T4 GPU or a 24GB RTX3090 gaming card!). FlexGen allows high-throughput generation by IO-efficient offloading, compression and large effective batch sizes.

• Neural Networks: Zero to Hero

  A course by Andrej Karpathy on building neural networks, from scratch, in code. We start with the basics of backpropagation and build up to modern deep neural networks, like GPT. In my opinion language models are an excellent place to learn deep learning, even if your intention is to eventually go to other areas like computer vision because most of what you learn will be immediately transferable. This is why we dive into and focus on languade models.

• GPT from scratch in 60 Lines of NumPy

  What is the magic of GPT? I want to understand how GPT works but can easily get lost in so many terms and papers so by looking at a minimal implementation give me some ideas.

• Bit Hacking (with Go code)

  Don’t let the “Go code” put you off if you aren’t a Go user. Bit manipulation is the fundamental in programming. It can do quite a cool tricks such as check same sign(positive/negative) by looking at the last bit of both number or a fast way to compute avg.

• The Performance Impact of SQL’s FILTER Clause

  FILTER and CASE can be use in an aggregate funcion in your SQL to caculate a value from a set of row that match the conditions. They looks similar in term of functionality, but FILTER is much nicer to read because its appear on the left of your aggregate function, not wrap around like case. So is there any perf impact? Generally, no, you can safely switch to FILTER.

• Visual design rules you can safely follow every time

  You do not have to follow these rules every time. If you have a good reason to break any of them, do. But they are safe to follow every time.

• Hardcoded Folder Icons in macOS

  There is a certain directory on MacOS has a specific icon in Finder. So let looks at how they work

• Reverse-Engineering YouTube: Revisited

  explain how YouTube works under the hood, how it serves streams to the client, and also how you can exploit that knowledge to download videos from the site

• SQL Window Functions: Ranking

  A window function performs a calculation across a set of table rows that are somehow related to the current row. It’s very flexible because it doesn’t need to pre-group row, and can work It’s useful to calculate extra data between the current row and a group of row related to current row by any condition that you can express in code. This post has many practical use of window function to calculate ranking

• The Guide To Responsive Design In 2023 and Beyond

  Introduce many approachs we can use to do responsive design in 2023. If you never heard of clamp, minxmax, Size Container Queries.. time to refresh your CSS skill for 2023.

• A gentile introduction to CRDT

Code to read

• Pathfinding Visualizer

  A pathfinding visualizer made in Python and Pygame. This project aims to provide a fun and interactive way to learn about popular pathfinding algorithms such as Dijkstra’s, A* and other supported algorithms.

• conc

  Better structured concurrency for go

• purego: A library for calling C functions from Go without Cgo.

• by

  a library preloader to speed up process startup, uses a client/server approach. The server loads the libraries and listens on a UNIX socket, and the client connects to that socket to run a process. For each client connection, the server forks a worker process, which uses the current directory, stdin, stdout, stderr, and environment of the client process. The worker process then processes the arguments provided by the client.

• Roo

  If you need to handle many common spreadsheet types, looks no more. It supports Excel, Libre, CSV, and even google sheet.

• resvg

  An SVG rendering library. It can do thing like accept SVG, render to PNG and customize SVG attribute. There is also a JavaScript binding call resvg-js

• boxxy

  boxxy is a tool for boxing up misbehaving Linux applications and forcing them to put their files and directories in the right place, without symlinks! It uses Linux namespaces for its functionality. Kind of like docker, but here simply prefix your command with boxxy and boxxy setup a namespace file system with dir mount etc then run your process inside that namespace. It’s Rust but you will learn a lot about this concept. Highly recomended to check it out.

• rxdb

  A fast, offline-first, reactive database for JavaScript. We linked to it on issue 122 but it’s still going strong

• minGPT

  A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training

Tools

• Will Postgres Use My Index? Hypothetical Indexing for Postgres

  a Postgres extension for adding hypothetical indexes to Postgres so you can do index planning. Read the introduction article to get a sense of it.

• vanillalist

  A directory of JavaScript lib/component without any dependencies.

• bookmarklet_editor

  To run a snippet of JS on any page, you can pop into the console

• JuiceFS

  a distributed POSIX file system built on top of Redis and S3.

• BlockNote

  A “Notion-style” block-based extensible JavaScript text editor built on top of Prosemirror and Tiptap. I always need these JS editor at some points and having a set of option to pick is great.

• plumber

  When working with message queue we sometimes need ability to peek into the queue from CLI. With so many solutions like Kafka, RabbitMQ, PubSub, SNS,Nats, Redis,… it’s hard to remember all the syntax. This tool rule them all using a consistent syntax. Support a lot of message queues system.

• tproxy

  A cli tool to proxy and analyze TCP connections.

• stackql

  Query, provision and operate Cloud and SaaS resources(such as AWS, GCP, Azure, Okta, Github,…) and APIs using an extensible SQL. Super useful when you want to have a big picture of what resources we have in a big org, not just for AWS, GCP but say how many users we have in Github and who belongs to which group

• pgbackrest

  aims to be a reliable, easy-to-use backup and restore solution that can seamlessly scale up to the largest databases and workloads by utilizing algorithms that are optimized for database-specific requirements.

• patroni

  a template for you to create your own customized, high-availability solution using Python and - for maximum accessibility - a distributed configuration store like ZooKeeper, etcd, Consul or Kubernetes.

• awesome-tunneling

  List of ngrok alternatives and other ngrok-like tunneling software and services. Focus on self-hosting.

• flowlogs-pipeline

  an observability tool that consumes logs from various inputs, transforms them, and exports logs and/or metrics to a chosen target.

• network observability operator

  An OpenShift / Kubernetes operator for network observability

• How a single line of code brought down a half-billion euro rocket launch

  How a rocket launch fail? Re-use of code written for 32-bit system on a 64-bit data range.

• Novo Bank

  Novo is the powerfully simple small business banking platform that’s resetting expectations of small business banking. Complete with integrations to tools small business owners are already using like Stripe, Shopify, Square, and Quickbooks Online.

• SSO-MONITOR: Fully-Automatic Large-Scale Landscape, Security, and Privacy Analyses of Single Sign-On in the Wild

  a paper on an open-source fully automatic large-scale SSO landscape, security, and privacy analysis tool. They automatically identified 1,632 websites with 3,020 Apple, Facebook, or Google logins within the Tranco 10k and analyse them. Some fact such as 10% of SSO is failed because of mis-configuring redirect_uris. The paper is also a remind for us to cleanup stuff and make sure our SSO, especially for internal tooling/documents is up to date with the right config.

• Rewrite, refactor, or reinvent?

  A new take on the age-old question: Should you rewrite your application from scratch, or is that “the single worst strategic mistake that any software company can make”? Turns out there are more than two options for dealing with a mature codebase.

• The life of a DNS query in Kubernetes

  DNS in kubernetes is more complex than in a normal server. Because there are service dns, and they have namespace and cluster domain as suffix, but namespace and cluster domain can be omitted. Therefore it creates more work to check wherether a name belongs to k8s resources.

• Features I'd like in PostgreSQL

  PostgreSQL is impressive with its featureset. But there are always gotcha and annoying. I also learn some useful pg extension such as pg_safeupdate to block update/delete without a where filter.

• Transaction ID Wraparound: A Walk on the Wild Side

  Many people know horror stories about anti-wraparound autovacuum tanking performance or databases that stop working, but who has ever seen actual data loss? The most high profile public incident with this that I know was belong to Sentry

• Writing a Custom SQLite Function (in C) - Part 1

  In this short series, we’re going to add a user defined function to SQLite in C. We’ll go through the process slowly and in a way which, I hope, a C novice can follow

• Carving the Scheduler Out of Our Orchestrator

  fly.io allow you to deploy your app as in a Heroku experience. They use Nomand behinds the scene, and out grow it so the build flyd. It’s quite different from K8S or Nomand. flyd operates like a market. Requests to schedule jobs are bids for resources; workers are suppliers.

• Examples of floating point problems

• So you want to write a package manager

  A very old post in 2016 laid out challenges to deal when building a packaga manager.

• Rundown of LinkedIn’s SRE practices

Code to read

• aocla

  A small stack based, written by Antirez, Redis author, to bring Advent of Code 2022 Day 13 puzzle to the extreme consequences

• gpu-io

  A GPU-accelerated computing library for physics simulations and other mathematical calculations

• pub_grub

  a must read for Ruby dev. This is the new bundler resolver to find gem version to install. Natalie is the original author that invented the algorithm

• goldmark

  A Markdown parser written in Go. Easy to extend, standards-compliant, well-structured.

• DiT

  Official PyTorch Implementation of “Scalable Diffusion Models with Transformers”

• driftdb

  a real-time data backend for browser-based applications. It can also deploy on CloudFlare Workers amd Diran;e Pnkects

Tools

• parseable

  a modern log store and analysis engine, lightweight, low latency, cloud native alternative to Elastic. Log can store on local drive or S3. Data can query using PostgreSQL compatible SQL.

• alasql

  JavaScript SQL database for browser and Node.js. Handles both traditional relational tables and nested JSON data (NoSQL). Export, store, and import data from localStorage, IndexedDB, or Excel.

• pgslice

  Postgres partitioning as easy as pie. Works great for both new and existing tables, with zero downtime and minimal app changes. No need to install anything on your database server. Archive older data on a rolling basis to keep your database size under control.

• ronin

  a free and Open Source Ruby toolkit for security research and development. It can perform many security tasks, such as encoding/decoding data, filter IPs/hosts/URLs, querying ASNs, querying DNS, HTTP, scanning for web vulnerabilities, spidering websites, install 3rd party repositories of exploits and/or payloads, run exploits, write new exploits, managing local databases, fuzzing data,

• Open-Assistant

  OpenAssistant is a chat-based assistant that understands tasks, can interact with third-party systems, and retrieve information dynamically to do so.

• tlbs

  a CI-Friendly tool for document a database, written in Go.

• pg-lock-tracer

  This project provides tools that allow you to gain deep insights into PostgreSQL’s locking activities and troubleshoot locking-related issues (e.g., performance problems or deadlocks).

• caretta

  Instant K8s service dependency map, right to your Grafana.

• sketch

  AI code-writing assistant that understands data content

• Start speaking a new language in just three weeks with Babbel

  Learning to speak a new language goes beyond just vocabulary: it’s about being able to hold a real-life conversation with a local, and understanding the culture and the people of each place. Consider Babbel your expert-led passport to learning, with 10-minute lessons that are so effective, many users feel confident speaking a new language in just three weeks. Supplement those with the podcasts, games, articles and live online classes for a well-rounded education in weeks. Start learning a new language (and culture) today for up to 55% off.

• Hello, PNG

  Only 2 formats that support transparent on the web is GIF and PNG. GIF is out of the option with its limited 256 colors. So how does PNG works, how can we hand write a PNG file. This is exactly that.

• Network Programming in Age of Empires and Beyond

  Multi player game is a distributed systems where they cannot wait for an acknolwedge write of the slowest node in the clusters. This artcile explains the design architecture, implementation, and some of the lessons learned creating the multiplayer (networking) code for the Age of Empires 1 & 2 games; and discusses the current and future networking approaches used by Ensemble Studios in its game engines.

• How I ruined my SEO

  In October 2022 traffic to my blog dropped like a stone. What happened? Somehow I ruined my SEO. Don’t be me. I’ll tell you what I got up to and hopefully you can avoid doing the same.

• The truth about CSS selector performance

  It’s likely faster for the browser engine to match just .link than it is to match this longer .wrapper .section .title .link selector. There are just fewer things to check. Turning not quite in practice.

• Fairness in multi-tenant system

  Multi tenant system is usually cheaper because they have higher resource utilization leads to less server. But in multi tenancy system, an increas load on a client may affect other. In this article we look at How AWS approach this problem with rate limiting.

• Internet Transport Protocols, Part I: Reliable Transports

  Most people who use the Internet just have some vague idea that it carries data from point A to point B (famously, through a series of tubes). Even people who regularly work on Internet systems tend to work with it through many layers of abstraction, without a clear understanding of the infrastructure components that make it work. This post is the first of a series about one such piece of infrastructure: the transport protocols such as TCP that are used to transmit between nodes on the Internet

• Fixing a Memory Leak in a Production Node.js App

  It’s a very interesting walk through of fixing memory leaks in a web app. Find leads, come up with theories, test it, rinse and repeat

• Why Not document.write()?

  why document.write() is so harmful? Actually not that harmful but only in a few specific condition

• No Start Menu for You

  What do you do if it takes a minute to launch NotePad on window? On a 32GB RAM, SSD laptop. In general if you want to understand why your computer is performing badly you need to record and analyze a trace. However if you want to see if you are hitting a particular problem then there are easier steps that you can follow.

Code to read

• OpenSnitch is a GNU/Linux port of the Little Snitch application firewall

  Interactive outbound connections filtering.

• Odigos

  Instant distributed traces without code changes. 🚀 Boost existing monitoring tools with higher-quality data

• ChatRWKV

  like ChatGPT but powered by the RWKV (100% RNN) language model, and open source.

• rough-notation

  Create and animate hand-drawn annotations on a web page. Very interesting to use this to add some nice animation when user finished an action.

• polars-ruby

  🔥 Blazingly fast DataFrames for Ruby

• sorcery

  Magical Authentication for Rails. Supports ActiveRecord, DataMapper, Mongoid and MongoMapper. Despite the name, there is no magic at all. Just code that you use inside your controller

• xilem

  An experimental Rust native UI framework, still small enough and straight forward to read and understand the code and the proceess of building a UI framework. For a production ready, more feature set, check out dioxus, a friendly React-like GUI library for desktop, web, mobile, and more

Tools

• security-guard

  Runtime security plug to protect user containers

• polar

  Lightweight & elastic kubernetes-native event streaming system

• rete.js

  Rete is a modular framework for visual programming. Rete allows you to create node-based editor directly in the browser. You can define nodes and workers that allow users to create instructions for processing data in your editor without a single line of code

• Sleuth

  an open-core enterprise search tool that allows organizations to search text from various workspace apps such as Slack, Jira, Salesforce, and Confluence.

• lotus

  Open Source Pricing & Packaging Infrastructure

• data-diff

  Efficiently diff data in or across relational databases

• The sqld (“SQL daemon”) project is a server mode for libSQL, which is fork of SQLite. Interesting is that it support PostgreSQL wire protocol and read replication.

• geoserver

  an open source server for sharing geospatial data

• Revolut: Sign up and get Revolut Premium free for 3 months!

  Banks will charge you when you spend or transfer money abroad. We’re not about that, and that’s why over four million people have switched to Revolut.

• What happens when you open a terminal and enter ‘ls’

  This could be the equivalent of the in famous “what happen when you type an address into the browser”, but for terminal which we interact on a daily basis but may forgot or not having a full understand of its architecture.

• Start speaking a new language in just three weeks with Babbel

  Learning to speak a new language goes beyond just vocabulary: it’s about being able to hold a real-life conversation with a local, and understanding the culture and the people of each place. Consider Babbel your expert-led passport to learning, with 10-minute lessons that are so effective, many users feel confident speaking a new language in just three weeks. Supplement those with the podcasts, games, articles and live online classes for a well-rounded education in weeks. Start learning a new language (and culture) today for up to 55% off.

• Making Sense of Hexdump

  hexdump is very versatile and allows you to look at the structure inside binary files as you see fit and once you learn how to use it you can apply it quickly to many problems. I think that easily qualifies as cool.

• Examples of floating point problems

  We keep hearing floating point issue and their issue but at a higher or abstraction level. Are there any specific examples that we can practical see how bad it’s

• How does rsync work

  Third article in a series of blog posts about rsync. This posts is about data transfering, split and checksum multi part of the files.

• Production Twitter on One Machine? 100Gbps NICs and NVMe are fast

  In this post I’ll attempt the fun stunt of designing a system that could serve the full production load of Twitter with most of the features intact on a single (very powerful) machine. I’ll start by showing off a Rust prototype of the core tweet distribution data structure handling 35x full load by fitting the hot set in RAM and parallelizing with atomics, and then do math around how modern high-performance storage and networking might let you serve a close-to-fully-featured Twitter on one machine.

• Origins, Sites and other Terminologies

  Cross origin request? What is that? Why loading an image work, including a JS from other domain works? but make ajax request won’t or cookie won’t get send? What are all the terms?

• The type system is a programmer's best friend

  Have you ever write this kind of code? copy(from_path, to_path) and accidentally paste the argument in reverse? If we use type we can avoid that? Or even typo, especially when using enum type.

• The gotcha of unhandled promise rejections

  NodeJS/JavaScript runtime run like a giant for loop and check for thing to do, these thing can in turn queue other things. What happen if they throw promise error which we forgot to handle? When that happen the execution halt in node or throw error in browser and stop the execution of JS right there, no more further JS after that point in your code get run. The point is, you want to avoid unhandled rejections.

• Faster virtual machines: Speeding up programming language execution

• Peeking into HTTPS Traffic with a Proxy

  This article is about configuring a web application, Appsmith in this case, to run correctly behind a firewall that does SSL decryption, as a Docker container. Instead of a firewall, we’ll use a proxy, which, for the purpose of the problem statement, should be the same. We usually don’t include these kind of “how-to” tutorial, but I feel like it helps a lot when debugging or reversing API traffic and that’s a really powerful tool to have in your toolbox

Code to read

• okaywal

  A Write Ahead Log (WAL) implementation in Rust

• YouPlot

  A command line tool that draw plots on the terminal.

• lazy-brush

  Smooth drawing with mouse, finger or other pointing device. I like these kind of code, it has a lot to learn about graphic, matrix and interesting algorithm to draw things that seem trivial.

• gnet

  high-performance, lightweight, non-blocking, event-driven networking framework written in pure Go. it makes direct epoll and kqueue syscalls rather than using the standard Go net package and works in a similar manner as netty and libuv, which makes gnet achieve a much higher performance than Go net.

• AIJack

  Security and Privacy Risk Simulator for Machine Learning

Tools

• sysm

  makes your system play custom sounds when any configured system or external event happens

• terranix

  Terrform support a syntax in JSON instead of HCL. So if we can generate that JSON, we can build another abstraction that work with Terraform. This tools allow you to use Nix as the language to generate JSON.

• appsmith

  Low code project to build admin panels, internal tools, and dashboards. Integrates with 15+ databases and any API.

• wildebeest

  an ActivityPub and Mastodon-compatible server written in TypeScript and runs on top Cloudflare’s Supercloud, uses Workers and Pages, the D1 database to store metadata and configurations, Zero Trust Access to handle authentication and Images for media handling. Essentially kind of like a full featured webapp deploy on serverless infrastructure including the database

• opal

  an administration layer for Open Policy Agent (OPA), detecting changes to both policy and policy data in realtime and pushing live updates to your agents. OPAL brings open-policy up to the speed needed by live applications.

• lotus

  Open Source Pricing & Packaging Infrastructure for SaaS

• caretta

  Instant K8s service dependency map, right to your Grafana.

• FastQL

  Spin up a blazing fast Rust GraphQL API to serve your ML model in one line of Python code.

• Create amazing content quickly with Canva – no design skills needed

  Canva Pro is the design software that makes design simple, convenient, and reliable. Create what you need in no time! Jam-packed with time-saving tools that make anyone look like a professional designer.

Hi everyone, this week is security week. First, if you used CircleCI, make sure to rotate any secret since they suffered a breach. Second, Slack private repository got hacked too, and there is no customer data there but regardless, if your company use Slack, best to be rotate as well. Third, Pytorch got a supply chain attack and compromise too, luckily only for nightly build, so if you don’t use nightly build, you’re fine

• Avoiding Double Payments in a Distributed Payments System

  How we built a generic idempotency framework to achieve eventual consistency and correctness across our payments micro-service architecture.

• Streamline the way you manage your business’s Apple devices with Jamf Now

  Managing your business’s Apple devices can be a headache if you don’t have a robust IT team—but you’re savvier than that. With Jamf Now, you bypass the confusion and head straight to streamlining Apple mobile device management. This integration makes it easy for biz leaders to deploy, oversee, and secure employees’ devices, while staff enjoys a simple and smooth user experience. Sign up for a free account now with the capabilities to manage three devices, and add more for just $2 per device each month.

• The infrastructure behind ATMs

  This isn’t quite a technical post but I think it’s amazing that when we walk to ATM and withdrawal our money, it just works and very reliable too.

• moving to cloud: How to do Migrations the wrong way

  Mercri ecently moved databases from local files in the codebase to an online Database. It took longer than expected but with good reason. The project started as POC and turn into a success, so they live with data in local files until traffic spike 10x and data is added at the 10x rate too

• Easy, alternative soft deletion: `deleted_record_insert`

  Soft delete is a technique where instead of removing data, we flag it as deleted by using a column like “deleted: true” or “deleted_at: time”. It seemed smart, cool, and very convenience since the data is there to refer back. However, it is easy to lead to mistake and complex our query. Every query has to check for it, index need to be built around, whoever join data need to check for it and so on. It adds up quickly and everytime I used it, I regret in the end, and just delete data for real. Here is another middle approach: use a completely new table with jsonb to store the whole record.

• Understanding Layout Algorithms

  Working with CSS layout is like just tweaking thing unil it works, exmaple the margin collasible I found is very confusing. The author share with us the mental model shift that makes CSS more intuitive

• Docker on MacOS is slow and how to fix it

  A tutorial to walk through understanding why docker is slow in term of OS on mac and a few solution. In short, Mutagen is what I plan to look into for my work too. We linked to Mutagen before and it’s a great tool. I heard good feedback

• Verifiable AES: encryption using zero-knowledge proofs

  Encryption is transforming messages into random-looking texts to ensure confidentiality between two parties. What is our objective here? We want to generate proof allowing us to verify an encryption algorithm, ensuring it does what it was designed for.

Code to read

• libreddit

  Reddit UI is very slow to load, constantly nagge us to use mobile app. Luckily they have a very openapi and people build UI on top of it. librreddit is built on rust, focus on content and run very fast. Part of my routine is source content from Reddit and IO found libreddit is a much easiser way to navigate reddit nowadays.

• gopher-lua

  a VM and compiler for Lua in Go. A good tool to add scriptable feature to your app. Another I approach I used is Otto

• nanoGPT

  The simplest, fastest repository for training/finetuning medium-sized GPTs. I like these small codebase to help me understanding more about these GPT and AI concept.

• osv.dev

  osv.dev is a vulnerability database and triage infrastructure for open source projects aimed at helping both open source maintainers and consumers of open source.

• ronin

  Ronin is a free and Open Source Ruby toolkit for security research and development. Ronin also allows for the rapid development and distribution of code, exploits, payloads, etc, via 3rd party git repositories. It’s quite a big codebase but we can learn how thing is structure in these kind of tool.

Tools

• FerretDB

  A MongoDB compatible server but use Postgres to persist data behind the scene. Most MongoDB users do not require any advanced features offered by MongoDB; however, they need an easy-to-use open-source database solution. Recognizing this, FerretDB is here to fill that gap.

• RustScan

  The modern port scanner

• wish: Make SSH apps, just like that!

  This probably should be in “Read code” but I think it could be a very valuable tools to design SSH-based tool. One of my use case is I have a very expensive build server which maynot being used often, let them run around is costly, so I setup a smaller node, and forward TCP traffic to the bigger node. When there is no connection, it shutdown the expensive node, and when the first request come in, it boot up the expensive server.

• cdc-file-transfer

  Tools for synching and streaming files from Windows to Linux extract from Google Stadia project.

• goalert

  Open source on-call scheduling, automated escalations, and notifications so you never miss a critical alert

• Start speaking a new language in just three weeks with Babbel

  Learning to speak a new language goes beyond just vocabulary: it’s about being able to hold a real-life conversation with a local, and understanding the culture and the people of each place. Consider Babbel your expert-led passport to learning, with 10-minute lessons that are so effective, many users feel confident speaking a new language in just three weeks. Supplement those with the podcasts, games, articles and live online classes for a well-rounded education in weeks. Start learning a new language (and culture) today for up to 55% off.

Happy new year everyone. 2022 was a brutal year for me. A few app and idea I was working on wasnt turning out well and my start-up also shutdown. At the end of 2022, I got a terrible COVID but luckily thing turn out well and here am I sending out this issue. Hope 2023 will be a better year. I’ll keep building software, trying out new ideas again, and keep sending newsletter :-). If you like my newsletter, consider sponsor me on https://www.buymeacoffee.com/betterdev. Thank you so much for being audience all these years.

• Challenging algorithms and data structures every programmer should try

  There are some really interesting algorithms and data structures out there. Not only have they come up during job interviews, but learning them changed how I think about problems. Plus they’re useful even if you don’t work on Google-scale problems. One of interesting data structures I learn from this is piece tables to represent text buffer of an editor. VScode use it too to implement their text buffer.

• Understanding UUIDs, ULIDs and String Representations

  an uuid looks like this ff78ae6b-7176-46e9-81c6-4bee60ba81bb? is uuid a string? how come it’s has 128 bit? At its core, UUID is just a byte array, we can re-present that byte array in whatever encoding we want, base10, base64, base36, or hex.

• Learn how CDNs work by coding one from scratch

  The CDN we’re going to design uses: nginx, lua, docker, docker-compose, Prometheus, grafana, and wrk.

• What is hexadecimal

  To represent raw byte data, we cannot use ascii printable alone, there are many char cannot be print. So we need raw byte, but writing all 0 and 1 next to each other in a 8bit sequence gonna be long and hard to see. Hexadecimal solve that problem very nice because it has a unique feature, one hex character can re-present 4 bit, which mean we need only 2 char to represent a byte.

• Raspberry Pi security alarm — the basics

  Thomas shows us how he use Raspberry Pi to build a DIY security alarm system, using a Raspberry Pi as the controller. His plan was to make a self-sustained system, using proper alarm hardware — like PIR sensors and sirens.

• Why does my SSH private key still work after changing some bytes in the file?

  A very interesting aspect of SSH private key and. How their optimization allow the key to not require a certain part of the file.

• Page Cache and basic file operations

  We linked to this site in last issue about pipe. This week we learn about page cache, file operation, and mmap syscall. What happen when we read or write data? how does that generate a syscall and how kernel do it? We know that the kernel cache data when you read a file, but how does we check what is cache, how do we inspect it? how does that data is invalidate

• my age+yubikeys password management solution

  Filippo is a security expert who design and implement age encryption. He’s probably most famous for his tool mkcert. So let’s learn how security use password manager to secure their password day to day. I belived for engineers, password manager shouldn’t be a cloud solution. we should self-hosted it and manage our own vault

• User interface design: 5 rules of thumbs

  Here are five rules of thumb you can apply to common UI design challenges when building web applications: Use defaults, Obvious settings, Show progress, Avoid interruptions, Flat hierarchy. Working on my second saas app(my first is mailwip, an email forwarding service), I like these kind of UI articles.

• Draw SVG rope using JavaScript

  Today, I’ll take you through the process I came up with in regard to transforming an SVG path into a vector rope drawing. You will learn how to turn a path into a true svg image with texture around the path.

Code to read

• mogo

  A collection of small DRY Go utilities to make life easier. DRY = Don’t Repeat Yourself.

• pa

  a simple password manager. encryption via age, written in portable posix shell. In the recent incident of LastPass, it maybe wise to roll out own password manager to ensure 100% our data is fully encrypted

• quokka

  Open source SQL engine in Python. Also checkout the why

• steep

  Static type checker for Ruby

• nextauth.js

  a complete open source authentication solution for Next.js applications. It is designed from the ground up to support Next.js and Serverless. The code is a monorepo of many small package so you may want to focus on next-auth package and the core

Tools

• reaction

  Mailchimp Open Commerce is an API-first, headless commerce platform built using Node.js, React, GraphQL. Deployed via Docker and Kubernetes.

• snapper

  a tool for Linux file system snapshot management. Apart from the obvious creation and deletion of snapshots it can compare snapshots and revert differences between them. In simple terms, this allows root and non-root users to view older versions of files and revert changes.

• directus

  an instant REST+GraphQL API and intuitive no-code data collaboration app for any SQL database.

• immich

  Self-hosted photo and video backup solution directly from your mobile phone.

• What every SRE should know about GNU/Linux shell related internals: file descriptors, pipes, terminals, user sessions, process groups and daemons

  Have you ever thought about how a shell pipe works, how pressing the CTRL+C combination delivers the interrupt signal to the currently running shell processes, or how vim rewrites the content of the console when you change the size of your terminal window?

• Finding JIT Optimizer Bugs using SMT Solvers and Fuzzing

  Compiler is hard. JIT is hard? How can we avoid bug, and how can we even pove its correctness? The author share with us how they uses the Z3 theorem prover to find bugs in the optimizer of PyPy’s JIT, in particular its integer operation optimizations

• How to estimate disk space

  Estimation doesn’t look like a skill. But lets say we stated a new project and is about to provision a database, how big should we need for disk space? Now if we break down, how many row per day we may add, what data type, how much is the index. We can roughly get an estimation. On top of that, by breaking down giving us more understanding of each component.

• How I found a bug in sqlite

  rqlite is a lightweight, open-source, distributed relational database written in Go, which uses SQLite as its storage engine. Recently I introduced a high-performance write-path into rqlite and, to my great surprise, it exposed a bug in SQLite.

• The GPT-3 Architecture, on a Napkin

  There are so many brilliant posts on GPT-3, demonstrating what it can do, pondering its consequences, vizualizing how it works. With all these out there, it still took a crawl through several papers and blogs before I was confident that I had grasped the architecture. So the goal for this page is humble, but simple: help others build an as detailed as possible understanding of the GPT-3 architecture.

• Optimize Interaction to Next Paint

  Interaction to Next Paint (INP) is an experimental metric that assesses a page’s overall responsiveness to user interactions by observing the latency of all click, tap, and keyboard interactions that occur throughout the lifespan of a user’s visit to a page To provide a good user experience, sites should strive to have an Interaction to Next Paint of 200 milliseconds or less

• Taming Names in Software Development

  What is a name? A name is a label, a handle, a pointer in your brain’s memory. A complex idea neatly encapsulated. A name lets you refer to “the economy” , or “dogfooding” mid-sentence without needing a three-paragraph essay to explain the term.

• How the 8086 processor's microcode engine works

  look at how the 8086 decides what microcode to run, steps through the microcode, handles jumps and calls inside the microcode, and physically stores the microcode. It was a challenge to fit the microcode onto the chip with 1978 technology, so Intel used many optimization techniques to reduce the size of the microcode.

• Sorting with SIMD

  SIMD stands for single instruction, multiple data. A single instruction is used to apply the same operation to multiple pieces of data. A single SIMD addition should be roughly 4 times faster than performing 4 individual additions. How can this be used for sorting? Sorting does not involve arithmetic, and the whole idea of sorting is that each element moves to its unique correct place in the output

• So, what can you do with a process ID?

  If you know a process ID, what information can we extract about it? Turning out a lot. You can know full command and argument, its environment variable when it runs.

Code to read

• phlex

  A framework for building object-oriented views in Ruby

• pdfreader

  Parse text and tables from PDF files

• marmot

  A distributed SQLite replicator built on top of NATS

• LaViLa

  Code release for Learning Video Representations from Large Language Models by Facebook. Given a video, how to generate its description?

Tools

• scenejs

  JavaScript & CSS timeline-based animation library

• smashing

  a fork of Dashing, an excellent framework created at Shopify for displaying custom dashboards on TVs around the office

• miraclecast

  provides software to connect external monitors to your system via Wi-Fi. It is compatible to the Wifi-Display specification also known as Miracast

• d2lang

  a diagram scripting language that turns text to diagrams. It stands for Declarative Diagramming. Declarative, as in, you describe what you want diagrammed, it generates the image.

• git-bug

  Distributed, offline-first bug tracker embedded in git, with bridges

This week there is a vulnerability in VS Code. If you used VS Code, make sure to upgrade your VS To latest version after v1.73.1.

• How Gitlab diagnosed and resolved Redis latency spikes with BPF and other tools

  It is hard to debug intermittent issue, since it is usually doesn’t follow any pattern. Usually to debug them we need low level tool to hook up and extract more metrics. In this post, Gitlab shared how they deal with an intermittent issue for 2 years, but getting worse recently so they go to bottom of its with BPF and other techniques

• The architecture of Mastodon

  take a high-level look at the architecture of a single Mastodon instance and how different instances communicate. Specifically, we’ll be looking at how the Mastodon network-of-instances might scale.

• Counting unique visitors without using cookies, UIDs or fingerprinting

  Building a web analytics service without cookies poses a tricky problem: How do you distinguish unique visitors?

• Color Formats in CSS

  Everytime it comes up I just bite the bullet and use hex format. But what is hsl, rgb, lch? how they really work?

• Nibbler 4 Bit CPU

  As a software engineer I always try to dabble into hardware, the knowledge of CPU/RAM how they work help me a lot when dealing with IO, performance in softwarwe. Nibbler is a 4 bit CPU built from standard 7400 series logic chips – individual counters, registers, buffers, and gates. It’s an educational example of a simple CPU that’s easy to understand and build, but still capable of running games and other interesting programs. Nibbler is built using wire-wrapping on a 5 x 4 inch (127 x 102 mm) perfboard. The CPU contains 17 chips in total.

• How to estimate an SSD’s working life

  Spin HDD working life is hard to predict because of their physical part movement. SSD on other hand have no such part, so should be easiser to predict based on Wear leveling and Total Bytes Written.

• Dealing with Flaky Tests

  Walk through tips for an app with 2000+ tests that does not have a severe flaky test problem

• optimizing performance for 1000 units

  How to get intense multiplayer combat with 1000 units to use just 50kb/s bandwidth. Push JS Perf to the extreme. The result is that with a simple “collision cells” optimization scheme, the overall CPU time of the server thread has been brought down by about 90%.

Code to read

• egui

  an easy-to-use immediate mode GUI in Rust that runs on both web and native

• typeless

  an interpreter for λ-calculus implemented in ruby

• tesseract.js

  Pure Javascript OCR for more than 100 Languages 📖🎉

• quicktemplate

  Fast, powerful, yet easy to use template engine for Go. Normally template engine work by parse a template file, process macro, replace certain placeholder with the right data So if there is an error in template engine, its hard to catch at compile time. What if we convert the template to Go function and execute it? This is the same approach Elixir/Erlang use.

• photon: an open-source incident light meter

  A hardware/firmware project

Tools

• sniffnet

  Cross-platform application to monitor your network traffic with ease

• vm2

  sandbox that can run untrusted code with whitelisted Node’s built-in modules. Securely! Very useful when you want to run untrusted JavaScript to process some interal ap data such as extension of an app

• olric

  Distributed in-memory object store. It can be used both as an embedded Go library and as a language-independent service

• guarddog

  a CLI tool to Identify malicious PyPI packages

Happy thanks giving for anyone that celebrate this holiday. I take a week off next week for thanksgiving so there will be no issue next week. I recommend some get away time from computer this weekend as well. I come from a third world country and I’m so thankful for what I’m able to do nowadays thanks to people who putting the foundation for the electric, computer, internet infrastructure. Internet changes the world forver IMHO.

• Who controls the internet

  For internet to work, the world need a DNS to translate domain to ip address. But who control those DNS servers?

• Argo Rollouts at scale: Bringing Automated Rollbacks to 2,100+ services at Monzo

  Monzo assively invested in the microservices philosophy and ship small and often. They have over 2,100 services and deploy over 100 times per day. With that much deploy they build very sophisicated tolling to deploy and rollback automatically.

• Explain that parameterized statement in postgresql!

  For detailed query performance analysis, you need EXPLAIN (ANALYZE, BUFFERS) output for an SQL statement. With a parameterized statement, it can be difficult to construct a run-able statement for EXPLAIN (ANALYZE). Sometimes, you don’t even know the parameter values.

• Reducing replication lag with IO concurrency in Postgres 15

  Database is the heart of an application. That’s why I always try to find some databse tips links for every issue. This is a quick tip to tune the IO when replicating data. The faster the replication, the better we can rely on read replicate to split load from primary

• What’s so great about functional programming anyway?

  Nobody cares what functional programming is, at least, not at first. “can we deliver better code, faster?”. It’s because functional programming allow us to write expressive code, with more confidence

• Reed-Solomon Error Correcting Codes from the Bottom Up

  Reed-Solomon forward error correction (FEC) is one such coding method. Until the discovery of better coding techniques (Turbo codes and low-density parity codes), it was one of the most powerful ways to make data storage or data transmission resilient against corruption

• Why you should never use px to set font-size in CSS

  Being a backend dev, I bited the bullet and did this anyway so it’s great to reflect back and see whats up.

• Building reliable services: A guide to setting SLOs

  Practical advice to setup SLOs. Other article on SLO from HoneyComb

• Is Everything Is A File?

  “Everything Is A File” is one of the most fundamental and widely influential pillars of Unix design philosophy.

• A Journey in Creating an Operating System Kernel

  This free book is intended to teach the reader how to create an operating system’s kernel called 539kernel.

Code to read

• varint

  fast & memory efficient arbitrary bit width integers in Go. The purpose of VarInt to provide the maximum memory compact way to use and store unsigned custom bits integers.

• tamarin

  A fun and pragmatic scripting language written in Go. May be used as a CLI or embedded as a library.

• forme

  HTML forms library for ruby by Jeremy Evans of roda fame

• happy-dom

  a JavaScript implementation of a web browser without its graphical user interface. It includes many web standards from WHATWG DOM and HTML. Essentially a quick way to learn and re-implement many browser concept in JavaScript itself.

• rsyscall

  a library which provides a interface to an ever-growing subset of Linux system call

• TinyBIOS

  A minimalist open source BIOS project for fun in Assembly

Tools

• SpyGuard

  a forked and enhanced version of TinyCheck. SpyGuard’s main objective is to detect signs of compromise by monitoring network flows transmitted by a device.

• pgsync

  Sync data from one Postgres database to another

• Pyrseas

  A tool to make a YAML represent the current database schema and verify against it to ensure schema consistency.

• ciao

  uptime monitoring for both of public or private service.

• hadmean

  an admin app generator. It generates powerful internal tools in seconds with just one command, no learning curve, no technical skills and finally zero maintenance.

• launchpad

  From Code to Kubernetes in One Step.

• Let's write a tiny chess engine in go

  In this article we will try to understand how chess engines work by porting the sunfish chess engine to Go. Sunfish is notable for its simplicity and small size, while still being capable of playing decent chess. Go is also known as a simple and highly readable language, so I hope the two of them could make a great couple.

• Creating fluffy trees with Three.js (part 1)

  It seems like a “useless” thing, just draw a tree but I really enjoy reading these kind of rendering/drawing graphic. I learn a bit of GLSL vertex shader today. Highly recommend you give this a try.

• How RegEx works

  This isn’t an article about how to use regexes; you’ve probably seen plenty of those already. It’s about how you would write a regex package from scratch, in a language like C that doesn’t already have regexes. The author use Perl but you got the idea of writing a RegEx engine.

• Reverse engineering an EV charger

  I like a lot article about hardware because as a software programmer, hardware is like a blackbox to me :-)

• Scaling PostgresML to 1 Million Requests per Second¶

  PostgresML engineering team discuss how they horizontally scale PostgresML to achieve more than 1 million XGBoost predictions per second on commodity hardware. Their architecture and benchmarking method.

• Seeing through hardware counters: a journey to threefold performance increase

  Netflix upgraded their servers to large instances and didn’t see the increaseing performance they are looking for and go hunt it.

• Tracing HTTP Requests with tcpflow

  Kind of like a short guide for tcpflow. it can analyze the data transmitted via tcp sockets. It can look at any live tcp socket and show the back and forth communications.

• Building a BitTorrent client from the ground up in Go

  Nothing is better than building a PoC of some established protocol. You’ll learn a lot.

• Accidental $70k Google Pixel Lock Screen Bypass

  Xdavidhu found a vulnerability affecting seemingly all Google Pixel phones where if you gave them any locked Pixel device, they could give it back to you unlocked. The bug just got fixed in the November 5, 2022 security update.

• Inside the mind of a frontend developer: Hero section

  it will be interesting to dig into the mind of a frontend developer while they are working on implementing a component. The most important thing here is the thinking process, not the CSS outcome as sometimes it can get fairly straightforward to understand

• Faster pagination in Hey

  Pagination on a single table with a sort key is easy, but when we add join and filter out query from that join table, the pagination can be slow. Here, Hey.com, an email app shares a technique they used to improve its performance.

• WebAssembly and Docker - Better Together

  A curated list of Wasm and Docker working togethe

Code to read

• tipi

  the All-in-one Web Server for Ruby Apps. I found its codebase is quite small enough to read.

• sunfish

  a Python Chess Engine in 111 lines of code. It’s amazing and have lot of comment to help understand it, code is very clean.

• perfect-shape

  a collection of geometric algorithms that are mostly useful for GUI manipulation like checking containment of a point in popular geometric shapes such as rectangle, square, arc, circle, polygon, and paths containing lines, quadratic bézier curves, and cubic bezier curves. Also, some general math algorithms like IEEE-754 Remainder.

• FusionAuth HTTP client and server

  A full featured, stand-alone, high-performance HTTP server and client written entirely in plain Java. I am not a Java dev but this library is from FusionAuth which I found very useful replacement for tools like okta or auth0 now that both of them got acquired by big corps.

• canvas

  a common vector drawing target that can output SVG, PDF, EPS, raster images (PNG, JPG, GIF, …), HTML Canvas through WASM, OpenGL, and Gio. It has a wide range of path manipulation functionality such as flattening, stroking and dashing implemented. I really love to learn how to draw thing with code.

• chaosbernie

  Azure as an external process source for psDoom-ng. a fun experiment please don’t use it for real.

• litedb

  Embedded NoSQL database for .NET.

Tools

• hishtory

  Your shell history: synced, queryable, and in context

• portmaster

  a free and open-source application firewall that does the heavy lifting for you. They are entirely open source, check out portmaster and safing private network

• slashbase

  Slashbase is an open-source minimal collaborative IDE for your databases in browser.

• katana

  A next-generation crawling and spidering framework.

• sargamicons

  A Collection of 262+ Open-source Icons Devoted to Match Your Exquisite Designs.

• uiverse

  A places that people create and share beautiful custom elements made with CSS and HTML which you can pretty much immediately copy and paste to apply to your website.

• mage

  modern replacement for Airflow

• hono

  Ultrafast web framework for Cloudflare Workers, Deno, and Bun. Fast, but not only fast.

• age

  Graph database optimized for fast analysis and real-time data processing. It is provided as an extension to PostgreSQL.

• scrutiny

  Hard Drive S.M.A.R.T Monitoring, Historical Trends & Real World Failure Thresholds

Hi everyone, welcome to first issue of November. This week, where I live, red leaves start to emerge and my family went outside, enjoy seeing the beautiful color of autumn. Social media had been a tough week with Twitter and Elon, same as finance system and crypto drama. The world looked so depressing, but no matter what, being a programmer, knowing how to code is really a privilege, we have a way to make a living while sitting at a deck doing thing we enjoy. No matter what happen, there will always a demand to make website, application or service and a continous stream of jobs require to run these things. We are thankful for the jobs and opportunities we had, be optimist and always leaning. on ward. have a good week everyone.

• How to interactively SSH into a build job on any CI system

  This week I shared a small tip to debug CI build failure easiser. If you ever debug some tricky CI failure and say to your self: “If I can SSH into this server or container that running the build, I will be able to figure out immediately”, then this could be very useful. It can also apply anytime we need to SSH back into some private infrastructure.

• Reducing BigQuery Costs: How We Fixed A $1 Million Query

  During the infrastructural exploration of a pipeline my team was building, we discovered a query that could have cost us nearly $1 million USD a month in BigQuery. Below, we’ll detail how we reduced this and share our tips for lowering costs in BigQuery.

• The Case for JPEG XL

  Recently, the Chrome developers announced their decision to remove the behind-a-flag support for JPEG XL. Cloudinary argue(to which I agree) JPEG XL can bring these benefits without introducing a regression in other areas, at least in terms of technical strengths. We can only hope that the Chrome developers reverse their decision and help JPEG XL catch up with the old formats in terms of software support, so we can all enjoy the benefits it brings.

• Wrap Third Party APIs in Service Wrappers to Simplify Your Code

  Almost web app needs to make call to third party API such as taking a payment, send out an email. By wrapping an adapter around each API, presenting only the features of that API the app needs, we can more easily manage and test the app.

• Visualizing Diffs

  when we run diff or git diff, we see a visualization of what were add, insert or delete? what algorithm power that?

• Motion Controls In The Browser

  If you’ve ever wanted to build a web app that you can control with hand gestures as if by magic, this article is for you. With a couple of APIs and some JavaScript, you can build apps that behave like sorcery.

• BTREE VS. BRIN: 2 options for indexing in postgresql data warehouses

  BTree is the most common index type we used every day. For exact lookup, BTree probably best choice most of time, but if you’re looking for a range query, Brin maybe an option

• Postgres Databases and Schemas

  We’re going to dig into some of the flexibility on architecting your database and terminology within Postgres and the trade-offs for different approaches.

• Can I exec an entirely new process without an executable file

• Internet Egress Filtering of Services at Lyft

• Tales from the Kernel Parameter Side

  When you tune performance for a system, very quickly you run into stuff like nofile, transaparent huge page which usually either has some high level command to set it, or sometime require you to change a certain parameters by writing to /proc/sys/dev. So today we will have a look around it.

• How to roll your kafka cluster with zero downtime and no data loss

• DevOps Roadmap

  This is a step by step guide on how to become a DevOps engineer, with links to relevant learning resources. If you plan to become a DevOps engineer, could be a great learning resource.

Code to read

• zip-it-and-ship-it

  AWS Lambda deployment requires a bundle of entire of your app and all dependencies, package in a certain structure. Zip it and ship it do exactly that, create a zip bundle for a lambda app in node go or rust and deploy to aws.

• polyphony

  a library for building concurrent applications in Ruby. Polyphony harnesses the power of Ruby fibers to provide a cooperative, sequential coroutine-based concurrency model. Under the hood, Polyphony uses io_uring or libev to maximize I/O performance.

• warp

  A Python framework for high performance GPU simulation and graphics

• xlwings

  a Python library that makes it easy to call Python from Excel and vice versa.

• rueidis

  A Pure-go redis driver use RESP3 protocol. If you curios about writing driver, this is a great example

• birdcage

  a cross-platform embeddable sandboxing library allowing restrictions to Filesystem and Network operations using native operating system APIs. It should be ued for education purpose only at this point

Tools

• medusajs

  The Open Source Shopify Alternative. Medusa is a composable engine that combines an amazing developer experience with endless customizations for merchants to scale

• markdoc

  a powerful, flexible, Markdown-based authoring framework to create blog or document site in markdown.

• cnquery: Open source, cloud-native asset inventory and discovery

  It answers thousands of questions about your infrastructure, and integrates with over 300 resources across cloud accounts, Kubernetes, containers, services, VMs, APIs, and more.

• cnspec

  finds vulnerabilities and misconfigurations on all systems in your infrastructure including: public and private cloud environments, Kubernetes clusters, containers, container registries, servers and endpoints, SaaS products, infrastructure as code, APIs, and more.

• blip

  A tool for seeing your Internet latency.

• pg_stat_kcache

  Gather statistics about physical disk access and CPU consumption done by backends

• pgdump-aws-lambda

  Lambda function for executing pg_dump and upload the output to s3. Can be useful to take a daily/hourly pg_dump of database

• pgMemento

  Audit trail with schema versioning for PostgreSQL using transaction-based logging

• icones

  an UI to search and explore many free or open source icon set. one UI to rule them all.

Hi, everyone, happy halloween. This issue is a haloween treat :-), it has a lot of short and small articles which you can apply to your daily work and small tips and tricks too.

• Time is an illusion, Unix time doubly so...

  We all though time is an ever increasing number from January 1st, 1970, we called this epoch. But reality kick in, leap second, 2039, y2k.. Lets dig in to see how it was handle and gotcha dealing with epoch and its data type through … time.

• Scaling Causal's Spreadsheet Engine from Thousands to Billions of Cells: From Maps to Arrays

  Simon shares with us how we help Causal Spreadsheet engine achive its performance. Simon pioneer the idea of first-principle thinking to estimate systems performance fast, by going to its root to understand and estimate it before writing any code. He moves the enginer from browser into node js into Go, scale from 10,000 rows to 100,000 then to 1,000,000.

• How slack uses terraform

  Using Terraform efficiently is a dark art. it’s very opinated and different among companies from what I see. terraform is often a mess because of the velocity of dev and workaround stuff we cannot do cleanly in terraform. Therefore learn from big company like Slack can be a great guidance to see how people handle big infra with it.

• Bit Twiddling Hacks

  There is a certain operation which can achive just by manipulation the binary represent of the data. It’s a bit C-heavy but understanding them make me has a better understanding of the lowest level that represent our data and how a certain operation is applied on these structure

• you might not need js

  I won’t drop JavaScript for these but it’s great to see how we can push the limit by leveraging CSS. Example, galery slide, modal box

• Design tips for developers

  Majority of BetterDev audiences are software engineers. And Im sure we love to build SaaS, when it comes to desigb, it is our painpoint. So here is a guide to help us design better.

• Shell Script Best Practices

  a short list which you can quickly do to write better shell script

• Wiring home with Fiber

  Stefan shareds how he wire his home network using fiber.

• Linux Security Hardening and Other Tweaks

  with the power of search engine such as Shodan, it can easy setup a unsecure server and have it discovered. Better to have some kind of check lists we can go through to harden our server.

• Which serverless edge platform has the fastest git deployments?

  deno is a new platform and a language at the same time written by NodeJS creator. In this post,they benchmark fly vs cloudflare vs vercel vs deno.

Code to read

• node-linux-pam

  pam allows you to delegate authentication to an external system. This is a nodejs package to show how we do PAM in nodejs.

• pam-onelogin

  a pretty complete pam/nss stack for using OneLogin as authentication source (with MFA) and user/group lookups. Primarily used for SSH.

• alba

  the fastest JSON serializer for Ruby

• tart: A Toy Gitlab CI Runner backed by Firecracker MicroVM.

  We usually want a CI runner to be very isolated, one way to achive is using VM but they are expensive and slow to boot. Other way is docker, but then we need to mount docker socket, solve “docker in docker” path issue, lot of gotcha. A new method recently is to use VM that boot very fast so we can have the best of both world.

• dataprofiler

  What’s in your data? Extract schema, statistics and entities from datasets

Tools

• quill

  Simple mac binary signing from any platform. Without this signing, we will have to go to “System Preference” to approve running unidentified code sign binary.

• FerretDB

  an open source MongoDB alternative but the underlying data is backed by … Postgres. So the best of both world

• ballerine

  Have you ever need to do KYC(Know your customer) in the app? The kind of system where you have to enter a certain information, upload a certain doc and get approve or reject. This is an open source system that do just that.

• Build FFmpeg filters without the headache

  I recently working on a video app and remember all the ffmpeg detail isn’t an easy job

• soul

  Soul is command line tool, after installing it, Run soul -d sqlite.db -p 8000 and it’ll start a RESTful API on localhost:8000 back by the sqlite db

Hi, everyone, welcome to our second issue of the come back. I had received many emails with encouragement about my resumt. I really really appreciate that. Thank you so much.

• Secure your connection by limit outgoing connection

  For years, I was just a reader, consume information and share with BetterDev audience. But that’s going to chance. I’m trying to make article and video screencast so I had some thing to offer to the world to share my limited knowledge and experience. This will be my first post ever, and I don’t even have blog setup yet so I’m leverage “buymeacoffee” link. I read about many hacks recently, some are very high profile hack on some dependencies, so how do we defend these hack? Since our app now run untrusted cost? I want to share my though and my approach on how I defense these attacks.

• SSH: More than secure shell

  SSH is the defactor to get a shell run in a remote host. But it can do more such as port forwarding or an intermediate host to access other host

• The HTTP crash course nobody asked for

  Explain HTTP at a very deep but at same time in a very practical manner so you can see and simule how thing work at plain text level. You will see the whole HTTP requests payload, how it was made and how to send it ray character by character.

• JavaScript metaprogramming with the 2022-03 decorators API

  One of Python great feature IMHO is its decorator system. Literally just by adding @decorator_name right before your function definiton, suddenly we can run custom code around the function. That power is being brough to JavaScript. The proposal are at stage 3. Once its reach stage 4 it will be ready for standarization.

• The wasted potential of CSS attribute selectors

  Every classes we added to our CSS has a cost in term of space and parse time. BEM standarize the modifier concept so we have class like menu--is-active. Modern HTML utilize ARIA for accessibility so stuff like screen reader can understand its current page/active syntax for example. So why don’t we use `menu[aria-current=‘page’] to customize its style. Make it broader we can even customize a link based on its domain, instead of adding extra parameter and at the same time it’s super flexible. Example: a[href^=https://specific-domain.com].

• From Development to Real Users: How to Create a Web Performance Story

  Some of the most common questions asked when it comes to work with performance are, How do you convince stakeholders that improving the performance of your project is actually worth the investment? How can you prove that the work is necessary to begin with? Or prove that you have shipped improvements? And what is the impact of certain changes on users in different scenarios?

• A Look at Postgres 15: MERGE Command with Examples

  Postgres update conflict is very powerful to update if row is already existed. But there is tricky case such as if data is already existed, delete it. Such as toggle a row by insert/delete, not easy to do with updatte conflict but now can do with merge. update conflict also rely on a contrain,

• Honeycomb Incident Review: Shepherd Cache Delays

  Honeycomb is an event logging platform where log and metric can be send to it for viewing and analyzing. They have an outage of 8 hours on Sep 8th, 2022. This post is an analysis of that incident. We will learn a lot from these real world incident, seeing how those world class term debug and troubleshoot their infra. What interesting is that Honeycomp cannot find the root cause and have accepted that they probably won’t find out how it all came together. The data simply isn’t there, their runway has been extended, and they’re now looking for longer-term fixes that circumvent the overall pattern.

• The Linux-PAM Module Writers' Guide

  PAM is an amazing feature of Linux that allows to customization the authentication process of an application on Linux. Such as when SSH to the node, we can add extra MFA verification. You can write PAM plugin by writing dynamic so lib that Linux can load, and PAM can call the specific funciton. Think of its like a set of function we have to expose to be called. They usually written in C, but in modern way we can easily hook up to Go, Rust, Zig and make it super easy to custimization the authentication process on a SSH server for example

Code to read

• sqlite3vfshttp

  sqlite3vfshttp is a sqlite3 VFS for querying remote databases over http(s). This allows you to perform queries without needing to download the complete database first.

• twgl

  A Tiny WebGL helper Library

• psd.js: a psd file parser

  If you curious how PSD files are structure and parse it to get all the object, layer then read on

• net-scp

  Pure Ruby implement of scp. great to read and understand all tcp detail of it

• an experiment to minimize the barrier between Python and Rust to unlock the full potential of high level languages. Build your own Runtime using standard CPython and extend it with Rust

• lapce

  Lightning-fast and Powerful Code Editor written in Rust

Tools

• spin

  an open source framework for building and running fast, secure, and composable cloud microservices with WebAssembly

• hayabusa

  Windows event log fast forensics timeline generator and threat hunting tool created by the Yamato Security group in Japan.

• sematic

  an open-source development toolkit to help Data Scientists and Machine Learning (ML) Engineers prototype and productionize ML pipelines in days not weeks.

• p5js

• mutagen

  Fast file synchronization and network forwarding for remote development

• appmap

  a free and open-source runtime code analysis tool with support for Ruby, Java, Python, Javascript. They had quite a bit of supported platform

• mirrord

  Have you ever debug some tricky thing only happen in prod in a K8S cluster and wish you can hook up local process as if it was a container run inside the prod cluster to connect to prod resources and debug it? Now you can do that with mirrord. Also read their introduction article

Hi, everyone. It has been almost 6 months with out any betterdev issue :-). It was due to my busy schedule. I finally started to get thing back on track. I will come back to my regular schedule of one issue every week again to capture the spirit of development every week. Thank you for being a supporter. If you are no longer interested in betterdev, click in the unsubscribe link at the bottom of the email to unsubscribe.

• Security headers quick reference

  There are certain of headers that just by adding them make browser protect the app from a certain kind of attack. Here is a quick walk through for some common ones.

• GCP Penetration Testing Notes

  Penetration teams start from the lowest level of the infrastrucuture, understanding every aspect of privilege and permission. We as a developer don’t have time to focus on all of them and just use one we most frequently use, which create blind spot for us. By understanding all of these aspect enable us to make information choice when designing infrastructure and make it more secure

• Postgres Full Text Search vs the rest

  In recent years, there are a bunch of search enginer appear to take on ElasticSearch such as TypeSense, MeiliSearch, Toshi. So how Postgress FTS compare to them in term of performanc for ingesting, query and accuracy.

• Six consideration when managing PostgreSQL indexes

  Needless to say what to index is an art. Over index and the app slow down on write, or make the engine harder to decide what index to use. Here, knock team shares with us their signal that they use to make decision on indexing.

• How to draw hexagons grid

  This guide will cover various ways to make hexagonal grids, the relationships between different approaches, and common formulas and algorithms

• WebKit Quirks

  Have you work on a very big app where a few old customers cannot upgrade their system for whatever reason and we have to maintain backward compatible and hack around? This is exactly what happen with WebKit browser engine, they have very ugly hack that check and hard code domain name, css selector to account for old websites.

• 10 best practices to containerize Node.js web applications with Docker

  There are quite good practices to apply to any container such as not using root, run multistage build etc.

• A tcpdump Tutorial with Examples — 50 Ways to Isolate Traffic

  Have you ever setup some tricky services where it cannot be exposed to the world directly and goes through a few proxy or tunnel, using a few iptables trick. It’s impossible to debug those without invoking tcdump to know where the traffic hit, how does it flow? But remember tcpdump syntax is tedious and I keep coming to this examples all the time