A queue pickup that ignored its limit, a plan cached while the table slept, and a consumer that deadlocked on its own success.

The System, Briefly

A staged pipeline processes events; each stage has a consumer composed of a fetcher goroutine and N = 10 workers per replica, two replicas per stage. Tasks live in a Postgres table, and a task’s stage is a column. A consumer picks up tasks by leasing them:

UPDATE tasks
SET picked_at = now() + $window, attempts = attempts + 1
WHERE id IN (
  SELECT id FROM tasks
  WHERE state = $state AND deferred = 'no' AND attempts <= $max
    AND (picked_at IS NULL OR picked_at <= now())
  FOR UPDATE SKIP LOCKED
  LIMIT $n)
RETURNING *;

After the UPDATE, the consumer takes a Redis lock per task, commits, publishes a notification per task on the event bus, and starts one goroutine per task that renews the lease and the lock every $window/3 until the task is released. Handlers advance the task to the next stage. If a consumer dies, leases lapse and another replica picks the task up with attempts + 1. This is a textbook Postgres queue, and it had run for months. Until it didn’t.

The Page: Nothing Is Wrong

An operator noticed 460 tasks in flight in one stage. Dashboards were fine, alarms were quiet. Everything was – as far as telemetry could tell – working as intended:

• The stage’s consumers were healthy and reported no errors;
• The database was healthy, with no long transactions and a connection pool that was nowhere near its limit;
• Tasks were not failing and not exhausted.

However, a backlog query told the real story. By splitting tasks into groups of waiting (eligible now), leased (a consumer holds them) and exhausted, I could observe the following scenario:

What we’re looking at is unexpected: all 460 were leased, with attempt 1 – and, beyond what any dashboard showed, their leases expired within half a second of each other. Two replicas with ten workers can hold about twenty tasks, not 460, and Redis agreed: 460 lock keys, one per task, with TTLs inside the refresh cycle – something alive was renewing all of them.

The go_goroutines metric named the process. One replica had 463 goroutines, while the other had 83; every other service in the namespace sat between 23 and 43 (the baseline). Each held task costs one refresh goroutine, so a replica with 420 extra goroutines was holding about 420 tasks that no worker was handling.

The Goroutine Dump

Although this wasn’t production, the pods already ran with a strict, hardened security posture – no shell, no user, nothing – and pprof wasn’t enabled. The last resort was to SIGQUIT the misbehaving process and read its goroutine dump. The cost was that the signal would also restart the pod, and everything would go back to “normal” until the same case was hit again, but it was the best available at the moment. It was telling:

The second replica’s dump had the same shape, with 40 refreshers. For completeness’ sake, here’s an ultra-simplified version of the protocol they follow:

// Worker
for {
    tasksWanted <- struct{}{}   // token: "I am idle". Capacity N. This is line
                                // 530: all ten workers were blocked here.
    t := <-tasksReceived        // Wait for a task. Capacity N.
    handle(t)
}

// Fetcher
for {
    wanted := takeTokens(tasksWanted) // up to N.
    tasks  := pickup(wanted)          // This runs the UPDATE … LIMIT $n
                                      // mentioned above.
    for _, t := range tasks { go refresh(t) }
    for _, t := range tasks {
      tasksReceived <- t              // This is line 418: the fetcher
                                      // blocked here.
    }
}

Every task the fetcher delivers is backed by a token it took, so with at most wanted tasks per pickup neither channel can fill. Hand the fetcher a pickup of 440, though, and the invariant is gone:

1. The fetcher takes ten tokens and receives 440 tasks. Ten go straight to the idle workers and ten more fill the tasksReceived buffer; the fetcher then blocks on the next send, holding the rest.
2. Each worker that finishes sends a token for its next task and picks up a buffered one. The fetcher is stuck in its delivery loop and takes no tokens, so the token channel fills after ten.
3. When that second wave finishes, the first worker to send a token blocks – the channel is full – and soon all ten do.
4. With no worker receiving, tasksReceived fills again and the fetcher stays blocked. The undelivered tasks have live refreshers and no handler: their leases and locks renew forever, the attempt stays at 1, and no timeout exists to notice the condition.

The protocol left a fingerprint: N tasks in the first wave plus N sitting in the buffer, so exactly 2N = 20 handled… then silence. Logs showed exactly that on both replicas, 20 starts after the burst, and nothing after. A small simulation of the channel logic reproduced it numerically: a first pickup of 440 gives 20 handled and 420 leaked refreshers; one of 60 gives 20 handled and 40 stuck. Those are the two replicas’ numbers.

This leaves us in a position where the deadlock was real and well-understood, and it needed a precondition that the design said was impossible: a pickup returning more than LIMIT.

Chasing the Impossible

The lock counters gave the first hard evidence: between two given timestamps, one replica acquired 440 locks, and the other 60. Two pickups, 500 tasks in total, which matches exactly the size of the batch that was inserted into the database moments before. Each replica’s first pickup after the 500-task burst had taken everything it could see. One more oddity: the first task started about two seconds after the insert committed. I’ll come back to that.

My hypotheses, with what happened to them:

• Connection pool exhaustion: an alert had fired (pool waits), and it is the first thing everyone looks at. It was a red herring: the metric counts acquires that had to open a connection, and the pool had ten workers against 36 connections, none in use.
• A leaked transaction – no: every path rolls back on exit, and the database showed only sub-millisecond transactions.
• Clock skew between the service that inserted the tasks and the consumers, to explain the two-second delay between the commit and the first start. I measured it: the pods agreed within 24ms.
• Slow Redis locks during a 440-task pickup. Measured: each acquire took under a millisecond.
• Slow publishing after the commit. Measured: each notification takes about 0.5ms.
• The query: I read it, ran its EXPLAIN on the environment where the application was, and tested it locally on the same Postgres version: stale statistics, fresh statistics, generic plans, eight concurrent pickers, 8,000 pickups, a pickup in the same shape as the real transaction. The largest batch was exactly 10 every time; another refuted hypothesis.

However, that last verdict was wrong, and the way it was wrong is the interesting part of this story. The tests were sound, but they sampled the wrong space: they only exercised the plans a healthy table produces, and the failure needed a particular plan.

The Query

Here is what EXPLAIN ANALYZE showed on the environment’s engine, on a scratch table in the state the real one was in, with the planner nudged away from its hash, sort, and materialise strategies:

Update on tasks (actual rows=500.00 loops=1)
  -> Nested Loop Semi Join (actual rows=500.00 loops=1)
       -> Seq Scan on tasks (actual rows=502.00 loops=1)
       -> Subquery Scan on "ANY_subquery" (actual rows=1.04 loops=502)
            -> Limit (actual rows=1.04 loops=502)
                 -> LockRows (actual rows=1.04 loops=502)
                      -> Index Scan ... on tasks tasks_1

actual rows=500 for LIMIT 10? The subquery is the inner side of a nested loop and runs loops=502 times. The same script on community Postgres 18.6 gave the identical plan, the same cost numbers and the same 500 rows, so this could not be an AWS Aurora – used by the application on its environment – defect. The issue was what the executor did with that plan.

The mechanism is in ExecLockRows: a row the current command has already updated comes back from the lock attempt as TM_SelfModified, and the executor skips it – the code comment cites the Halloween problem. Each re-execution of the locking subquery therefore skips every row this statement has already updated. Outer and inner scan the table in the same order, so the first surviving row is always the current outer row: the semi join matches on it and stops, and LIMIT 10 never binds. That is the rows=1.04 per loop above and, in the repro at the end, rows=250.50 loops=500 on the inner scan: on average 249.5 skipped rows plus the one returned. Every outer row therefore finds itself in a fresh set, and the statement updates all of them. With two statements running at once, each also skips the rows the other has locked, which then splits the table between them. Two sessions racing under that plan, 250 and 250 locally. In production, 440 and 60: the replicas started 463ms apart.

The documentation is candid about SKIP LOCKED:

Skipping locked rows provides an inconsistent view of the data, so this is not suitable for general purpose work, but can be used to avoid lock contention with multiple consumers accessing a queue-like table.

It also says that:

If a LIMIT is used, locking stops once enough rows have been returned to satisfy the limit (but note that rows skipped over by OFFSET will get locked).

Read carefully: that is a guarantee about one evaluation of the query. Nothing promises that a LIMITed locking subquery inside IN (…) is evaluated once, and as we know, the planner may choose any join strategy. The documentation is explicit that, for a locking clause inside a sub-SELECT, conditions from the outer query may be used to optimise the sub-query’s execution. For this specific query, some of those strategies are catastrophic.

Why That Night

Forcing a plan with planner flags proves a mechanism, not a cause. I still had to explain how Aurora came to run that plan with default settings.

Three facts combined:

1. The driver caches prepared statements per connection: pgx’s default mode prepares a statement server-side and reuses it. For the first five executions the server plans with the actual parameters (custom plans). After that, plan_cache_mode = auto may switch to a generic plan, planned once without parameter values, and cache it on the connection.
2. The fetchers polled the table back to back: the loop, I found, had no sleep after an empty pickup, so each connection ran this statement thousands of times an hour against a table that could, for long stretches, be idle and empty. The generic plan was built from the statistics of an empty table: one estimated row.
3. For a one-row estimate the planner sees no reason to materialise the subquery: Materialize appears when the outer side is estimated at two rows or more, which is why a table with two live rows was safe in every test. At one row, the nested loop with the bare subquery is the cheapest plan.

A fresh custom plan, planned after the burst, would have scaled its estimate by the table’s current size and been safe. Only a cached plan carries the empty table’s estimate forward, and it is invalidated only by something that refreshes those statistics. Then 500 rows arrived in one transaction, and until Aurora’s autovacuum analysed the table, every pickup ran through a plan built for an empty table. With no live tuples recorded, the scale factor contributes nothing, so the trigger is the base threshold (autovacuum_analyze_threshold, 50 rows by default), which 500 inserts crossed at the next autovacuum_naptime tick – five seconds here (the community default is a minute); in my lab runs the analysis landed within a few seconds, about three in one of them. ANALYZE invalidates the cached plan, and the next execution was safe once again.

That explains why the bug needed a burst after an idle period to show itself. It also needed the burst to exceed N: with ten rows or fewer, the bad plan is indistinguishable from a good one, so a single task enqueued five minutes earlier and picked up one millisecond after its commit proves nothing either way.

I could reproduce it with no planner settings, through the real driver. A prepared statement was executed eight times against an empty table, then 500 rows were inserted with no ANALYZE:

The Fixes

1. The Query

Pick the rows in a CTE that is evaluated exactly once, and update from it. The documentation notes that a locking clause on the outer query does not reach into a WITH query, and that to lock within one, one must put the clause inside it, which is what this does:

WITH picked AS MATERIALIZED (
  SELECT id FROM tasks
  WHERE state = $state AND deferred = 'no' AND attempts <= $max
    AND (picked_at IS NULL OR picked_at <= now())
  FOR UPDATE SKIP LOCKED
  LIMIT $n)
UPDATE tasks
SET picked_at = now() + $window, attempts = tasks.attempts + 1
FROM picked
WHERE tasks.id = picked.id
RETURNING tasks.*;

Under the same forced plan that made the old statement update 500 rows, this one updates 10. The plan reads CTE Scan on picked with the work done once under CTE picked, so the join can iterate however it likes without side-effecting anything else.

Two alternatives are worth naming. = ANY(ARRAY(SELECT …)) has the same effect, since an uncorrelated ARRAY() subquery is evaluated once; I prefer the CTE because it states the intent. Setting plan_cache_mode = force_custom_plan removes this particular trigger, but leaves a statement whose correctness depends on which plan it gets, and I would rather fix the statement. I keep MATERIALIZED: by default a side-effect-free WITH query used once may be folded into the outer query, and I would rather not depend on whether a locking SELECT counts as side-effect-free.

2. The Consumer

A fetcher that deadlocks when a producer breaks its contract is brittle, even if the contract is “obviously” kept. The surplus tasks must now be delivered one per fresh token, so the fetcher keeps taking tokens while it has tasks to hand out:

for i, t := range tasks {
    if i >= wanted {
        select {
        case <-tasksWanted:
        case <-ctx.Done():
            releaseRest(tasks[i:])
            return
        }
    }
    select {
    case tasksReceived <- t:
    case <-ctx.Done():
        releaseRest(tasks[i:])
        return
    }
}

I chose to drain an oversized pickup instead of rolling it back: a rollback would have stalled a replica for as long as the database kept over-returning rows, and I did not yet know why it had. The code also logs an error with the row count, the distinct ids and the milliseconds spent in the query, the locks and the commit, which is the evidence this incident lacked. Shutdown now releases every undelivered task, where before only the one being delivered was released.

Draining has a cost: the replica that wins the race holds the whole surplus, which is exactly the 440/60 split of this incident. The alternative is to keep wanted rows and release the rest before committing (picked_at = NULL, attempts - 1), which bounds the damage without stalling a replica. It trades a second write path for a smaller blast radius, and I did not carry it in the same change.

3. The Tests

The most useful is the one that changes nothing about the planner:

func TestPickupHonoursItsLimitAfterIdlePolling(t *testing.T) {
    // analysed, empty table; autovacuum disabled so nothing re-analyses it
    for range 8 {
        require.Empty(t, pick()) // same connection: the plan is cached per connection
    }
    insertBurst(500)
    require.Equal(t, 10, len(pick()))
}

On the old query it fails with expected: 10, actual: 500; on the fix it passes. Next to it sits a test that forces a 440-task pickup through the fetcher and checks that all 440 are handled. With the old delivery, it fails with “20 of 440 tasks handled”: the production fingerprint.

What I Couldn’t Prove

1. The incident’s plan: the cluster keeps no plan history (the engine’s plan capture is off, there is no slow-statement log, and pg_stat_statements was preloaded but never created in the database). I had a faithful reproduction on the same engine, not a recording.
2. The two seconds: the first task started about two seconds after the insert committed. Clock skew, lock acquisition and publish cost were each too small, and they stay small when summed: for the 440-task pickup, 440 locks (under 440ms in total), 440 publishes at about 0.5ms each (about 220ms) and the 47ms statement come to under a second at worst, about 0.7s. The other replica is further off still: 60 tasks add up to roughly 0.15s, against an observed 2.5s. The two delays barely differ despite a sevenfold difference in pickup size, which points to a roughly fixed cost rather than a per-task one. I don’t know what it is.

Lessons Learned

1. IN (SELECT … LIMIT n FOR UPDATE SKIP LOCKED) is a trap: it is correct under most plans, and the plans that break it are exactly the ones a young table, a stale statistic or a cached plan produce. Materialise the locked set, or use = ANY(ARRAY(SELECT …)). Then test with the planner against you.
2. Prepared statements, idle tables and bursts are a hazard class: a plan’s lifetime is not the statistics’ lifetime. A generic plan built during a quiet hour can serve a burst for as long as nothing invalidates it. Hot tables that go empty deserve a deliberate plan-cache strategy.
3. Idle polling without backoff has side effects beyond load: it guarantees the switch to a generic plan within minutes, and the connection then keeps that plan for as long as it lives. Backoff alone would not have prevented it – any sane interval reaches five executions within the hour – but a loop that never sleeps is a bug in a trenchcoat, and no spec ever covered the empty queue.
4. A consumer should survive a producer that exceeds its contract. Bound every blocking operation, and keep the invariant between your channels local and obvious. The application’s protocol was correct if and only if a database query kept a promise – one that was treated as an invariant and never verified.
5. A liveness failure that looks like work is the worst kind. Tasks that are leased and not progressing look busy: healthy probes, growing metrics, no errors. Alert on liveness (goroutine counts, handled rate against leased count), not only on backlog age.

Procedures Pinned

Process lessons, separate from the technical ones:

1. Alerts name candidates, not causes. The pool-waits alert pointed at the database first, and the fastest way past it was to read what the metric actually counted.
2. Write hypotheses down together with their refutations, and be ready to retract them. I retracted my own conclusions, including “the query is safe”, when a better experiment arrived. The record made that cheap.
3. Build the reproduction on scratch objects, in transactions you roll back, with a snapshot behind you. The experiments that cracked this ran on tables created for the purpose, on the same engine as the environment, and were safe because nothing they did was permanent.
4. Measure what you would otherwise infer. Clock skew, lock latency and publish cost each had a plausible story and each was ruled out with a number.

Reproducing the Issue

The following self-contained script reproduces it on community PostgreSQL 18.6 (the actual rows=500.00 decimals are the version 18 output format); the same sequence on tables shaped like mine gave the same result on Aurora PostgreSQL 18.6. It updates 500 rows for a limit of 10. Plan caching and MATERIALIZED CTEs date from 12, but the one-row estimate for an analysed empty table depends on the reltuples = -1 convention introduced in 14, so I’d expect 14 and newer to be affected. I have not tested earlier versions:

CREATE TABLE tasks (
  id        uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  state     text NOT NULL,
  picked_at timestamptz,
  attempts  int NOT NULL DEFAULT 0
);
CREATE INDEX ON tasks (state, attempts, picked_at);
ALTER TABLE tasks SET (autovacuum_enabled = false); -- holds the statistics still,
                                                    -- as in the first seconds
                                                    -- of the incident

ANALYZE tasks; -- an idle, empty table. This matters: a table never vacuumed
               -- or analysed (reltuples = -1) is assumed to hold at least ten
               -- pages, which hides the bug

PREPARE pick(text, bigint) AS
  UPDATE tasks SET picked_at = now(), attempts = attempts + 1
  WHERE id IN (SELECT id FROM tasks WHERE state = $1 AND picked_at IS NULL
               FOR UPDATE SKIP LOCKED LIMIT $2)
  RETURNING id;

-- a worker polling an empty queue (the sixth execution onwards may use a
-- cached generic plan)
EXECUTE pick('pending', 10); EXECUTE pick('pending', 10);
EXECUTE pick('pending', 10); EXECUTE pick('pending', 10);
EXECUTE pick('pending', 10); EXECUTE pick('pending', 10);
EXECUTE pick('pending', 10); EXECUTE pick('pending', 10);

-- a burst arrives, and nothing has re-analysed the table yet
INSERT INTO tasks (state) SELECT 'pending' FROM generate_series(1, 500);

BEGIN;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, BUFFERS OFF)
  EXECUTE pick('pending', 10);
ROLLBACK;

 Update on tasks (actual rows=500.00 loops=1)
   ->  Nested Loop Semi Join (actual rows=500.00 loops=1)
         Join Filter: (tasks.id = "ANY_subquery".id)
         ->  Seq Scan on tasks (actual rows=500.00 loops=1)
         ->  Subquery Scan on "ANY_subquery" (actual rows=1.00 loops=500)
               ->  Limit (actual rows=1.00 loops=500)
                     ->  LockRows (actual rows=1.00 loops=500)
                           ->  Seq Scan on tasks tasks_1 (actual rows=250.50 loops=500)

Run ANALYZE tasks; and execute the statement again, and the plan is replanned and updates 10. Replace the statement with the MATERIALIZED form above, and it updates 10 in both cases.

Final Thoughts

None of the individual pieces here was exotic: the documentation was accurate, the driver cached a plan as drivers do, and the loop – a bug, but a quiet one – polled an empty table. The bug lived in their product, and the failure window was the few seconds between a burst and the next ANALYZE. That is the part I keep coming back to – a contract that held for months without ever being tested under a bad plan. Two questions remain open (the plan itself, and two seconds of latency), and I’d rather leave them open than paper over them. The repro above takes a minute to run; if it prints something different on your version, I’d like to know.

Nothing like debugging what was assumed as an invariant for a couple of hours.

But Vito, everyone moved to GitHub, that broke the “decentralised” part!

It may have. But Git, per se, is still decentralised; one may maintain copies in multiple places. The option to keep everything in one place was a choice people made freely, for convenience or visibility, and that’s fine. Git was Git. GitHub was GitHub.

For a long time, they earned the position. Here’s the shape of what they built:

• April 2008: GitHub launches.
• July 2009: 100,000 users.
• July 2010: 1 million hosted repositories. 60% codebases, 40% Gists.
• April 2011: 2 million repositories. 33 employees.
• September 2013: 7 million repositories, 4 million users, 234 employees.
• March 2015: 21 million repositories, 9 million users, 330 employees.

330 people carrying 21 million repositories: the ratio alone is worth sitting with. And yet, the platform was stable, the product made sense, and the surrounding ecosystem was alive in the way that healthy ecosystems are: every service did one thing and did it well. Want automated tests? Travis CI. Static analysis? Codacy. Coverage? Coveralls. GitHub was Git hosting with a social layer on top. The Unix-shaped approach worked – and no surprises there.

I treated my GitHub feed as a homepage for years. The signal from starred-by-people-I-respect was worth the visit on its own. That company was a place I wanted to be part of: large-scale systems, helping people all around the world, contributing back to the community.

June 4, 2018

Microsoft announces the acquisition. The developer community is caught off guard, and although everyone knows very well how Microsoft operates, nobody is quite sure what to make of it. The first months are quiet, quieter than expected. Then the displacing begins:

• May 2019: Dependabot is acquired. Third-party dependency scanners lose their reason to exist on the platform.
• September 2019: Semmle is acquired to become CodeQL, which becomes the engine of GitHub Advanced Security. Same playbook.
• November 2019: GitHub Actions launches, scrapped from Azure Pipelines. Travis CI, the de facto standard for open-source CI, is sold to Idera that same year and guts itself within two years.
• March 2020: npm is acquired. It has been visibly under-maintained since.
• 2021 onward: Copilot ships, trained on the public corpus its own users uploaded under licenses that did not contemplate this use. The lawsuit (Doe v. GitHub) is still working through the courts.

The pattern is Microsoft’s EEE playbook with a fourth E bolted on. Embrace the ecosystem. Extend it with first-party equivalents. Extinguish the third parties that depended on integration access. Then enshittify what remains.

What the Numbers Look Like

Pre-acquisition: a flat line at effectively four nines. Post-acquisition: a noisy descent that has never recovered. The worst drops – dipping below 99.6% – cluster in 2020, 2022, and 2023. There has not been a clean year since 2019, and the most recent data point, January 2026, is among the worst in the entire dataset.

This is not subtle, and it does not recover.

What’s Left

The product I log into now is slow, the UI nags about Copilot in places where you’re reading issues, reviewing PRs, managing settings. Outages that used to be quarterly are now a weekly occurrence, and the chart above is what that looks like in aggregate, stripped of any individual incident’s ability to be explained away.

I don’t think this is recoverable from inside the company, and I don’t see incentives pointing that way.

What I’m Doing

All my projects now live on Forgejo instances I run myself. Maintenance is minimal: it’s a Go binary and a database. The server does what I need and stops there. No feed, no notifications about features I didn’t ask for, and the uptime is whatever I make it — and it’s higher than GitHub’s.

For anyone who doesn’t want to self-host, Codeberg is doing serious work, and Forgejo is building federation so instances can communicate without a central host.

I’ll keep a presence on GitHub for work that requires it. Nothing new goes there.

git remote set-url origin was a two-second command, but the decision to run it took way longer than the migration.

If the first post was about dismantling assumptions, this one is about replacing them with certainty. Back then, we had a decoder that worked, a pile of strange patterns, and a growing suspicion that the name RLE was little more than a historical artifact. Now, after walking the bitstream one bit at a time and validating every rule against a real corpus, the format is no longer a black box. It has structure, semantics, and — finally — a specification.

When I started digging into this format, I expected some variation of classic run-length encoding. What I found instead was closer to a tiny virtual machine for pixels. Yes, there are runs, but there are also signed deltas, literal spans, and a single bit-cursor shared across colour channels. Nothing is byte-aligned, nor accidental: every bit participates in reconstructing UI artwork that was never meant to be seen outside its original renderer.

The Turning Point: Transparency

Up until then, decoding felt linear: read tokens, expand pixels, move forward. Alpha changed the rules. Transparent spans emit pixels without consuming colour data. Buffers persist across segments; the “previous value” never truly resets. Suddenly the format stopped behaving like a simple stream and revealed itself as a stateful decoding model layered on top of a bit-level language. Once that behaviour became clear, the rest of the structure — row navigation, token width B, palette post-processing — just snapped into place.

Verification

Hundreds of files were decoded and compared pixel-for-pixel against the original renderer. No heuristics, no guesses — just reproducible behaviour. At that point, the reverse engineering effort had effectively crossed a boundary: the goal was no longer to make a decoder work, but to describe why it works.

So instead of shipping another tool, I wrote the document I wish I had at the beginning: a complete, clean-room specification of the format. It defines the container, the payload, the bitstream grammar, channel interleaving, the alpha meta-decoder, palette semantics, DPI scaling, and the precise arithmetic rules that govern decoding. Every field is explained, every algorithm is explicit. Every edge case that once looked like noise now has a name.

The format is no longer folklore, no longer that “strange Messenger blob.” It is a fully described image format with a deterministic decoding model — and you can read it here:

RLE-DIB Format Specification

The mystery is over. From here on, this format is something one can implement, reason about, and build against — and, perhaps most satisfying of all, the decoder is now far smaller than the puzzle that led to it. I validated the spec by implementing a Swift parser — aside from the Python parser included in the spec itself — that faithfully rendered assets on macOS.

Legal & Research Disclaimer

This material is intended strictly for academic research, reverse engineering for interoperability, and software preservation.

The author does not distribute, reproduce, or provide any proprietary software, including components originally developed by Microsoft.

This work is independent, clean-room in nature, and is not affiliated with, endorsed by, or supported by any original vendor.

Any trademarks, formats, or software referenced remain the property of their respective owners.

The first step to make this way easier is to enable logging for troubleshooting, which happily dumps really interesting stuff into ~/Library/Logs/Microsoft-Messenger.log.

The Login Dance

Once an email and password is provided, the client immediately opens a raw TCP connection to messenger.hotmail.com, port 1863. This seems to be the server that speaks the MSNP protocol, given the very first message from the client:

VER 1 MSNP16 MSNP15 MSNP14 MSNP13 CVR0

The protocol is fairly simple: text-based, each message ending in CR-LF. It has the following format:

  VER 1 MSNP16 MSNP15 MSNP14 MSNP13 CVR0
  ━┳━ ┳ ━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   │  │  │
   │  │  └──▶ Payload
   │  └──▶ Message ID
   └──▶ Operation

The message begins with an operation code, followed by the message ID, and a payload. The server always responds with the same message ID, and a corresponding operation code and payload.

The VER operation is a handshake between the client and server. The client announces the protocols it supports, and the server answers with one of the protocols it supports. So the fake server can just pick the highest one and be done with it.

I suppose CVR stands for two different things during the flow. When sent by the client, it may be a Client Version Report, meaning “I use the base CVR capability schema, revision zero”. For the server reply, CVR may be a Capability Vector Revision, but I can’t confirm that. Echoing the CVR0 back to the client is enough to make it happy.

This leads to the following response from the server:

VER 1 MSNP16 CVR0

Once the version is agreed between parties, the communication continues with the client sending:

CVR 2 0x0409 mac 10.14.6 i386 macmsgs 8.0.1 macmsgs hey@vito.io

This is the Client Version Report, containing the following fields:

• 0x0409: Microsoft’s LCID (Locale ID) in Hex. 0x0409 is simply en-US; Italian, for instance would be 0x0410.
• mac: Platform/OS family.
• 10.14.6: OS version string.
• i386: Architecture. This seems to be hardcoded in the binary.
• macmsgs: Client product identifier. Just identifies as Microsoft Messenger for Mac.
• 8.0.1: The client version.
• macmsgs: Again. Not sure why.
• hey@vito.io: User identity. That’s me!

Even though the client waits for a response, what the server returns doesn’t seem to be used at all. My theory is that this may be just telemetry, or a mechanism for Microsoft to block old clients or architectures.

For now the fake server is replying with CVR 2 1 2 3 4 5, and Messenger is happy. After getting the reply, the client then starts the real authentication flow:

USR 3 TWN I hey@vito.io

From observation, USR is the User Authentication OP. Decoding it:

• TWN: I assume this means Ticket With Nexus, considering what happens later.
• I: Seems to be the authentication phase the client is currently at. I may indicate Initiate? Initial?
• hey@vito.io: That me, again! This just includes whatever the user inputs in the login “Email” field.

The server then replies with some metadata that’s opaquely handled by the client and appended to an HTTP Authorization header. So the server can reply with literally anything:

USR 3 TWN S fake=1

Then things shifts a little:

The Passport Backend

While still connected to the TCP server, the client opens a separate connection entirely – a concurrent, out-of-band flow to the defunct Passport.net server. Not without caveats, though.

All communication with Passport.net uses TLS – Actually, not TLS. SSL. – A really old one: SSLv2 Hello, TLS 1.0. And the best part is that OpenSSL dropped that a while ago. What I need now is a CA Root installed on Mojave, and a TLS terminator so I can continue observing traffic.

So I got a certificate for nexus.passport.com, and wrote a little TLS shim in C to route TLS traffic from Mojave to my alpine box running the fake servers. Just a matter of compiling OpenSSL 1.0.2u, accepting the connection with the right certificate, and proxying the raw TCP data. OpenSSL was linked statically with the final binary, so we don’t break the whole OS.

The first request is a GET to /rdr/pprdr.asp. With some guesswork we can assume rdr stands for Redirect or Redirector, and pprdr may stand for Passport Redirect(or) – naming those things don’t really matter, but it’s nice to give names to those pesky abbreviations Microsoft uses.

The response has no body, but the client expects an HTTP header named PassportURLs with a few fields:

• DARealm: The realm name. Not used by the client, but it requires it to be present.
• DALogin: The realm host and login path. Microsoft seemed to use nexus.passport.com/login2.srf. The lack of schema is intentional, the client will use HTTPS. If you are wondering, DA stands for Domain Authentication.
• Properties: Not used by the client, but it requires it to be present.

Those pairs get concatenated as K=V separated by commas.

Then, the actual login magic happens. The client POSTs to DALogin without a body, but with an interesting header:

Authorization: Passport1.4 fake=1,OrgVerb=GET,OrgURL=http%3A%2F%2Fmessenger.msn.com,sign-in=hey%40vito.io,pwd=asdf

Breaking it down:

• Passport1.4: The authentication mechanism version. Hardcoded.
• fake=1, the opaque value provided by the TCP server on the S phase of the TWN operation. Everything connects!
• OrgVerb=GET: I honestly have no idea what that’s supposed to be. Maybe something for federation, or to allow Messenger servers outside Microsoft?
• OrgURL=http://messenger.msn.com: Same as OrgVerb.
• sign-in=hey@vito.io: The identity of the user being authenticated
• pwd=asdf: The password provided by the user on the login window.

The response can again have an empty body, but the client expects a specific header to be present:

Authentication-Info: Passport1.4 da-status=success,from-PP='a=HENLO'

Breaking it down:

• da-status: The status returned from the Domain Authentication. Here we are indicating the authentication was successful.
• from-PP='a=HENLO': Assuming this as From Passport. The value is opaque to the client, it just returns this to the TCP server.

This is the last interaction with Passport – for now! The dance continues with the TCP server again, where the client sends the following message:

Back to TCP

USR 4 TWN S a=HENLO 235411cb78234a468feb6f1bccca43fd

Same OP, different message ID – as it’s a new message, – and the value returned from passport is handled opaquely and just returned to the TCP server. The final hex is 32 characters – MD5-length – but I haven’t confirmed what it represents. Correlation token, telemetry, session nonce; take your pick.

With this, the TCP server can just FINALLY complete the authentication flow by replying with:

USR 4 OK hey@vito.io 1 0

• OK: Authentication passed. Hooray.
• hey@vito.io: The user’s identity
• 1: Authentication flags. 1 seems to be enough to keep the client happy.
• 0: Capability flags. Just echoing from CVR.

Should we expect something else over TCP now? Absolutely not. The plot thickens:

Microsoft SOAP Services

Microsoft loves SOAP for reasons one can’t even reason about. As I’m just trying to boot the app, and not actively use it, the idea is to just provide the minimal amount of stuff to make it get to the main window post-login.

Looking at how the client uses those endpoints, I can again just provide what it deems required. One interesting thing to notice, however, is that for each SOAP endpoint, the client seems to perform authentication once again against the Passport service, and relays whatever the authentication endpoints provides in response to a TicketToken field.

SOAP goes – at least initially – to one endpoint:

byrdr.omega.contacts.msn.com receives a POST request for /abservice/SharingService.asmx, with an HTTP header SOAPAction: http://www.msn.com/webservices/AddressBook/FindMembership, followed by another POST, this time to /abservice/abservice.asmx, with the same header, but this time containing the value http://www.msn.com/webservices/AddressBook/ABFindAll.

On those, FindMembership seems to be responsible to indicate the membership graph. The MSN Address Book service (aka abservice) has several logical lists, where each one corresponds to a different kind of “membership” in a materialised view for the current user. Those lists are:

• FL: The Forward List, representing one’s contacts (people one added)
• AL: The Allow List, representing who can see one’s presense status.
• BL: The Block List, containing blocked contacts.
• RL: The Reverse List, containing who added the user (reverse of FL).
• PL: The Pending List, containing pending invitations.

FindMembership provides the AL, BL, and RL lists, while ABFindAll returns what’s enough for the client to rebuild the FL list.

ABFindAll is a different story; it returns the full contact records for the authenticated user. Each record contains several fields, including:

• contactId: a GUID representing the contact.
• contactType: the contact’s type. Regular is the only value that’s of my interest right now.
• passportName: the user’s Passport name. Basically their email.
• displayName and quickName: the user’s display name. Not sure why it is sent twice.

The object may contain lots of other fields, but those are the most relevant to achieve our objective.

Once those SOAPs are returned, the client resumes communication with the TCP server:

The Final Dance

The next operation requested by the client is a new kind we haven’t seen yet: a two-part operation. This is what is sent:

ADL 5 245\r\n
<ml l="1"><d n="local"><c n="away" l="3" t="1" /><c n="brb" l="3" t="1" /><c n="busy" l="3" t="1" /><c n="hidden" l="3" t="1" /><c n="idle" l="3" t="1" /><c n="lunch" l="3" t="1" /><c n="online" l="3" t="1" /><c n="phone" l="3" t="1" /></d></ml>

This will need some explanation. Through the previous SOAP, I’m setting a few users to have each one showing a different status. Here are the users:

  GUID                                 Passport
  ------------------------------------ -------------
  11111111-1111-1111-1111-111111111111 online@local
  22222222-2222-2222-2222-222222222222 busy@local
  33333333-3333-3333-3333-333333333333 idle@local
  44444444-4444-4444-4444-444444444444 away@local
  55555555-5555-5555-5555-555555555555 brb@local
  66666666-6666-6666-6666-666666666666 lunch@local
  77777777-7777-7777-7777-777777777777 phone@local
  88888888-8888-8888-8888-888888888888 hidden@local

ADL is the Address List command. The payload 245 indicates how many bytes follow after the terminator (\r\n), comprising a second payload, that’s formatted below:

<ml l="1">
  <d n="local">
    <c n="away"   l="3" t="1" />
    <c n="brb"    l="3" t="1" />
    <c n="busy"   l="3" t="1" />
    <c n="hidden" l="3" t="1" />
    <c n="idle"   l="3" t="1" />
    <c n="lunch"  l="3" t="1" />
    <c n="online" l="3" t="1" />
    <c n="phone"  l="3" t="1" />
  </d>
</ml>

The XML contains a single ml (Membership List) element with an l attribute with value 1, indicating this is the first list exchanged through the ADL OP. Each sub-list is split by domain, and since all my users are @local, we get a single d (Domain) element, with the n attribute set to local. Inside we have each user as a c element, their name in the n attribute, l (List) set to 3, which is a bitmask indicating that the contact is both on FL and AL lists and t (Type) set to 1, indicating a Passport-type identity.

The bitmask values are:

• 1 = Forward List (FL) – contacts the user added
• 2 = Allow List (AL) – contacts allowed to see the user’s status
• 4 = Block List (BL)
• 8 = Reverse List (RL)
• 16 = Pending List (PL)

For now we just acknowledge the message, so it can continue with its process:

ADL 5 OK\r\n

The next OP we get from the client is PRP:

PRP 6 MFN hey@vito.io

PRP stands for Profile Property. The client is requesting a property to be set: MFN, also known as My Friendly Name. Again, we just need to acknowledge it, but this time with an ACK OP:

ACK 6 PRP\r\n

The client continues with:

BLP 7 AL

BLP is the Blocking Policy. It controls the default privacy rule for people not explicitly on AL and BL. The last token we get is the policy to be applied, which can be one of:

• AL: Allow by default. Unknown users can see and contact the current user.
• BL: Block by default. Only allowed users can see and contact the current user.

The client is setting the policy to Allow. Ok. We just acknowledge it again:

ACK 7 BLP\r\n

Next up: Status update. The client sends:

CHG 8 NLN 1085276192:131088

CHG requests a status update, NLN indicates Online, and is followed by an optional capability bitmask (two numbers, colon-separated). For now I’ll ignore those, but they will be important in the future, depending on how this progresses.

For this one we will echo everything back to the client:

CHG 8 NLN 1085276192:131088\r\n

But this change is also an important trigger for the TCP server. From this point onwards, the client accepts the Initial Status List, (ILN) which assigns statuses and Display Names for each one of its contacts, followed by an UBX OP that provides extra information regarding their profiles, such as a Personal Status Message, and Media Status.

Here we reuse the Message ID from CHG to send one ILN for each contact, along with an UBX. Notice, however, that UBX takes no Message ID. From the previous contact list, I’ll also assign a random UUID for their PSMs:

ILN 8 NLN online@local 1 Online%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX online@local 1 89\r\n
<Data><PSM>0b48cf1c-b9a5-4793-b8d4-a8ddd6b13c6c</PSM><CurrentMedia></CurrentMedia></Data>

ILN 8 BSY busy@local 1 Busy%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX busy@local 1 89\r\n
<Data><PSM>72b6dca4-811b-4c6a-8b2a-c87cfa6e3299</PSM><CurrentMedia></CurrentMedia></Data>

ILN 8 IDL idle@local 1 Idle%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX idle@local 1 89\r\n
<Data><PSM>88360ef7-5152-4ada-8bc8-e0343736e830</PSM><CurrentMedia></CurrentMedia></Data>

ILN 8 AWY away@local 1 Away%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX away@local 1 89\r\n
<Data><PSM>d430617a-17bf-4437-a8f2-54aa4d33444e</PSM><CurrentMedia></CurrentMedia></Data>

ILN 8 BRB brb@local 1 Brb%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX brb@local 1 89\r\n
<Data><PSM>ecd12daf-0c11-445f-b1ae-4751a509c25c</PSM><CurrentMedia></CurrentMedia></Data>

ILN 8 LUN lunch@local 1 Lunch%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX lunch@local 1 89\r\n
<Data><PSM>9bd609bc-d2f8-4e1a-be72-06804bf73183</PSM><CurrentMedia></CurrentMedia></Data>

ILN 8 PHN phone@local 1 Phone%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX phone@local 1 89\r\n
<Data><PSM>b1482981-ad81-4a2a-872f-ba3331c6bb91</PSM><CurrentMedia></CurrentMedia></Data>

The ILN command has the following parameters after the Message ID:

• The status (NLN, BSY, IDL, AWY, BRB, LUN, PHN)
• Their email address
• Their Network ID (1 = Passport/Windows Live)
• Their display name, URL Encoded
• Their capabilities flag. Here I’m sending the same caps the client provided to the server, except that it only takes the first portion (before the colon).
• A msnobj payload. Here it is empty, and has no significance to my objective.

UBX has the same format as UUX (see below).

As a side note, the required msnobj portion was what took most of the time during this session. Without it the client just refused to assigned statuses and failed silently. Codepaths leading to this portion were also quite complicated, so getting to the point “oh, that’s what’s missing” took a while.

Then, more state is sent by the client:

UUX 9 53
<Data><PSM></PSM><CurrentMedia></CurrentMedia></Data>

UUX (User eXtended data) is a set of metadata announced by the client. Here it’s setting the PSM (Personal Status Message), and the CurrentMedia to empty values. Again, we just acknowledge it:

ACK 9 UUX\r\n

Following, we get two extra UUX messages:

UUX 10 75\r\n
<EndpointData><Capabilities>1085276192:131088</Capabilities></EndpointData>

and

UUX 11 92\r\n
<PrivateEndpointData><EpName>Rhodes</EpName><ClientType>1</ClientType></PrivateEndpointData>

The first one contains the same capability set we got during the CHG OP. The second is part of the “multi-endpoint” implementation that was being rolled, allowing users to sign in in multiple places at the same time. It contains the endpoint name EpName, that’s carrying the name of the computer running Messenger (Rhodes), and a client type. We can assume 1 represents a Desktop.

Then, the client continues asking for a service URL using the URL OP:

URL 12 MOBILE

There it is asking specifically for the Mobile Messaging / SMS Service MSN used to support. For now, let’s just route it to our server:

URL 12 MOBILE http://10.0.10.126:8080/\r\n

And continues with further URL ops:

URL 13 PROFILE 0x0409
URL 14 CHGMOB

PROFILE is requesting the profile webpage/service URL for a LCID in hex, again, 0x0409 is en-US. CHGMOB is the Change Mobile settings URL, part of the MOBILE capability. For both we will reply the same URL as MOBILE, as the client wouldn’t care anyway.

Followed by:

UUN 15 hey@vito.io 8 9
ignore me

I think UUN is User Update/Notification the value 8 is unknown, and 9 represents the amount of bytes following. For some reason, the client is sending ignore me. Weird. After some Trial-and-Error, the client is happy with a simple ACK:

ACK 15 UUN\r\n

And finally, the client rests sending periodic PNG (Ping) OPs:

PNG 16\r\n

QNG 60\r\n

The result is nothing less than satisfying.

And there it is. Eight contacts, eight statuses, no Microsoft infrastructure involved. A fake TCP server, a TLS shim, a handful of SOAP responses, and a lot of trial and error – enough to convince a 2008 Mac app it’s talking to servers that haven’t existed for over a decade.

That was just a little detour try to relax from the amount of Assembly and C/C++ on Windows. Did it work? No, I had to disassemble the macOS binary, but at least that is known territory.

Later on I will continue on the main project.

But this one… This one lied to me in the first three bytes.

52 4C 45   →   "RLE"

It was not RLE.

Not byte-aligned runs, not (count, value), not even something you could generously call RLE-like. What sits behind that header is a bit-packed predictive codec with interleaved channels, delta predictors, a meta-decoder layered above colour decoding, and a rendering pipeline buried under more code than anyone should have to read for a small icon.

What you are reading is about how I went from opaque binary blobs to a bit-exact decoder, validated against the original renderer. No Microsoft code used. Only observation, instrumentation, patience, and stubbornness.

This is the first step toward a side project I was thinking about for way too long: recreating the Windows Live Messenger 2009 look-and-feel. Same visuals, entirely new engine, open, secure, cross-platform. Not nostalgia, not preservation. Reconstruction.

The most important part of this endeavour would be recreating the assets, or at least extracting them. WLM2009 had stunning visuals that made it look great, both on PCs and Macs. So the first task was set: Let’s get those assets. That couldn’t be hard, right? Right?

Where are the assets?

The UI clearly contained intricate elements: icons, frames, gradients, control visuals, but there were no images anywhere. One may expect to at least find PNGs, BMPs, and such. But no PNG, no BMP, no JPEG, nothing that looked like pixel data, nothing in the resources that resembled bitmaps. It was madness! How could that amount of transparency and effects be nowhere?

Well, let’s make a small detour to a quick history lesson: Microsoft’s internal UI frameworks.

Everything on WLM2009, and actually, in all Windows Live™ products of the same era used an internal framework called DirectUI. After some research, it seems that while people were suffering with MFC and WPF, Microsoft was using a custom UI framework to make Windows Live, Office, and as far as I understood, parts of Windows as well. That was DirectUI. People seem to think they gave up on that, but I haven’t looked deeper, sorry.

So that meant that all that fancy stuff was being drawn by something else, but as WLM also worked on Windows XP, Windows Vista, and Windows 7, binaries should provide those facilities. Awesome, but where are the assets?

Interestingly, what those binaries lack in PNGs and other image formats, they compensate with lots of blobs containing the same three-byte magic header: RLE – Great! Everyone (okay, not everyone) knows RLE stands for Run-Length Encoding, so decoding that would be easy: extract the RLE blob, strip the header, run the algorithm, and we should get some kind of image.

No. Not this time. Doing this yielded nothing but garbage. Again, wrong assumption. Now it feels like even Occam was also lying to me; a big conspiracy.

Jokes aside, one thing is clear: that was no RLE, but something else! Something internal that only Microsoft knew, and that poked that small piece of my brain that really, really likes to dig into this kind of thing – so it took the wheel. Entropy analysis, heuristics, pattern matching, anything that could give any kind of direction on what those blobs were about. Nothing.

That was the moment the situation became clear. These were not compressed images. These could only be the images. There was no fallback, no original bitmap stored somewhere else. The blob itself was the canonical representation of the artwork. But if we cannot decode it, we cannot reproduce the UI. There is no other source of truth.

Finding the real decoder

Searching for RLE on the binaries leads to a forest of misleading symbols. Helpers, wrappers, palette code, rendering glue, dead paths, UI plumbing. Most of those never touch pixels. They prepare, route, or post-process, but they do not decode.

Ghidra helped map the topology, but the generated C was often fiction. Registers reused in unexpected ways confused the decompiler, calling conventions were occasionally guessed wrong, stack frames collapsed, and control flow flattened into something that looked plausible but was not what the CPU executed. The assembly was the only reliable source of truth.

So the process became mechanical: Ghidra to understand structure, x32dbg to observe reality, follow registers and observe the process’s memory, confirm assumptions against execution, and treat decompiled C as a hint rather than documentation; instead of trusting the C, the only source of truth was the generated assembly from the binaries. And that hit hard.

But no problem, nothing we can’t solve, right? Eventually one function stood out: RLEDrawToDIB. That’s the one responsible for converting the mysterious RLE into something observable, and even better, on a DIB (Device-Independent Bitmap), which can be exported.

Ok, we have the entrypoint, but what now?

Building an Oracle

The main executable helped a lot finding where and what was responsible for drawing, but the amount of noise after that was just too much, after all, the main binary did way more than just rendering. x32dbg was constantly pausing execution at way too many places, and analysing the stack at those points became tiring.

The plan now was to devise an oracle. Its job is to exercise only the hot path concerning parsing and drawing. For that, one must first find the entrypoint – which is already done, we have RLEDrawToDIB, – and make it work isolated from the host binary. As it may be known, dear reader, I am no Windows developer; most of my life I used Linux, Unix, OS X, and now macOS, so this is a whole new world. But as I usually say, in the end it’s just code, so let’s jump in and understand how to make this work.

The idea is dumb in the best way: load uxcore.dll into a process we control, resolve internal functions by raw offset – no exports, no symbols, just hard RVAs pulled from Ghidra. Reconstruct just enough of the calling sequence to drive the real renderer into a DIB section we own. Then dump every pixel.

The shim is small: a macro computes function pointers as base + offset into the loaded binary. A handful of typedefs describe the calling conventions Ghidra helped identify, a thin wrapper mirrors the original dispatch logic, including the nine-slice path, and routes execution through the same code paths the real UI would take. The renderer writes into a top-down 32bpp DIB section, BGRA in memory, which gets converted to RGBA and dumped as raw bytes alongside a JSON sidecar with dimensions, pixel format, and colour-source metadata.

That raw RGBA dump is the oracle. Not the PNG, as PNGs go through GDI+ encoding and are useful for eyeballing, but byte-level comparison needs raw pixels. Every single byte my clean-room decoder produces can now be diffed against what the original renderer actually drew. No ambiguity, no interpretation, no room for “close enough.”

But the oracle was not just a validation tool, but an exploration tool. With the real renderer running under our control, we can instrument it, set breakpoints, watch which functions get called and in what order, and trace the actual execution path through those functions. Instead of staring at a forest of dead code in Ghidra trying to guess what matters, we can observe what matters. The oracle narrowed the search space from everything to exactly this.

The process ran across 516 files exported from Messenger. For each one: load the blob, ask uxcore.dll for the frame size, allocate a DIB, clear to transparent, render, flush, dump. A system-colour table gets serialised at the end so palette-mode files can be validated deterministically. The whole thing is a few hundred lines of C++ and the ugliest #define you have ever seen, but it worked.

A note aside, outputting PNG using GDI+ is ugly as ugly gets, not to mention Windows ABI. If there’s a hell, I’ve found it. I’m also skipping a whole part where the oracle crashed, memory was corrupted, NULL pointers were dereferenced, and everything that could go wrong, did. Just for context, here what I tried once it refused to work:

• Do we need to initialize COM for this process? Maybe. Would it hurt if we did it? Hardly. So COM initialization was done. As the oracle was a “Console Application”, in Windows lingo, COINIT_APARTMENTTHREADED seemed enough. That didn’t change anything.
• Maybe Common Controls were missing? Did the DLL expect this to be ready? InitCommonControlsEx to the rescue. Guess what? Nothing changed.
• Maybe GDI+ needed to be initialized? GdiplusStartup was called, and guess what? That wasn’t it either.

Only then I took the time to look at the host binary again, only to find two interesting functions being called on uxcore.dll: UXCoreInitProcess and UXCoreInitThread. And ed ecco! No crash.

This exposed the whole hot path the decoder used: every function could now be isolated, inspected, and observed. “But how many?” you may be asking yourself, and I’m glad you asked. From there the hot decoding path expanded into one hundred and twenty-one internal functions, excluding Win32 calls. That was the actual renderer. Now to get the gist of the format, we just need to extract all of them as both C code and assembly. The result? ~20k SLOC of C code, ~41k of Assembly. I hoped I had enough coffee in the kitchen for that and for the blessing of Our Lord Turing, cause I knew that would hurt. Not that Assembly is a problem, but after optimisations and some weird stuff we know Microsoft compilers do, I knew I was signing up for headache and frustration.

Ghidra vs reality

One may wonder why export both C and Assembly. Well, at several points, the decompiler confidently produced code that looked reasonable and also was completely wrong. And not wrong in subtle ways, but wrong in ways that obliterate decoding on the first byte. Some examples (we will get to each of these):

• Bit order: The decompiled logic strongly suggested MSB-first reading. The stream is LSB-first. That single inversion produces garbage immediately, and it is the kind of mistake that looks correct on paper until you watch actual execution and realise every bit is landing in the wrong position.
• The predictor: Decompiled output implied frequent resets. In reality, it carries across runs within a row. That persistence is exactly why gradients compress well, and why breaking it turns smooth ramps into staircases.
• Row navigation: The decompiled view made it look absolute. It is forward-relative, with row zero starting at the last index entry. Miss that, and the decoder walks into the middle of someone else’s bitstream.
• Alpha: The reconstructed C placed it inline with colour. In execution, it sits above colour decoding, as a meta-layer that controls when colour tokens are consumed. Without modelling that correctly, colour buffers desynchronise and pixels drift. This one took the longest to untangle.
• Token alignment: Loops in the pseudocode looked byte-aligned. They are not. Everything is bit-aligned. Every. Single. Field.

For those starting in reverse engineering: The lesson is simple, but important. Decompiled code is not truth. Execution is.

The Trampoline

The format remained opaque until a small piece of assembly changed everything. There was a dispatch trampoline routing execution into different meta-decoders depending on header flags. That explained why streams behaved differently even when they looked similar.

Some used palette. Some used alpha runs. Some consumed colour continuously, others stalled. Some interleaved channels, others did not. The structure finally emerged. This was a bitstream, not a byte stream. Channels decoded independently. Predictive delta replaced repetition. Alpha was not inline; it existed in a meta-layer. Rows were not sequential but navigated via a pointer table. Bits were read least-significant-bit first.

This was not RLE. This was a “small” image codec wearing an RLE badge.

The Format

With the oracle, the trampoline, and 41k lines of assembly in front of me, the format started to take shape. What follows is a high-level overview, enough to understand why calling this “RLE” is an insult to the codec. A full specification with worked examples exists, but that deserves its own post.

The Header

Every blob opens with six bytes. Three magic bytes (RLE), an image count, a flags byte encoding the tile topology and payload size width, and a DPI byte. Six bytes. That is the entire file header.

The tile topology is a 4-bit nibble packed into the flags byte. Each bit controls whether a fixed border exists on that edge — left, top, right, bottom. The result is a nine-patch system: a 1x1 grid when no bits are set, a full 3x3 when all four are. Sixteen possible configurations, all encoded in half a byte. After some research, I noticed Android also have those nine-patch PNGs! Same idea, different decade. Microsoft got there first. (Is that a first?)

Sub-Images

Each sub-image carries a 4-byte header dword packed with geometry and flags. Width and height, yes, but also: palette mode, 5-bit or 8-bit precision, alpha presence, and the width of the row-navigation index entries. Thirteen bits for width, twelve for height, and the rest is flags. Every bit accounted for.

The row-navigation index is where things get even weirder. Rows are not stored sequentially. A table of forward-relative offsets sits at the start of the pixel data, and row zero begins at the position of the last index entry. It is elegant once you see it, and completely opaque until you do.

The Bit-Cursor

Everything past the row index is a bitstream. Not bytes. Bits. Read LSB-first within each octet, shared across all channels in a row. The decoder maintains a bit-cursor: a byte pointer and a bit offset, and every field, from token types to run counts to delta widths, is read through it. Nothing is byte-aligned unless it happens to be by coincidence.

Tokens

Each colour token starts with a 2-bit type and a B-bit run count, where B is derived from the image width. We have four token types:

• Solid fill: one value, repeated.
• Delta-positive: each pixel is the previous value plus a small positive offset.
• Delta-negative: same, but subtracted.
• Literal: raw values, one per pixel.

The deltas are the reason this format compresses well. Gradients, anti-aliased edges, smooth colour transitions: they all become tiny increments instead of full pixel values. The predictor carries across tokens within a row, which is what makes it work. And what makes it break if your decompiler tells you it resets. I’m looking at you, Ghidra.

The Alpha Meta-Decoder

And then there is alpha. It does not sit alongside colour: it wraps it.

A 2-bit alpha type is read before each group of pixels: transparent run, opaque run, or explicit alpha. Transparent runs emit zeros and skip colour decoding entirely. Opaque runs consume colour normally. Explicit alpha decodes an alpha token first, then pulls the corresponding colour values from the channel buffers.

This meta-layer is why the format was so hard to reverse. Colour and alpha share the same bitstream, the same cursor, but alpha controls when colour is consumed. Get that wrong and everything desynchronises. Ghidra placed it inline, and guess what? It is not inline. It is a state machine sitting above the channel interleaver.

Validation

With the format understood and a clean-room decoder written in Python (no dependencies beyond the standard library) it was time to answer the only question that matters: does it match?

The oracle produced raw RGBA dumps for 516 files extracted from Messenger’s binaries. The decoder ran against the same inputs. Byte-level comparison. No thresholds, no visual inspection, no “close enough.”

516 files. 380 single-tile, 136 multi-tile nine-slice. 516 exact matches. Not a single byte off.

What’s Next

This post covered the archaeology: finding the format, building the tooling, understanding the structure. The next one will go deeper: a full worked decode of a real file, bit by bit, token by token, with annotated hex dumps and the complete specification. If you enjoy reading binary formats the way some people enjoy crossword puzzles, you will like it.

There is also the matter of the DPI scaling pipeline and the palette system, both of which deserve their own attention. And eventually, the actual goal: using all of this to reconstruct the Windows Live Messenger 2009 interface, from scratch, on modern platforms.

But that is for another day.

New Findings

In the last post, I mentioned that I had omitted a few DHCPACK and DHCPINFORM packets, assuming they weren’t particularly important. My DHCP knowledge is a bit rusty, and I naïvely thought: “DHCPINFORM? The client already has an IP. No big deal.”

That assumption was completely wrong.

What was actually happening is that the Ray was requesting someone to inform it about some knobs so it could finish bootstrapping and start the boot sequence! That should have been obvious, especially given that it was retrying the same packet over and over since no one was answering it, but yes, I missed that. Oops!

Now, back on track. The packet clearly indicated what it was looking for:

• Option 53 (DHCP Message Type): DHCPINFORM (0x08)
• Option 55: Parameter Request List
  • Interface MTU (0x1A)
  • X Window System Display Manager (0x31)
  • TFTP Server Name (0x42)
  • Vendor-Specific Information (0x2B)

The choices there are interesting indeed! MTU makes sense, but, XDM? Odd at least.

At this point, we know that:

• The Sun Ray successfully acquires an IP
• It explicitly requests vendor-specific parameters
• It stalls indefinitely without them

Looking for some information on the Web I could find some piece of documentation that Oracle haven’t burned yet, and oh my, it’s telling. For pure archival purposes, I’m mirroring it here: Sun Ray Client Initialization Requirements Using DHCP, but what’s interesting there is the Alternate Vendor-Specific DHCP Options section, listing exactly what we need. The response for the DHCPINFORM SHOULD carry a bit of extra information in the Vendor-Specific Information TLV, but the Ray could piggyback on “Standard” DHCP Options, if your DHCP server didn’t support them. Clever, very Sun-ish.

What’s relevant for us is this: We could add a Vendor-Specific Information to the DHCPACK of the DHCPINFORM packet, and the Ray would just use it. And we also have A LOT of knobs that can be turned. Below is a quick summary:

• AuthSrvr, Type: IP - Mandatory. Single Sun Ray server IP addresses.
• AuthPort, Type: NUMBER - Sun Ray server port.
• NewTVer, Type: ASCII - Desired firmware version.
• LogHost, Type: IP - Syslog server IP address.
• LogKern, Type: NUMBER - Log level for kernel.
• LogNet, Type: NUMBER - Log level for network.
• LogUSB, Type: NUMBER - Log level for USB.
• LogVid, Type: NUMBER - Log level for video.
• LogAppl, Type: NUMBER - Log level for firmware application.
• NewTBW, Type: NUMBER - Bandwidth cap, value is bits per second.
• FWSrvr, Type: IP - Firmware TFTP server IP address.
• NewTDispIndx, Type: NUMBER - Obsolete. Do not use.
• Intf, Type: ASCII - Sun Ray server interface name
• NewTFlags, Type: NUMBER - Obsolete. Do not use.
• AltAuth, Type: IP - List of Sun Ray server IP addresses.
• BarrierLevel, Type: NUMBER - Mandatory. Firmware Download: barrier level.

Lots of interesting stuff, but let’s begin with what can really help us: Logging. It seems the Ray is capable of emitting log data to a Syslog server, and we can set the level for several facilities, including firmware application, network, and kernel! That’s just awesome! Thanks, Sun!

Now, given it wants a Syslog server, one can assume that the log levels are also standardised to the levels Syslog supports, and one would be absolutely right. From RFC 3164 - The BSD syslog Protocol, page 9 contains Table 2. syslog Message Severities, listing from 0 (Emergency) up to 7 (Debug). I think the box may use 6 (Informational) as default, so setting it to 7 may make it be more chatty!

Encoding Vendor-Specific Information

The archived page also explains with less-than-desired details the encoding format, but it’s nothing more, nothing less than a plain old TLV structure. Adorable!

The structure is composed of repeated TLVs: one byte for the tag, one byte for the length, followed by n bytes of value. The only important thing to keep in mind is that any IP type is a simple representation of the IP octets, meaning that 192.168.100.1 becomes 0xC0, 0xA8, 0x64, 0x01.

Now let’s make the Ray talk to us through syslog, and make it chatty. The DHCP Option payload for Vendor-Specific Information would then be:

• Option 0x2B:
  • 0x18, 0x04, 0xC0, 0xA8, 0x64, 0x01. (LogHost, 4 bytes, 192.168.100.1)
  • 0x25, 0x01, 0x07. (LogKern, 1 byte, value 7)
  • 0x26, 0x01, 0x07. (LogNet, 1 byte, value 7)
  • 0x27, 0x01, 0x07. (LogUSB, 1 byte, value 7)
  • 0x28, 0x01, 0x07. (LogVid, 1 byte, value 7)
  • 0x29, 0x01, 0x07. (LogAppl, 1 byte, value 7)

I don’t want to bring up a full syslog daemon, so listening to UDP port 514 is enough.

And surely, we got the little box to talk with us:

[syslog from 192.168.100.50:514] <15> 0x0.0x48b 0:14:4f:85:f2:b Network: soff 0 n 1
[syslog from 192.168.100.50:514] <15> 0x0.0x48b 0:14:4f:85:f2:b Network: GetNextAltAuth 0 0 0
[syslog from 192.168.100.50:514] <15> 0x0.0x48b 0:14:4f:85:f2:b Application: Stop tokens 8009df20
[syslog from 192.168.100.50:514] <15> 0x0.0x48b 0:14:4f:85:f2:b Application: sessionMonitor TcpOpen timeout to 0.0.0.0:7009
[syslog from 192.168.100.50:514] <15> 0x0.0x48b 0:14:4f:85:f2:b Network: soff 0 n 1
[syslog from 192.168.100.50:514] <15> 0x0.0x48b 0:14:4f:85:f2:b Network: GetNextAltAuth 0 0 0
[syslog from 192.168.100.50:514] <15> 0x0.0x873 0:14:4f:85:f2:b Application: getNextServer: no message - timeout

Gorgeous! It can talk, and the log levels do map cleanly to syslog severities. That <15> is a PRI value in the syslog protocol, obtained by (facility * 8) + severity, as defined by the spec. Working backwards:

15 / 8 = 1 remainder 7

Meaning:

• Facility: 1 (aka user-level messages)
• Severity: 7 (aka Debug)

The Authentication Server

It seems to be trying to do something, but failing. That’s expected, since the essential value is still missing from the Vendor TLV: AuthSrvr. It’s even marked as mandatory! One funny detail about that table is that, although the BarrierLevel is also marked as mandatory, the client continues the process normally. More on that later.

Now let’s set the AuthSrvr value. It’s the same as LogHost, but with a different tag:

• 0x15, 0x04, 0xC0, 0xA8, 0x64, 0x01. (AuthSrv, 4 bytes, 192.168.100.1)

After updating what’s mimicking the DHCP server the Ray expects and adding the potentially last flag it wants, we have new stuff on the syslog!

[syslog from 192.168.100.50:514] <15> 0x0.0x1e9 0:14:4f:85:f2:b Network: GetNextAltAuth 0 c0a86401 0
[syslog from 192.168.100.50:514] <15> 0x0.0x1e9 0:14:4f:85:f2:b Application: Stop tokens 8009df20
[syslog from 192.168.100.50:514] <15> 0x0.0x1e9 0:14:4f:85:f2:b Application: sessionMonitor TcpOpen timeout to 192.168.100.1:7009
[syslog from 192.168.100.50:514] <15> 0x0.0x1e9 0:14:4f:85:f2:b Network: GetNextAltAuth 0 c0a86401 0
[syslog from 192.168.100.50:514] <15> 0x0.0x1fd 0:14:4f:85:f2:b Network: GetNewtBandwidth: status down speed 100000000 mode full

There’s one little thing that may catch your eye: Application: sessionMonitor TcpOpen timeout to 192.168.100.1:7009. TCPOpen? To the IP of the AuthSrvr? Port 7009? That’s oddly familiar! We had a mysterious UDP packet going to that port in the last post!

The Mysterious serverQ

To be honest, serverQ was a dead-end. I couldn’t find any references on what it means, how it should respond, and why that happens. But hey! Now we have the Ray trying to open a full-fledged TCP connection with us! I mean, it’s trying, and that’s progress! I’m calling this out explicitly because the previous post ended on a bit of a cliffhanger, and I hate those as much as you probably do. For now, serverQ remains unexplained, but it no longer blocks progress.

Moving on!

A Sun Ray walks into a bar…

Since the Ray wants to speak, let it speak! Let’s spin a TCP server listening on port 7009 and dump whatever we get there. Here’s a few bytes of output from the hacky TCP server:

DummyAuthSrvr: 696e666f526571204d54553d31353...

For the well-trained eye, that hex looks an awful lot like an ASCII blob. And guess what? It is! Here’s the ASCII version of it:

infoReq MTU=1500 _=1 barrierLevel=310 cause=insert clientRand=nrlifRNCnn4xe.F5Be/DA7gkwFF9cOXQOmxxzu8SM3e ddcconfig=1 event=insert firstServer=c0a86401 fw=MfgPkg_4.15_2006.07.20.16.57 hw=SunRayP8 id=00144f85f20b initState=1 namespace=IEEE802 pn=53622 realIP=c0a86432 sn=00144f85f20b startRes=1400x1050:1400x1050 state=disconnected tokenSeq=1 type=pseudo

And that’s A LOT! Holy guacamole!

Let’s try to break it up:

• infoReq: Possibly an operation identifier?
• MTU: The MTU we already provided to it.
• _: No idea. Maybe a placeholder, maybe something deprecated. Who knows!
• barrierLevel: Possibly the BarrierLevel we didn’t provide it. Something related to firmware download, but let’s leave that for another time.
• cause: Perhaps what made the Ray request that operation?
• clientRand: A Base64! Sweet! Given the entropy, it’s really random! Perhaps for some kind of negotiation? I think we will eventually figure that out.
• ddcconfig: No idea.
• event: The same value of cause? We will need some other packages to figure this one out as well.
• firstServer: That’s the server’s IP in hexadecimal. Nothing interesting so far.
• fw: A firmware version! Seems like this one is from 2006. Cute!
• hw: A hardware identifier! It is a SunRayP8. Duly noted.
• id: I don’t know, but this value is static across all messages it sent. (And resent.)
• initState: Not a clue!
• namespace: IEEE802 is a fancy name for Ethernet. Maybe this thing (or other revisions) could support other connection methods?
• pn: This value changes in a non-monotonic fashion. Packet number? We will figure out.
• realIP: This is the Ray’s IP in hexadecimal.
• sn: Also static across all packets. Possibly a serial number.
• startRes: It’s initial resolution?
• state: Disconnected? Maybe because authentication hasn’t completed?
• tokenSeq: I’m sure we will eventually figure out as well.
• type: pseudo is an interesting choice. I’m interested, but not a clue.

Then, it immediately sends another packet:

DummyAuthSrvr: 6b656570416c697665526571205f3...

Again, more ASCII:

keepAliveReq _=1 fw=MfgPkg_4.15_2006.07.20.16.57 hw=SunRayP8 namespace=IEEE802 pn=53622 sn=00144f85f20b state=disconnected

And we have more details now!

1. Seems like we have another operation: keepAliveReq
2. pn is the same across both messages. Possibly not a packet number!
3. Everything else stays the same.

Since we’re not answering after that point it just eventually times out and retries:

[syslog from 192.168.100.50:514] <15> 0x0.0x16ca1 0:14:4f:85:f2:b Network: TcpReceive: FIN received 805bdd78 1
[syslog from 192.168.100.50:514] <15> 0x0.0x16ca1 0:14:4f:85:f2:b Application: restartAuth
[syslog from 192.168.100.50:514] <15> 0x0.0x16ca1 0:14:4f:85:f2:b Network: TcpClose: 805bdd78
[syslog from 192.168.100.50:514] <15> 0x0.0x16ca1 0:14:4f:85:f2:b Application: TcpClose1 failed
[syslog from 192.168.100.50:514] <14> 0x0.0x16ca1 0:14:4f:85:f2:b Network: FreeTcb:Freeing Tcb before the connection is closed

It’s not rebooting, though. Only restarting the authentication phase. It may be worth noticing that whenever it retries, even without rebooting both clientRand and pn changes, but everything else stays the same. There’s one small detail, though! Here’s a new detail in syslog:

[syslog from 192.168.100.50:514] <13> 0x0.0x544 0:14:4f:85:f2:b USB: rdd: unable to contact device manager on 192.168.100.1:7011  retrying

It’s ALSO trying to establish a connection to port 7011! Looking at my packet dumps, it tried to open a TCP connection to that port. Let’s also see what happens if we let it! It seems to be something called a Device Manager, so let’s do the same we did with the AuthSrvr:

DummyDeviceManager: 636f6e6e6563740a
[syslog from 192.168.100.50:514] <15> 0x0.0x1f270 0:14:4f:85:f2:b Network: TcpClose: 8059c538
[syslog from 192.168.100.50:514] <15> 0x0.0x1f270 0:14:4f:85:f2:b Network: TcpReceive: FIN received 8059c538 8
[syslog from 192.168.100.50:514] <14> 0x0.0x1f2d8 0:14:4f:85:f2:b Network: TCBDaemon: packet receive error
[syslog from 192.168.100.50:514] <14> 0x0.0x1f33c 0:14:4f:85:f2:b Network: TCBDaemon: packet receive error
[syslog from 192.168.100.50:514] <14> 0x0.0x1f3a0 0:14:4f:85:f2:b Network: TCBDaemon: packet receive error
...

Yeah, it’s not happy. Again, an ASCII payload. This time it only says connect, followed by a newline.

Eventually it will close the connection and send another connect, while the AuthSrvr continues receiving keepAliveReq.

For now, that’s it. I’ll eventually rewrite the C stuff in Golang just for the sake of it, and keep it updated in a Git repository. This way we can isolate the AuthSrvr, Device Manager, and the hacky DHCP server in a single place.

I said I hate cliffhangers, and I meant it, but unfortunately the Sun Ray just gave me a really good one.

In the next post, we’ll respond to its requests and see what happens when we get it wrong.

The Ray is a thin client that connects to a remote Sun server. I already ordered a Sun Fire, but until it arrives, I decided to poke the little thing and see what makes it tick!

After powering on, and having a network cable connected, the Ray shows a little screen with its MAC address, the network cable speed, and if it’s reached a local server. Since the Sun Fire is not here, the last part just does not happen, so let’s begin by observing how it tries to reach the server.

DHCPDISCOVER

The Ray starts with a DHCPDISCOVER to try to reach its server, which is no big surprise:

0000   ff ff ff ff ff ff 00 14 4f 85 f2 0b 08 00 45 00   ........O.....E.
0010   01 48 00 28 40 00 ff 11 7a 7d 00 00 00 00 ff ff   .H.(@...z}......
0020   ff ff 00 44 00 43 01 34 00 00 01 01 06 00 ff 4e   ...D.C.4.......N
0030   c3 a7 04 7e 00 00 00 00 00 00 00 00 00 00 00 00   ...~............
0040   00 00 00 00 00 00 00 14 4f 85 f2 0b 00 00 00 00   ........O.......
0050   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0060   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0070   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0080   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0090   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00a0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00b0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00c0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00d0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00e0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00f0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0100   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0110   00 00 00 00 00 00 63 82 53 63 35 01 01 3c 0e 53   ......c.Sc5..<.S
0120   55 4e 57 2e 4e 65 77 54 2e 53 55 4e 57 3d 07 01   UNW.NewT.SUNW=..
0130   00 14 4f 85 f2 0b ff 00 00 00 00 00 00 00 00 00   ..O.............
0140   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0150   00 00 00 00 00 00                                 ......

DIX / Ethernet II

The first portion is a classic DIX Ethernet frame (or Ethernet II, if you’re still litigating DIX credit). From 0x00 to 0x05 we have the broadcast address ff:ff:ff:ff:ff:ff, the second one is the Ray’s address, 00:14:4f:85:f2:0b, 0x0C to 0x0D indicates an IPv4 packet (0x0800).

IPv4

The second portion, as expected, is the IPv4 header. Nothing new there as well, the only relevant information there is that the destination address is 255.255.255.255, and it’s a UDP packet; this basically covers bytes 0x0E to 0x21.

UDP

From 0x22 to 0x29 we can observe the UDP header. Interesting points are source port 68, and destination port 67.

BOOTP/DHCP fixed header + DHCP options

From 0x2A onwards we have the DHCPDISCOVER payload. Finally something interesting:

• Message Type is 0x01, meaning it’s a Boot Request,
• Hardware Type is 0x01, or “Ethernet”
• Hardware Address Length is 0x06
• Hops is set as 0x00
• There’s a Transaction ID (0xff4ec3a7)
• A “Seconds Elapsed” field, from the BOOTP era (which the Ray is actively using!). This is an interesting find: nowadays many implementations don’t do much with this field, and plenty of clients just leave it at zero, but this was important back then for a few different reasons! This field aided:
  • Relay agents decide whether to forward a request
  • Servers decide whether to prioritize a client that has been waiting longer
  • Debugging slow or failing DHCP negotiations
• BOOTP flags all zeroed; basically indicating the client is capable of receiving unicast replies
• Zeroed IP addresses
• The Ray’s MAC address, again
• A lot of padding for the Ray’s hardware address
• An empty server hostname
• An empty boot file name
• The Magic Cookie
• And FINALLY what I’m looking for:
  • Option 53, indicating the discovery packet
  • Option 60, containing the vendor class identifier
    • The value is SUNW.NewT.SUNW
  • Option 61 identifies the client using a hardware-type-prefixed identifier (Ethernet + MAC address), effectively restating the Ray’s MAC in a DHCP-standardized form. Again.
  • Option 255, indicating no more options follow
• And a lot of padding.

So what’s interesting so far?

The big oddity is what’s missing: Option 55 (Parameter Request List). Most DHCP clients use it to ask for things like DNS servers, routes, NTP, etc. The Sun Ray doesn’t ask, it just identifies itself (Vendor Class + Client ID) and waits to see what comes back. That suggests this client either runs with strong defaults, or expects a Sun-aware environment to “just know” what to provide.

Also: SUNW.NewT.SUNW looks like a Sun vendor-class marker; possibly a New Terminal generation marker rather than a generic Sun Ray family identifier.

This is the first part of the whole negotiation, and lots can already be observed and speculated!

Now, let’s make something that replies the Ray as it expects. I could just leverage dnsmasq for that, but let’s go the fun way(tm) and write a bit of C:

//
// Created by vitosartori on 12/13/25.
//

// Minimal DHCP responder (DISCOVER->OFFER, REQUEST->ACK) for RE labs.
// Build:  gcc -O2 -Wall -Wextra -o ray_dhcp ray_dhcp.c
// Run:    sudo ./ray_dhcp enp2s0 192.168.100.1 192.168.100.50 255.255.255.0 192.168.100.1

#define _GNU_SOURCE
#include <arpa/inet.h>
#include <errno.h>
#include <netinet/in.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/types.h>

enum {
    BOOTREQUEST = 1,
    BOOTREPLY   = 2,

    HTYPE_ETHERNET = 1,
    HLEN_ETHERNET  = 6,

    DHCP_COOKIE = 0x63825363,

    // DHCP Options

    OPT_PAD        = 0,
    OPT_SUBNETMASK = 1,
    OPT_ROUTER     = 3,
    OPT_REQ_IP     = 50,
    OPT_LEASETIME  = 51,
    OPT_MSGTYPE    = 53,
    OPT_SERVERID   = 54,
    OPT_VENDORCLASS= 60,
    OPT_CLIENTID   = 61,
    OPT_END        = 255,

    // DHCP Message Types

    DHCPDISCOVER = 1,
    DHCPOFFER    = 2,
    DHCPREQUEST  = 3,
    DHCPACK      = 5
};

#pragma pack(push, 1)
typedef struct {
    uint8_t  op;
    uint8_t  htype;
    uint8_t  hlen;
    uint8_t  hops;
    uint32_t xid;
    uint16_t secs;
    uint16_t flags;
    uint32_t ciaddr;
    uint32_t yiaddr;
    uint32_t siaddr;
    uint32_t giaddr;
    uint8_t  chaddr[16];
    uint8_t  sname[64];
    uint8_t  file[128];
    // Following is cookie + TLVs
} bootp_t;
#pragma pack(pop)

typedef struct {
    uint8_t  msgtype;       // Option 53
    uint8_t  has_msgtype;
    uint8_t  vendor[256];   // Option 60
    uint8_t  vendor_len;
    uint8_t  has_vendor;
    uint32_t req_ip;        // Option 50 (optional)
    uint8_t  has_req_ip;
} dhcp_opts_t;

static void die(const char *msg) {
    perror(msg);
    exit(1);
}

static int parse_ipv4(const char *s, uint32_t *out_nbo) {
    struct in_addr a;
    if (inet_pton(AF_INET, s, &a) != 1) return 0;
    *out_nbo = a.s_addr;
    return 1;
}

static void print_mac(const uint8_t mac[6], char *buf, const size_t buf_len) {
    snprintf(buf, buf_len, "%02x:%02x:%02x:%02x:%02x:%02x",
             mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
}

static int parse_dhcp_options(const uint8_t *p, const size_t n, dhcp_opts_t *o) {
    memset(o, 0, sizeof(*o));

    // First the cookie
    if (n < 4) return 0;
    uint32_t cookie;
    memcpy(&cookie, p, 4);
    if (ntohl(cookie) != DHCP_COOKIE) {
        fprintf(stderr, "drop: cookie mismatch (got %08x)\n", ntohl(cookie));
        return 0;
    }

    size_t i = 4;

    while (i < n) {
        const uint8_t code = p[i++];
        if (code == OPT_PAD) continue;
        if (code == OPT_END) break;
        if (i >= n) break;

        const uint8_t len = p[i++];
        if (i + len > n) break;
        const uint8_t *val = p + i;

        if (code == OPT_MSGTYPE && len == 1) {
            o->msgtype = val[0];
            o->has_msgtype = 1;
        } else if (code == OPT_VENDORCLASS && len > 0 && len <= 255) {
            memcpy(o->vendor, val, len);
            o->vendor[len] = 0;
            o->vendor_len = len;
            o->has_vendor = 1;
        } else if (code == OPT_REQ_IP && len == 4) {
            memcpy(&o->req_ip, val, 4);
            o->has_req_ip = 1;
        }

        i += len;
    }

    return 1;
}

static uint8_t *opt_put(uint8_t *w, const uint8_t code, const void *val, const uint8_t len) {
    *w++ = code;
    *w++ = len;
    memcpy(w, val, len);
    return w + len;
}

static size_t build_reply(uint8_t *out,
                          const size_t outcap,
                          const bootp_t *req,
                          const dhcp_opts_t *reqo,
                          const uint8_t dhcp_type,
                          const uint32_t server_ip,
                          const uint32_t yiaddr,
                          const uint32_t subnet_mask,
                          const uint32_t router_ip,
                          const uint32_t lease_seconds) {
    if (outcap < sizeof(bootp_t) + 4 + 64) return 0;

    bootp_t rep = {0};

    rep.op = BOOTREPLY;
    rep.htype = req->htype;
    rep.hlen = req->hlen;
    rep.hops = 0;
    rep.xid = req->xid;
    rep.secs = 0;
    rep.flags = req->flags; // Echo client flags

    rep.ciaddr = 0;
    rep.yiaddr = yiaddr; // Offered/acked IP
    rep.siaddr = server_ip; // the mothership hint
    rep.giaddr = 0;

    memcpy(rep.chaddr, req->chaddr, 16);

    memcpy(out, &rep, sizeof(rep));
    uint8_t *w = out + sizeof(rep);

    const uint32_t cookie = htonl(DHCP_COOKIE);
    memcpy(w, &cookie, 4);
    w += 4;

    // Options
    w = opt_put(w, OPT_MSGTYPE, &dhcp_type, 1);
    w = opt_put(w, OPT_SERVERID, &server_ip, 4);

    const uint32_t lt = htonl(lease_seconds);
    w = opt_put(w, OPT_LEASETIME, &lt, 4);

    w = opt_put(w, OPT_SUBNETMASK, &subnet_mask, 4);
    w = opt_put(w, OPT_ROUTER, &router_ip, 4);

    // Optional: echo vendor class back, may help with “vendor-aware feel”
    if (reqo->has_vendor && reqo->vendor_len > 0) {
        w = opt_put(w, OPT_VENDORCLASS, reqo->vendor, reqo->vendor_len);
    }

    *w++ = OPT_END;

    return (size_t) (w - out);
}

int main(const int argc, char **argv) {
    if (argc != 6) {
        fprintf(stderr,
            "Usage: %s <ifname> <server_ip> <offer_ip> <subnet_mask> <router_ip>\n"
            "Example: %s enp2s0 192.168.100.1 192.168.100.50 255.255.255.0 192.168.100.1\n",
            argv[0], argv[0]);
        return 2;
    }

    const char *ifname = argv[1];
    uint32_t server_ip, offer_ip, mask_ip, router_ip;

    if (!parse_ipv4(argv[2], &server_ip) ||
        !parse_ipv4(argv[3], &offer_ip)  ||
        !parse_ipv4(argv[4], &mask_ip)   ||
        !parse_ipv4(argv[5], &router_ip)) {
        fprintf(stderr, "Invalid IPv4 argument\n");
        return 2;
    }

    const int s = socket(AF_INET, SOCK_DGRAM, 0);
    if (s < 0) die("socket");

    const int yes = 1;
    if (setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes)) < 0) die("setsockopt REUSEADDR");
    if (setsockopt(s, SOL_SOCKET, SO_BROADCAST, &yes, sizeof(yes)) < 0) die("setsockopt BROADCAST");

    // Bind to port 67 on all addresses
    struct sockaddr_in bindaddr = {0};
    bindaddr.sin_family = AF_INET;
    bindaddr.sin_port = htons(67);
    bindaddr.sin_addr.s_addr = htonl(INADDR_ANY);

    if (bind(s, (struct sockaddr*)&bindaddr, sizeof(bindaddr)) < 0) {
        fprintf(stderr, "bind(:67) failed: %s\n", strerror(errno));
        fprintf(stderr, "Tip: run as root or grant cap_net_bind_service.\n");
        return 1;
    }

    // Pin to interface so we don't respond on the wrong NIC.
    if (setsockopt(s, SOL_SOCKET, SO_BINDTODEVICE, ifname, strlen(ifname) + 1) < 0) {
        fprintf(stderr, "SO_BINDTODEVICE(%s) failed: %s\n", ifname, strerror(errno));
        return 1;
    }

    fprintf(stderr, "Listening on %s UDP/67. server=%s offer=%s\n", ifname, argv[2], argv[3]);

    uint8_t inbuf[2048];
    uint8_t outbuf[2048];

    for (;;) {
        struct sockaddr_in src;
        socklen_t srclen = sizeof(src);
        const ssize_t n = recvfrom(s, inbuf, sizeof(inbuf), 0, (struct sockaddr*)&src, &srclen);
        if (n < 0) {
            if (errno == EINTR) continue;
            die("recvfrom");
        }
        fprintf(stderr, "got %zd bytes from %s:%d\n",
        n, inet_ntoa(src.sin_addr), ntohs(src.sin_port));
        fflush(stderr);
        enum { BOOTP_FIXED_LEN = 236 };

        if ((size_t)n < BOOTP_FIXED_LEN + 4) continue;
        const size_t opt_len = (size_t)n - BOOTP_FIXED_LEN;

        const bootp_t *req = (const bootp_t *)inbuf; // prolly OK for chaddr/xid/etc.
        if (req->op != BOOTREQUEST) { fprintf(stderr,"drop: not BOOTREQUEST\n"); continue; }
        if (req->htype != HTYPE_ETHERNET) { fprintf(stderr,"drop: htype=%u\n", req->htype); continue; }
        if (req->hlen != HLEN_ETHERNET) { fprintf(stderr,"drop: hlen=%u\n", req->hlen); continue; }

        // Options start at offset 236
        dhcp_opts_t o = {0};
        if (!parse_dhcp_options(inbuf + BOOTP_FIXED_LEN, opt_len, &o)) continue;
        if (!o.has_msgtype) {
            fprintf(stderr,"drop: no option 53\n");
            continue;
        }

        char macs[32];
        print_mac(req->chaddr, macs, sizeof(macs));

        fprintf(stderr, "opts[0..15]: ");
        for (int i = 0; i < 16 && (size_t)i < opt_len; i++) fprintf(stderr, "%02x ", inbuf[BOOTP_FIXED_LEN + i]);
        fprintf(stderr, "\n");

        uint8_t reply_type = 0;
        if (o.msgtype == DHCPDISCOVER) reply_type = DHCPOFFER;
        else if (o.msgtype == DHCPREQUEST) reply_type = DHCPACK;
        else {
            fprintf(stderr, "drop: Unknown msgtype 0x%02x\n", o.msgtype);
            continue;
        }

        // Honor Option 50 Requested IP if present.
        uint32_t yiaddr = offer_ip;
        if (o.has_req_ip) yiaddr = o.req_ip;

        size_t outlen = build_reply(outbuf, sizeof(outbuf), req, &o, reply_type,
                                    server_ip, yiaddr, mask_ip, router_ip, 3600);
        if (!outlen) continue;

        // Send to broadcast:68, as we noticed the client doesn't have an IP
        struct sockaddr_in dst = {0};
        dst.sin_family = AF_INET;
        dst.sin_port = htons(68);
        dst.sin_addr.s_addr = htonl(INADDR_BROADCAST);

        if (sendto(s, outbuf, outlen, 0, (struct sockaddr*)&dst, sizeof(dst)) < 0) {
            fprintf(stderr, "sendto failed: %s\n", strerror(errno));
            continue;
        }

        fprintf(stderr, "Sent %lu bytes\n", outlen);

        if (o.has_vendor) {
            fprintf(stderr, "%s xid=%08x -> %s (vendor=%s)\n",
                    macs, ntohl(req->xid),
                    (reply_type == DHCPOFFER) ? "OFFER" : "ACK",
                    (const char *)o.vendor);
        } else {
            fprintf(stderr, "%s xid=%08x -> %s\n",
                    macs, ntohl(req->xid),
                    (reply_type == DHCPOFFER) ? "OFFER" : "ACK");
        }
    }
}

This replies with what the Ray expects, offering it an IP leased from the Linux box. I only forgot to mention how things are wired here at my lab:

Sun Ray ↔ USB Ethernet ↔ Linux box (DHCP responder) ↔ rest of LAN

So the magic is happening on interface enx00e04c3f6690, I’ll give myself an IP 192.168.100.1, and give 192.168.100.50 to the Ray. First, let’s get an IP on that interface:

sudo ip addr add 192.168.100.1/24 dev enx00e04c3f6690
sudo ip link set enx00e04c3f6690 up

And finally compile and run (as sudo). The result is positive! The Ray updated the UI indicating it has communicated with the server! Then, a wild UDP packet appears:

0000   ff ff ff ff ff ff 00 14 4f 85 f2 0b 08 00 45 00   ........O.....E.
0010   00 27 00 06 40 00 40 11 15 e6 c0 a8 64 32 ff ff   .'..@.@.....d2..
0020   ff ff 1b 61 1b 61 00 13 00 00 73 65 72 76 65 72   ...a.a....server
0030   51 3d 31 0a 00 00 00 00 00 00 00 00               Q=1.........

Ignoring the framing information and other headers we get 11 payload bytes plus some padding. The payload is:

0000 73 65 72 76 65 72 51 3d 31 0a 00                   serverQ=1..

That’s… Literally serverQ=1\n\0. But there’s more! Here’s what the packet gives:

1. Both source and destination ports are 7009
2. The source IP is the IP we gave to the Ray! (I omitted a few DHCPACK and DHCPINFORM packets)
3. The destination IP is STILL the broadcast address

This might be Sun Ray’s appliance protocol territory (UDP/7009 shows up in Sun Ray land), but I’m treating it as an unknown for now. The protocol is proprietary and not documented, but let’s wrap up for now and just be happy for the little, still important, progress.

Next: what happens if we answer serverQ=1?

1. I despise LLMs. I don’t think they should replace or even try to replace people.
2. I am a big fan of well-written RFCs and specifications.

The first point is not a surprise; being a part of this industry before the advent of LLMs gave me enough resources to be able to compare how things used to work, and how they work now. The field is clearly divided, though, but I see it as as binary as ever: either engineers love it, or despise it. I feel that people are still in awe of the possibility to have something else capable of, for instance, to write unit tests for a feature, but they fail to understand that the one who wrote the implementation must also be the one who writes the tests, as albeit manual, the process is still important, since it forces one to use what they implemented. In this step, several extremely relevant points can still be observed:

• The ergonomics of the API. Is it cohesive? How hard is it for someone to use it?
• Its documentation. Is it documented at all? Will someone that has absolutely no context of the whole be able to still understand it and use without friction?
• Is the domain properly segregated? Is the API requiring more than it should?
• Are names descriptive enough? Do they make sense when used with other components?

Those are just a glimpse of what writing both tests and documentation can provide to the one who is implementing it, but the list goes on. Now, delegating the job to a tool whose reasoning is not even visible blinds implementors from those nuances.

I also see how this is deeply affecting those who are entering the field. For instance, when mentoring anyone, the most important thing I keep repeating on and on is: Read and understand error messages, and do not ignore warnings. I see often people just acknowledging message boxes on UIs, hitting OK without even reading it, and then asking themselves “Why isn’t this working?”; the same applies to developers. One must be able to understand what an error message means, and how to read a stack trace. Now, imagine someone in university, or in an internship blindly writing code, and asking an LLM to interpret any errors and fix them. What will this person be able to accomplish without it?

But this is not only affecting newcomers. I often see interesting cases coming from people who are part of the field for a while:

• “Your code is wrong, see what said: ..."; meaning "I haven't read your code, I have no idea what it does, and I'm acting as a proxy to an LLM".
• “I think we should use , see my chat transcription": meaning "I have no domain nor property, but you should use what the machine told me"
• I opened a Pull Request, but I have no idea what it does, as seen in the infamous Mesa Issue.
• “There’s something wrong, see what I got regarding this issue using ".

Sometimes I feel like people can’t even read and maintain their own code, let alone code generated by an LLM, copy-and-pasted into the codebase; I don’t need to say that’s dangerous not only on a security point-of-view, but on a business continuation one.

This ties directly into my second point about well-written specifications, which exist precisely to prevent this kind of ambiguity and dependency on individual authors.

The second point is also not a surprise. I’m a big fan of LaTeX, and a bigger fan of IETF and everything they achieved. I work with the premise that every RFC and specification must be written in a fashion that allows anyone reading it to fully understand what it is about, and even more important: to be able to implement it no matter the language or technology they want to use. This way, even if I leave the company I’m working on, the documentation is there, all the design decisions are still there, and are both navigable and clear to the point where they don’t need me, the original author, to be able to update or reimplement it: everything is within a single documentation unit.

Now, where can LLMs help us here? Not designing documentation, not writing code, but reviewing the documents. It turns out that LLMs are great reviewers, since they can keep a lot in their context, and are also able to comprehend documents in a matter of seconds. However, there’s an important pitfall in leveraging hosted LLMs (like ChatGPT or Claude): one must be absolutely sure of their data policies, and even more important, that the company will not use anything you provided to train future versions. So that is how I use LLMs: for reviewing what I write.

Given their ability to quickly cross-examine a whole document in seconds, I mostly use it as part of my writing process. First, I draft the document, put everything I need, and make the first review. When I’m happy with the result, it’s time to get a critic. I allow the LLM to read it, and ask for feedback on sections that need development, for points that are not clear or ambiguous, for contradictions and typos. And they are excellent at this. They can pinpoint which section makes a contradictory statement against another, potentially incomplete definitions, examples or small details that are present in an ABNF section, but explained incorrectly in the body of text, along other small nits and inconsistencies that can truly elevate the level of the document. But do notice how it didn’t replace the creation process, it is not replacing the brain, only augmenting it to the point where one would forget that a small sentence ten sections earlier didn’t match what’s written in the end of the document.

That’s what people should be using LLMs for. For augmenting our abilities. Not replacing them. Renata Martins wrote a few months ago a post with a provocative title: LLMs are the Lobotomy of the 21st Century, and time and time again I fear that it may be true.

This post diverged from what I’m used to writing, but I had this on my head for so long, that I thought it was worth putting out.

For now, let’s continue using our brains as we should, and avoid atrophying them; if we use LLMs thoughtfully, they can strengthen (rather than weaken) our engineering practices.

According to the git-commit manpage, it’s “a good idea to begin the commit message with a single short (no more than 50 characters)”. This commit message lands at 71 characters. Personally, I would have put something like “Fix array parsing when input has multiple spaces”. Smaller, and also complies with recommendations from the git documentation.

The type of a commit here is only really useful for automated tools, usually the type can be inferred from the summary of the commit. As such, I think the type would be better off as a field in footer of commit, under a key such as Commit-Type or Type.

But why make an exception for BREAKING CHANGE ? Why not have it as Breaking-Change ? It’s not like it adds or remove any character, all that would change is it would stay consistent with other keys.

That makes so much sense. And folks, we’re just a --trailer away from making this work as it should.

Now for the facts: I do use conventional commits, as it is — as Sarah said — useful for automated tools, and every time I make a commit message larger than it should be, I do feel bad. However, adding --trailer for each commit and for each property would be cumbersome. So why not make a wrapper for git commit to handle that?

The Idea

What if we could do something like git commit --chore --breaking -m "Message here" and get those fancy trailers? Of course, we have some commits like chore(some-component): Some description, so let’s also allow --chore to accept an optional parameter. I’m low on imagination today, so let’s call this “Pretty Commit”, and alias it as pc. Here are some possibilities:

git pc --chore component -m "Commit message"
# or
git pc --spec -m "Commit message"
# or even
git pc --feat "Tokenizer" --breaking -m "Allow tokenizer to do a backflip"

And the result from the last commit:

commit 4decde3d186a834a092676d3143fae6f7a54945a (HEAD -> master)
Author: Vito Sartori <hey@vito.io>
Date:   Mon Jul 7 10:50:22 2025 -0300

    Allow tokenizer to do a backflip

    Type: feat
    Scope: Tokenizer
    Breaking: true

All parseable! Let’s, for instance, get the meta from this commit:

 λ | git log -1 --pretty=format:"%B" | git interpret-trailers --parse
Type: feat
Scope: Tokenizer
Breaking: true

That’s so much easier to parse than extracting stuff from the commit message alone. Git plumbing has everything everyone ever needed.

I’ll try using this approach and also use it as a reason to bump heyvito/semver-releaser.

Can I use it?

Of course! Git is composable, meaning you just need to have git-pc in your PATH. Then git pc is immediately available. The source is available at github.com/heyvito/git-pc, and is a (very) simple bash script — as far as ‘simple’ and ‘bash’ can go together.

Contributions are welcome!

What else?

If you didn’t, read Sarah’s post. It’s full of insights and was what made me do this. Also, Tyler Cipriani posted about Git Notes, which is another feature that goes unnoticed.

And that’s it for today. Let’s see if we can push this forward!

Now it’s been a while since The Browser Company released a compelling update that’s not just a bump to the internal Chromium version, and filling their release notes with irrelevant content. Of course, they are focusing on their AI-based browser Dia, but I feel betrayed, considering all the promises they made.

There’s one thing that kept me on Arc, though: its sidebar. The sidebar was a great concept, and I could simply drag-and-drop tabs to read later, and it would just sync across my computers. That was great — until it became a mess. My sidebar is long. I mean really long. And searching/exploring it became hard. Adding all that up, I felt it was time to switch back to Firefox — and switch back I did.

But, what about my sidebar? Well, Arc does not offer a way to export the sidebar, but luckily, it keeps it in a JSON file in its Application Support directory. So a quick Ruby script later, all my links were exported. The point then was: where to place those items?.

Fixing my Bookmarks Issue

The first place I looked was raindrop.io, which I used for a while after it launched. Raindrop is great — really — but it has one small issue that I can’t stand: It does not permanently store YouTube videos — which makes sense, I think.

So I started working on my own solution to that: Rio (Retain Information Onshore). Rio takes a URL or file from my system, processes it, and permanently stores it, be it a web page, PDF, docx, or video. With a little help from Llama/Mistral, PDFs are correctly summarized and have their title extracted, so everything stays organized.

I have plans to also add a Solr integration for full-text search, but that will have to wait longer.

This is somewhat a big project, and may be interesting to other people who like to self-host stuff, so eventually it may become open-source. Who knows :)

So, so long Arc. It was good while you lasted. Let’s rock Firefox Nightly and keep the web balanced.