muki.gg

Where are the old posts?

2025-12-22T00:00:00Z

These old posts were too much of a pain to port over to the new blogging infrastructure. I also don't think they are necessarily representative of the things I want to talk about and the depth I want to go into. This blog now has a dedicated RSS feed available here.

Slab Allocating in a Web Browser

2025-12-25T00:00:00Z

Introduction #

If you didn't know from the index page, I spend my days working as an engineer on Lightpanda, a headless web browser that is written in Zig. We've recently been working on a rework of our DOM layer on a branch aptly named zigdom. This will be about a recent optimization that took place on the zigdom branch, focused on improvements of allocator behavior.

Too Many Allocations #

One of the new additions with this branch was the Factory, which handles the creation of objects shared between Zig and the Javascript running from the web page. This Factory creates objects and their prototypes and wires them up together correctly, allowing for the object oriented nature of Javascript to correctly function.

pub fn createT(self: *Factory, comptime T: type) !*T {
  // Chooses a pool based on size and alignment of T.
  // Allocates in that pool and returns a ptr to T.
}

pub fn eventTarget(self: *Factory, child: anytype) !*@TypeOf(child) {
    const child_ptr = try self.createT(@TypeOf(child));
    child_ptr.* = child;

    const et = try self.createT(EventTarget);
    child_ptr._proto = et;
    et.* = .{ ._type = unionInit(EventTarget.Type, child_ptr) };
    return child_ptr;
}

pub fn node(self: *Factory, child: anytype) !*@TypeOf(child) {
    // ...
    child_ptr._proto = try self.eventTarget(Node{
        ._proto = undefined,
        ._type = unionInit(Node.Type, child_ptr),
    });
    // ...
}

pub fn element(self: *Factory, child: anytype) !*@TypeOf(child) {
    // ...
    child_ptr._proto = try self.node(Element{
        ._proto = undefined,
        ._type = unionInit(Element.Type, child_ptr),
    });
    // ...
}

pub fn htmlElement(self: *Factory, child: anytype) !*@TypeOf(child) {
    // ...
    const html = try self.element(Element.Html{
        ._proto = undefined,
        ._type = unionInit(Element.Html.Type, child),
    });
    // ...
}

Imagine there is a HTMLDivElement being created. The Factory will call into the methods above where each one will allocate one of the structures required in the prototype chain, resulting in a total of 5 allocations.

The initial approach used a list of MemoryPools, each one corresponding to a unique size and alignment pair. This minimized the impact of internal memory fragmentation as each item takes up only the space it needs (along with the padding required for alignment). This method however comes with a few tradeoffs.

Each object in the prototype chain is allocated in a different memory pool.

The result of each of these allocations residing in a seperate pool is significantly worse memory locality. Additionally, whenever we end up wanting to free one of these, we must iterate through all of the prototypes, marking each of those slots as free in whatever pool it belongs to. When creating and working with tens of thousands of objects created through the Factory, memory locality tends to become more important.

_size_8_8: MemoryPoolAligned([8]u8, .@"8"),
_size_16_8: MemoryPoolAligned([16]u8, .@"8"),
_size_24_8: MemoryPoolAligned([24]u8, .@"8"),
_size_32_8: MemoryPoolAligned([32]u8, .@"8"),
_size_32_16: MemoryPoolAligned([32]u8, .@"16"),
_size_40_8: MemoryPoolAligned([40]u8, .@"8"),
_size_48_16: MemoryPoolAligned([48]u8, .@"16"),
_size_56_8: MemoryPoolAligned([56]u8, .@"8"),
_size_64_16: MemoryPoolAligned([64]u8, .@"16"),
_size_80_16: MemoryPoolAligned([80]u8, .@"16"),
_size_88_8: MemoryPoolAligned([88]u8, .@"8"),
_size_96_16: MemoryPoolAligned([96]u8, .@"16"),
_size_128_8: MemoryPoolAligned([128]u8, .@"8"),
_size_144_8: MemoryPoolAligned([144]u8, .@"8"),
_size_152_8: MemoryPoolAligned([152]u8, .@"8"),
_size_160_8: MemoryPoolAligned([160]u8, .@"8"),
_size_184_8: MemoryPoolAligned([184]u8, .@"8"),
_size_232_8: MemoryPoolAligned([232]u8, .@"8"),
_size_648_8: MemoryPoolAligned([648]u8, .@"8"),

Another issue with the initial Factory approach was the manual management of the Pools, where one must create a new MemoryPool and update some of these core Factory methods whenever an object of unique size and alignment was implemented.

The Slab Allocator #

With the problems of the initial implementation highlighted, the solution exists in the form of a Slab Allocator. This allocator internally manages a list of Slabs, where each Slab represents a unique size and alignment pair. This echoes the strategy used previously by the MemoryPools but eliminates the need to have a manually managed list of Pools.

The object and all of its prototypes are allocated together with good locality.

The Slab Allocator also came with a variety of changes to the Factory itself, mostly in the form of amoritized allocations for objects. We can now allocate one contiguous slice of memory instead of N slices in different Pools, reducing the number of allocations required compared to the previous Pool approach.

The Slab Allocator also amoritizes allocations for Slabs that are considered hot, using an exponentially growing chunk count (with a limit at 128 slots). This allows us to balance fragmentation and the minimization of allocations, ensuring that hot slabs allocate less often and cold slabs use less memory.

Below are two benchmarks of Factory memory usage, one with the old Pool Factory and one with the new Slab Factory.

=== Factory Memory Statistics ===
Total Allocations: 13527
Total Allocated: 576306 bytes (0.55 MB)

Per-Pool Breakdown:
              Name |        Bytes |  % Total
  -----------------+--------------+---------
         _size_8_8 |          196 |     0.0%
        _size_16_8 |        77165 |    13.4%
        _size_24_8 |          280 |     0.0%
        _size_32_8 |        68786 |    11.9%
       _size_32_16 |         8578 |     1.5%
        _size_40_8 |       118052 |    20.5%
       _size_48_16 |       145357 |    25.2%
        _size_56_8 |       145357 |    25.2%
       _size_64_16 |         1272 |     0.2%
       _size_80_16 |          176 |     0.0%
        _size_88_8 |            0 |     0.0%
       _size_96_16 |         3275 |     0.6%
       _size_128_8 |         3864 |     0.7%
       _size_144_8 |            0 |     0.0%
       _size_152_8 |            0 |     0.0%
       _size_160_8 |            0 |     0.0%
       _size_184_8 |            0 |     0.0%
       _size_232_8 |          392 |     0.1%
       _size_648_8 |         3556 |     0.6%

=== Slab Allocator Statistics ===
Overall Memory:
  Total allocated: 587784 bytes (0.56 MB)
  In use:          549656 bytes (0.52 MB)
  Free:            38128 bytes (0.04 MB)

Overall Structure:
  Slab Count:    27
  Total chunks:    153
  Total slots:     7997
  Slots in use:    7691
  Slots free:      306

Overall Efficiency:
  Utilization:     93.5%
  Fragmentation:   6.5%

Per-Slab Breakdown:
   Size | Algn | Chunks |  Slots |  InUse |      Bytes |  Util%
  ------+------+--------+--------+--------+------------+-------
    272 |    3 |      1 |      1 |      1 |        272 | 100.0%
     48 |    3 |      2 |      3 |      3 |        144 | 100.0%
     32 |    3 |      1 |      1 |      1 |         32 | 100.0%
    408 |    3 |      1 |      1 |      1 |        408 | 100.0%
     72 |    3 |      1 |      1 |      1 |         72 | 100.0%
      8 |    3 |      4 |     15 |     10 |        120 |  66.7%
    128 |    3 |      1 |      1 |      1 |        128 | 100.0%
     16 |    3 |     20 |   1791 |   1780 |      28656 |  99.4%
    144 |    3 |     17 |   1407 |   1300 |     202608 |  92.4%
     48 |    4 |     31 |   3199 |   3145 |     153552 |  98.3%
    112 |    3 |     13 |    895 |    887 |     100240 |  99.1%
    264 |    3 |      4 |     15 |     14 |       3960 |  93.3%
     40 |    3 |      6 |     63 |     57 |       2520 |  90.5%
     80 |    4 |      6 |     63 |     52 |       5040 |  82.5%
    152 |    3 |      2 |      3 |      3 |        456 | 100.0%
    168 |    3 |      2 |      3 |      2 |        504 |  66.7%
    184 |    3 |      6 |     63 |     51 |      11592 |  81.0%
     96 |    3 |      4 |     15 |      8 |       1440 |  53.3%
    176 |    3 |      2 |      3 |      3 |        528 | 100.0%
     24 |    3 |      4 |     15 |     14 |        360 |  93.3%
     64 |    4 |      4 |     15 |     13 |        960 |  86.7%
    784 |    3 |      2 |      3 |      2 |       2352 |  66.7%
    176 |    4 |      9 |    383 |    308 |      67408 |  80.4%
    160 |    4 |      1 |      1 |      1 |        160 | 100.0%

These were both run on a Reddit page we use for end to end testing and clearly highlights the difference between the two implementations.

The old Pool Factory (left) makes a total of 13527 allocations with a total allocated size of 0.55 MB. There are pools that are hot and cold, where the (size 48, alignment 16) and (size 56, alignment 8) make up 50% of the total allocations while others like (size 144, alignment 8) make up 0%. Allocating for each item has an advantage however, it ends up using less memory overall because there is never unused space as N is always divisible by 1.

The new Slab Factory (right) makes a total of 153 allocations^[1] with a total allocated size of 0.56 MB. This Factory also has hot and cold slabs but they retain a high utilization due to the capped exponential growth of the chunk sizes. The Slab Factory will always have some form of fragmentation because N is not always divisible by the current chunk size.

This shows a 98.86% decrease in total memory allocations while paying only paying a cost of ~2% increased memory usage. There are additional benefits such as improved memory locality due to all of an objects prototypes being colocated, freeing entire object slots at once, and better observability as to how our Factory is behaving with different web pages.

[1]: The Slab Factory allocates in chunks instead of individual items so total chunks == total allocations.

Post Mortem #

The PRs (1, 2) have been merged into the zigdom branch for a couple of weeks now with no real issues. I've personally added additional types to the Factory recently when working with various Events and their prototypes and felt generally positive.

Lightpanda Now Supports robots.txt

2026-02-20T00:00:00Z

Web Bot Auth Support in Lightpanda

2026-03-20T00:00:00Z

Getting Data from the Farmtek Polaris

2026-04-28T00:00:00Z

Farmtek Polaris? #

If you don't know what the Farmtek Polaris is, their own website describes it as "a full featured, multi-event timer for timing of equestrian events including barrel racing, pole bending, roping, team penning, bull riding, cutting, show jumping and more. The Polaris timer is also used in a wide variety of non-equestrian events including lap timing, autocross, drag racing, track and field events, speed training, and dog agility trials."

This is what the FarmTek Polaris looks like. (farmtek.net)

My personal experience with this device is as a motorsport timer so I will be documenting my experience capturing the timing data from it in that context.

Connecting to the Polaris #

Before we get data out of the Polaris, we need to figure out how to connect to the device. These devices are sometimes used to directly drive a motorsport lap printer. When someone completes a lap, these devices communicate with the printer and you will get a little slip with the time of your latest lap.

We are more interested in ingesting the times digitally than printing out times. We can look into what the Polaris sends over the "Output" port. Since the Farmtek company sells these "Computer Interface Cables", we know that the data we need is coming from here.

The 3.5mm Output Jack Pinout

The "Output" port on the Polaris is a 3.5mm jack with a special pinout that features a couple different things. It is a three conductor setup where the base is GND, the ring is UART TXD and the tip is some Scoreboard output. The one we are interested is the UART pin as it is how these devices communicate with the printers.

Seeing the Data #

The easiest thing to do now is to just connect our GND and UART TXD pins to an oscilloscope and see if we can capture the frames that the Polaris transmits.

If you are following along, my Polaris device is in "Autocross" mode with the interface type set to "Tag Heuer".

I will start at a baud rate of 9600 and an idle state of high to see if we get any data on start and stop triggers.

We can see that we are getting some obviously structured binary data so this baud rate might be correct. The easiest thing to try here is just to switch to an idle state of low before we start messing with baud rates.

This looks good now. These seem to be ASCII characters with 54 being 'T' and 20 being ' ' (space). We can also see that the UART is using a TTL of 5V.

Ingesting the Data #

So now that we know that our data is ASCII, the baudrate is 9600, it is idle low, and that the TTL is 5V, we can begin the process of ingesting the data digitally. I personally used an FT232RL adapter that I had laying around but pretty much any UART to USB adapter should work given that it can invert RXD (or set it to idle low) and it supports 5V TTL. The FT232RL can do this if you use ft232r_prog on Linux or the official program for Windows.

Make sure the Polaris UART TXD on the 3.5mm jack is connected to the UART RXD on your adapter.

Once you have the Polaris hooked up to your UART to USB adapter and the USB adapter to your computer, you can use much any serial program to dump the data that it is outputting. I personally like picocom. Once connected and you have your serial port open, you can press the trigger a few times and get some data.

T           01 00:17:11.509000
T           02 00:18:37.216000
...

Here's a sample of what I got from a start and a stop.

Parsing a Timing Frame #

Now that we have the data reliably arriving, we just need to parse these timing frames that we get over serial. The format seems pretty simple: starts with a 'T', has a few spaces, has a status indicator and then a timestamp.

I'm going to write a simple parser in Rust but this is possible in any language and should be simple to follow.

pub struct PolarisFrame {
    /// This is the trigger value.
    trigger: u8,
    /// This is the timestamp in milliseconds.
    timestamp: u64,
}

#[derive(Default)]
pub struct PolarisParser {
    // We have this a String because we know that our data is always ASCII.
    buffer: String,
}

impl PolarisParser {
    // We feed bytes into the parser and it can return an arbitrary number of parsed frames.
    pub fn feed(&mut self, bytes: &[u8]) -> Vec<PolarisFrame> {
        self.buffer.push_str(&String::from_utf8_lossy(bytes));

        let mut frames = Vec::new();

        // We find any new line characters.
        while let Some(pos) = self.buffer.find(['\r', '\n']) {
            let total_line = &self.buffer[..=pos];
            let trimmed_line = total_line.trim();

            if !trimmed_line.is_empty()
                && let Some(frame) = Self::parse_frame(trimmed_line)
            {
                frames.push(frame);
            }
            
            self.buffer.drain(..=pos);
        }
        
        frames
    }

    // This parses a single frame of data from the Polaris.
    fn parse_frame(line: &str) -> Option<PolarisFrame> {
        let mut parts = line.split_whitespace();

        let tag = parts.next()?;
        // This ensures that we are getting a proper frame.
        // If it doesn't start with 'T' then we can assume we got garbage.
        if tag != "T" {
            return None;
        }

        let trigger_str = parts.next()?;
        let trigger: u8 = trigger_str.parse().ok()?;

        // We only support 01 or 02 as the trigger which becomes 1..=2.
        if !(1..=2).contains(&trigger) {
            return None;
        }

        let time_str = parts.next()?;

        // HH:MM:SS.mmm000
        let (hms, ms_part) = time_str.split_once('.')?;

        let mut hms_parts = hms.split(':');
        let hours: u64 = hms_parts.next()?.parse().ok()?;
        let minutes: u64 = hms_parts.next()?.parse().ok()?;
        let seconds: u64 = hms_parts.next()?.parse().ok()?;

        let millis: u64 = ms_part.get(..3)?.parse().ok()?;

        let timestamp = millis + seconds * 1000 + minutes * 60 * 1000 + hours * 60 * 60 * 1000;

        Some(PolarisFrame { trigger, timestamp })
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_parse_start() {
        let mut parser = PolarisParser::default();
        let frames = parser.feed("T           01 00:17:11.509000\r".as_bytes());
        assert_eq!(frames.len(), 1);
        let frame = &frames[0];
        assert_eq!(frame.trigger, 1);
        assert_eq!(frame.timestamp, 17 * 60 * 1000 + 11 * 1000 + 509);
    }
    
    #[test]
    fn test_parse_stop() {
        let mut parser = PolarisParser::default();
        let frames = parser.feed("T           02 00:18:37.216000\r".as_bytes());
        assert_eq!(frames.len(), 1);
        let frame = &frames[0];
        assert_eq!(frame.trigger, 2);
        assert_eq!(frame.timestamp, 18 * 60 * 1000 + 37 * 1000 + 216);
    }
    
}

There's a Rust Playground available here.

Summary #

The Polaris in Autocross mode with interface type "Tag Heuer". The "Output" port uses a 3 pin 3.5mm jack port. It uses the base as ground and the ring as the UART pin. It uses 9600 baud and idle low and can be ingested using commodity UART to USB converters. It outputs a simple ASCII format that can be parsed, involving a 'T', a few spaces, trigger status, and timestamp.