Lobsters

I guess everyone should know SIMD but I would argue that not everyone needs to know SIMD. Most programmers work on much higher levels of abstractions where this is not only not needed but would also constitute as premature optimization.

Feel like Zig make this more pleasant to work with than some other languages.

Sometimes I contemplate using SIMD but then I wonder:

• Is there some restriction on the buffer alignment?
• How to build my software once and run it on heterogenous hardware? CPUID dispatch does not seem straightforward.
• I'd often want to bench/test the SIMD implementation against the baseline implementation but it means they need to coexist at runtime.

Well, picking the baseline target at compile time may not be such a bad idea after all, it makes writing SIMD straightforward.

Title reminds me of What every computer scientist should know about floating point arithemetic, and What every programmer should know about memory.

Tangential to the article, but I think it's so cool: if you find yourself forced to use pure Python, but wishing you could also do fast data processing with SIMD, you'll love this innovative technique by @retr0id.

The summary with spoilers is that SIMD is used in CPython for integer arithmetic, so if you pack your data into big integers, it will use SIMD under the hood for fast processing.

One thing I really don't understand in the example is why it's using u32 for each lane when u8 is enough precision and range for all the steps? @mitchellh?

Input sizes in relation to SIMD vector sizes remains a tricky issue though. In particular, SIMD isn't really worth the trouble unless your input sizes are at least one multiple of your vector size. The smallest vector size today (that I know of at least) is 128 bits/16 bytes.

The reason for this is that for inputs up to 16 bytes you can instead read/process them as integers of different sizes (128 bits, 64 bits, etc), i.e something like this:

while size(input) >= 16 {
  value = read(input, bytes: 16) as Int128

  ... do something with the value ...
}

while size(input) >= 8 {
  value = read(input, bytes: 8) as Int64

  ... do something with the value ...
}

... repeat ...

This is for example how Inko's equality for strings and byte arrays is implemented (see here), though Inko certainly isn't unique in this regard.

Even if your input size is large enough to fit into a vector, I think that due to the setup cost (i.e. loading data in and out of vectors) of SIMD you realistically need at least two vector's worth of input. So if your target vector size is 128 bits, ideally you have at least 256 bits of input to process.

Apart from that, I think the issues that hold SIMD back today are:

1. The mess that is different CPUs supporting different vector sizes, or newer CPUs not supporting something, or it causing your CPU to overheat
2. Compile-time checks for different vector sizes being a pain as a result, while runtime checks are a pain because you have to compile the same code a bunch of times (something generally not well supported and thus requiring lots of manual plumbing), in addition to runtime checks being potentially more expensive than the work to perform, depending on where the checks take place
3. SIMD instruction documentation being the equivalent of "haha go screw yourself", the names alone should be enough to send those who came up with them to jail
4. A lot of SIMD algorithms are in fact technically unsound in that they often read beyond buffer boundaries, something that mostly works by coincidence but is in fact very much unsound/relying on undefined behavior. Translating those algorithms to something that is sound is often a challenge

I collected some more findings on this for Inko in this issue, with my conclusion being that at this stage I'm just not sure how to make it work without ripping my hairs out, though part of that is due to how Inko itself works and not just because of SIMD.

I don't know how effective it is, but Julia's @simd macro is a fantastic interface. Just write a normal loop and put it in front!

Great article and something I was curious about when exploring the ghostty codebase. Also while exploring the foot codebase I noticed they use PGOs for compiler-optimizations. It looks like zig doesn’t support PGOs at this point but I’d be curious if ghostty could get some gains out of employing a similar optimization strategy on top of simd

Neat article, thanks! One question:

Could this:

        if (@reduce(.And, greater_than_threshold)) continue;
        const mask: std.meta.Int(.unsigned, lanes) = @bitCast(greater_than_threshold);
        end += @ctz(~mask);
        break;

just be simplified to:

        if (@reduce(.And, greater_than_threshold) == false) break;

which then lets the final loop grab the index of the interesting item. (apologies if that's not valid Zig but you know what I mean)

Or is that clever ctz/bitmask so much faster than a for loop going over, at most, <lanes> items that it's definitely worth it?

The reason I ask is that for the purposes of this article, trying to demystify simd, it would make things much simpler.