Welcome to the first feature article on our site. We’re going to cover an ongoing problem with x86 emulation that affects every application that we emulate. This comes down to a single over-arching term that has wide-reaching ramifications; Emulating the x86 Total Store Ordering memory model (x86-TSO).

The problems with emulating this memory model on the weak ordering memory model that ARM defines is multi-faceted and covers multiple issues. We’re going to go over all the problems that we can encounter and the ways we solve (or in some cases can’t solve) in this article. Get yourself a snack and a warm drink to enjoy, this is going to be a long one.

• What exactly is x86-TSO?
• The humble beginnings of ARMv8.0-a
• I thought accessing memory was the easy bit?
• Oh no, what are these atomic instructions?
• What do you mean split-lock is mandatory?
• Wait, uncached memory needs to work?
• Looking towards a brighter future

What exactly is x86-TSO?

Before diving in to how we work around the x86 memory model problem, we need to first discuss exactly what it is. A memory model is a set of rules for how memory accesses in a system behave in relation to each other. The rules will dictate how loads and stores interact in a single-threaded or a multi-threaded environment. There’s a handful of popular memory memory models implemented in various forms of hardware, but the two we care about today is ARM’s relaxed (or weak) consistency model, and the x86 variant of Total-Store-Ordering consistency model. These two models are basically the two extremes of the spectrum; where ARM is the most relaxed, allowing significant hardware optimizations; and x86 is the most strict, enforcing a very strong coherency model that doesn’t allow a lot of room for optimization. One thing to be careful about when discussing memory models is the difference between consistency and atomicity. While these are related, they are not the same nor guaranteed in all cases.

The best way to explain how the differences in memory models work is to start with how x86 handles this. With TSO being very strict in how it operates, the programmer can assume that when a memory store occurs, that this will be coherently visible to all other processors in the system. This additionally means that when a memory load occurs, all stores before it “logically” will have been completed, or at least visible. This matches programmer expectations, you write to memory, it becomes visible as at the point of writing, as this is intuitive to think about when programming. The stores are effectively ordering the visibility of the loads, thus the name of the model. There’s a bit of nuance with how this operates but isn’t strictly necessary to understand.

The weak memory model that ARM has is a bit less intuitive about how it operates. By default the regular memory loads and stores that ARM uses aren’t strictly coherent across processors in your system, allowing the CPU to operate more efficiently most of the time. When a store instruction executes, that piece of memory (the cacheline) isn’t immediately visible to other processors in the system. Saving on precious power and efficiency because it’s expensive in hardware to invalidate other core’s cachelines, or allow them to snoop another processor’s caches. Relatedly if a processor is loading data from memory that another processor has written to, it’s not guaranteed that this load will even see this updated memory. This sounds like it would cause some significant problems in a multi-threaded application right? Older versions of ARM (ARMv7 and older) used a memory barrier instruction to ensure ordering, which had significant performance implications.

To get around this limitation of consistency, ARM also introduced load-acquire, and store-release memory instructions. In C++ parlance this maps to std::atomic’s memory_order_acquire and memory_order_release definitions respectively. In ARM’s terminology, these instructions also aren’t technically considered to be atomic operations, but programmers conflate the two. FEX has used the terms atomic-load and atomic-store to mean the same thing! The distinction usually doesn’t matter, but when discussing these topics it may be better to be pedantic about it.

The primary use case for these instructions is to force memory ordering between these class of instructions. ARM calls this the “Release Consistency sequentially consistent (RCsc)” model. Without getting too far in to the weeds about how this model operates, the basic gist is that the load-acquire instructions must be observed sequentially without reordering, and the store-release instructions must as well while fulfilling “barrier-ordered-before” semantics. Removing the costly memory barrier instruction required in older ARM architecture versions.

The humble beginnings of ARMv8.0-a

This is the premise of where we start in ARMv8.0-a when we’re emulating the x86-TSO memory model. We make all x86 memory loads turn in to ARM’s load-acquire instructions, and x86 memory stores turn in to store-release instructions. This gives FEX effectively the same memory semantics as x86, although we are actually being more strict than what is necessary. This is because we had no middle-ground which exactly matches behaviour. As one might think, it is exceedingly costly to emulate TSO wth this instructions and we have microbenchmarks that can show this. As ARM CPUs weren’t designed to have these relatively rare acquire/release instructions suddenly become the vast majority of instructions executed.

First let’s start with something easy and use a microbenchmark that is fairly nice to the hardware. No tricky edge-cases, just accessing memory in the common case. This gives us some baseline numbers for what the best-case situation should be.

Let’s break down this graph as it tells us a few interesting stories. The Load and Store columns of each machine is representing our baseline performance number that our hardware should be attempting to achieve. These aren’t trying to max out the memory bandwidth of each system, but do the same amount of work for each type of operation. If we turn our attention to the acquire-load results, we can see that out of the five CPUs tests, three of them have their performance hindered quite a bit by using acquire-loads! Additionally we can see that the AmpereOne CPU has release-store instructions that are strikingly low compared to the other results, and the M1 Acquire/LRCPC load instructions are quite a bit lower than the baseline as well.

The AmpereOne results in particular showcase how bad this legacy path can get. These instructions were never designed to be used this way. Using acquire-release semantics for every load for x86 emulation actually imposes some really strict limitations on ARM CPUs in that the load instructions can no longer be ordered around each other at all. So when you have millions of them in flight per second, the performance isn’t really expected to be good. But because these are the only instructions we had with ARMv8.0-a, it’s what we had to use. While Cortex-X4 and Cortex-X925 have amazing performance for these, you can see how the Oryon-3 has deprioritized their importance.

Where do we go from here?

Let’s take a closer look at the LRCPC-load instructions, which is mandatory since ARMv8.3. This extension adds a bunch of new load instructions to the ARM ISA and adds a new memory model on top of ARM’s RCsc model from before. This new “Release Consistency processor consistent (RCpc)” memory model is what we’ve been wanting! This extension is designed around the requirements that x86 emulation requires, and is expected to get utilized heavily on hardware that implements it. As you can see from the graph, almost all of the platforms have their LRCPC-loads matching their regular loads in performance.

With this new extension that is mandated by newer ARM versions, we basically get solved memory performance. At least according to this microbenchmark that seems to be the case. Once FEX detects this extension we stop using Acquire-Load instructions entirely and switch over to LRCPC-Load instead. But what’s going on with that Apple M1 result..?

This is where we need to commend Apple’s path towards solving this problem. With their Apple Silicon processors they directly added support for the x86-TSO memory model. When the CPU feature is toggled, their regular load/store ARM instructions change behaviour to match what x86 requires. They went this route knowing that they will need a high performance solution for their hardware when switching to the ARM ecosystem exclusively. This is why on their hardware the LRCPC-load instructions are actually aliases of their acquire-load instructions, because their x86 emulator doesn’t even use these instructions! Because they implement the x86-memory model, they just use regular load/store instructions, which can be seen in our microbench results as indiscernable performance overhead. To be fair to the other platforms, this thread-wide TSO mode toggle does have some performance impact, we just don’t see it here. When FEX detects this CPU feature from Asahi Linux we will also enable this and get the “free” performance improvement. A potential concern is that when jumping between x86 emulation and ARM code, that the ARM code will pay unnecessary overhead due to all its accesses being TSO now. While this is a reasonable concern, the amount of ARM native code executing under emulation approaches 0%. As a developer, you don’t care about 1% of memory accesses becoming 10% slower, you care about 99% of accesses becoming 15% of the “ideal” (As shown in AmpereOne results).

As a note, we think a TSO mode is the best path forward for ensuring high performance x86 emulation on the platform. Because this ensures that every memory access instruction behaves how we want or expect. This is shown with the official FEAT_LRCPC extension actually having three versions that apply bandages to the implementation each time.

• FEAT_LRCPC - Adds basic GPR TSO load instructions
• FEAT_LRCPC2 - Adds small offset immediate to TSO load instructions
• FEAT_LRCPC3 - Adds basic vector and stack-based TSO load & store instructions

Even with these three extensions, there is edge-case behaviour that can’t be emulated as nicely as if we had a TSO hardware toggle. We are expecting there to be additional extensions versions as time goes on, trying to fix some of the additional problems we’ll discuss later in the article.

I thought accessing memory was the easy bit?

In the previous section, we were being nice to the ARM hardware and playing along with the underlying hardware’s alignment requirements to get a baseline for what the performance should look like. When emulating x86 although, we run face first in to a glaring problem right from the start. Your favourite x86 applications don’t care about alignment! They’ll access memory however they please, crossing cacheline granularities, doing atomics that aren’t aligned. You think of the alignment problems, these games are doing it. This problem is so bad that we have a term associated with it, called split-locks. These are such a big deal that even the Linux kernel will capture when these occur and slow down games when they do it! Causing many gamers to tinker with kernel options to avoid the slowdown!

But we aren’t going to talk about full on split-locks yet, let’s get started with just load-store instructions in an environment that doesn’t care about alignment. x86 makes certain guarantees to the programmer; if you do a load-store and it is inside of a cacheline then that load-store will be both atomic and still match the coherency model as described before. However, to be a little bit nice to the hardware developers, if the load-store does cross a cacheline, the data isn’t atomic and other threads can and will see it tear. So the programmer needs to be careful as a basic load-store is not a split-lock.

The problem with emulating these basic accesses with load-acquire/store-release is that ARMv8.0 requires what is known as natural alignment. This means that for whatever size of data being accessed, the offset in memory must match the size. So for an 8-byte access, it must be at offsets; 0, 8, 16, 24, etc. This works well for native ARM applications, but what happens when we don’t obey natural alignment requirements? For ARM, this means the instruction with raise an alignment fault. The hardware validates that the alignment requirements are fulfilled and if they are not then the CPU will fault. This usually results in a crash but FEX does special handling.

Inside of FEX’s JIT mechanism we keep track of memory load-store instructions that are emulating the x86 load-stores. When we know that a load-store can cause an alignment fault we have what is known as a patchpoint in the code. For load-store instructions, this shows up as a NOP instruction either before or after the load-store. When a alignment fault occurs as one of these patchpoints, FEX will capture the fault, patch the code from a load-acquire/store-release instruction to a basic equivalent load-store, and wraps the instruction in a data memory barrier. Then it continues executing!

Before then after patching

That entire discussion from before about how ARMv8.0-a added these new fancy load-acquire, store-release instructions? We immediately fall back to the classic memory barrier instruction instead when alignment behaviour doesn’t match. Our previous chart didn’t show this bad case, so let’s bring in some fresh data.

Oh, that’s a lot of data to sift through. While again good to see how far away the hardware is from the “optimal” path while emulating TSO, it’s not what we care about here. It is interesting to note that this microbench doesn’t showcase much of a difference between aligned and unaligned for regular load/stores so we just calculated an average between the two. We’ll be removing the x86 CPU and the regular load-store data from the ARM columns, as these aren’t the common FEX paths. This way we’ll have a more targeted view about how badly unaligned memory accesses hurt under emulation.

Now that we have a much more reasonable graph of data, let’s walk from left to right on this and discuss what is going on.

AmpereOne

This one is pretty interesting, both the aligned and unaligned load instructions are roughly equivalent and fall within noise. This means that even though the unaligned loads are getting hit with a data memory barrier penalty, the CPU just handles it. This might be the case that the benchmark is bottlenecked by other things, considering how much lower the performance is compared to other platforms.

Meanwhile the store side is not looking to be in a good shape even without unaligned. It nearly isn’t visible on the chart! When hitting unaligned stores we’re looking at ~8.5% of a performance hit, but because we are already starting so low it is hard to notice. This is also in stark contrast to regular store instructions getting ~28GB/s in this bench.

The only conclusion we can come to here is that Ampere is optimizing for some server class workload and doesn’t really match consumer hardware behaviour. It’s an interesting datapoint, but our users aren’t typically running games on this class of hardware.

Cortex-X4

This is a highly popular CPU core that is living inside the Qualcomm Snapdragon 8 Gen 3. We only tested this one core from the SoC to not overwhelm the chart with data. Quite a large number of handhelds ship with this so it’s an interesting target. This CPU actually does surprisingly well considering it’s the only cellphone SoC on this list. Overall this core kind of falls in line with what we would expect from it and the graph trends follow with the next-generation Cortex in that chart.

The main topics for this CPU are that its aligned loads and stores are reasonably powerful, getting around 11.5GB/s and 6.7GB/s respectively. What’s interesting is the performance falloff when it needs to deal with unaligned loadstores, hitting the DMB instructions penalizes the core roughly evenly between loads and stores at around 50% in this benchmark.

This seems to imply that the CPU can keep a decent number of LRCPC-release loadstores in flight so the DMB instructions hurt more when they are encountered, but it isn’t causing world-ending performance. Just that a 50% performance hit due to alignment isn’t an amazing result.

Cortex-X925

Following up the X4, let’s stop by the DGX Spark and its X925 cores. Not only is this a newer CPU core from ARM, it’s running on a system with dramatically more memory bandwidth. 273GB/s in the platform versus the previous 76.8GB/s. This means that we get fairly similar results to the X4 even, just the graph scales a little higher. Interestingly enough, the performance penalty for unaligned accesses roughly match the X4 even. Although it looks like the stores can recover a little faster, likely due to the faster memory helping out. No surprises here, just consistently matching performance across the generations.

Oryon-3

This CPU core design is hot off the presses from Qualcomm. Linux support is still in the process of coming up but it already has a strong showing. The most interesting result from this actually comes from the fact that aligned LRCPC-load instructions are matching the performance of regular loads! That means in the case of a well-behaved application we can typically expect full performance. This continues onward to the release-store instructions being quite capable, although it doesn’t quite match regular stores with only 68% of the bandwidth. Not a bad showing in the slightest.

This CPU also can’t escape from the penalty of unaligned LRCPC-release loadstores. The load side is roughly matching the ~70% performance penalty of the Cortex-X925, likely because the Snapdragon X2 Elite also has tons of bandwidth. But the store side actually gets off a little worse at ~43% of the performance. Even with these performance hits of unaligned accesses, this platform is actually faster than the aligned accesses from the Cortex offerings.

One of the weird things about this platform is that it was advertised to have “Fully coherent 96KB 6-way L1 cache with 64B coherency granules.” Which to our reading implied that unaligned accesses should have dramatically less of a performance impact. Interesting… keep that in mind.

Apple M1

This is the big one we need to talk about. This is the one that was a game changer, it was the “Apple moment.” It showed everyone that ARM was not only feasible, it could be faster. These numbers on this chart are amazing and it’s the result of Apple sticking the TSO memory model directly in to their hardware. Instead of using LRCPC-release accesses for this one, we just enabled their TSO feature and the aligned versions basically match the unaligned version. Maybe a 5% performance hit on the stores? Compared to every other device on that chart, it’s effectively nothing. This primarily comes down to unaligned accesses no longer requiring DMB instructions to be backpatched in to the code, as the hardware just handles it directly.

For us, this is what it means to take x86 emulation seriously on ARM and it really shows that Apple cared that their customers would have a good experience running software both natively and emulated. They saw the problem and just “solved” it, making it go away. That said, when the TSO mode is enabled, you do get a performance hit. Comparing to the previous graph it’s only getting 76% of the regular store performance, and the load performance basically matches; that’s much more tolerable to bear when everything is so much faster.

Wrapping up unaligned LRCPC/release accesses

Wrapping up this section, we need to talk about one of the performance improvements that all of these vendors actually support. This is an extension that ARM whipped up called FEAT_LSE2 which all of these tested platforms implement. We previously talked about how acquire/LRCPC/release memory accesses require natural alignment in order to not incur the wrath of the CPU raising alignment faults. ARM actually thought about this problem and implemented this extension which helps x86 emulation (and probably other workloads). This extension loosens the alignment requirements of not only acquire/LRCPC/release load store instructions, it also loosens the requirement for read-modify-write atomics!

That sounds all well and good, but here’s the kick to the teeth: that means it only provides marginal performance gains for x86 emulation. This extension only loosens the alignment requirements to allow unaligned memory accesses inside of a 16-byte granule. Any access that crosses that 16-byte granule still receives an alignment fault. x86 applications don’t really care about the alignment of their memory accesses, so we get unaligned accesses across the entire cacheline. It’s only read-modify-write atomics that try to avoid crossing a cacheline on x86!

So thanks for the attempt, it’s nice to see, but it doesn’t really move the needle. Since we’re already talking about it, let’s dive in to those RMW atomics shall we?

Oh no, what are these atomic instructions?

Like most modern instruction sets, x86 supports atomic memory operations. These are instructions that execute an ALU operation on data in memory atomically, allowing no intermediate state to be visible. In x86 terms this operates on memory that is both atomic and coherent, while ARM lets you choose to be only atomic or both atomic and coherent. We touched on this briefly before but there is actually a difference between operating on data atomically, and coherency of that data. What difference does it make?

For all of the previous x86 memory model discussion we have been talking about the coherency implications of loads and stores being visible to other processors in the system. What we entirely glossed over is the atomicity requirements of these memory accesses. In the world of x86 a load or store usually completes atomically even when unaligned. This means that if you’re storing 8-bytes of data, and another thread is loading those 8-bytes in a race condition it will never suddenly see a mix of the data from before the store and after the store. In ARM these atomicity guarantees are significantly weaker, meaning if you do an unaligned store instruction the specification of the ISA has zero guarantees about reading a tear in the data. Thankfully for naturally aligned load-store instructions, ARM has a specification called “single-copy atomicity” which guarantees you don’t get a tear for these accesses. Also good news; that FEAT_LSE2 extension from before? It actually extends the single-copy atomicity guarantees to any unaligned access inside of a 16-byte granule! The downside is that x86 has single-copy atomicity guarantees across a full cacheline, so once again the extension still didn’t solve anything completely, just reduced the number of occurences.

Enough about the differences in atomicity and coherency. Where’s the actual atomic instructions? What do they do? Starting in ARMv8.1-a, our ISA has gained instructions that mostly matches x86 atomic instructions in behaviour. Let’s just give the full list to show how they map directly in our JIT.

Well would you look at that, we have a full list of the 19 atomic RMW operations and they basically map directly to some ARM instructions. Ignore the questionable one as it’s not used in real workloads and we would get far too in to the weeds talking about it. We have a pretty clear 1:1 mapping between the architectures, job’s done right? That’s the funny thing about x86 emulation, just because we have these instructions doesn’t mean we get to wire them up without problems. We spent all this time talking about how unaligned accesses can really hurt performance of regular loads and stores, this same problem also applies to RMW atomics!

With this graph, we are looking at a single atomic instruction with its memory address landing somewhere within a cacheline. If we included all of the data for all 19 atomic operations then this data would be even more overwhelming than it already is. All these atomic operations behave roughly equivalent so it would be redundant and wouldn’t matter for what we’re discussing here anyway. This is also the first graph in this post that is actually using logarithmic scaling, so when reading it make sure to understand that the performance difference from the fastest to slowest result is on the scale of around 1000x.

Starting with the x86 Zen processor on this graph; these are the results that our emulation should be striving to achieve. As we can see, if the access is fully contained within a cacheline then the latency of the instruction is the same at 1.44ns. This can be explained by x86 having “atomic cachelines” or “coherent cachelines”, where as long as an unaligned atomic operation stays within a cacheline then it roughly costs the same. This is a really powerful feature of x86 that has been supported for decades at this point so games end up relying on this heavily without even realizing it. The stand-out result for x86 is the final result that is crossing a 64-byte granule and taking ~660ns! That’s an amazingly slow result at ~458x slower compared to the other results because this is finally the hardware using split-locks.

We need to take a moment here to shout out an article that Chips and Cheese wrote while we were preparing to write our article. They do a great deep dive in to why these split-locks are so dramatically slower and is worth the read if you’re unaware of how they work. Specifically we need to mention that x86 split-locks maintain the atomicity and coherency requirements of x86-TSO and will never tear the data even when crossing a cacheline. This is kind of nuts and we’ll explain this more later.

Now for our ARM processors, let’s start with the natural alignment latency numbers. As we can see, all of our platforms perform fairly well but even the latest cores don’t get anywhere near x86. Even our fastest ARM platform is ~3x the latency compared to x86; This directly impacts performance of games but usually isn’t the direct bottleneck so it’s hard to measure exactly how much. Continuing onward to the next data point, we can actually combine the results for 16-byte granule and 64-byte granule crossing with most of our ARM platforms. Due to how the ARM specification defines how unaligned atomics work, both of these results are roughly equivalent and FEX treats them the same as the x86 split-lock problem.

We keep bringing up this split-lock problem but how exactly does FEX emulate them and what makes it so slow? “I thought Apple M1 added x86-TSO support in the hardware, why is it still slow?” If you recall how we brought up before that FEAT_LSE2 introduced support for unaligned memory accesses within a 16-byte granule; these split-lock operations end up hitting the same alignment problems as before but are dramatically slower. FEX can’t backpatch any of these instructions to just do a DMB operation, so we cause an alignment-fault every time one gets executed. This means that we do a kernel -> userspace signal handler -> kernel -> original code dance. every—single—time one of this split-lock operations execute. Jumping between kernel-space and userspace is slow on every platform and when you’re executing thousands of these per second it adds up very quickly. This is why the emulation of these feature is so terribly slow on ARM.

One ARM platform today actually partially resolved this problem although. The Oryon-3 CPU cores introduced what they advertised as “coherent cachelines” and we can see this in our microbenchmark results here. Just like with x86, if the atomic memory access in anywhere inside of the 64-byte cacheline, the performance matches the natural alignment version! This is a tremendous improvement that means the CPU is on par with x86 in feature support until the point it tries to cross a cacheline. We need to applaud Qualcomm on implementing this feature, as it resolves a major performance and correctness problem around split-locks for x86 emulation. The hardware still doesn’t support 64-byte split-locks so we still fall down the FEX emulated path in that instance although.

Continuing on to the Apple result; even though they added x86-TSO memory accesses to their hardware for some reason they neglected to implement full cacheline unaligned atomics like Oryon did. It seems like they should have expected this edge case to surface and implement it but that’s just speculation. This is why you can see the cross 16-byte granule behaving the same as other platforms even with the TSO hardware toggle enabled.

You might have also noticed another little data quirk in the graph. We have an asterisk on the Cortex-X4 result in this benchmark and the performance of the unaligned atomics are dramatically faster than significantly newer CPUs. It is somehow managing to have only ~209ns latency, while the X925 is latency is 1060ns; that’s a 5x perf improvement! How can this possibly be the case? This is actually some fun “special sauce” that is shipping on the platform we’re testing on, which is of course the Valve Steam Frame. Because Valve cares about the performance of their existing gaming catalogue, they are shipping a kernel patch that one of the FEX developers whipped up. This allows the Linux kernel itself to handle the unaligned atomic without that slow dance with FEX and userspace, allowing it to be dramatically faster. If other platforms want to ship this patch in the kernel then we recommend picking it up as and FEX will automatically start using it.

Speaking of kernel intervention, we need to talk about how split-lock emulation is not actually quite correct under FEX due to limitations in the hardware. In order to implement this mandatory feature of x86 correctly, any time we do a 16-byte or 64-byte split-lock, the only way to handle it is to have the kernel implement the feature. Right now FEX implements this as a “best-effort” attempt that can actually tear the data in some cases. You’ll recall that before we said split-locks on x86 will never tear right? Not even the Oryon-3 with its “coherent cachelines” have resolved this problem yet.

What do you mean split-lock is mandatory?

Implementing split-lock emulation with today’s ARM hardware in a performant matter is actually really difficult to do. A naive implementation is to use a global mutex and whenever a split-lock occurs we will ensure to acquire the mutex before doing the operation. This means that any participating split-lock operation will funnel through this mutex. This is correct except for the issue that any aligned atomic operation isn’t a split-lock and won’t participate. Due to the split-lock emulation code needed to be implemented as two 64-bit compare-exchange operations with each half straddling the granularity boundary, we can get a tear with a non-participating atomic still. A trivial example is one thread constantly modifying an atomic in the middle of the cacheline, and then another thread modifying only the integer on one half. This might sound like a contrived example initially, but there are lock-less linked-list implementations that behave exactly like this! Depending on which half the aligned thread is modifying, either the first or second CAS in the split-lock code will fail. If the first CAS fails, then that’s safe and the code can retry, if the second CAS fails that means the data has torn and we can do nothing but hope it doesn’t corrupt data and crash. This will entirely depend on the algorithm that the guest application is using so we don’t control it.

An alternative approach that is completely untenable is to have the kernel track all processes and threads that are sharing memory with each other, then when a thread needs to emulate a split-lock the kernel can halt every process that is sharing memory with that process, do the split-lock in isolation, and then restart the world. The performance implications of this approach aren’t viable. Applications and games can end up doing thousands or more split-locks per second and halting the world will have an intractable performance hit that is dramatically worse than even x86 native.

If we want to ensure correctness in the emulation of split-locks FEX needs to have hardware support in some form to support these. Although we’re not saying that all atomic operations should now support split-locks like x86, that would also not be viable. The good news is that ARM actually has an extension for this that does exactly what we want. ARM has an extension call Transactional Memory Extension that could solve our problem. This extension allows our code to do some number of operations inside of a transactional region, then commit that work atomically; if the commit operation fails, then we can simply retry. The downside of this extension? ARM has officially deprecated the extension and no one ever shipped it. This is likely for the best as the x86 version of the extension has had an abundance of problems that caused it to be disabled on many platforms.

So we need something else to emulate split-locks correctly. For a solution that we believe works for both FEX needs and ARM vendor needs, we have come up with the idea that a 128-bit CASP instruction can be given the ability to have each half of the CASP perfectly straddle the atomic granule boundary, 64-bits on the lower half, and 64-bits on the upper half. Then only in that case does the instruction not raise an alignment-fault and tries to do the CAS operation. This works because x86 only has up to 64-bit unaligned atomic operations, so both halves of the operation can always be fully enclosed by our single operation.

But you may be asking yourself, “how is this any better than the hardware just supporting split-locks?” That’s a good thought and we need to be careful with the how exactly we describe this operation. For x86 their atomic operations must always succeed without tear. For our emulated approach, we can have this ARM CASP instruction fail safely and then we can try again. This is one of the benefits of CAS is that the operation can fail for any reason and it must be tried again. The instruction then also returns the data that it loaded from memory in that time so the program has the latest up to date memory. This is an important distinction since that means FEX can retry the CAS operations infinite times until it inevitably succeeds! This is a benefit of ARM LL/SC architecture that basically allows this to work. A tricky thing is that the hardware does need to guarantee forward progress at some point but it already has support for that for other reasons so it’s completely viable! The only newly added failure mode to the CAS instruction is purely if one of the two cachelines got acquired by another core before it could do the full operation. Even if the hardware still requires up to a couple thousand cycles to guarantee forward progress, that basically matches x86 behaviour.

We think this would be the best way forward for x86 emulation of split-locks on ARM platforms, but we’re not hardware architects so all we can do is complain and hope someone solves it for us. We’ll leave the split-lock discussion there for now so we can move on to another interesting problem.

Wait, uncached memory needs to work?

Before we get in to this topic we need to talk about the term “uncached” because it can mean a couple of things depending on your view of the world. For the purposes of this article, we are using Vulkan terminology because we care about games primarily. In Vulkan terms we have VK_MEMORY_HOST_CACHED_BIT which means that the host CPU caches this memory. The lack of this bit is what we care about here, and what we refer to as “uncached.” As for what this means to the memory subsystem, it gets a little more complicated than you would think. In particular when the memory is living on a GPU, potentially over PCIe, when the memory is “uncached” it will also typically (but not always!) also gain the flag VK_MEMORY_HOST_COHERENT. This means that because of the uncacheable property of the memory, the CPU and GPU always have a coherent world memory view with each other.

For the CPU this typically means the memory can be mapped up to three ways. When asking for “cached” memory, this typically has a memory type of Write-back which is also what regular memory mapping types are. “uncached” mapping can be either Write-Combine or “Strong Uncacheable”. The “Strong Uncacheable” implementation is basically non-existant for userspace applications so we can ignore that for today’s discussion. This limits us to effectively WB (cached) and WC (uncached) memory types. Cached is what games typically use for staging buffers, and then uncached is what we use when passing data directly to the GPU.

This is code-ified in many game engines that if you don’t expose support for uncached buffer types then some don’t work. This comes down to a behaviour detail around the differences of UMA systems like APUs and PCIe GPUs. UMA systems will typically expose the ability to allocate memory that is cached, coherent, and GPU visible. Where PCIe GPUs can’t guarantee that behaviour so game developers need to either use a staging buffer and an async copy of the data over to the GPU, or use “uncached” memory to very carefully shuffle the data over to the GPU through PCIe. Because of how ubiquitous PCIe is with PC gaming, some engines won’t even do UMA specific code paths and will do the uncached approach regardless!

With that little introduction out of the way for what uncached means for us. Let’s bring up a benchmark for how fast cached memory is on some UMA Snapdragon systems. This will let us get a baseline for how the performance should be regularly.

For both the Steam Frame and Snapdragon X2 Elite these are some really good results. As we would expect, the Oryon-3 platform has more memory bandwidth so it is able to scale higher in the chart, but both are hitting dozens of gigabytes per second in their results. This graph sets a good baseline for what “normal” write-back memory can achieve. Let’s now show uncached results to see the performance differences.

There’s some strange things happening here so we had to use logarithmic again on this graph. Let’s talk about the good first that has shown up. Due to uncached memory buffers being write-combine, we can see that the regular stores for our ARM platforms match the cached benchmark results. This comes down to write-combine memory using what is coined as write combine buffers that actually very temporarily keep around a cacheline of data so that write-combine can burst a cacheline of memory at a time. Interestingly enough it looks like the Zen 4’s WCB can’t quite keep up with cached, but considering this is expected to be going over a PCIe bus it’s probably fine.

Now let’s get in to the really ugly results that we have here. Starting off with the easier to explain is the load bandwidth from write-combined memory is abysmal on all platforms tested. If we’re using Zen as our baseline for performance, then our regular load instructions are ARM are winning, but the LRCPC loads are worse. What’s going on here? This is a quirk of how write-combined memory operates, because it is uncached our load instructions are required to go out to system memory for every single access to maintain semantics. Then when we add LRCPC-loads on top of that, it just compounds the problem even further. But the worst case out of all of this is just how badly the store performance is, compared to the performance that Zen gets on the stores, this is basically a showstopper. Up to 816x worse bandwidth! We had games like Hollow Knight: Silksong and Subnautica 2 run at less than 1FPS because of this performance cliff.

As we were saying above, when there are PCIe GPUs in the mix then games will need to use uncached memory to pass data to the GPU. When emulating x86 games on platforms with a dedicated PCIe GPU then we are in an unwinnable situation and we are guaranteed to run dramatically slower. Remember how ARM has added the family of FEAT_LRCPC1/2/3 extensions from before to improve x86 memory model emulation? This is what happens when we hit an edge-case that isn’t supported. All of these extensions add new instructions to handle loading memory using x86-TSO memory model semantics but none of them solve storing to write-combine memory with x86-TSO semantics. All the way from ARMv8.0-a our store instructions use the regular store-release instructions regardless of the backing memory type. The only way for FEX to work around this problem is to selectively disable TSO-emulation when it becomes an issue, so x86 emulation platforms with PCIe GPUs will always be a worse experience than UMA. At least until we get another FEAT_LRCPC4 or similar to resolve the issue.

For users on UMA systems then rejoice, there’s a workaround for gaming that we use to improve performance. Because we know when a platform supports cache-coherent CPU and GPU combinations, we can have the video driver always use cached buffers and never encounter this problem. NVIDIA already does this on their Tegra platforms, Snapdragon has been supporting this since at least Adreno 600 class GPUs, and there are many Mali platforms where this is also the case. We have a Adreno Turnip patch that ensures when FEX is running, we never hit uncached memory for platforms that support it. A funny thing is that since Asahi users have a hardware TSO bit, they just naturally don’t encounter this problem in the wild, but getting a PCIe GPU on to that platform is a different story altogether. There’s also a fun quirk where Radeon GPUs on ARM platforms hide all write-combine memory to instead be write-back but we’ll talk about that another time.

Looking towards a brighter future

After that marathon of an article we hope you have a better understanding of some of the challenges that emulating the x86-TSO memory model brings. Where we started with ARMv8.0 as a minimum spec and where the hardware has provided dramatic improvements over the years in nothing short of astounding. While not all of the edge-cases are yet resolved at the architecture level, it looks like there is a genuine commitment across the ecosystem for trying to improve the worst cases. We have various vendors solving some parts of the problem and moving the needle forward for better compatibility. Maybe in another decade as we look back at this time we’ll laugh about the problems we were encountering now, while enjoying some quality x86 games that will never see a port to ARM hardware. Keeping the legacy of the PC gaming ecosystem alive, regardless of where we might end up playing it.

We’re not going to sugar coat it, this last month was very busy and time just kept moving faster than expected. A ton of work landed this month so we’re going to just need to pick some of the highlights. Let’s hop in!

Jit work you say?

A bunch of random bug fixes and performance improvements landed this month. Primary things here are that there were various behaviour fixes for some instructions. Undefined and defined flag fixes that FEX was messing up, some deadlock fixes, handling more cases that were previously asserts. There was an optimization for the PMULHRSW instruction family that increased some Geekbench benchmark numbers. Surely some game uses it somewhere to notice the 2x perf uplift. We also made the “full” self-modifying-code detection faster, which is used for Unity games in some instances. So that’s nice. We also reduced some lock-contention in the JIT so multiple threads jitting code at the same time should block each other less frequently.

There’s more of course but come on we gotta go!

Removed hacky unixlib fallbacks

For Wine we had some fallbacks for when our unixlib helper library wasn’t installed. This last month we removed these fallbacks and require the unixlib for these features now. Currently all of these features technically are optional, so it isn’t a big deal if unixlib is missing. We also discovered when removing these that we would accidentally be doing Windows syscalls if someone was replacing XTAJIT/Prism with FEX. Oops! Well those fallbacks are removed so that won’t happen anymore at least.

Disk-cache as an option

Through an absolute flurry of commits this last month, disk caching of JIT code is in a place where we think it can be tinkered with more readily. What does disk caching mean for FEX? Once the configuration option is enabled; any code that FEX’s JIT generates is cached to the disk in a FOZ file format database. Then when FEX needs to JIT code subsequently, it will attempt to look up code from that FOZ database first before going through the work of actually jitting. This database lookup happens during the initial run but also subsequent runs of FEX to reduce JIT stutter when running games.

Some downsides today are that if the disk caching is enabled on applications that JIT code itself, then the cache can technically grow unbounded. We haven’t yet implemented any caching size limits or any way to remove stale cache entries. Additionally if any of the file contents change, there might be conflicts today where the cache doesn’t get invalidated in those instances. So there’s still some growing pains but we don’t think these are a showstopper for tinkering today. To give it a try you can set the `FEX_DISKCACHE=1` environment variable or the `“DiskCache”: “1”` json option to enable everything. This works with both the FEX linux interpreter program and also WINE, it maybe be a bit more stable in WINE for now. The cache files will end up in WINE’s %APPDATA% directory or $HOME/.cache/fex-emu/ by default. So if you are running out of space make sure to delete those directories to recover some space!

Let us know how it goes and if there are any games that completely break when this feature is enabled! We’ll be spending continued effort making this feature better as time goes on.

Oh right, the new logo

If you missed the random post, we got a new logo! You’ll find that it’s not dramatically different, but the rainbox gear has definitely changed!

See the 2609 Release Notes or the detailed change log in Github.

That is all, see you next month.

Removal of deprecated FEXInterpreter binary

Back in the FEX-2510 release we completely removed a FEXLoader binary with a deprecation of the FEXInterpreter binary, renaming it to just FEX. We kept the FEXInterpreter binary around for a while because downstream projects had hardcoded the binary in their scripts and we wanted to give some time for them to migrate. After nearly a year we’re hoping that everyone has migrated to the new binary name. If your project is still using the old binary name then we recommend fix them!

The typical JIT improvements

As typical there are a bunch of JIT changes this month. Let’s breakdown what was fixed.

• Use SVE XAR for SHA1 emulation as a minor optimization.
• Fix JIT telemetry value unconstrained predictable problem on ARMv8.0
  • Would only cause some missed telemetry on ARMv8.0 platforms, very minor.
• Optimize VMOVMASKPD/PS instructions.
  • Cutting from 11/10 instructions down to 7.
• Fix incorrect emulation of BFP store instruction
• Fixes crash if ProcessorID IR op was encountered on WINE
  • Silly little assert check that crashed.
• A bunch more improvements to hardware with 256-bit SVE
• Actually a ton of little bug fixes and edge cases getting patched up.

Fix VirtualProtect on win32

This change is fairly spicy because it means some of our memory protections were silently failing under WINE. With this fixed, it should fix some weird edge case crashes that weren’t able to be explained.

Fix overwriting memory region in allocation tracking

This is a bit of a weird case where FEX’s memory allocations on 48-bit VA systems were silently only ever using one allocation region. It was accidentally overwriting all regions with whatever the smallest allocation happened to be. Luckily on 48-bit systems, the smallest region would still be very large and unlikely to run out, but a bug fix is a bug fix.

Fix inactive JIT guard pages not being ignored

A funny little quirk of our JIT is that we rely on SIGSEGV to handle memory overflows safely. Where we try to calculate a size of code memory that gets generated from IR through a heuristic and if the size is calculated incorrectly we safely fault and longjump to try again. The times in which the heuristic is incorrect isn’t very frequent so the performance win of not range checking as heavily is a big win.

One problem is we forgot to turn off the JIT guard page checking when outside of the JIT, so if an application happens to access from [0, 4096), we would long jump back in to our JIT and hilarious crashes would occur. Oops!

Fixes AVX signal state saving and restoring

While signals are fairly rare, we implementing saving the AVX state incorrectly, which could cause us the fail to restore it if the application jumps to a new location. This is now fixed which should probably resolve a few crashes.

Spinloop using WFE on WINE

On WINE we had some spinloops that are required when an application is in the process of suspending a thread. We now spin using the WFE instruction which puts the ARM CPU in lower-power state while waiting on an atomic. This can shave off watts of power in this edge case, so it’s worth supporting!

Fix untracked thread handling

When a process creates a new thread using the Clone syscall API, there’s a lot of tricky edge cases to support. In one of those paths we were accidentally losing tracking of a thread in our frontend. Usually this is pretty benign but it could result in some broken TLS tracking due to a new thread data structure actually pointing to the parent thread. Oops!

See the 2608 Release Notes or the detailed change log in Github.

Optimizations and fixes for 256-bit SVE2 hardware

While this hardware doesn’t exist yet, we know it is an inevitability that it will at some point. While we switched gears a couple years ago to implement AVX using 128-bit operations, we never removed this code. In-fact we’ve been spending a bunch of effort on it fixing bugs and optimizing it so that once hardware ships it won’t be broken. We have now validated extensively that all AVX instructions now zero extend their results as expected and optimized a bunch of the instructions so they generate faster code. We think it’s now at a point that in the common case the code will be faster than our 128-bit emulation, but there is definitely still some more work todo. SVE2.1 provides a significant improvement to how shuffles operate and we haven’t gotten those implemented yet. Because there aren’t any 256-bit SVE2 hardware on the market, we might even require SVE2.1 or SVE2.2 for this class of hardware. We’ll of course still maintain our 128-bit path for lower-end hardware of course.

Various JIT fixes/changes

Once again, too many to go through individually, let’s throw it in a list. Most of these are just bug fixes but there are a handful of optimizations as well.

• Fixed corruption with back to back PMOVMSKB instructions and full SMC detection
  • Fixes Vivado in this situation
• Handle incorrect LOCK prefixed instructions correctly
• Allow larger CPU context state
  • Fixes compilation error with musl
• Fixes vsyscall page tracking
  • Was accidentally a NOEXEC page
• Only use DC ZVA for Ampere when clearing AVX state
  • It’s the same or slower on other hardware
• Fix zero extension in VCVTPS2PH
• Fixes CRC32 with high 8-bit registers
• Fixes 64-bit LODS instruction with address size override
• Fix Mafia 3 in Arm64ec Proton
• Fix Bioshock and other 32-bit games that disable DEP on Arm64ec Proton
  • Requires bleeding-edge Proton or WINE
• Optimize x87 FYL2X/FPREM/FPREM1
• Fix incorrect RSP update on 16-bit LEAVE instruction
• Handle some new CPL0 only instructions correctly
• Fix JIT allocations ending up in 32-bit VA space
  • Fixes some 32-bit games under Proton

Implement support for a unixlib under Proton/WINE

For a long time FEX’s WINE support works by building all of FEX as a DLL file which WINE loads at runtime. This has worked for us pretty well but constrained some of our design decisions at times, causing us to take less optimal or hacky paths. We also have some upcoming changes that would be dramatically harder to implement without a unixlib so we decided to start using one. Now FEX for WINE ships two DLL files, and two unixlib SO files.

Right now we are duplicating some code between the unixlib and the FEX WINE DLL file to make sure everything works correctly but within a few months we are likely to remove the duplicated code and rely entirely on the unixlib. Currently all the functionality provided is optional, but we would recommend using it because we may eventually not have it be optional.

Partial support for CUDA thunking

Some people were asking for CUDA thunking on the DGX Spark and we decided to implement partial support for it. It’s not 100% coverage but depending on the workload it can potentially work. We know today that if you try to execute applications with the static CUDA runtime linked, it won’t work well. Other than that give it a try and you might be surprised at what works.

See the 2607 Release Notes or the detailed change log in Github.

Various JIT fixes and performance improvements

We can’t keep getting away with it. An emulator optimizing and fixing bugs in its JIT? Unheard of. This month we have various improvements littered around. We optimized some more x87 instructions in our reduced precision path again; This time hitting ATAN, FYL2X, FSCALE, and F2XM1, the typical 2x-4x improvement on these instructions depending. We’re starting to run out of x87 instructions to optimize at this point and 32-bit games can only run so fast!

Thanks to a new contributor bringing up this issue, we learned that cmpxchg8b/16b was setting some CPU flags incorrectly! This is a pretty big mistake and likely caused some games to spin on memory agressively. Luckily these two instructions are relatively rare, so it wouldn’t be a common sight.

Another bug fixed was with SSE MAXPS and MAXPD instructions not breaking ties correctly on Inf/NaN inputs, which could have caused some issues. Some x87 operations also had some trivial denormal result checks in the same vein that were resolved. An additional bug fix was pushing and popping of 16-bit segment registers wasn’t actually working correctly. No idea who uses that feature on 64-bit CPUs today, but we at least know it’s fixed!

We also optimized this month the MMX PSHUFW instruction, finding more cases in which we can reduce a heavy table lookup in to either one or two instructions. We also modified the x87 class of FIST instructions to only access memory without TSO emulation to improve performance.

Fixed a crash with ARM64ec and controllers

This month we found a nasty bug that cropped up in an edge case. We only really noticed this because SDL behaves different when a DualSense controller was plugged in for whatever reason.

The core of the issue comes down to a feature in ARM64ec called a suspend doorbell. When WINE wants to pause an emulated thread so it is in a safe location, it will set a doorbell which FEX watches and will collaboratively suspend if it is set. Turns out we had a bug in the code that would always crash in this particular edge case, but because it is so uncommon we never really noticed it.

When a controller was connected, WINE was asking for our emulated threads to suspend so commonly, that it basically became a unittest by itself and immediately caught the issue. Easy fix, but it took a couple days of trying to figure out with this wacky race condition between multiple threads and signals getting passed around!

Query DCZID_EL0 on ARM64ec so CLZERO works

This is a fairly simple fix, we were forgetting to test if ARM’s DC ZVA instruction matches CLZERO behaviour, causing a crash on applications that unconditionally use the instruction. This just got lost with the initial implementation and we didn’t really notice it because it’s fairly uncommon in today’s games. Luckily we have some microbenchmarks that abuse it and found out the issue!

Snapdragon X2 Elite fixes

This month we finally got our hands on some new Snapdragon X2 Elite platforms for testing on! While they don’t yet support Linux out of the box, Qualcomm is pushing commits each kernel release to move the needle. With 7.1 or 7.2 we might even have a working GPU!

Since this was the first time we actually got our hands on the platform, we had to fix some new bugs and features we noticed in the hardware.

• RNDRRS is still broken just like X1E, so disable the RNG feature
• Still ships a 19.2Mhz cycle counter, so it’s ARMv9.0-a compliant hardware
  • ARMv9.1-a mandates a 1Ghz cycle counter
• It supports SVE2 (with 128-bit registers) and SME, and KVM virtualization!
• The hardware supports some new atomic behaviour!

Let’s dive in to that last bullet point, because that’s highly interesting for FEX’s interests. As usual for emulating x86, the hardest problem is emulating the memory model. Regular loads and stores turn in to acquire load and release store instructions, for a programmer we tend to call these “atomic” operations. Even the C++ standard wraps these in std::atomic<>. Then of course any x86 instruction that has a LOCK prefix on it, is an atomic memory operation that does a RMW of the location in memory. x86 has this extremely cool feature where the alignment of your accesses in memory don’t really matter, as long as they don’t cross a cacheline which is always 64-byte in size. If you cross a cacheline, then you invoke your CPU’s wrath and engage what is known as a split-lock, which is very slow and also slows down other processes on your system. Chips and Cheese actually had a really nice article recently on the performance of these split-locks. tldr the latency of an atomic goes from something like 1-2 nanosecond inside of a cacheline, up to around 700 nanosecond when a split-lock is engaged (Or worse depending on hardware).

FEX needs to emulate these split-locks in a way that works on ARM, but turns out that the latest ARM CPUs are even worse off. When your atomic operation crosses a 16-byte granularity, the operation will raise a SIGBUS. FEX then needs to catch this SIGBUS and emulate the split-lock functionality. This happens dramatically more frequently because of the 16-byte granularity instead of 64-byte granularity. An additional problem that FEX has, is that emulating the split-lock isn’t actually safe on ARM because we will always have the potential to tear.

Luckily! Qualcomm has partially solved this problem on X2E! With the latest iteration of their hardware, RMW atomic operations will now only SIGBUS when crossing a 64-byte cacheline granularity instead of every 16 bytes! This is a boon to correctness for the common case, which is to be commended. Somewhat interestingly, although they support RMW atomic operations without SIGBUS within a cacheline now, the acquire load and release store instructions still SIGBUS when crossing a 16-byte granularity. This is sadly the more common experience and what tends to cause performance issues with hand-coded memcpy/memset functions inside of games.

We also still don’t have a way to correctly emulate split-locks crossing a 64-byte granularity without tearing, but maybe the ARM-lords will grant us CASP cacheline straddling split-lock support some day.

Post by @FEX_Emu@mastodon.social

View on Mastodon

See the 2605 Release Notes or the detailed change log in Github.

More memory savings

This month we have had some memory saving changes land, which is vitally important for 8GB and 16GB systems. Primarily we have now enabled our Dynamic L1 lookup cache and disabled our L2 lookup caches by default. We talked about this more in the FEX-2511 release post, but this can save hundreds of megabytes by changing these default options.

Additionally we have fixed a pseudo-leak in one of our thread-pool allocators. It wasn’t quite a real leak because each thread only ever held a single allocation, but it is supposed to share allocations between threads which means this ballooned pretty heavily for games that create a lot of threads. For our test game, ENDER LILIES: Quietus of the Knights, this meant going from consuming 409MB of memory down to 6MB for this pool.

Another change that occured this month is being more aware of Transparent Huge Pages potentially causing us to consume more memory than expected. When the operating mode is set to always instead of madvise then we were consuming significantly more RAM than expected. ArchLinux currently defaults to always which caught us by surprise in our testing. FEX will now actively ask for THP or non-THP buffers depending on their use-case which can dramatically reduce memory usage for our sparse buffers on these systems that default to always. As a side-effect, our JIT code buffer now always asks for a THP buffer, which cuts iTLB misses in half in our testing which dramatically reduces pressure on CPU’s L2 TLB lookups.

A smattering of bug fixes and performance improvements

As usual we have a large number of bug fixes and performance improvements. Each one being small enough that it would be hard to list them all, but we do have some highlights.

Inline SIN/COS/TAN for x87 reduced precision

One of the most costly things that our JIT can do is x87 emulation and jumping out of the JIT for a helper. Unfortunately they tend to come hand-in-hand. This month we have optimized these three transcendental operations to no longer jump out of the JIT which has sped up the operations by an average of 3.7x! This makes games that hit these x87 transcendentals go quite a bit faster, like Bayonetta and Fallout: New Vegas. Improving their playability on a larger set of systems.

Additional changes as follow:

Performance

• Replace a code invalidation mutex with our hand-rolled implementation that is dramatically faster
• Wire up FEAT_MOPS support. The Samsung Exynos 2600 is one of the first SoCs with support
• Rearrange some Arm64EC dispatcher code for performance
• Optimize a vector broadcast a game was hitting
• Skip ELF parsing when code caching is disabled

Bug fixes

• Fix prefetch encoded nop instructions
• Ensure MXCSR is saved and restored correctly on signal
• Reset relocation data on JIT restart

Workaround a Docker seccomp filter bug

A user has been tinkering with FEX inside of a Docker environment and they uncovered an issue where FEX was crashing for really bizarre issues. We eventually tracked down some syscalls that were ending up returning broken results due to a bug in Docker’s seccomp filter rules. It turns out that their filter doesn’t follow the AAPCS64 nor the SystemV x86-64 ABI rules around zero extending arguments that are smaller than the register size. This causes problems because it is on the callee to do the zero extension of the argument, and if you have garbage in the upper bits of the source register, it should get ignored.

Because Docker’s seccomp filter only ever compares the values passed to system calls to the full size of the register, 8/16/32-bit arguments can have garbage in the upper bits and incorrectly return -EPERM for perfectly valid data. We manually worked around the one instance we saw this causing problems locally, but Docker needs to audit their seccomp filters and correctly handle this for a real fix!

Add option to FEXGetConfig to show fault granularity

One of the major struggles with emulating x86’s TSO is that because our memory accesses fault when unaligned, they have dramatic overhead compared to x86 basically never faulting due to alignment problems. This is slightly improved on newer ARM CPUs where the FEAT_LSE2 extension removes a percentage of faults by allowing unaligned accesses inside of a 16-byte granule. With this new run-time test, we can visualize when instructions are going to fault to showcase how bad it is.

First a system that doesn’t support FEAT_LSE2

Then a system that supports FEAT_LSE2

The green pips show which byte-aligned memory accesses don’t fault and cause problems, while the red pips show where a fault will occur and we end up needing to backpatch the code with a memory fence, or simulate the operation in the signal handler. As seen, there is still quite a bit of red on the graphs even with best hardware for this. Meanwhile if we had a similar test for x86, all the pips would be green except the 128-bit result, which matches behaviour between the two architectures. (Except vector accesses which this doesn’t test).

We added this test capability so that if any hardware in the future does decide to fix this performance and correctness problem, then we get a very quick test we can run to detect it.

See the 2604 Release Notes or the detailed change log in Github.

Steamwebhelper crash - Known Bug

One of the bugs we encountered that caused us to cancel last month’s release was a spurious crash that occurs with Steam’s steamwebhelper process. This ends up behaving like Steam is crashing constantly and coming back. What’s actually happening is one of the steamwebhelper processes crashes, and then steam restarts it consistently. This isn’t actually a regression on the FEX side, but a change that occured in Steam late last year that we didn’t notice initially. The CEF version that Steam is shipping has updated to change some behaviour around FD handling that FEX interacts badly with. We haven’t fully worked around the issue so Steam’s GUI may still crash fairly frequently, but as long as FEX’s logging is disabled then it is less likely to occur.

Hide big.LITTLE layout by default

We determined this month that some games with anti-tamper break if we expose the different CPU core names in the “product string” of CPUID. To combat this we are now hiding the big.LITTLE nature of the CPUs by default and now replicate CPU zero’s name across all cores. This fixes a fairly significant number of of anti-tamper games with just a trivial change.

Convert vzeroupper/vzeroall to use ZVA

This month we found out that some CPU cores are significantly faster at zeroing a couple of cachelines of memory using the dc zva instruction. This hadn’t become an issue on most consumer class CPUs because they can typically saturate their store pipelines using regular stp instructions already. vzeroupper is quite common when executing AVX code, so we want to make sure it is as fast as possible. This change alone increased Death Stranding’s FPS from 55FPS to 70FPS on AmpereOne CPUs in our test scene.

Fix build with upstream Clang/LLVM 22 and Mingw

This month we fixed our builds for both the latest LLVM 22 release and the newest LLVM-Mingw toolchain. There were some minor changes to the clang API and libc++ that required some work on our side to resolve. With these changes in place, this will now more easily allow package managers to build our Wine DLL files and use a newer compiler.

Switch one allocator to RPMalloc

This is a fairly big change that we did this month. We ripped out one of our JEMalloc allocators and instead replaced it with RPMalloc. The driving force behind this change is that RPMalloc uses significantly less RAM for its internal state tracking, which in turn means that FEX itself is using less RAM for the emulation. We’ve seen some dramatic examples where this change would shave hundreds of megabytes of memory off of FEX’s memory usage. This allocator is also quite a bit smaller so it is easier to read and see what it’s doing, which is good when JEMalloc is no longer maintained.

Various JIT changes

Another month of JIT changes that would take too much time to dive in to directly, so we’ll just list them off.

• VEX compare operations fixed
• Optimize x87 conversion instructions
• Fix undocumented x87 instruction alises
• Implement uncommon instruction ARPL
• Switch over to ankerl::unordered_dense instead of tsl for cache
• Fix initial PF flag state

Various Linux frontend changes

This month we had too many frontend changes to dive in to as well.

• Override glibc program_invocation name, so mesa can see application profiles
• Works around execveat Linux kernel bug with MFD_CLOEXEC
• Ensures seccomp gets inherited correctly
• Ensure personality gets inherited correctly
• Moves DRM LRU FD cache to be per-thread
• Updates sigaltstack minimum size requirements
• Ensures XSTATE_MAGIC2 is saved correctly

See the 2603 Release Notes or the detailed change log in Github.

Update thunks for Vulkan 1.4.337

This update is fairly important as Proton and Mesa have started using some new extensions that we didn’t previously support. So if your system had a new driver with these extensions then dxvk/vkd3d-proton would assert out. With these updated it is no longer a problem and thunking is working again as normal!

Fix a couple rare hangs on Wine mutex handling

FEX has a custom mutex implementation that has “writer” priority that was implemented a few months ago. We needed to implement this to help reduce stuttering in our code cache implementation. When implementing it we actually had two bugs in the implementation for Wine that went unnoticed until now. Our implementation is fairly smart and will spin on the mutex using ARM’s highly efficient “Wait-For-Event” instruction for 1/10th of a millisecond before deferring to the kernel mutex implementation. Because we block the mutex for such a small amount of time, it was /highly/ unlikely to hit the kernel implementation. When we did defer to the kernel (Or Wine’s implementation anyway) we had a bug in our anti-stampeding behaviour which would cause reader threads to never wake up. Then additionally we had a bug in the calling API for the “RtlWaitOnAddress” function declaration where it was only waiting on a 32-bit address. Causing the process to never wake up and usually crash.

A very long winded way to say we had two simple bugs that were rare and infuriating to debug because of a race.

JIT fixes

This month there weren’t actually that many JIT fixes. We found a bug that was breaking Ubisoft’s UPlay program which is now fixed. Additionally we resolved some handling of self-modifying code on our Wine implementation that could fix some spurious hangs or incorrect invalidations.

Minor Linux syscall fix

This month we noticed that Steam was using some new fcntl syscall operations that we didn’t handle. This caused Steam to crash in some rare edge cases when it actually called this syscall. We have now resolved this, and future proofed any new commands getting emulated by passing directly to the kernel.

More code caching implementation work

This is definitely the task that has the most code land for it. There was code caching support wired up for both Linux side and Wine side. While still a heavy Work-In-Progress, we have code caches getting generated and loaded at runtime to reduce the amount of time spent running in the JIT. In particular, this can be thought of as trading CPU time for disk space, theoretically reducing JIT stutter if the code already existed in the cache. Still lots of work to go to make this viable for the user, but we’ll be trucking along as usual!

39C3 talk from @neobrain

This last month, neobrain had a talk going through some of the architecture of FEX-Emu. It’s very informative and definitely worth the watch!

See the 2601 Release Notes or the detailed change log in Github.

Remap procfs cmdline using PR_SET_MM_MAP

This has been a thorn in our side for a while. When an application reads the cmdline FEX would need to rewrite the file contents to remove the FEXInterpreter argument. Turns out the kernel has had this feature for quite a while to remap this file, we just weren’t utilizing it. Now instead of mangling the data, we are using the correct interface from the kernel. This means that things like Mesa application profiles and KDE Plasma see the correct application name in all instances.

Big shoutout to the external contributors that implemented this for us!

Implement support for JIT codebuffer guard page based restart

This one takes a bit to explain what this is and why it is necessary. When writing our AArch64 code emitter, we made the decision not to do range checks for how much memory is remaining in our JIT code buffer. We instead used a heuristic to determine how much space is required which usually worked. The problem with heuristics of course is that they can fail and our “fallback” case was to crash. This was a known problem that we would need to resolve at some point, and that was finally this month that we go around to it. Due to us utilizing larger “multiblock” JIT blocks, we had started having a more likely chance of hitting this crash, which usually ends up being due to x87 heavy code because the JIT translation is heavy.

Now when the heuristic fails, our code emitter will try writing to our guard page and we will catch the SIGSEGV and restart the JIT with a larger code buffer. Fixing these edge case crash behaviours and making our JIT more robust in the process.

Initial code caching features landing

There’s an absolute ton of work that is going in to this and it’s not yet ready for users yet, but it would be remiss to not call out all the effort on this front. This month we landed initial support for “code maps” and offline “code cache” generation. There is not yet any way for a user to actually utilize these code maps and caches but these are the required steps to get us to the transparent code-caching that we are expecting to have. Watch out for the coming months as we finish fleshing out this feature fully wired up.

Fixes APICID count

This is a bit of a weird feature that we had accidentally missed. When reading CPUID processes get what is called an APIC ID, which is essentially just a core index. Some applications will use this ID as a way to determine how many unique CPU cores are available on the system. We were accidentally always returning zero which was causing some applications to only think the system had one CPU. With this fixed, the FPGA software that this was detected in now generates the correct number of worker threads for the cores in the system. This of course improves their synthesize time dramatically since they scale well with the number of cores in the system.

Disable io_uring syscalls

Our good friends over at felix86 alerted us to an issue around io_uring causing infinite loops in node.js and libuv. Upon further investigation we determine that there is an ABI break in io_uring between x86 and Arm64 that we previously didn’t know about. This comes down to how the user submission queues in io_uring can embed epoll_event structures and these have different layouts between the architectures.

Because we can’t safely rewrite the queue data to handle this layout difference, we have determined the only course of action is to disable the syscalls. Luckily most games don’t rely on this syscall interface or applications will have a legacy fallback for when it is unsupported. In that vein, node.js now works again.

FEAT_LRCPC2 performance errata

This month we found out that a large number of Cortex and Neoverse CPU cores have an errata that only affects the instructions added in FEAT_LRCPC2. We have disabled this extension on the affected CPU cores, which can give a reasonable performance improvement in games that were TSO emulation bounded.

JIT and emulation bug fixes

There were a bunch of bug fixes in both our JIT and Linux syscall emulation this month as usual, but this month’s report is already running long so if you’re interested, take a peek at our pull requests to find out more.

See the 2512 Release Notes or the detailed change log in Github.