… and 23 years of tedious yak shaving.

 Ten years ago my rather anemic post on the Arcan project started to circulate. It got enough attention that I considered stopping right there and then. I do not care much for attention; most of my other open source work has been anonymous or with throwaway handles, and I still regularly run into reflections of my previous misdeeds.

To mourn the occasion, it seems appropriate to pen down some of the history before I manage to repress it entirely. For this round I will omit the funding story; the commercial projects it has been hiding inside; the start-up close calls or that one time the gang nearly sold out to a VC.

Even 10 years ago the project had been open and around for several years, masquerading as a ‘frontend’ for DIY arcade cabinets, but we will get back to that in a little bit.

The actual story starts around 2003. At that point I was mostly ‘done’ with the programming ‘languages’ part of the computing journey — it is the single most boring topic I can think of; the joy is in the music, not the sheet notes. When thinking of something to compose, I was struck by a nasty and incurable case of ‘oscilloscope envy’.

While the cool electronics kids could attach a probe here and there and see some of the circuitry in action, us computing folks were still stuck with caveman style ‘print’ logging or the chainsaw assisted surgery that is symbolic debuggers. I wanted to hear, feel and see the computing machinery as it spun the information weave.

If something like that is even possible, it would take rather precise control of the visual, the aural and the tactile so that what you sample and represent is what you intended to. That takes wrestling some of the worst companies in tech history. But you need to start carving out your space somewhere, so I blew the dust from the old von Clausewitz and sketched out a plan.

First of all, I surely lacked the proper skills. Those should be acquired somehow. You don’t know what you don’t know and all that, and ideas are a dime a dozen. Get to building the thing to see where and why it is flawed. Answers lie in calluses, not in prompts.

The plan was to setup a diary project to chronicle this caving expedition and split it up into three phases. At each transition, evaluate if it is still worth going, or if I should cut my losses and do something less irresponsible.

Each phase should have a handful of actual applications that filled some personal need for more grounded projects. Waypoints of sorts to avoid losing track and burning out as well as a ‘ok this was dumb, but at least I got a t-shirt’ defense mechanism against the sunken cost fallacy.

Being a diary also meant jotting down other parts of the experience, such as loves gained and lives lost. It turned out to be many more of the latter. Every time there is a release post it means someone died, with one or more tributes hidden inside.

The three phases are ‘Fun and Games’ (Arkanoid), ‘Challenge the Established Order’ (Arcane) and ‘Asylum’ (Arkham). Smash them together and you get Arcan.

Sidequest

To remedy the skill situation, and postpone the whole ‘growing up’ thing, I sought sagely advise. As bad luck would have it, I got an audience with the local bigwig CS professor; a bearded fellow of some repute with an unrequited love for mathematics and a nose for systems thinking.

After some back and forth we agreed that it would be best (but perhaps not for me) if I took aim for the whole PhD thing. He just so happened to have funding for that in the pipeline. Not the whole debugging thing, heavens no. That is something for people with dirt under their nails, not chalk. But debugging is basically security, if you squint a little, and that is a lot easier to fund.

Monitoring and surveillance are two sides of the same security coin, and power grids do love some monitoring as part of the SCADA (Supervisory Control and Data Acquisition) acronym. This was basically corporate IT security but constantly 15-20 years behind the times.

Smart Micro Grids were to become all the research-rage and to be built from COTS (Common, Off The Shelf) components from dubious sources of surely compromised supply-chains. It also implied turning the whole thing inside out, and monitoring all of that was much closer to what I was looking for anyhow, so fine, that will be the research target.

There is a lot more to this tangent that we do not have the time for, nor could I handle the blood alcohol levels necessary to revisit those memories, so lets move on to something more playful.

Phase One – Fun and Games

The point of this phase was to establish the necessary groundwork – Media Processing Pipeline, Input Devices, Privilege Separation and a scripting API to tie it all together. So what would be a fun target application that ticks all the boxes and could drive the design? An unconventional emulator frontend.

Backstory

I had a thing for Arcade Machines and their natural habitat, the arcades of yore. I still do, even though it has drifted more towards the softer aspects that cannot be easily preserved or reproduced. A current side project is a motion simulation rig that just needs a few more welds before I can relive some of the glory of Metal Hawk in the comfort of my own basement.

Back then my interest was in the nitty gritty of emulation and emulator development. Going all the way from a physical thing in front of you into a working software replica is a full-blown computing crash course. There’s reversing schematics to understand how things are wired up; dumping the ROMs where code and data hide out; Digging through old and inaccurate specifications for ancient CPUs and DSPs, then discovering and replicating all their little bugs that some crafty developers took advantage of; writing and debugging the first flawed processor implementation; cracking bespoke copy protection schemes, exotic cryptography and so on. It is a very long hike before you even get to see real signs of life. Wear comfortable shoes.

The gold standard of arcade emulation, MAME, is a massive work of art and open source cultural heritage; its code should be obligatory reading for anyone seriously interested in computing and its legacy. I hold it in much higher regard than something as trite and dull as the Linux kernel.

It does leave some things up to the imagination for actually using it, and the community of early adopters had compensated by writing graphical launchers, frontends, for it.

Now, I had previously built an arcade cabinet of my own from scraps found dumpster diving. This in a spacious 16m² apartment. In the process I stepped on a soldering iron; hot glued all the fingers on my hand; electrocuted myself on the anode of a CRT and slept in enough MDF dust for it to be a cause of concern.

First Appl: Gridle

The ‘frontend’ part to my bespoke arcade machine was just some list of games and a cursor. Selecting an item launched an emulator with some game specific settings and that was it. So I set out to write a new one with all the bells and whistles: video previews; related art and historical description; sound effects and background music; filters; video recording and streaming; save state management; input mapping and so on. Basically what Steam does today, excluding the ‘purchase and download’ part, which is apparently how you make money. Oh well.

The unconventional part was that instead of just launching the emulator and letting it do its thing, the frontend scripts should route input and control composition effects. You know, like a (foreshadowing) display server. In the end, Gridle had the game running on the monitor portion of a 3D model of the matching cabinet itself; networked remote control; on-screen keyboard; dynamic LED controls; CRT and Vector display simulation and more.

I don’t have any recordings or demonstrations of it live anymore, but the code is still around (yuck) and here is a photo of it in action on the cabinet:

It is also briefly shown in this slide from the first public presentation:

A detail here is in the screenshot of the login screen from World of Warcraft: Mists of Pandaria (so around 2012). It is actually running WINE with some process injection hacks, but composed inside Arcan:Gridle with the CRT effect added. This was another sidequest where I was coding bots using only injected mouse/keyboard events and computer vision to automate finding game-logic bugs without triggering the ‘Warden’ anticheat system.

That’s a story for another time — Gridle had run its course; about 90% of the actual API had landed, the IPC system did what it was supposed to, interposing emulators gives you a fantastic test-set for latency thanks to tool-assisted speedruns, and it was time to step things up a notch.

Second Appl: Arcan Workbench (AWB)

The Gridle frontend part worked fine for my gaming habits and for my movie and music collection on the home theater PC, but the desktop computers were still running Window Maker on Xorg, so that was next.

My second computer was an Amiga 500, and I had some lingering nostalgia for the earlier versions of the ‘Workbench’ windowing system with its oddly shaped icons and eye piercing colour-scheme. Could I recreate that, but bring in all the bits needed for my actual day to day work?

There is less that I need to say about it since the demo video is still alive:

In the end there was a native web browser via NetSurf; the video and recording tool could do interactive sharing of the tagged windows over VNC; it ran natively on the proprietary stack of the first generation Raspberry Pi; It was still portable across OSX, Windows and BSD along with a non-public fork that substituted much of the Android ~4.3 userspace, and worked on a handful of Sony-Ericsson devices.

It became my daily driver both at home and at work. Now, it was time for the project to move on to the next phase.

Phase Two – The Established Order

The ‘fun and games’ phase was a convenient but fleeting retrocomputing romance. Convenient because you get to do things without thinking too hard. The less romantic form where you do have to exercise the old noodle a bit is understanding legacy to try and improve things without ending up just slowly repeating history.

You can try and cheat and just make fun of the elderly without actually doing the work — “you looked but you did not see” as Dr. Lecter put it. The downside is that it will eventually come back and bite you in the rear, repeatedly. The real work takes understanding why something worked a certain way, not just copying what it did.

I set sail for another voyage of the reverser and headed into battle. After scouring through millions of lines of Xorg and Chromium and Xterm code; disassembly; related specifications and mailing list flame wars, returned a scarred man. Perhaps even a wiser one, but I have my doubts.

The choice in targets was deliberate. To the uninitiated these things seem different. They are not. They are instances of the same ‘feature archetype’ and SCADA- class systems.

I have already written more than enough on the matter in older posts; should I cherry-pick some posts, It’d be:

Sunsetting Cursed Terminal Emulation

Arcan vs Xorg: Feature Parity and Beyond

Arcan as Operating System Design paired with Arcan Explained – A browser for different webs.

The actual applications for this phase are much more diverse and latch into each other in some less than obvious ways, so lets break those down a bit.

First Appl: Durden

The workbench had outstayed its welcome. Sneaking a peek at a colleague’s workflow that was centered around tiling, there was a brief moment of ‘this is nicer’ but also a number of rough edges — and I wanted something else. It just so happens that I have the right sanding equipment.

I took a step back and considered the situation. All window managers are just an event trigger, like a button press, into re-stacking- and re-configuring- boxes of pretty pixels around in all kinds of wonky ways. The building blocks the windowing system provides will steer how hard that turns out to be.

I am normally deeply allergic to ‘everything is a XYZ’ shtick, especially so for ‘files’. Pipes and filters designs are appealingly simple at first glace, but degrades fast as the number of steps in a pipeline increases. Here there was actually a case for it, as you don’t need to stitch that many operations together to mimic any normal window management scheme.

So that became the building block for the Durden appl. It’s a file system where the files are either of a value type, like global/config/visual/font/size=14, or an action target/window/clipboard/paste. Every input action, decoration button and so on are just bound to paths. There is a user facing interface for navigating it and binding things, but can just as easily be mounted or accessed over a socket; a bash script into socat would be sufficient to implement most traditional schemes.

The old clip below shows the use of the in-UI menu to navigate and active one out of, currently, ~700 different paths: slice-region to new window. The workspace is running in a tiled mode where media windows are set to scale to fit, tile allocation, preserving aspect ratio.

Second Appl: Senseye

The Senseye project is the first iteration of what I actually wanted to build way back when. The last signs of public life was covered in Digging for Pixels and Senseye 0.3. It had sensors, translators and representations.

Each discrete sensor samples a data source, like a file, process memory or network stream. They come with basic controls for seeking, sample window size controls, pre-processing filters and so on.

The translators takes a sample buffer of some size and packing, and generates a higher level version of it, like image decoding or machine code disassembly. Since these are parsers that are working on untrusted data, they are strictly sandboxed and any crashes act as signals for parser vulnerabilities that should be inspected further.

The representations are visualization and statistics tools, like mapping to histograms; coloured patterns; 2D- bigrams; hilbert curves; 3D point clouds; delta comparisons to reference patterns etc.

For development, the testing targets were things like process parasites (also known as disk-less malware) randomly injected into virtual machines hosting Chrome processes all tuned to the same website. The task then was to try and locate the tainted ones using only the tools above.

The released version had a user-interface with pieces from AWB joined together with some duct tape. The slightly improved version carried the same sensors and translators, but integrated as plugins (‘tools’) in Durden. Neither was a good fit.

The project got some attention from organisations on my naughty list, so unfortunately it was for the best to bow out and continue with this offline. It will stay that way.

Before finding a better windowing and control scheme, there was a slight detour with the next application.

Third Appl: Safespaces

This was a complicated one, but not for technical reasons. Virtual Reality (VR) had started its next cycle towards become the supposed ‘next great thing’. The odds for it succeeding this time around were not great, but at least it meant access to hardware that did not cost an arm and a leg just to test the waters. VR from a display server perspective is also a good validation ground for load balancing (mitigating denial of service) and latency (“motion to photon”) as mistakes here can be felt and even cause physical harm.

There were two things I wanted to cover.

The first was again related to Senseye and of a highly questionable nature; I might just have damaged my cognition in ways I still cannot put my finger on. In short, it was basically overloading my vision by projecting large (hundreds of GiB) unknown data streams unto point clouds, and see if patterns started to emerge. Oh boy, did they ever.

I was laying in bed for hours strapped to a VR headset staring at brightly coloured dots zipping past. The dots quickly turned into clusters that turned into types that turned into hierarchies and locations. After taking the headset off, I struggled with edge detection. A doorway felt like a solid object, and other shapes were simply fused together. It took a few minutes for that to subside, but it left a strong feeling of unease for much longer.

The second thing was different states of ‘memory palaces’. Our computers are still terrible at actually organizing all the little snapshots of life that we collect over time. They are even worse at presenting and navigating these collections. They are much better at collecting and sifting through other people’s collections for signs of guilt.

VR could provide the means for putting these artifacts together in ways that paint a larger and much more intimate picture. Natural compartments; rooms and spaces. First for you to be able to recall as a means of coping with the many pains of getting older. Later the same would serve as something to leave for future generations to see and get a stronger idea of who you were, a place of remembrance — the virtual cemetery.

Something this intimate should be in your strict possession and under lock and key. To that end I drafted Safespaces. The complications came from Facebook Meta sucking all the air out of the room, and bringing all kinds of opportunistic fuckwits with them.

So I framed the Release Post in about as dry and mellow way as I could, showing only the dumbest thing I could think of — “tiling window management in space”.

As an aside, the Arcade cabinets from before can be spotted at the very end of the demo clip, and the much maligned background music selection, ‘Den Indre Kilde’ has a dark and depressing story to it, to stress the ‘not so safe’ part.

The post still got way too much traction and brought all kinds of abuse, including, I kid you not, being sent “I know where you live” kind of photos from the outside of my apartment for apparently being ‘woke’.

I decided yet again to continue offline. It will stay that way.

Fourth Appl: Pipeworld

Back to the ‘better windowing and control scheme’ then. The next application is hopefully the answer I was looking for, and something I still wish I had more time to chip away at. That application is ‘Pipeworld‘.

The backstory is that during a semi-regular lunch thing with a ragtag band of adorable misfits, someone joked that ‘in the end, everything computing degrades into Excel’. My rusty cogwheels started turning — what if we start there then? where will we end up?.

Each cell is a command-line that morphs into the result of running it. Expand function evaluation with process execution and profit? Sure there will be problems with the devil, synchronization, but that is something for future me to worry about.

A quick scan for prior art lead to Hishahm’s Doctoral Thesis, “Dataflow Semantics for End-User Programmable Applications” and its related “Userland” project. I played around with it for a few hours and concluded that the concept itself is more than viable, but the cell management quickly becomes visually crowded and the processing itself scaled poorly.

As a long time fan of infinite canvases, layering in a non-uniform Zoomable User-Interface (ZUI) seemed workable enough. A few weeks later and I had most things in place. It was a big improvement for my sensors and translators. I put it on the back-burner and continued with the last two big ticket items on my checklist; a better networking protocol and removing the terminal emulator middle man.

Terminating Terminals: Cat9

I will not waste time repeating all the terminal emulator painpoints; the Sunsetting Cursed Terminal Emulation article already covered enough of that terrible grind.

The main result of that long chain of work was that I finally had:

1. A client – server buffer transfer format that was text and formatting hints.
2. Server-side sharing of glyphs binned by font, style and size regardless of client, as well as on-demand transfer and ‘race-the-beam’ single-buffered scanout.
3. A replacement to ncurses/libreadline/… with appropriate Lua bindings that was actually nice to use and cooperated with the outer windowing system.

These were necessary parts for building Command-Line Interfaces and Textual User Interfaces that are so much more capable, faster and efficient than anything else out there that it is not even funny — and no exploitable in-band signalling hacks.

The final boss was a shell to actually take advantage of it. This is much more involved than one might think from a quick glance at a humble command-line prompt; the default data processing is now asynchronous and event driven.

To illustrate that with a clip of something completely trivial:

Here you have a Cat9 shell to the left, and a terminal emulator to the right. Both run find /usr. It’s the same actual command execution that happens on both sides. With Cat9 you can see how it gets a compartment of its own. Since it is a long running and noisy process, it automatically switches to a summary view of the ongoing data collection and expands when completed.

That cuts down on pointless updates which would otherwise take up precious network bandwidth when running remotely, and a lot of processing and re-rendering in the local graphics stack. Even with the terminal variant getting a head start it completes the job in last place.

I can toggle the summary on and off; continue to work on other things in the same session; interact with the collected data by attaching content triggers, filters, search, copy in/out and so on — all while it is still running and the prompt is alive and well. Had the accessibility features been activated, my screen reader would just give me the essentials and not get back-pressured to hell and back trying to read out some 73MiB of text.

This is my current baseline for absorbing automation and building workflows, and has been that for the last few years. Alas, just like with Durden, the distance between the current state and what has been published, is hopelessly long.

Phase Transition

I am glossing over a lot here, as this phase alone was, conservatively put, over 30.000 man hours of work. Some things worth referencing, at least, is the work on Accessibility, Crash Resilience, the IPC System and the Networking Protocol.

The high level conclusion to this entire phase was summarized in the recent post on Arcan Explained: A Browser for Different Webs.

Now we have almost caught up with the story so far. It is time for the next phase transition, some of which was foreshadowed in the post on Arcan-A12: Weaving a New Web.

Phase Three – Asylum

This brings us to today and the self-evaluation part I mentioned at the beginning. Do I have it in me anymore? Yes and No: The intellectually lazy overuse of neural nets killed my interest in Computer Vision. The crypto-bros took my interest in Cryptography Engineering away. Now, the slop jockeys take away my interest in general computing as they cover everything in a thick, oily residue of Silicon Valley soot, smearing the collective art project that was free and open-source into a blurry mess. This will force me and those of my ilk to adapt and pick another route.

To see your freely shared and well-intended hard work be chewed up, ruminated and repeatedly spat back in your mouth leaves a discouragingly bad aftertaste. To add insult to injury, you are then asked to pay up in tokens. While this is going on, a deafening horde of dowsers keep yapping in your ear about how great their particular slop machine does the job. That is incredibly rude, and discourtesy is unspeakably ugly to me.

There is still a time window in which to archive and preserve what once was, but the noise floor is reaching the ceiling. Affordable parts will become scarce as the powers that be try to five finger death punch personal computing into an age verified, identity proven hellscape of short form content.

I still intend to carry this limp sled dog of mine across the finish line before bowing out and retreating back into the shadows; for my friends, for my brothers. I had three applications planned for the final phase, but that will be reduced to one and a half. Too bad, the others were pretty nifty. The set of optimization and hardening techniques I wanted to try out for 0.8 and 0.9 will also be cut short. NLnet (A12-Endpoints) has graciously agreed to sponsor a substantial part of what is missing, so there remain some incentive in all of this.

But onwards and upwards, thanks for reading this far. See you next time in ‘A Farewell to Hubs’ where we will cover how the project will be maintained and managed in a post-Github-post-Discord world.

As the four horsemen of the applpocalypse rode off into the sunrise after five days of frolicking in the middle of nowhere, a spiritually Danish part of southern Sweden, Bjorn the #808080 was left standing staring into the dust cloud they left behind, contemplating the how’s and why’s of the event. The following video clip is an artist’s rendition of that.

In terms of actual work, this time around was less flashy than the last (part 1, part 2) as everyone already had their thing to work on and we are in more of a ‘spit and polish’ phase than one of ‘adding more fuel to the fire’. Newcomers (Mirco) had to be on-boarded, and Valts ran around stomping bugs like a kid from that one recruitment ad in Starship Troopers.

The first noteworthy update comes from Magnus and his works on the QT-QPA that is getting much closer to being complete (as far as the leaky platform abstraction in QT permits). The screenshot below shows some of the more complex clients up and running.

Alex continued with the Gamescope backend towards being able to sandbox the myriad of sideband IPC it is so enamoured with (X11, Wayland, Pipewire, D-Bus, …). This time he had succeeded with redirecting Pipewire audio output into shmif so network transparency and recording/streaming doesn’t stay silent, hopefully in a way that can be re-used elsewhere.

Björn opened with a hearty “great news everybody”; Just as the current NLnet grant is close to finished, we received confirmation about a new one. The details can be found at https://nlnet.nl/project/Arcan-A12-endpoints/ . The downside to this is that it adds another year before he can pack his bags, disconnect from the Internet, forget about 40 years of computing and live out what little remain of his life woodworking and farming chicken.

For actual hacking he poked at ways of using arcan-net to submit tickets and drive-by patches as well as bringup of “consort”, the Arcan appl we will use to replace Discord with in the near future. At the last breath of the final day it had passed the IRC- like stage of things, which was celebrated with everyone trying their favourite variants of SQL and prompt injections and generally trying to break things — to no avail. This lead to further celebrations with breakout singing and dancing to our spiritual theme-song of “Støj Snak — White Male Middle Class Blues“.

The gory details and master-plan will be kept for a while longer, look for an article on “A farewell to hubs” in the near future.

Also 4 out of 5 people spent way too much time troubleshooting Nix related issues. The one that didn’t simply wasn’t using Nix.

On the social side things were also a bit more scattered than usual. Even though people were physically in the same time-zone, individual circadian rhythm was not as forgiving. Most of the time at least one part of the house was snoring. Björn being the gracious host he is, dealt with the problem by simply not sleeping. A solution that surely will work out long term with no adverse health effects.

To assist with that task, a reasonable amount of club mate had to be consumed:

Tacky sci fi movies were watched, and pinball was played:

We will less-than-eagerly async-await certain depressing world-events before planning the next one, but there is a non-zero chance that there will be some project presence at BornHack and elsewhere before a fall session into 40c3.

This is the concluding follow up to the article on ‘Arcan as Operating System Design‘. When combined, we have reach the end of trying to explain what Arcan actually ‘is’ on a higher level. After this we can revisit the topic in a first order form as ‘Implementation’.

In the first slides from the only public presentation (2015) we claimed that the idea behind Arcan was to find the crunchy middle between a display server, a game engine and a multimedia processor.

The control plane to this “desktop engine” was designed for a scripting API targeting entry-level developers. Lua remain as the weapon of choice for that role as a better follow up to the ‘BASIC’ of the home computer era, but also as an intermediate for other tools to compile to.

To make a long story short: That, my friends, is a browser; just average in a few areas, yet create something interesting out of orchestrated mediocrity — not unlike musical theatre.

In order to get anywhere on such a broad topic we will need to limit the scope a bit or we will be here all day. The web itself is as much, or more, social structures growing and evolving as a bacterium on a technological substrate than it is any singular piece of technology.

Even if some cry out in doom and gloom about the dominance of Chrome ‘the browser’ (albeit also Chrome as Chrome OS and Chrome as Electron), do remember that it is still subject to other pieces like the server, the protocols, the app and the structural glue that is ‘the link‘. If you want to offset one part, you ought to consider how well it plays with the others as well.

This article will only focus on the technical side to the user facing access to this substrate. That is the browser software itself. An example of the makings of another ‘web’ to go with it, sponsored by NLnet (https://nlnet.nl/project/Arcan-A12/), is covered in ‘Arcan-a12: Weaving a Different Web‘ but it is architecturally loosely coupled.

To make things a little easier than what is normal around here, I will split it into a simplified ‘short form’ and as two heavier text sections.

The first section, Browser Breakdown, covers the problem space and some cherry picked history. The second, Arcan as Browser Design, goes into how we account for it.

The browser broken down is the end result of “the document” trying to envelop the computing world. The browser as proposed is “the desktop” trying to reach out.

The short form can be found here:

To demonstrate that this is existing technology, the presentation is also provided as an Arcan appl, viewable through “arcan-net arcan.divergent-desktop.org explain” (it is running on a potato used for active development).

That one happens to have been published a while ago. According to the logs, about two people have managed to find it. It seems there are lessons from the past about other meanings of “Search” that might be useful to rediscover now when more and more people and places alike are all llmpty on the inside and the noise floor is reaching the ceiling.

Some suggested extended reading on the topics that will be covered here and in the next section:

• The History of PDF (historical, light read)
• Information Management: A Proposal (1990) (historical, intermediate)
• NeWS: A Networked and Extensible Windowing System (video, pdf) (historical, advanced)
• Inside look at a modern web browser (current, multiple parts, light read)
• SOK: On the Analysis of Web Browser Security (current, advanced)
• A File Structure for the Complex, The Changing and the Intermediate (historical, advanced)
• Five Walled Gardens: Why Browsers are Essential to the Internet and How Operating Systems are Holding Them Back (current, political)

Browser Breakdown

The big question to unpack is if the browser is supposed to ‘present and navigate interlinked documents?’ in the spirit of the original much-too-simple hypertext, or be ‘an operating system agnostic substrate for networked applications’.

Lets start with a hypothetical speedrun through history.

You are back in the cosy times of the early 1990ies when computing was slow, storage was small and floppy disks got corrupted simply by staring at them for too long. You are struck with the brilliant idea of developing a software product that is supposed to just show very simple documents with formatted text in various typefaces, sizes and styles. Throw in the odd image or two. Maybe some media clip, but that can come later.

These documents should naturally be hosted and shared across networks of computers as that is faster and more convenient than copying things to floppy disks.

It turns out great with a lot of satisfied users that soon wants more. At the same time, windowing systems become more common. These force your simple document viewer to support moving things around to react to the window resizing. Users naturally want to handle different outputs, such as printers.

The documents these users have created started to form packs and people wanted ways for them to cross reference each-other and navigate these references with the click of a button. You oblige by adding ‘links’ to other documents and the option to open them in a new window.

Networks at this time are slow and servers are quite busy, so the next logical step would be some caching to make sure you only retrieve a linked document or its embedded images when something has actually changed.

A ‘popular’ kind of document turned out to be ‘forms’ for people to fill out, print and post because those are like cocaine to governments and businesses alike. With that you add some way for the user to type into parts of these forms to fill them out before printing. This will save many from wasting time deciphering messy handwriting. Both throughput and accuracy improves greatly.

Still people manage to fill them out wrong — to err is human and all that. It would save everyone time if you just add some easy way to have presets to chose from; a list box here and a radio box there. Soon enough every UI ‘widget’ imaginable has managed to join in, but you still think of it as a ‘document’ in the end.

Why even go through the pains of printing and posting? We can just sneak in some magic to POST the filled form as an e-mail or send it to the same place the document was hosted. Even better, we can embed some code to make sure that some of the inputs the user provides can be checked for errors locally and tell the user about any errors.

If something is made programmable, it follows that some crazy hacker kids will figure out creative ways to take advantage of this to do all sorts of unintended things. It did not take long before the documents started spawning little cats chasing the mouse pointer around.

Documents reshaping themselves as they were being viewed became the new normal. That is a big change in expectations for all stakeholders.

You should be able to see where this is going by now — It doesn’t take that many feature requests for a static document to become a dynamic application, similar to how few wrong turns it take to make a regular language grammar into a recursively enumerable one.

To keep up with new discoveries in everything from image compression to network communication lots of new code got written in order to support new media formats and more advanced protocols.

The easiest way to make software do new things is to add more code, and the path of least resistance is typically the one people follow. This is where software architecture matters and stops being just boxes and arrows on a whiteboard somewhere; it sets the constraints for how new code combines with the old.

Just adding more code to solve for new features rather than refactoring to be more clever about things comes at a cost. More code means more potential bugs. Mix in economic incentives and some of those bugs turn into opportunities to exploit which calls for mitigation strategies.

One of the more potent ones are sometimes called sandboxing (a terrible term but that is another can of worms best left unopened) and comes with heavy requirements on security boundaries. Re-architecting to do that successfully is not for the faint of heart or light of wallet.

In the browser case, such a project comes with another interesting side effect: you are now compositing and routing inputs and outputs between a number of interactive sources and sinks across several privilege boundaries.

That is how you end up with something like this:

Congratulations, you’ve made a display server with the weakest windowing system in town. Instead of leveraging the IPC system for better ways of collaborating with local software they stuck to crude left-overs from the outer windowing system like clipboards, drag and drop or, sigh, “portals”.

A historically interesting tangent is that Microsoft was on this trajectory way back when and got close to a stronger solution through its COM/OLE/ActiveX path before falling into the antitrust trials pit (not that the idea went away).

Now every few years there is some take on the “Java Applet” form of assimilation by forced cross compilation to an embedded executable format referenced from the document (which turns the document into a de-facto format for linking and loading yet with few of the insights embedded in proper executable formats) to target a virtual machine. Shockwave Flash being a notable one, and “WebAssembly” a more recent attempt.

What if, instead of trying to regress towards “simple” in the browser domain (just to see the same story repeat itself) or in the data model (and still be stuck with the same browser story) – we assume the networked application endgame from the start and design for that?

This leads us to …

Arcan as a Browser Design

To summarise the previous section – the browser started as linked documents breaking down into a networked application with ‘the document’ organically turning into ‘an executable format’.

In a spider graph form they went from this:

To this:

These are, of course, rough estimates. Improvements along each feature axis is exponentially more difficult and expensive.

The expected and required feature-space span the entire gradient:

It is worthwhile to note that it can transition from any point on the gradient to another. You get no guarantees about where you start, or if it will stay that way, unless you disable expected features with unpredictable results.

We can say the processing is iterative and the linking (as in program execution) is dynamic and mutable (events triggering code pulling in remote code hooking events).

In contrast, Arcan started at roughly this back in the early 2000s when hope was high and life worth living:

It is now here:

The processing looks like this:

We can say the processing is planar, static and recursive. With planar we mean that if there is a document stage, there is a discrete and decoupled transition from that into the app form. You know, a build system: compile, link/package, execute.

This means that the “as document” part of the runtime are separate tools not part of the browser itself. It only concerns itself with being an execution environment for the networked application. This keeps the browser part from having to walk an endless trail of outdated formats. Some are already conditioned to this approach with that of “static site generators”, it is just the output format that needs some change to move away from document to document translation.

With static it means that the compiled app is not capable of loading code outside of its own package, except for a set of preset helper scripts. This is the default with a user controlled opt-in rather than an opt-out as in “install extension to disable javascript”.

With recursive it means that navigating a link starts another instance of the browser that would either embed itself as any other data source in the parent, or take its place.

The emphasis on the display server aspect throughout the years here is no coincidence: it’s the weak-point of the dominant browser model and one that has very little documentation. It is also the reason why the apps developed and demonstrated here has mainly been desktop oriented — to politely point out that “you missed a spot”.

Chrome splits its app execution into a number of processes that are roughly “Network”, “Browser”, “UI”, “Storage”, “GPU”, “Device”, “Render” and “Plugin”. This is a logical breakdown of the internal division of responsibility. For privilege separation, each process role has a configured policy on which system resources it can access.

Arcan splits up its app execution into a number of processes by information transfer boundaries:

• Decode – computer to human representation (images, videos, music, voice synthesis, …) into GPU buffers, pixel transfers and audio sample streams.
• Encode – human to computer representation (recording, streaming, language models, object-character recognition, …).
• Network – name to address resolving, discovery and retrieving remote resources.
• Terminal – outer operating system space controls.

These come as a set of external clients that are picked for each instance – meaning that the supported formats, protocols and so on can be swapped out without any changes to the engine itself and per application basis. They are dependency and license “sponges”.

Their security model comes from a combination of least-privilege and capabilities. “Decode” is security wise the most sensitive one as that is where parsing of untrusted inputs go. “Encode” is privacy wise the most sensitive one as that is where the real ‘you’ distil into digital form.

After the connection is setup, it needs no file system access at all. The system calls required match an OpenBSD pledge of “stdio sendfd recvfd” which is a hard surface to break through. Even then we have microarchitectural attacks to consider in the most hardened setups. This is where the ability to live-migrate any shmif client to a networked form comes in — you can compartment to cheap one-off devices like single board computers and the architecture still holds. Memory is never safe, wear a helmet.

State store and list of permitted integrations with local software are in an SQLite database that is also picked for each instance. That local software can, of course, also be a different browser.

This becomes a hierarchy that looks like this:

Not pictured here are the ‘hook-scripts’ facility which corresponds to ‘browser extensions’ in the web sense and the ‘developer tools’ sense.

The key component is the inter-process communication system (IPC). The one in Chrome, Mojo, is highly generic: an interface description language (IDL), a binding code generator and there you go. The qualities of the IDL limits what can be communicated and the binding generator affects the developer effort it takes to use. Part of the interface description links in ancillary specifications. An important one is the sandbox policy to apply.

We have seen it before: MIDL, CORBA, Fuchsia IDL, D-Bus, Wayland, Binder (AIDL) etc. ad nauseam. One may wonder how many generic ways we may need in order to eventually be specific.

The pitch for those are always ‘extendable / flexible’. I read that as ‘afraid to commit to anything’. It is a tempting model if you are unsure what you are supposed to be building (or organisation decoupling where you want formal barriers for communication between teams) and kick the bucket down the road. It is less tempting if there is ample history and plenty of examples to learn from.

The generic approach come at a cost. Code generated creeps into, or even dictates, the design of both sides and it ends up much bulkier and more verbose. It is rare for ancillary tooling support for debugging, report generation, testing, profiling etc. to be included.

In a previous life as a security auditor, a very successful path to victory was to locate any IDLs in use; patch to integrate fuzzing; deploy and wait for things to blow up. They typically do.

The IPC in Arcan, SHMIF, is highly specific. It is a lock-stepped data model with a C API. The message types and fields came from the needs of a full desktop OS, which has changed very little over the years. There is no reason to be clever, flexible or extendable. Just look at what others have done before and generalise from that. Validate against existing examples and use cases. Easy peasy.

Where we did put effort into being clever was two parts. Crash resilience and Alternate Representations.

Both are heavyweight topics in their own right, a dissertation or three, but for this context the motivation for resilience is easy enough – it allows us to redirect user workloads to relaunched instances (recovery), to others or other devices across the network (redirection) for collaboration or stronger security compartments, or to a future version of ourselves (upgrade).

For alternate representations, well we don’t all look at the world with the same eyes or desires now do we? The one that should be obvious is accessibility where the screen reader tradition is to snapshot composition state back into a document form and then navigate through voice synthesis and keyboard controls.

A less obvious one is for debugging – be able to visually inspect and interact with the dynamics of the software rather than what it presents itself as. The other side of the same coin comes from malware reversing as that kind of work leverages the same kind of mechanisms. Process parasites (or as some call it, ‘Fileless malware‘) have been a thing for a very long time now, yet the generalised tools to shake them out have been sorely lacking for just as long.

In closing, there is a lot of technical detail that had to be omitted for this piece to not be completely indigestible but at the high level, there is not much more to say about the Arcan project as such. There are things left to do, but the end is in sight. Midnight draws near.

Closing the year before heading to 39th Chaos Communication Congress we decided to tag a release for the main project for those too conservative to run from the main branch.

For those on site that want to have a chat, our DECT extension is 6681. Incidentally the same number as the control port for when Arcan was used to provide overlay, texting, memes, scheduling etc. for a pirate university TV station back in the early 2000s.

Before going into the project updates, there is tragic news to relay that put a massive damper on spirits and overall productivity for the last few months: Our most beloved of project members, Elijah “moon-child” Stone died on the 9th of September at the young age of 22. Elijah had been with the project since his early teens and was consistently kind, caring, courteous and clever well beyond his years. The 0.8 topic branch on one of his favourite subjects, performance engineering, will be dedicated to his memory. Our thoughts are with his family and partner.

Community Updates

It has been quite a while since we switched from GitHub to Fossil for development. Since we don’t expect people to tool around an uncommon tool we also mirror to git hosted on Codeberg.

A final friendly warning to packagers is to use those repositories. The ones on GitHub will no longer receive any mirror updates and any changes over there are likely to be of a more incendiary nature.

As covered in more detail, (part 1, part 2) we have had a longer in-person hackathon. The main outputs from those is that Alexander has a port of Gamescope to save us from keeping up to date with all the special quirks that running Steam over Xwayland takes. The clip below shows it running Baldur’s Gate 3.

Magnus keeps chipping away at his platform plugin for Qt5/Qt6 that largely works as intended for the likes of Qbittorrent and Binary Ninja but struggles still with hybrid 2D/3D complex window managed applications like FreeCad.

He also created patches to KeepassXC and a script (added upstream) to Durden to integrate them. Both are recommended reading.

Valts is still busy with his portable viewer for the A12 protocol that should soon be usable with some of the neat bits we cover further below.

In the bin of experimental applications, we have Atro with “Lasso” that is a hybrid ‘interactive canvas’ form of window manager.

On top of providing a number of bug fixes across the board, Bohdan created Xkbd2Lua for statically translating X Keyboard Layouts to our own format, removing the need to lug libxkbcommon around for those that want even fewer traces of X11 around in their lives.

Ariel has been toiling away at one approach for a bootable complete Arcan+Durden+Cat9 setup from as static a build as possible matching Arcan as OS design (there will be more). As part of that there is a nix oneliner that should hopefully work on a few setups:

nix run --impure 'git+https://codeberg.org/ingenieroariel/arcan?ref=nix-flake-build&dir=nix'

Status Update: Arcan

As normal, check the changelog for the fine grained changes.

A lot has happened on the network side, thanks in large part to the continued support from NLnet and a longer write up for that is coming up in a little while.

To start with, there is now support for ML-KEM as Post-Quantum cryptography to protect against ‘collect now, decrypt later’ (should the fantasy computers ever materialise). This is implemented as part of the forward secrecy ratcheting rekeying process. The same has been extended to set a signature verification key for file transfers and a way to limit a high number of search requests during high load.

Connection resumption and Casting

Clients that act as sources now has connection resumption support. This means that if the network connection is lost, the source application is kept alive and re-paired when you connect back.

In the following clip I first host arcan running pipeworld on my networked machine and connect. The window pops up and I create a few cells to show that there is data/state remotely. I close the window and reconnect. The window reappears just as I left it.

There is also a –cast option added. This lets the first user that connects become the “driver” over the hosted application. Any subsequent connections gets a read-only copy of the stream.

Unified and Referential Links

The other major network changes have to do with the directory server part. To recap, any endpoint can have one out of three roles: Source, Sink or Directory. Normally a source hosts some kind of application and provides, either inbound or outbound, access to a Sink. The Directory works as a self-hosted rendezvous for discovery, but also state, file-store and coordination between multiple clients using the same appl.

For instance, if I have a directory server hosting the Durden desktop appl and multiple clients download and run it from the directory, they can use the messaging domain that the directory provides to synch clipboard state or share input devices like one would do with Synergy/Barrier in the past.

The admin API for configuring the directory server has received two new functions, reference_directory for referential links and link_directory for unified links.

A referential link lets users with access to directory server ‘myserver’ access a referenced directory ‘myfriend’ and so on, letting them browse a larger network. Using the command-line tool, the following example:

arcan-net --path myfriend myserver@ myappl

Would have the client connect to myserver, then download and run myappl from the referenced directory myfriend. The path can/be/arbitrary/long. Connection primitives are negotiated by each step in the chain, which takes us into the very interesting space of transitive trust-discovery models.

In the example above, myserver gets a DIROPEN request for myfriend from the client. Myserver knows that this is a referential link, and asks myfriend for connection primitives on behalf of the client – forwarding the public key previously used by the client to authenticate to myserver. Myfriend returns with either direct connection information (ip, port etc.) or a request that myserver tunnels the traffic if it is not directly reachable.

The unified link is invisible to the user and is a more privileged connection. It lets multiple directories form a shared namespace such that they can access/host/mirror the same resources as if they were one logical server.

Say that you have a server in your home and another hosted on a VPS somewhere. With a unified link between the two, your devices can access the one when you are at home, and hand over to the other when you go outside.

Dynamically Hosted Directed Sources

For every directory hosted appl it is possible to slot in a controller. A controller is a set of server side scripts that regulate messaging and resource access for more advanced networked applications. The scripting API has received some new functions that are worth looking into.

The first is launch_target. This is best explained through an example. Say my directory server configuration database has a launch target defined, like:

arcan_db add_target "chromium" /usr/bin/Xarcan -redirect -exec chromium

I then slot in a controller for the ‘durden’ appl by doing this:

# mkdir durden; echo '
function durden_join(cl)
  launch_target("chromium", {}, cl)
end ' > durden/durden.lua

# arcan-net --sign-tag mytag --push-ctrl durden myserver@

This would package the controller for durden, sign it with whatever is assigned to mytag in the local keystore, and upload it to the directory server marked as myserver in the local keystore.

The server verifies permissions and that the signing key match previous signatures, and assigns a runner VM. Now the next time any client would run the durden appl:

# arcan-net myserver@ durden

It would spawn an instance of chromium that connects as a source only visible to the specific client, with a polite suggestion to sink it immediately. The client does so automatically and the window pops up.

There are more controls to add here for state management, sandboxing details, letting the controller script inject events and so on. The point of the feature is that the beefier server now has a mechanism for fine grained application hosting.

Speaking of, if the client end doesn’t have the full Arcan stack but something simpler like the Smash viewer, it is now possible (opt-in) to let the directory server host the arcan side of the equation turning the client into a very thin one:

# arcan-net --host-appl myserver@ durden

External Resource Resolver

Other changes to the controller development side is that the event handlers used to to list/download/upload files can now be hooked up to an external resolver.

The way it normally works is that you add an event handler, like this:

function durden_load(cl, name)
    return name
end

This does nothing fancy, it just forwards whatever the client requested to the server-side file-store. The function in this form is just to add any client specific block/name translation. Another option would be to return the result of open_nonblock to dynamically generate the data to be transferred.

If I modify the server’s config.lua to this:

function ready()
    launch_resolver("durden", "/usr/bin/myresolver")
end

Any resource requests forwarded from the runner scripts to the ‘durden’ appl will be routed to an external process. This can be any shmif client that implements a handler for the BCHUNK_IN / BCHUNK_OUT events. A simple test client can look like this: https://codeberg.org/letoram/arcan/src/branch/master/tests/core/a12resolve/a12resolve.c

Had this been the assigned resolver, storage requests would be rejected outright and load requests would get the ‘test.mp4’ file contents regardless of what was asked for. The point of this feature is to provide caching and translation to other file providers. Ones currently being looked into covers regular URLs, Magnet-to-torrent and IPFS.

Custom Debugger Integration

The remainder of changes are mostly on the developer side of things. Recall that we once wrote a debugger frontend to Cat9. This implemented the Debug Adapter Protocol, but was designed in such a way that we could support other protocols. DAP is too heavy for our own needs, so a lighter one was thrown together to match how we use the Lua VM in both the engine and the directory server. This lets us debug locally as well as with a corresponding directory controller in lockstep mapped as remote threads.

The following clip shows the current state of that in Cat9:

It can also be attached on-demand on a scripting error. There is some heavy lifting left on the frontend side before this is completely seamless, but in a not too distant future we will be at a point where you can pause-edit-update-continue a fleet of devices running a single Arcan application at once.

With that we have all the building blocks in place for more interesting networked Arcan applications. The first target for that will be a community chat application to move us away from Discord.

The second half of the week-long session started with us waking up with a hole in our collective hearts: Alex had absconded in order to participate in a meeting way up north. He bravely overslept (brave brave Sir Alex) and missed that, regretting most decisions along the way. Such is life.

48 of the bottles of Mate did arrive, so to fill the hole we speed-ran through them and none us will have another bottle again, or until the next time — whichever comes first.

For show and tell it was finally Björn’s time to shine like a very distant and faint star. First squishing a number of long standing bugs, such as primary screen sometimes not getting the right identity/serial/make/model at startup breaking monitor settings persistence; unreliable transition between 16-bit, 24-bit, 30-bit and fp16 composition modes; cat9 subwindows failing to materialise when hosted over a network and directory server hosted sinks randomly shutting down (spoiler: always check your signal masks when execve()ing).

Then, apparently, he had snuck in a little edit mode into cat9, promptly overdoing it by adding vim style input modes, folds and language support. The following clip captures the current state.

The tangent was justified by some long rant about feature archetypes and a pilgrimage in search for the answer to the existential question `if a text editor is predestined to eventually grow a shell, and a shell is predestined to become a text editor, will they end up in the same place?`

With the aforementioned fixes we saw first signs of arcan-net -l 6680 -- /usr/bin/gamescope supertuxkart running, with a little caveat left to fix:

The details are fairly interesting. When trying to pass accelerated buffers from a shmif source, it is always possible for the other end to reject acceleration. The buffer passing routine then tries to gracefully convert it into pixels and try a slow path. This does not currently work when the shmif context structure is not actually managing the GPU, as the screenshot illustrates. The network tool (arcan-net) currently always rejects accelerated buffers and tuxcart-on-psychedelics it is.

To celebrate its 30 year anniversary we watched the aptly named hacking movie ‘Hackers’. Magnus confessed to not having seen it even once. Oh to be young again, and also a robot.

Speaking of that, the QPA 3D support commits materialised, then proclaimed to be working flawlessly, only to promptly break as soon as others tried it out. Somewhat disheartened by the fact and followed by “I fixed the race condition now but sometimes it goes black for some reason” on repeat, he asked for help and help was given.

The following clip is an artist’s rendition of Björn providing sagely advise about swap-chains, fencing and buffer lifecycles during resize.

Valts, meanwhile, started his days with the work from the previous night unavailable to him. Swallowing a large dose of what the kids would call copium, “It is not Nix. There is no way it is Nix. It was Nix.” could be heard.

This was promptly followed by adding a UI for configuring smash that we will opt to not show here to spare any would-be-UX designers from feeling overwhelmed.

All in all the participants confessed to having mild to moderate fun, with faint hopes of meeting up again at the next Chaos Communication Congress – should we manage to score some tickets.

We are a few days into the 3rd Arcan IRL hackathon.

The theme this time around is “Project Wayhem” with hem meaning home in Swedish, and it taking place in my renovation project/eternal money sink fort out in the middle of nowhere. Three brave travellers fought the perils and outrageous ticket prices of public transport in Sweden and came out victorious.

After an unhealthy amount of greets, rants and assorted beverages, we eventually managed to get to work.

First up in the show and tell was Alexander with his SHMIF backend for Gamescope that got massaged into working shape. The repository can be found here: https://codeberg.org/sashabjorkman/GameScope

The following clip shows it in action, playing Baldur’s Gate 3.

He also commissioned an artist for an event-appropriate mascot:

Scandinavian hedgehog being the only non-rubbish wild animal currently living in my backyard.

While we have paths for running x11 and wayland clients already, such as Xarcan and our native wayland support, nobody wants to work on the wayland implementation and the Xorg DDX works well enough for clients with reasonable use of desktop resources. Steam is not such a client and definitely not reasonable.

Since the goal-posts for anything “”Wayland”” these days are mounted on roombas powered by jet engines, the best effort / reward solution is Gamescope: write a backend for that and let them exhaust their resources chasing down the complementary IPC system of the day or the next ‘protocol’ extension to join the animal farm.

We celebrated this victory with a few runs of the true eternal game of the year (tuxkart) on the only machine out of 10 or so with functioning bluetooth support for the controllers. Instead of the 2.4 meter projection screen in a calibrated viewing environment, we thus huddled around the tiniest laptop screen around.

Next up was Magnus with his Qt QPA (found here: https://codeberg.org/vimpostor/qtarcan). He fought bravely, and eventually won, getting 3D accelerated windows working. This is one of the few remaining items before the fabled feature completion.

After a number of hours staring into the void and the void staring back, “fucking hell I can finally see my application, this is huge” could be heard from several kilometres away.

While everyone is waiting for the commits to materialise, the following screenshot is proof that he is not a pathological liar:

Valts (thanks NLnet!) being true to his nature, was much less dramatic and kept toiling away at the alternate, portable and lightweight implementation of our network protocol, that can be found here: https://smash.tase.lv — inching closer to supporting multiple windows. He was also trying his best to say good things about Nix and that was promptly ignored.

Björn (thanks NLnet!) while not busy making good use of his many academic degrees by cooking; writing this blog post; crafting cocktails and preparing snacks for each movie night, spent his time trying to debug why every n’th modeset between SDR and HDR caused the amdgpu kernel module to crash. He got absolutely nowhere and remain a disappointment to most everyone.

To salvage some remaining shreds of dignity, the 8+ year old backend for SDL2 got resurrected.

The purpose of that is threefold: One is to pretend to actually do something useful. The other is making sure there is an reference to compare to when eventually repeating the process for SDL3. The third reason is simply that we need more varied clients to test network transparency and local/remote migration with.

With several days to go, there might be more things to report on in the near future. With any luck, the 60 bottles of Club Mate ordered weeks ago might actually arrive before everyone heads back.

This is the final part concluding the long journey on how to migrate away from terminal emulation as the main building block for command-lines, text-dominant shells and user interfaces in general.

First, a few links to previous parts:

• Chasing the dream of a terminal-free CLI (2016, rant)
• Dawn of a new comand line interface (2017, architecture)
• Day of a new command line interface: shell (2022, design)
• Lash#Cat9 (2022, proof and development)

On top of those, and the many related demos, we have specialised tangents on accessibility and networking. The only remaining thing we have yet to cover is the programmer’s interface to all of this.

Before slipping into that dark night, I would like to summarise the mental model I use for ‘terminals’:

Terminal emulators emulate a machine that has never existed, a fantasy computer. They are constructed just like other emulators for the weakest of devices; a big old state table and switch (*pc++){case OP_1: implement OP_1(); case OP_2 ...} with a quirk being that the real horrors are hidden elsewhere. Many indulge in expanding this fantasy computer, adding all kinds of crazy expansion boards for faster video modes and co-processors. This is nothing new in itself — we have done that to practically every computer ever.

The tragedy is that it is taken seriously as a default.

Another day at the office is just running a terminal emulator (xterm) inside a terminal emulator (console) so you can use a terminal emulator-translator (ssh) to emulate more terminals (tmux). It is emulation all the way down instead of as a last resort for compatibility.

The other quirk is the flora of instruction sets to emulate and the subset selection heuristics. These are downplayed as ‘just text’ but everything that touches American Standard Code for Information Interchange are raw instructions for this machine. Programs for it are everywhere and most of them transform or expand the code, without being treated with the same scrutiny as one would for other code generators such as compilers.

A related take worth highlighting is this quote:

It feels a little unsettling that what we now use, effectively, as a data encoding format (ASCII) continues to hold – and it’s up to us to handle when writing software, too – things that correspond not to textual data, but TTY key press and serial link events, and that they follow you everywhere, even in places where the are no teletypes, no serial links, and no keys. Like most people who poked at their innards I used to think that terminal emulators are nasty, but for the longest time I thought that’s just because we took the “emulator” part a bit too seriously and didn’t bother to question it because we’re intellectually lazy. Now I realize it’s legitimately hard to question it, because a lot of assumptions are actually baked into the damn data model – terminal emulators aren’t leaky abstractions, they’re quite literally the abstraction being leaked :-D.

x64k@lobste.rs

Computer people have tried to tame this beast before, using abstraction libraries such as TurboVision and Curses (hence the title of this post). One might be tricked into thinking that keeping these interfaces and substituting the implementation with something befitting the real computing environment would be enough. That is where the ‘leaky abstraction’ part meets ‘just text’ pops up like a little demon and plants itself firmly on your shoulder. Since it is so easy to sneak in arbitrary instructions in-band, people have done a lot of that, all over the place — going so far as to publishing cutesy little articles encouraging the malpractice.

Which building blocks do we have to plug the leaks with?

The Arcan project is suspiciously large in scope. Much of it is not needed for this particular problem. There are a few parts to single out, and a simplified build-mode (-DBUILD_PRESET=client) to assist with that. This will produce:

• libarcan-tui
• libarcan-shmif
• libarcan-shmif-server
• libarcan-a12
• afsrv_terminal
• arcan-net

[libarcan-tui] is the meat in this dish. It fits the ‘ncurses’ bin (.. and ‘libreadline’ and other widgets). There are also Lua bindings with some extra conveniences to smooth over other related POSIX details to help escape the clutches of /dev/tty.

Its reference implementation uses [libarcan-shmif] to talk to the desktop, using some variant of [libarcan-shmif-server] for the other half of that equation. One could substitute in [libx11/win32/…] for those, but some features make that quite difficult. For an technical walkthrough how those work, see the recent writeup of A deeper dive into the SHMIF IPC system.

To fill the portability gap across mobile and desktop, we have a (young) separate project, smash, managed by Valts Liepiņš. For the Zig capable, this is a good place to support.

[libarcan-a12] provides the reference implementation for [A12], the wire transport for this across devices. [arcan-net] providing a standalone binary wrapper for common use cases.

[afsrv_terminal] is the last piece in the equation. It is a standalone binary which, at first glance, is a terminal emulator(!) that outputs into [libarcan-tui]. That is needed for compatibility and proof of feature parity. When passed an environment (ARCAN_ARG=cli=lua) it ignores the terminal emulation part; enables the Lua bindings with some added functions useful specifically for shell work; and runs a bootstrap script that loads the real shell. An example of such is the Lash#Cat9 link from before.

Enough babbling. Time for some code.

We start with a low level C ‘hello world’, then repeat it using the Lua bindings:

#include <arcan_tui.h>

static void redraw(struct tui_context *C)
{
  arcan_tui_erase_screen(C, false);
  struct tui_screenattr text =
      arcan_tui_defattr(C, TUI_COL_ALERT);
  arcan_tui_writestr(C, "hello world", text);
}

static void resized(struct tui_context *C,
                    size_t px_w, size_t px_h,
                    size_t cols, size_t rows, void *T)
{
  redraw(C);
}

int main(int argc, char* argv[])
{
  struct tui_cbcfg cbcfg = {
   .resized = resized
  };

  arcan_tui_conn *con =
    arcan_tui_open_display("myname", "hello");

  struct tui_context *tui =
    arcan_tui_setup(con, NULL, &cbcfg, sizeof cbcfg);

  if (!tui)
    return EXIT_FAILURE;

/* draw something immediately */
  redraw(tui);
  arcan_tui_refresh(&tui);

/* main processing loop */
  int inf[] = {STDIN_FILENO};
  while(1){

/* block (-1) until there are events happening on a set of contexts or inbound data on a set of descriptors (inf) */
    struct tui_process_res result =
      arcan_tui_process(&tui, 1, inf, 1, -1);

/* synch any changes imposed by the event handlers */
    arcan_tui_refresh(&tui);
  }

/* shutdown, with no error message */
  arcan_tui_destroy(tui, NULL);

  return EXIT_SUCCESS;
}

There are quite a few nuances to cover here already.

For starters, opening a display matches the pattern used in EGL and elsewhere in using an opaque pointer to allow the implementation to support different outer display systems. It takes an immutable ‘name’ of the application and a mutable ‘identity’ (e.g. the current open document, path or other resource).

With the display, we can acquire a context (window) and set a table of event handlers. The table also lets the implementation know which features you support/need and which ones you do not.

Some of the possible handlers include:

• Input: Text codepoint, Abstract announced label, Native keys with modifiers
• Input: Mouse buttons and motion
• Window Management State: Visibility, size and colourscheme changes
• Acquiring new window contexts
• Language and location preferences
• Clipboard paste
• State transfer requests
• File transfer requests
• Seeking requests (horizontal / vertical scrolling)

Note: Passing the size of the vtable doubles as an additive version identifier. This makes it possible for features to be appended to the table without breaking backwards compatibility.

A tui context represents a single window. You can request additional ones from a set of types, and have ones pushed to you as a user request for alternate representations (mainly accessibility and debugging).

Calls to arcan_tui_refresh will atomically forward changes to the contents of the window. If nothing has changed it will return immediately.

When drawing we have a lot of shaping attributes to apply, from the regular bold/italic/underline/strikethrough to shaping hints, double-width characters, ligature substitutions, border drawing and foreground/background colours.

Note: Colours are a special case. The terminal tradition is ugly - there used to be a small set of reference colours, the emulator remapping them as desired (RED is now YELLOW) and several different extensions to specify explicit red/green/blue values or a wider palette. Here there is a semantic palette (ALERT, LABEL, ...) as well as the legacy one. The values they map to are tracked per window and can be dynamically updated by the outer display system (this triggers a 'recolor' event). They are resolved to red/green/blue values when the window is packed on refresh.

The process function takes a set of tui contexts, a set of file descriptors, their sizes and a timeout. The set of contexts is for handling multiple windows and the set of file descriptors and timeout for the common pattern of being input triggered.

Note: Since the UI processing is all out of band, STDIN and STDOUT are left intact and you can mask all signal handling. This removes the need for isatty() and you get both pipeline processing through an intact STDIN/STDOUT and an interactive UI. The choice of colouring or not colouring output is up to the final presentation.

In Lua, it would look like this:

tui = require 'arcantui'
local function redraw(wnd)
 local attr = tui.attr({fg = tui.ALERT})
 wnd:write_to(0, 0, "Hello World")
end

local handlers = 
{
  resized = redraw
}

wnd = tui.open("myname", "hello", {handlers = handlers})
redraw(wnd)

repeat
  wnd:refesh()
until not wnd:process()

There are a lot of bells and whistles in order to cover everything people have ever done with terminals, far too many to go through here. You have helpers for asynchronous transfers between file descriptors with progress updates; arbitrarily many cursors; verifying if a certain unicode codepoint has a representation in the current set of fonts; spawning new processes with embed-able windows; window manager integration controls; per row attributes; notifications and much more. Then there are widgets for showing buffers, navigating lists of items and so on.

More Windows

Since there is no restriction on a single screen we can be more adamant about avoiding premature composition – the desktop and its window manager should naturally be responsible for such decisions.

To request a new window, you do this:

static bool on_window(struct tui_context *T,
                      arcan_tui_connection *C,
                      uint32_t id,
                      uint8_t type,
                      void *T)
{
/* we use a custom id to distinguish between multiple
 * request and windows pushed to us */
    if (id != 0xcafe)
        return false;

    struct tui_cbcfg cbcfg = {
/* fill in as needed */
    };

    struct tui_context* new = 
          arcan_tui_setup(T, C, &cbcfg, sizeof(cbcfg));

/* window allocation can always fail */
    if (!new)
       return false;

/* do something with new */
  return true;
}

/* set in initial handler table before context creation */
cbcfg.subwindow = on_window;

...

struct tui_subwnd_req req = {
 .hint = TUIWND_SPLIT_LEFT,
 .rows = 80,
 .cols = 25
};

arcan_tui_request_subwnd_ext(tui, TUI_WND_TUI,
                             0xcafe, req, sizeof(req));

As you can see it is very similar to how the original context was created. [new] should be added to the processing loop, or spawned into a thread with its own separate one.

Note: The context setup now uses a parent as a reference. This inherits all dynamic properties regarding locale, colours and text so there is a starting point for the implementation to work with.

Among the nuances is the HINT. This is a way to communicate to the outer display system how its window is intended to be used. Among the possible values are for setting it to a discrete tab, but also for embedding into other windows or be ‘swallowed’ (take the parents place until closed).

In this clip from Cat9 you can see that in two forms:

In the first form we take the contents of a previous job, request a new window and move the job output into it.

The second one is something more advanced. The requested type is actually HANDOVER, and we use the helper:

pid_t arcan_tui_handover(struct tui_context*,
                         arcan_tui_conn*,
                         const char *path,
                         char *const argv[],
                         char *const env[], int flags);

This helper creates a new process that inherits window ownership. The context can still be referenced as a proxy and used with the arcan_tui_wndhint function to reposition or reanchor.

This is useful for embedding as done in the clip, where we also get feedback on scaling and dimensions. This is where some of [shmif] bleeds through; if the new process is graphical and uses that library, such as the video player and PDF viewer in the clip, we can embed and even interact with graphical applications even though the TUI process itself has no concept of bitmap graphics.

In Lua, it would look like this:

wnd:new_window("handover",
    function(wnd, new)
        if not new then
           return
        end
        local in, out, err, pid = 
           wnd:phandover("/usr/bin/afsrv_terminal", "")
    end
)

Reading a line

Lets take a look at the common case: a command-line, i.e. our ‘libreadline’ replacement. This is a rather complicated affair since it involves text editing; suggestions; prompt; auto-completion; help information; keybinding controls; masking characters for password entry; searchable command history and so on.

Some of that you can see in this clip, also from Cat9.

The prompt is live updated (shown by the ticking clock), feedback on syntax/validation errors are highlighted, toggle-able extended help as to what a completion does and error messages explaining the validation failure.

#include <arcan_tui_readline.h>

...
struct tui_context *T;
/* context has been setup like before */
...

struct tui_readline_opts options = 
{
  .anchor_row = 1
  .allow_exit = true
};

arcan_tui_readline_setup(T, options, sizeof options);
int status;
char* out;

while (1){
    struct tui_process_res result =
      arcan_tui_process(&T, 1, NULL, 0, -1);

    status = arcan_tui_readline_finished(T, &out);
    if (status)
        break;
}

if (status == READLINE_STATUS_DONE){
/* do something with *out */
   free(out);
}

This swaps out relevant parts of whatever previous handler table you have for ones provided by the readline widget, and reverts it once finished. There are a lot of options to pick from when setting up the widget, including callback handlers for requesting history, validating current input, password masking, filtering undesired characters and so on.

For Lua, we can go with a more involved example:

local readline = 
  wnd:readline(
               function(_, message)
                   print("you entered", message)
               end
              )

readline:set_prompt("type something: ")
readline:set_history({"things", "from", "the", "past"})
readline:suggest({"some", "thing", "to", "add"})

while (wnd:process()) do
    wnd:refresh()
end

Networking

All this would be only marginally useful if it did not also work across the network. Remote administration and similar tasks are, of course, passport holding citizens in this world and even more work has gone into the networking protocol itself.

Say that you have written a new application using libarcan-tui and you want to share it with others or access it from another computer. From the developer angle, there is nothing to do, the existing tooling provides everything.

We will omit authentication key management here, since that is an big topic in its own right. That’s done with the ‘soft-auth’ argument for the pull apprach, and a12:// rather than a12s:// for the push one.

Speaking of push and pull, running something like:

arcan-net --soft-auth -l 6680 -- /path/to/my_thing

Would set the device up to serve an instance of my_thing to every inbound connection. This is what we mean with ‘pull’, the display end connects, gets served an application and pulls the output to its display. This is what you would be most familiar with through ssh.

Push flips the direction around and is what you would expect from X11 style remoting with something like DISPLAY=some.host:port xterm.

Here ‘my.other.host’ would be setup with arcan-net --soft-auth -l 6680. To push my_thing: ARCAN_CONNPATH="a12://my.other.device" /path/to/my_thing.

Both have its uses. Normally I have a number of devices fuzzing, doing traffic monitoring, or monkey testing with a core dump crash handler installed. All of these attach their corresponding TUI interaction tool and on-demand push to whichever device I am currently working on directly, or to a fallback multiplexing server that keeps them around until I check in.

What is interesting in the network case is how it differs from terminal times over the wire. This is nuanced with many factors in play as congestion control isn’t exactly XON/XOFF or the RS-232 RTS/CTS of yore. Is the priority interactive latency? memory consumption locally? memory consumption remotely? low network bandwidth?

To our disadvantage there is potentially more data per packed cell, but ZSTD compresses that very well. At the same time it is only the visible changes that are transferred, not some theatre piece with ASCII-character LOGO turtle on acid playing the lead. Any back-pressure propagates from the network tool to the source so pending updates merge there.

In a simple line-mode screen, ssh + your shell doing a ‘find /’ would start sending a colossal amount of data on the wire. Here the strategy is partly up to the shell.

In the following clip from Cat9, you can see that it detects a data heavy job and switches to a slow updating statistics view until it terminates or I instruct it to show what is going on. Importantly I can keep on working and I am not shoving 127MiB of data over the wire.

For an alt-screen tool, like vim and tmux, they instead get to pay the price for cursor movements to draw at non-continuous locations; style changes; border drawing characters and so on. Then suffer all the crazy things ncurses does (like padding or delaying writes to align with the baudrate of the connection). Then suffer the crazy things emulators have to do (no incoming data last n miliseconds, then it’s probably safe to draw). Then suffer the crazy things they themselves have to do to understand if a key is a key or a control character.

This is a technical description of the IPC system used throughout the Arcan project, from both a designer and developer perspective, with annotations on legacy and considerations along the way. It’s one of a few gems inside of the Arcan ecosystem, and thousands of hours have gone into it alone.

The write-up assumes a basic computer science background, while the sections prefixed with ‘comment’ are more advanced.

History

SHMIF, or “SHared Memory InterFace” has a long history, dating back to around 2007. It was first added to cover the need to least-privilege separate all parsing of untrusted data in the main engine, simply because the ffmpeg libraries couldn’t stop corrupting memory – generally a bad thing and we had more than enough of that from GPU drivers doing their part.

With parsers sandboxed, it evolved to also work as a linker interposed- or injected shellcode- way of manipulating 3rd party audio/video processing and event loop without getting caught doing so. Rumour has it that it was once used to automate a lot of the tedium in games such as World of Warcraft, and was not caught doing so.

It was written to be portable across many operating systems. The initial version ran on both Windows, OSX, BSDs and Linux. There were also non-public versions that ran on Android and iOS. These days the focus remains on BSDs and Linux, with the networking version of the same model, “A12”, intended to retain compatibility with the others.

Its design is based on lessons learned from emulating arcade games of yore, as they represent the most varied and complex display systems to date. The data model evolved from increasingly complex experiments, up to- and beyond- the point of painstakingly going through every single dispatch function in X11 to guarantee that we did not miss anything. The safety and recovery aspects come from many lessons learned breaking and fixing control systems for power grids. The debugging and performance choices came from working on a last-resort debugging tiger team on (mainly) Android.

Layout

There is a shared memory region, and a set of OS specific primitives to account for inadequacies in how various kernels expose controls over memory allocation and use. Combined we refer to these as a segment. The first one established is referred to as the ‘primary’ and it is the only one that is guaranteed on a successful connection. Additional ones are negotiable, and the default is to reject any new allocation. This decision is ultimately left to the window management policy.

Comment: In this form, there is no mechanism for a client to influence allocations in the server end. Further compromises are possible (as we come to later) in order to gain more features, but for a hardened setup, this is one out of several ways we reduce the options for exploiting any vulnerabilities or staging a denial of service attack.

The shared memory is split into a fixed static region and a dynamic one.

The following figure shows the rough contents of these regions:

The order of the fields in the static region is organic, it has been extended over time. To avoid breaking compatibility, changes have been appended as more metadata was needed. The region marked ‘aux’ is 0- sized by default; it is only used for negotiating advanced features e.g. HDR metadata and VR device support.

Some of the more relevant and non-obvious members of the static regions are:

• DMS – Dead Man’s Switch. If it is ever modified the segment is considered dead. After this point no modifications to the page will be processed by the other side. (See ‘Safety Measures’ section).
• Verification Cookie. This is a checksum comprised of calculating the offsets and values of other members in the region. Both sides periodically calculate and compare this value to detect version mismatches or corruption.
• Inbound/Outbound event buffers. – These are fixed slot ring buffers of 128b events. They can be thought of as asynchronous ‘system’ calls (See ‘Event Processing’ section).
• Segment Token. A unique identifier for this specific segment. This can be used by the client end to reference other events if the identifier has been shared by some other mechanism. The ‘VIEWPORT’ event, for instance, instructs window management for repositioning or embedding segments owned by other clients or threads.

The entire memory region is treated as an unsafe contested area; one side populates it with changes it wants to see done and through some synchronisation trigger, and the other side verifies and applies or rejects them.

Comment: For debugging and inspection, this means a single snapshot of the mapped memory range is sufficient to inspect the state of the connection and trivial to write analysis, fuzzing and reporting tools for.

The raw layout is not necessarily exposed to the consumer of the corresponding library. Instead a context structure (struct arcan_shmif_cont) contains the developer-relevant pointers to the corresponding subregions.

Comment: While the implementations for this interface live in userspace, the design intent was to be able to have the server end live completely in a kernel, and have this act as the sole system call interface.

Each segment has a type that is transferred once from the client to the server during the REGISTER event (or when requesting a new one through a SEGREQ event). This is mainly a window management and server hint to control event response, but also determines if video and audio buffers are for (default) client to server or (screen recording and similar features) server to client.

First Connection (Client)

The client end comes as a library, libarcan-shmif. The rough skeleton that we will unpack here looks like this.

#include <arcan_shmif.h>
int main(int argc, char **argv)
{
  struct arg_arr args;
  struct arcan_shmif_cont C =
    arcan_shmif_open(SEGID_APPLICATION,
                     SHMIF_ACQUIRE_FATALFAIL,
                     &args);

  struct arcan_shmif_initial *config;
  arcan_shmif_initial(&C, &config);

/* send audio/video */
/* event processing */

  arcan_shmif_drop(&C);
}

The SEGID_ part is a hint to the server as to the intended use of this connection and how it could manage its resource allocation and scheduling. There is a handful of types available, but APPLICATION is a safe generic one. A video player would be wise to use MEDIA (extremely sensitive to synchronisation but not input), while a game would use, well, GAME (high resource utilisation, input latency and most-recent presentation more important than “perfect” frames).

The FATALFAIL part simply marks that there is no point to continue if a connection can’t be negotiated. It saves some error checking and unifies 'fprintf(stderr, "Couldn't connect")' like shenanigans.

The arg_arr ‘args’ is a form of passing command line arguments to the client without breaking traditional getopt/argv. It can be used to check for key=value pairs through something like ‘if (arg_lookup(&args, "myopt", 0, &val)){ ... }‘ .

A good question here would be, how does the client know to find the server? The actual mechanism is OS dependent, but for the POSIX case there are two main options that the library is looking for: the ARCAN_CONNPATH and ARCAN_SOCKIN_FD environment variables. The value for CONNPATH is the name of a connection point and is defined by the server side.

Comment: The connection point name is semantic. This stands in contrast to how Xorg does with its DISPLAY=:[number] where number normally came from the virtual terminal the user was starting Xorg from. The server end can spawn multiple connection points with different names and apply different policies based on the name.

ARCAN_SOCKIN_FD is used to reference a file descriptor inherited into the process running arcan_shmif_open. This is used when the server itself spawns the new process. It is also used in the special case of ARCAN_CONNPATH being set to “a12://” or “a12s://”. This form actually starts arcan-net to initialise a network connection to a remote host, which creates a single-use shmif server for the client to bind to. This is one of the ways we translate from local IPC to network protocol.

The 'arcan_shmif_initial' part gives the information needed to create correct content for the first frame. This includes user preferred text format (size, hinting), output density (DPI aware drawing over ‘scaling’ idiocy), colour scheme (contrast for accessibility, colour blindness or light/dark) and even locale (to migrate away from LC_… /getlocale) and location (latitude/longitude/elevation).

Comment: For accelerated graphics it also contains a reference to the GPU device to use for rendering, this lets the server compartment or load-balance between multiple accelerators.

Now for the ‘send audio/video’ part.

shmif_pixel px = SHMIF_RGBA(0x00, 0xff, 0x00, 0xff);
for (size_t y = 0; y < C.h; y++)
  for (size_t x = 0; x < C.w; x++)
    C.vidp[Y * C.pitch + x] = px;

arcan_shmif_signal(C, SHMIF_SIGVID);

This fills the dynamic video buffer part of the segment with a full opaque green pixel in linear RGB in whatever packing format the system considers native (embedded in the SHMIF_RGBA macro).

Comment: While on most systems (these days) that would be 32-bit RGBA, it is treated as compile time native as CPU endianness would be. Low-end embedded might want RGB565, special devices like eInk might want RGB800 and so on. 

There are a lot of options available here, but most have to deal with special synchronisation or buffering needs. These are covered in the ‘Special Case’ sections on Synchronisation and on Accelerated Graphics.

For this example we omitted the aural representation, but if you have a synthesizer core or even tone-generator the same pattern apply; switch vidp for audp and SHMIF_SIGVID for SHMIF_SIGAUD (it is a bitmask, use both if you have both).

Comment: The common distinction between audio and video is something we strongly oppose. It causes needless complexity and suffering trying to have one IPC system for audio, then another for video and then trying to repair and synchronise  the two after the fact. It is one of those historical mistakes that should have ended yesterday, but the state of audio on most systems is almost as bad as video.

At this stage we are already done (13 lines of code, zero need for error handling) but for something more polite, we will flesh out the ‘event processing’ part.

struct arcan_event ev;
while (arcan_shmif_wait(&C, &ev)){
 if (ev.category == EVENT_IO){
/* mouse, keyboard, eyetracker, ... handling goes here */
 }

 switch(ev.tgt.kind){
 case TARGET_COMMAND_EXIT:
/* any custom cleanup goes here */
 break;
 case TARGET_COMMAND_RESET:
/* assume that we are back where we started */
 default:
 break;
 }
}

This will block until an event is received, though more options are covered in the section on ‘Synchronisation’. No action is ever expected of us, we just get polite suggestions ‘it would be nice if you do something about this’. The category part will only be EVENT_IO or EVENT_TARGET and the next section will dip into why.

Comment: The _RESET event in particular is interesting and will be covered in the 'Recovery' Special Case. It can be initiated by the outer desktop for whatever reason, and just suggest 'go back to whatever your starting state was, I have forgotten everything' but is also used if the server has crashed and the implementation recovered from it, or is shutting down and have already handed responsibilities over to another.

The event data model cover 32 different server to client possibilities, and 22 client to server. Together they cover everything needed for a full desktop and more, but it is descriptive, not normative. React to the ones relevant to you, ignore the others.

First Connection (Server)

There are two implementations for the server end; one inside the arcan codebase tailored to work better with its more advanced resource management, crash resiliency and scripting runtime. The other comes as a library, libarcan-shmif-server, and is mainly used by the arcan-net networking tool which translates this into the A12 network protocol.

Let’s walk through a short example which accepts a single client connection, and in the next section do the same thing for the client application end. Normal C foreplay is omitted for brevity.

#include <arcan_shmif.h>
#include <arcan_shmif_server.h>

struct shmifsrv_client *cl = 
  shmifsrv_allocate_connpoint("demo", NULL, S_IRWXU, fd);

shmifsrv_monotonic_rebase();

This creates a connection point for a client to bind to. There are two alternatives, shmifsrv_spawn_client and shmifsrv_inherit_connection. Spawn takes care of creating a process with the primitives inside. Inherit takes some preexisting primitive and builds from there. Both return the same shmifsrv_client structure.

Comment: For a controlled embedded or custom OS setting, the spawn client approach is the safest bet. The inherit connection approach is for when there is a delegate responsible for spawning processes and reduces the number of system calls needed to a bare minimum.

The shmifsrv_monotonic_rebase() call sets up the internal timekeeping necessary to provide a coarse (25Hz) grained CLK (clock) signal.

Now we need some processing that interleaves with rendering/input processing loops, which is a larger topic and out of scope here.

int status;
while ((status = shmifsrv_poll(cl) <= CLIENT_NOT_READY)
{
/* event and buffer processing goes here */
}

if (status == CLIENT_DEAD)
{
  shmifsrv_free(cl, SHMIFSRV_FREE_FULL);
  exit(EXIT_SUCCESS);
}

It is possible to extract an OS specific identifier for I/O multiplexing so that _poll is only invoked when there is some signalled inbound data via shmifsrv_client_handle().

Comment: The flag passed to _free determines the client side visibility. It is possible to just free the server side resources and not signal the dead man's switch. This can be used to transparently pass the client to another shmif server process or instance.

Before we get to the event and buffer processing part, there is also some timekeeping that should be managed outside of a higher frequency render loop.

int left;
int ticks = shmifsrv_monotonic_tick(&left);
while (ticks--){
  shmifsrv_tick(cl);
}

This provides both integrity and liveness checks and manages client requested timers. The (left) returns the number of milliseconds until the next tick. This is used as feedback for a more advanced scheduler, if you have one (and you should).

Now to the event processing:

struct arcan_event buf[64];
size_t n_events, index = 0;

if ((n_events = shmifsrv_dequeue_events(cl, buf, 64)){
  while (index != n_events){
    struct arcan_event* ev = &buf[index++];
    if (shmifsrv_process_event(cl, ev))
      continue;

/* event handlers go here */
  }
}

This will dequeue, at most, 64 events into the buffer. Each event is forwarded back into the library in order to allow a subset of internally managed ones. They are just routed through the developer to allow complete visibility. You can use arcan_shmif_eventstr() to get a human readable representation of its contents.

Comment: The reason for having a limit is that a clever and malicious client could set things up in a way that would race to stall the server or exhaust its file descriptor space as part of a denial of service, either to affect the user directly or as part of trying to make an exploitation chain more robust.

Now for the ‘event processing’ part.

if (ev->category != EVENT_EXTERNAL){
  fprintf(stderr, "unexpected event category\n");
  continue;
}

switch (ev->ext.kind){
case EVENT_EXTERNAL_REGISTER:
/* only allow this once on the client */
  arcan_shmifsrv_enqueue(cl, &(struct arcan_event){
   .category = TARGET_COMMAND,
   .tgt = {
     .kind = TARGET_COMMAND_ACTIVATE
   }
  }
break;
default:
break;
}

The event data model has a lot of display server specific nuances to it, neither is necessary except for the one above. This unlocks the client from the ‘preroll’ state where it accumulates information received into the “arcan_shmif_initial” structure as covered in the client section. Any information necessary for a client to produce a correct first frame goes before the ‘ACTIVATE’ one. The most likely ones you want is DISPLAYHINT, OUTPUTHINT and FONTHINT to instruct the client about the size it will be scaled to, the density, colourspace and subchannel layout it will be presented through, as well as the preferred size of the most important of primitives, text.

Comment: There are a number of event categories, but only one reserved for clients (EVENT_EXTERNAL). The other categories are for display server internals. The reason they are exposed over SHMIF is for the 'server' end to be split across many processes and still interleave with event processing in the server. This allows us to have external sensors, input drivers etc. all as discrete threads or processes without changing anything else in the architecture. It also allows a transformation from using it as a kernel-userspace boundary to a microkernel form.

The last part is to deal with ‘buffer processing’ part of the previous code.

/* separate function */
bool audio_cb(shmif_asample *buf,
              size_t n_samples,
              unsigned channels, unsigned rate,
              void *tag)
{
  /* forward buf[n_samples] to audio device or mixer
   * configured to handle [channels] at [rate] */
  return true;
}

if (status & CLIENT_VBUFFER_READY){
  struct shmifsrv_vbuffer vbuf = shmifsrv_video(cl);
  /* forward vbuf content to GPU */
  shmifsrv_video_step(cl);
}

if (status & CLIENT_ABUFFER_READY){
  shmifsrv_audio(cl, audio_cb, NULL);
}

The contents of vbuf is nuanced. There is a raw buffer or opaque GPU system handle + metadata (timing, dirty regions, …), or TPACK (see section on ‘Text Only Windows’) and a set of flags corresponding to the ‘presentation-hints’ on how buffer contents should be interpreted concerning coordinate system, alpha blending and so on.

Comment: Most of the graphics processing properties are things any competent scanout engine has hardware acceleration for, excluding TPACK (and even ancient graphics adaptors used to have those as 'text mode' display resolution). There are support functions to unpack this into a compact list of text lines and their colouring and formatting in "arcan_tui.h" as arcan_tui_tunpack().

Comment: For accelerated GPU handles it is possible to refuse it by sending a BUFFERFAIL event. This will force the client implementation to convert accelerated GPU-local content into the shared pixel format on their end. This is further covered in 'Special Case: Accelerated Graphics'. It doubles as a security measure, preventing the client from submitting command buffers to the GPU that will never finish and livelock composition that way (or leverage any of the many vulnerabilities GPU side). On a hardened system this would be used in tandem with IO-MMU isolation.

In total this lands us with less than 100 lines of code with very granular controls, a fraction of what other systems need to just boilerplate graphics alone. If you only want a 101 level – take on how SHMIF works, we are done; there is a lot more to it if the topic fascinates you, but it gets more difficult from here.

Synchronisation

While one might be tempted to think that ‘display servers’ are about well, providing access to the display, its real job is actually desktop IPC with soft realtime constraints. The bread and butter for such systems is synchronisation. If you fail to realise this you are in for a world of hurt and dealing with it after the fact will brew a storm of complexity.

Comment: It is also the hardest problem in the space - figuring out who among many stakeholders knows what; when do they know it; when is something said relevant or dated. All of those are difficult as is, but it gets much worse when you also need to factor in resonance effects, malicious influence and that some stake holders reason asynchronously about some things, and synchronously about others. As icing on an already fattening cake you need to weigh in the domain specific (audio / video) nuances. Troubleshooting boils down to profiling, and problems manifest as 'jitter' and 'judder' and how those fit with human cognition. Virtual Reality is a good testing ground here even if you are otherwise uninterested in the space.

Comment: beginner mistakes here are fairly easy to spot; if someone responds to synchronisation problems by increasing buffer sizes (landing in a version of the network engineering famous 'buffer bloat' problem) or arbitrary sleep calls (even though some might be necessary without adequate kernel level primitives) they are only shifting the problem around.

Recommended study here is ‘queuing theory’ and ‘signal processing’ for a deeper understanding.

To revisit the previous code examples on the client end:

arcan_shmif_signal(&C, SHMIF_SIGVID);

This is synchronous and blocking. The thread will not continue execution until the server end has said it is ok (the shmifsrv_video_step code). The server can defer this in order to prioritise other clients or to stagger releases to mitigate the ‘thundering herd’ problem.

For normal applications, this is often sufficient and comparable to ‘VSYNC’. When you have tighter latency requirements and/or it is costly to produce a frame, you need something more. The historically ‘easier’ solution has been to just add another buffer:

arcan_shmif_resize_ext(&C, C.w, C.h,
                       (struct shmif_resize_ext){
                         .vbuf_cnt = 2
                       });

_resize and _resize_ext calls are both also synchronous and blocking. This is because the server end needs the controls to guarantee that enough memory is available and permitted. It will recalculate all the buffer offsets (vidp, audp, …) and verification cookie in the context and possibly move the base address around to satisfy virtual or physical memory constraints.

Comment: Some accelerated display scanout controllers have hard requirements on physically continuous memory at fixed linear addresses and those are a limited and scarce resource and why such resize request might fail, especially in tight embedded development settings. Same applies when dealing with virtual GPUs in virtual machines and so on. The other option to still satisfy a request is to buffer in the server end, causing an extra copy with increased power consumption and less memory bandwidth available for other uses.

The request above would make the first arcan_shmif_signal call return immediately, and only block if another signal call happens before the server is able to consume the buffer from the first. Otherwise the context pointer (C.vidp) will be changed to point to the new free buffer slot. This also has the added cost of adding another display refresh period of latency.

Comment: It is possible to increase the buffer count even further, but this changes the semantics to indicate that only the most recently submitted buffer should be considered and others can be discarded. This counters the latency problem of the double buffering problem at the expense of memory consumption. This has historically been called 'triple buffering' but, to much confusion, has also been used for the 'double buffering' behaviour with just deeper buffer queues and is thus meaningless.

Not every part of a buffer might have actually changed. A common optimisation is to annotate which region that should be considered, and for regular UI applications (blinking cursor, single widget updates, …) this substantially cuts down on memory transfers. To cover this you can mark such regions with calls to arcan_shmif_dirty() prior to signalling.

Comment: While some might be tempted to annotate every pixel, there are strong diminishing returns as soon as you go above just one region due to constraints on memory transfers. Internally the shmif client library implementation will just merge multiple calls to _dirty into the extents of all changes. For the triple buffering behaviour mentioned in the previous comment, dirty regions won't have any effect at all as changes only present in the one buffer would not guarantee to transfer over in the next and the cost of trying to merge them on the composition end would cancel out the savings in the first place.

There are more synchronisation nuances to cover, but to avoid making this section even more exhausting, we will stick to the two most relevant. The first of these look like this:

arcan_shmif_signal(&C, SHMIF_SIGVID | SHMIF_SIGBLK_NONE);

This returns immediately and you can chose to check if it is safe to draw into the video buffer yourself (through arcan_shmif_signalstatus), or to simply continue writing into the buffer and risk ‘tearing’ in favour of lower latency. This amounts to what is commonly called ‘Disable VSYNC’ in games.

Comment: For those that have written games in days of yore, you might also recall 'chasing the beam', unless dementia has taken over by now. Since game rendering can have predictable write patterns and display scanout can have predictable read patterns it is possible to align your writes such that you raster lines up to just before the one the display is currently reading. This is neither true for modern rendering nor is it true for modern displays, 'the framebuffer' is a lie. Still, for emulation of old systems, it is possible, but impractical, to repeatedly access the 'vpts' field of the static region to figure out how many milliseconds are left until the next VBLANK and stage your rendering accordingly.

The last option is to to keep the SHMIF_SIGBLK_NONE behaviour, but adding the flag SHMIF_RHINT_VSIGNAL_EV to C.hints prior to a _resize call. This will provide you with a TARGET_COMMAND_STEPFRAME event and you can latch your rendering to that one alone and let your event loop block entirely.

Comment: Enabling STEPFRAME event delivery by sending a CLOCKREQ request provides a secondary path for advanced latency management as it enables feedback in presentation timing. Sampling the 'vpts' field of the static region would provide information about upcoming deadline and STEPFRAME contains metadata about presentation timing as to when the contents was actually presented to scale up/down effects and quality. Current game development is full of these kinds of tricks.

Event Processing

Event loop construction is an interesting and nuanced topic. We start by returning to the naive one introduced in the client section:

struct arcan_event ev;
while (arcan_shmif_wait(&C, &ev)){
/* interpretation goes here */
}

Responding to each event that arrives here should be as fast as possible. This is easy most of the time. Whenever a response includes rendering however, response times can vary by a lot. Some events are more prone to this, with mouse motion and resize requests being common offenders.

What happens then is that the number of events in the incoming queue starts to grow. If the rate of dispatched events is lower than that of the incoming one, we get buffer back-pressure.

This applies to both the server and the client side. Each side has different constraints and call for different mitigation. The server end is more vulnerable here as it has multiple clients to process, and higher costs for processing events as most prompt some form of managerial decision.

Comment: Events from client A can directly or indirectly cause a larger number of responses from clients B and C (amplification), which in turn can cascade into further responses from A. This can snowball fast into 'event storms' and back-pressure building up in others as 'resonance effects'.

One small change to the loop improves on the client end of the equation:

bool process_event(struct arcan_event *ev)
{
/* interpretation now goes here */
}

struct arcan_event ev;

while (arcan_shmif_wait(&C, &ev)){
  bool dirty = process_event(&ev);
  size_t cap = PP_QUEUE_SIZE;

  while (arcan_shmif_poll(&C, &ev) > 0 && cap--){
    dirty |= process_event(&ev);
  }

  if (dirty){
    render();
  }
}

This will flush out as much of the inbound queue (or up to a cap corresponding to the size of the ring buffer) as possible, and only render after all have been applied. This prevents earlier events in the queue from being overdrawn by responses to later ones in the queue.

Comment: Since the event queue is a ring-buffer visible to both sides, it is possible for either party to atomically inspect the head and tail values to determine the current state of the other end, as well as incoming workload. This is a powerful advantage over other possible carriers, e.g. sockets.

The full data model would take another lengthy post to flesh out, so we will only look at one event which highlights library internals. That event is ‘TARGET_COMMAND_DISPLAYHINT’. This event is used to indicate the size that the server end would prefer the window to have. The client is free to respond to this by resizing to the corresponding dimensions. If it doesn’t, the server can still scale and post process – it has the final say on the matter.

As mentioned earlier, resize is synchronous and blocking due to its systemic cost so it makes sense to keep them at a minimum. Some of that responsibility falls on the window manager to ensure that a drag resize using a 2kHz mouse doesn’t also result in 2000 DISPLAYHINTs. Even if that would happen, the implementation has another trick up its sleve.

There is a small number of events which are considered costly and can be coalesced. When _wait or _poll encounters such an event, it sweeps the entire pending queue looking for similar ones, merging them together into the one eventually returned, providing only the most recent state.

Comment: There is a tendency for IPC systems to be designed as generally as possible, even if their actual narrow use is known. This defers the decision to commit to any one domain specific data model, making optimisations such as this one impossible -- you can't coalesce or discard what you don't know or understand, at least not responsibly with predictable outcome. This does not make complexity go away, to the contrary, now every consumer has increased responsibility to manage queuing elsewhere. The problem doesn't magically disappear just because you have an XML based code generator or some-such nonsense.

Most of this article has been about a single segment, though in ‘rich’ applications you would have more: child windows, popups, custom mouse cursors and so on. We already mentioned it is possible to request more, even though only one is ever guaranteed. This is not more difficult than setting up the primary one. You submit a SEGREQ event with a custom identifier and type. Eventually you either get a NEWSEG event or REQFAIL event back with the same identifier. For the NEWSEG you forward the event structure to arcan_shmif_acquire and you get a new arcan_shmif_cont structure back.

What does this have to do with queue management? Well, each new segment has their own separate queues and each segment can be processed and rendered on separate threads independent of each other. There is a monotonic per-client global counter and timestamp as part of each event to account for ordering requirements between ‘windows’, but in practice those are exceedingly rare.

A final part about the events themselves. SHMIF is an IPC system, it is not a protocol, it doesn’t cross device boundaries. We have a separate and decoupled networking protocol specifically for that. As an IPC system we can and should take advantage of device and operating system specific nuances.

Two such details is that each event has a fixed size, 128 bytes (64 did not cover all cases) which amounts to 2 cache lines for the vast majority of architectures out there. They are in linear continuous buffers at a native aligned base with access patterns that prefetch very well. The packing of different fields is tied to the system ABI which is designed to be optimal for whatever you are running on.

Safety Measures

We are almost done with the overall walkthrough, then we can finish off with some special cases and exotic features. Before then there is one caveat to cover.

In previous sections we have brushed upon a few tactics that protect against misuse; the validation cookie to detect corruption, version and code generation mismatches as well as the dead man’s switch. There is still one glaring issue from shared memory event management and audio/video signalling solution: what happens if the other end livelocks or crashes while we are locked waiting for a response?

In a socket based setup the common ‘solution’ for a crash in the other end would cause it to detach and you can detect that. For a live lock it is to resort to a kind of ping-pong protocol and somehow disambiguate between that and a natural stall for some other part of the system (very often, GPU driver).

By default (there is a flag to disable this) each segment gets a guard thread. This guard thread periodically (default: every second) checks the aliveness of a monitoring process identifier that the server filled out, as well as if the dead man’s switch has been released. If that happens, it immediately unlocks all internal semaphores causing any locked call into the shmif library to release and any further calls to error out so the natural event loop takes hold. This setup is also used to not only detect- but recover from- crashes (see ‘Special Case: Recovery and Migration’).

This might not be enough for troubleshooting or even communicating to a user that something is wrong. For this we have the ‘last_words’ part of the memory region. This is an area the client can fill out with a human presentable error message that the server end can forward to relevant stakeholders.

The Arcan engine itself splits out into two parts. One potentially privileged parent supervision process that is used to negotiate device access, and the main engine. This supervision process also acts as a watchdog. Every time the engine enters and exits a dangerous area, e.g. the graphics platform or the scripting VM, it registers a timestamp with the parent. If this exceeds some threshold, the parent first signals the engine to try and gracefully recover (the scripting VM is able to do that) and if the problem persists, shuts down the engine. This will trigger the guard threads inside clients and they, in turn, enter a reconnect, migrate or shutdown state.

Special Case: Handover Allocation

As we covered previously, requesting a new segment to use as a window takes a type that indicates its role and purpose. One such type that sticks out is ‘SEGID_HANDOVER’. This means that the real type will be provided later and that the segment will be handed over to a new client.

To better illustrate, let’s take a code example:

arcan_shmif_enqueue(&C, &(struct arcan_event){
  .category = EVENT_EXTERNAL,
  .ext.kind = ARCAN_EVENT(SEGREQ),
  .ext.segreq.kind = SEGID_HANDOVER
});

...

uint32_t new_token;

/* in event handler, 'ev' being the inbound event */
case TARGET_COMMAND_NEWSEGMENT:
  arcan_shmif_handover_exec(&C, ev,
                            "/path/to/something",
                            argvv, envv,
                            0 /* detach-options */);  
  new_token = ev.ext.ioevs[4].uiv;
break;

This would launch “/path/to/something” so that when it calls arcan_shmif_open it will actually use the segment we received in order to connect. We can then use new_token in other events to manage some of it, e.g. reposition its windows, inject input and more. All of this retains the chain of trust: the server end knows who invited the new client in and can treat it accordingly.

This can be used to embed other clients into your own window, to build an external window manager and so on. In our ‘command lines without terminal emulation’ shell, Lash#Cat9, we use that to manage graphical clients while still being ‘text only’ ourselves. For other examples, see the article on Another Low Level Arcan Client: A Trayicon Handler.

Special Case: Recovery and Migration

A key feature of SHMIF is that it can redirect and reconnect clients manually. Through this we can even transition back and forth between local and networked operations. The section on ‘Safety Measures’ covered how it works in SHMIF internals. There is also an article on ‘Crash Resilient Wayland Compositing’ (2017) that demonstrates this.

When a client connects, the library enqueues a ‘REGISTER’ event that contains a generated UUID. This can be leveraged by the window manager to persist location on the desktop and so on. At any stage it can also send a ‘DEVICEHINT’ event back.

This event is used to provide an opaque handle for GPU access in operating systems which requires that (which can further be used to load balance between multiple GPUs), but it can also mention a ‘fallback connection point’. Should the server end die (or pretend that it has died), the library will try to connect to that connection point instead.

If it is successful, it will inject the ‘TARGET_COMMAND_RESET’ event that we covered earlier. We will use the following clip from “A12: Visions of the Fully Networked Desktop“. as a starting point.

In it, you see Lash#Cat9 CLI shell inside the ‘Durden’ Window manager having a video clip as an embedded handover allocation. This has previously used the local discovery feature of the network protocol to detect that the tablet in front of the screen (a Surface Go) is available as a sink and has added it as an icon in the statusbar — unfortunately occluded by the monitor bezel in the clip.

When I drag the window and drop it on that icon, Arcan sends a DEVICEHINT with the connection primitive needed for the network embedded into the event. It then pulls the dead man’s switch, forcing the shmif library to go into recover. Since it remembers the connection from the DEVICEHINT, it reconnects and rebuilds itself there.

This feature is not only leveraged for network migration as shown, but also for compartmentalisation between multiple instances; for crash recovery; for driver upgrades and for upgrading the display server itself. All using the same code paths.

Special Case: Accelerated Graphics

Many ‘modern’ clients unfortunately have a hard dependency to a GPU, and unfortunately the mechanisms for binding accelerated graphics between display server and client are anything but portable.

Comment: Khronos (of OpenGL and Vulkan fame) tried to define a solution of their own (OpenWF) that failed miserably. What happened instead is even worse; the compromise once made for embedded systems, 'EGL' got monkey patched with a few extensions that practically undoes near all of its original design and purpose, and it is suddenly what most are stuck with.

There is a lot of bad blood and vitriol on the subject that we will omit here and just focus on the SHMIF provided interface. Recall the normal way of starting a SHMIF client:

#include <arcan_shmif.h>
int main(int argc, char **argv)
{
  struct arg_arr args;
  struct arcan_shmif_cont C =
    arcan_shmif_open(SEGID_APPLICATION,
                     SHMIF_ACQUIRE_FATALFAIL,
                     &args);
}

This does still apply. A client is always expected to start a normal connection first, and then try to bootstrap that to accelerated, which can fail. The reasoning for that is iff permissions or GPU driver problems stop us from providing an accelerated connection, the regular one can still be used to communicate that to the user rather than have them dig through trace outputs for the answer.

To extend a context to being accelerated you can do something like this:

struct arcan_shmifext_setup cfg = {
  .api = API_OPENGL,
  .major = 4,
  .minor = 2
/* other options go here */
};

int status = arcan_shmifext_setup(&C, &cfg);
if (status != SHMIFEXT_OK){
/* configuration couldn't be filled */
}

There are a number of options to provide in the config that requires some background with OpenGL etc. to make any sense so we skip those. If you know, you know and if you don’t, enjoy the bliss. If the setup is OK, the passed ‘cfg’ is modified to return the negotiated values, which might be slightly different than what you requested.

Afterwards, you can then continue with arcan_shmifext_lookup() to extract the function pointers to the parts of the requested API that you need to use, bound to the driver end of the created context.

When writing platform backends to existing applications, they often provide their own way of doing all this and we do need a way to work with that. If there already is a context living in your process and you want to manually export and forward a resource, it is possible through:

size_t n_planes = 4;
struct shmifext_buffer_plane planes[n_planes];
n_planes = arcan_shmifext_export_image(&C, (uintptr_t) MyDisplay, (uintptr_t) my_texture_id, n_planes, planes);

if (n_planes){
  arcan_shmifext_signal_planes(&C,
                               SHMIF_SIGVID,
                               n_planes, planes);
}

There are a several more support functions with similar patterns for context management, importing already exported images and so on, but this should be enough to get an idea of the process.

Special Case: Debugging and Accessibility

We have already shown how the client end can request new segments through SEGREQ events and how those are provided through NEWSEGMENT events coming back. Another nuance to this is that the server end can push a NEWSEGMENT without having to wait for a SEGREQ in advance.

This can be used to probe for support for things, such as custom client mouse cursors, or to signal a paste or drag and drop action (clipboard is just yet another segment being pushed), as the server end will know if the client mapped the segment or not.

There is nothing stopping us from actually mapping and populating the segment from within libarcan-shmif, and there are two cases where that is actually done. One is for SEGID_DEBUG and another for SEGID_ACCESSIBILITY.

There are two longer articles related to how this works in more depth, one on ‘Leveraging the Display Server to Improve Debugging’ and another on ‘Accessible Arcan: Out of Sight’.

If one of these are received, libarcan-shmif will (unless the feature is compiled out) internally spawn a new thread. In the debugging case it will provide a text interface for attaching a debugger, for exploring open files, inspecting environment and memory from within the process itself. In the accessibility case it will latch on to frame delivery (SHMIF_SIGVID) in order to overlay text descriptions that gets clocked to the video frames being delivered and the dirty regions being updated.

Special Case: Text-only Windows

There are more reasons as to why a set of fonts and desired font size is provided during the preroll state and why there is a ‘text rows and columns’ field in the static region.

Among the hints that can be set for the video region, there is SHMIF_RHINT_TPACK. This changes the interpretation of the contents of the video buffer region to use a packing format (TPACK) which is basically a few bytes of headers and then a number of rows where each row has a header covering how it should be processed (shaping, right-to-left) along with a number of cells with the formatting, colour and possible font local glyph indices (for ligature substitutions).

The format is complete enough to draw anything a terminal emulator would be able to throw at it, but also do things the terminal emulator can’t, such as annotation layers or borders that fall outside of the ‘grid’, saving precious space.

This approach lets the server end handle the complex task of rendering text. It also means that the costly glyph caches, GPU related acceleration primitives like distance fields and so on can all be shared between windows. It means that the server can apply the heuristic for ‘chasing the beam’ style minimal latency or tailor updates to the idiosyncrasy of eInk displays when appropriate, and that the actual colours used will fit with the overall visual theme of the desktop while letting the client focus on providing ‘just text’.

While it is possible to build these yourself, there is a higher level abstraction support library, ‘libarcan-tui’ for that purpose. The details of that, however, is a story for another time.

Our reference desktop environment, Durden, rarely gets covered here these days. This is mostly due to the major features are since long in place and that part of the project is biding its time with smaller fixes while waiting for improvements in the rest of the stack.

Recently the stars aligned and I had some time over to work on the accessibility story, particularly on the ‘no vision’ parts. Other forms (e.g. limited mobility, low vision) will be covered in future articles but most things are already in place, including eye tracking and bespoke devices like stream decks.

Here is a short recording of the first run of a clean installation, setting things up.

This is enabled by default during first setup. The first question presented will be to disable it, but there is no hidden trapdoor combination or setting to enable it for the first time.

The following recording shows using the builtin menu system to start a terminal emulator and open a PDF and the mouse cursor to navigate, with OCR results as I go. These will be elaborated on further in this article.

There is a previous higher-level article which covered a rougher outline of what is intended, but this one is more specific about work that can be used today — albeit with some rough corners still.

One detail to take away from that article is how the architecture splits user data processing into specific one-purpose replaceable programs. These can be isolated and restricted to a much stronger degree than a more generic application.

These are referred to as frameservers. They have a role (archetype) and the ones of interest here are decode and encode. Decode translates from a computer native representative to a human presentable one, like image loading into a pixel soup or annotated text into sound via synthesised speech. Encode goes from potentially lossy computer to human translation such as pixel soup to text via OCR, image description or transcribing audio.

Another detail is that the “screen reader” here is not the traditionally detached component that tries to stitch narration together through multiple sidebands. Instead, it is an “always present first class mechanism” that the window manager should leverage. There is no disconnect between how we provide visual information from aural and they naturally blend with the extensive network transparency as part of our ‘many devices, one desktop’ design target.

Key Features

Here is a short list of the things currently in place:

• Multiple simultaneous positional voice profiles for different types of information
• On-Demand Client requested Accessibility windows
• Force-injected client accessibility fallback with default OCR latched to frame updates
• Buffered Text Input with lookup oracle
• All desktop interaction controls as a file-system
• Command-line shell aware
• Keyboard controlled OCR
• Content aware mouse- audio feedback
• Special controls for text-only windows tracking changes

There is a lot more planned or in progress:

• Premade bootable live / VM image and generator
• Bindings to additional client support (AccessKit)
• Compatibility preset profiles to help transition from NVDA/Orca
• Extend accessibility fallback with description oracles (LLM:offline, …)
• Extend speech process format with waveform/tone/samples for formatting
• Extended lookup oracle
• Stronger language controls/toggles
• Scope based ‘few-line’ text editor extension to shell
• Haptic support
• Braille Terminal output (lacks hardware access currently)
• Indexer for extracting or generating alt-text descriptions

Screen Reading Basics

Let’s unpack some of what happens during setup.

The TTS tool in Durden uses one or many speech profiles that can be found in the durden/devmaps/tts folder. Each profile describe one voice, how its speech synthesis should work, which kinds of information it should convey and any input controls.

They allow for multiple voices to carry different information at different positions to form a soundscape, so you can use ‘clean’ voices for important notifications and ‘fast robotic’ for dense information where actions like flush pending speech buffer doesn’t accidentally cancel out something important.

A compact form starts something like this:

model = "English (Great Britain)",
gain = 1.0, gap = 10, pitch = 60, rate = 180, range = 60, channel = "l", name = "basic_eng", punct = 1, cappitch = 5000

These are just the basic voice synthesis parameters one would expect. Then it gets better.

actions = {
 select = {"title", "title_text"},
 menu = "menu",
 clipboard = "clip",
 clipboard_paste = "clip-paste",
 notification = "notify"
}

bindings = {
 m1_r = "/global/tools/tts/voices/basic_eng/flush",
 m1_t = "/global/tools/tts/voices/basic_eng/slow_replay"
}

These tell what this voice gets to do. The action keys mark which subsystems it should jack into, and some custom prefix announcement as value. The profile above would present all menu navigation, system notifications, clipboard access and window title on selection.

The bindings are keyboard binding overlays that take priority when the voice is active. Just like any other binding, they map to paths in the virtual filesystem that Durden is structured around. The two examples shown cancels all pending speech for the specific voice, or turns down the speech rate to low and repeats the last message then returns it back to the voice default. There are, of course, many others to chose from including things like key echo and so on.

Holding down any of the meta or accessibility bound buttons for a few seconds without activating a specific keybinding will play the current ones back for you to ease learning or refresh your memory.

By adding a position = {-10, 0, -10} attribute to the profile the system switches to 3D positional audio and, in this example, positions the voice to your back left. This feature also introduced the /target/audio/position=x,y,z which lets you move any audio from the selected window to a specific position around you, along with /target/audio/move=x,y,z,dt which would slide the voice around over time.

With an event trigger, e.g. /target/triggers/select/add=/target/audio/position=0,0,0 and /target/triggers/deselect/add=/target/audio/position=10,0,-10 the soundscape around you also match window management state itself.

The cursor profile has a few more things to it:

cursor = {
  alt_text = "over ",
  xy_beep = {65.41, 523.25},
  xy_beep_tone = "sine",
  xy_tuitone = "square",
  xy_tuitone_empty = "triangle"
  ...
}

The alt_text option would read any tagged UI elements with their text description. XY beeps specifies the frequency range to map the mouse cursor coordinates based on their screen position so that the pitch and gain changes as you slide it across the screen.

The waveform used to generate the wave will also change with the type of contents the cursor is over, so that text windows such as terminal emulators will get a distinct tone and distinguishes between empty and populated cells.

The following clip shows navigating over UI elements, an browser window and a terminal window. You can also hear how the ‘select text to copy to clipboard’ doubles as a more reliable means of hearing text contents in uncooperative windows.

There are also more experimental parts to the cursor, such as using a GPU preprocessing stage to attenuate edge features and then convert a custom region beneath the cursor into sounds. While it takes some training to decipher, this is another form of seeing with sound and applies to any graphical content, including webcam feeds. After some hours I (barely) managed to play some graphical adventure games with it.

Kicking it up a notch

Time to bring out the spice weasel and get back to the OCR part I briefly mentioned earlier.

The astute reader of this blog will recall the post on Leveraging the “Display Server” to Improve Debugging. A main point of that is that the IPC system is designed such that the window manager can push a typed data container (window) and the client can map and populate it. If a client doesn’t, there is an implementation that comes along with the IPC library. That is not only true for the debug type but for the accessibility one as well.

This means that with the push of a button we can probe the accessibility support for a window, and if none exists, substitute our own. This is partly intended to provide support for AccessKit which will complete the solution with cooperative navigation of the data model of an application.

The current fallback spawns an encode session which latches frame delivery to the OCR engine. The client doesn’t get to continue rendering without the OCR pass completed so the contents is forced to stay in synch.

This also where our terminal replacement comes in, particularly the TUI library used to provide something infinitely better than ncurses. The current implementation turns the accessibility implementation into a window attachment with a larger font (for the ‘low vision’ case), the shell populates it with the most important content and the text to speech tool kicks in.

Such text-only windows also get added controls, /global/tools/tts/voices/basic_eng/text_window/(speak_new, at_cursor, cursor_row, synch_cursor, step_row=n) for one dedicated reading cursor, that you move around separately.

Our custom CLI shell, Cat9, probes accessibility at startup. If found, it will adjust its presentation, layout and input control to match so that there is instant feedback. There is still the normal view to explore with keyboard controls, but added aural cues. The following clip demonstrates this both visually and aurally:

All this is, of course, network transparent.

A final related nugget covers both accessibility and security. The path /target/input/text lets you prepare a text locally that will be sent as simulated discrete characters. These are spaced based on a typing model, meaning that for the many poor clients that stream over a network, someone with signal processing 101 doing side channel analysis for reconstructing plaintext from encrypted channel metadata will be none the wiser.

This is useful for other things as well. For an input prompt one can set an oracle which provides suggestion completions. This is provided by the decode frameserver through hunspell, though other ones are just around the corner for filling the role of Input Method Engines, Password Manager integration, more complex grammar suggestions and offline LLM.

The following clip shows how I first type something into this tool, then read it back from the content of the window itself.

The current caveat is that it still does not work with X11 and Wayland clients due to their embarrassingly poor input models. Some workarounds are on their way, but there are a lot of problems to work around, especially for non-latin languages.