What does a compiler job look like in the year 2024?

2024-04-27T00:00:00+00:00

When people think about compiler development, they usually imagine a job working on a general purpose compiler/runtime that they use in their day-to-day work such as Clang or V8. While those jobs do exist, they make up just one slice of the pie that is the compiler job market.</p>

There are resources out there listing companies that do compiler work (namely the invaluable Compiler Jobs</a> page), however I thought it'd be useful to write up a post talking about what types</strong> of roles are out there.</p>

In this post, I'm necessarily talking about parts of the industry that are outside of my own personal experience. If you feel that I've omitted or mischaracterised something, shoot me an email and I'll fix it.</p>

General purpose programming languages</h1> These are the technologies that we mentioned earlier. Think compilers and runtimes for C++, Java, JavaScript, Python, etc. I'd put compiler infrastructure such as LLVM</a>, Cranelift</a> and GraalVM Truffle</a> under this category too.</p>
Toolchains for general purpose programming languages are, generally speaking, not directly monetisable. This means that this type of development is usually part of a broader company strategy and, with a few exceptions, tends to be the domain of larger companies that have a heavy investment in a particular language/stack and want to represent their interests in the open-source community.</p>
One way to get an idea of what companies hire contributors is to clone one of these projects and see what corporate emails turn up in `git log</code>.</p>`
`Examples:</p>`
`Google, Apple, Microsoft, et al</li>`Red Hat</a></li> Mozilla</a></li> Igalia</a></li> </ul> Embedded</h1> There are a number of companies developing specialised compiler toolchains for embedded applications. For the most part, these are C/C++ compilers although Pascal and Ada do exist.</p> These compilers are often from the hardware vendors themselves and come with an IDE, build system, etc. There are also third-party toolchains that have a focus on code size optimisation or are formally certified in some way to justify their pricetag over the hardware vendor's toolchain or open-source alternatives like GCC. Interestingly, this is one niche where the practice of selling compiler licenses still thrives.</p> Examples:</p> ARM</a></li> Qualcomm</a></li> Green Hills Software</a></li> AdaCore</a></li> SEGGER</a></li> </ul> Hardware accelerators</h1> With improvements in processor performance diminishing in recent years, there's an incentive to develop domain specific hardware</em> that is designed to tackle certain applications more efficiently. Chris Lattner did a great talk on this topic here</a>. The development of new hardware and ISA</abbr>s necessitates new compilers to target them.</p> The most common application is machine learning. These compilers are unusual in that they often don't have a source language in the conventional sense as they're trying to lower ML models (Tensorflow, PyTorch, ONNX, etc) to hardware.</p> I'll go ahead and throw GPU compiler stacks such as NVIDIA's CUDA</a> and AMD's ROCm</a> under this category too as they share some similarities.</p> Examples:</p> Google TPU</a></li> Amazon Inferentia</a></li> Groq</a></li> Tenstorrent</a></li> Fabric Cryptography</a></li> </ul> Web3</h1> There are a number of companies in the Web3 space that are developing programming languages for writing smart contracts and dApps</abbr>. The most widespread of these is Solidity</a>, a language designed to target the Ethereum network's EVM</abbr>.</p> Since the introduction of the EVM, there have been a growing number of networks with their own virtual machines, smart contract languages, toolchains, etc.</p> Examples:</p> Solana</a></li> DFINITY</a></li> Aleo</a></li> </ul> Static analysis</h1> There are a number of companies with a SaaS</abbr> offering for statically analysing code. These often come as part of a larger suite of security-focused tools.</p> Unlike the basic static analysis implemented in most compilers, these dedicated tools usually have cross-language support and provide more in-depth analysis such as: detecting security vulnerabilities (e.g. SQL injection), checking code quality or adherence to style guides, and analysis that would be too intensive to run as part of a developer's edit-compile-run workflow.</p> Most of these tools are heavily customisable and in the case of CodeQL, can be extended with a query language.</p> Examples:</p> Synopsys Coverity</a></li> Perforce Klocwork</a></li> Snyk</a></li> GitHub CodeQL</a></li> Praetorian</a></li> </ul> Security research</h1> DARPA</a> funds a lot of security research that involves compiler-adjacent problems such as: program analysis, decompilation, binary patching, etc.</p> These projects are a blend of academia and industry as they are often about validating novel ideas and producing proof-of-concepts to pave the way for further research rather than building industry-grade implementations.</p> Examples:</p> Trail of Bits</a></li> Galois</a></li> GrammaTech</a></li> </ul> Domain specific languages</h1> There's a certain school of thought that believes that the most efficient way to solve a problem is to write a specialised language for a given domain. The most widespread example of that is SQL! These small languages or DSLs</em> are used for everything from querying databases to scripting game engines.</p> Database query languages are a particularly interesting example as, internally, most database systems contain a query compiler that performs optimisations to the query and outputs a query plan</em> that outlines how to access and transform data in an efficient way. If you squint a bit, this looks a lot like traditional compiler optimisation.</p> This category is a bit of a catch-all as a lot of these use-cases don't have much in common but the point is that these small specialised languages exist everywhere</strong>. I've expanded on the examples below to give you a sense of just how diverse this category is.</p> Examples:</p> Roblox</a>: Building a Lua variant called Luau</a> for scripting the Roblox game engine.</li> Snowflake</a>: Building a query compiler for the Snowflake SQL Engine.</li> Lokad</a>: Building a DSL for representing supply chain optimisation problems.</li> Workday</a>: Building in-house proprietary programming languages for development of the Workday platform.</li> </ul>

New year, new blog

2024-01-31T00:00:00+00:00

One of my New Year's Resolutions for 2024 was to start a technical blog. Since it's already the last day of January and I've yet to write anything, I felt obliged to kick things off with an introductory post describing what I'm intending to write about as well as some guiding principles that I have in mind.</p>

If any of this sounds interesting to you, please bookmark my blog or add my feed</a> to your RSS reader to keep up to date!</p>

Ideas</h1>
In a few weeks, I'll be beginning a new job working on a software stack for a chip designed to accelerate cryptographic workloads (in particular, ZKP</abbr> and FHE</abbr>). This is another step in a journey that I've been on since 2019 to become a compilers specialist.</p>
Naturally, I'm planning to document some of the skills that I have to learn on the job. Aside from general compiler knowledge, this will involve diving deeper into cryptography and mathematics.</p>
I'd also like to write some more industry focused content about what compiler jobs are available, how someone might go about getting one of these jobs and perhaps talk a bit about my own experience attempting to make this transition from a generalist systems programmer to a compilers specialist. Given how niche these jobs are, there understandably isn't a great deal written on this topic so it can feel daunting to figure this all out yourself.</p>
More generally, I love to learn how things work under the hood and I suspect that this is a large part of why I'm drawn to low-level programming to begin with. Apart from compilers and runtimes, there are a lot of different types of software that I'd like to understand on a deeper level: debuggers, linkers, compression algorithms, fuzzers, etc. I'd like to be able to describe what makes them tick and how someone would go about building one themselves, ideally accompanied by a GitHub repo with working code.</p>
Principles</h1>
When it comes to learning how to build computer programs, I believe that the most important thing is to be prolific. As I was starting out in the industry, I wasted a lot of time bike-shedding on personal projects and trying to figure out the optimal way to design a given piece of code. The reality is that the time I spent didn't improve the code much as I simply didn't know</strong> how to design software and that, even with unlimited time, I probably wouldn't have arrived at a good solution. I'm now of the opinion that it's more important to just churn it out, get as much experience as possible and observe firsthand what works well and what doesn't.</p>
I'm trying to apply this philosophy to writing as well. That is, prioritise writing frequently over obsessing about the form, and be at peace with the fact that my writing won't be great at first. My hope is that by writing consistently, producing quality writing will start to feel normal and become part of my modus operandi.</p>
Another principle that I want to stick to is to write in an approachable way that is understandable by the working programmer. One of the difficulties I had learning about compilers was that most learning materials are couched in academic language. I'd like to write in a common sense way that makes these topics feel more friendly and do my part to perhaps dispel some of the mystique around compilers and programming languages.</p>

Alex Cameron

What does a compiler job look like in the year 2024?

New year, new blog