Fuzzing¶

Random input data¶

This is probably the most obvious fuzzing method: You have code that processes data and you provide it with random or mutated data, hoping that it will uncover bugs in your implementation. Examples are media formats like JPEG or H.264, but basically anything that involves processing a “blob” of data can be a valuable target. Countless security vulnerabilities in a variety of libraries and programs have been found using this method (the AFLFuzz bug-o-rama gives a good impression).

Common tools for this task are e.g. libFuzzer and AFLFuzz, but also specialized tools with custom logic like LangFuzz and Avalanche.

Random API Usage¶

Randomly testing APIs is especially helpful with parts of software that expose a well-defined interface (see also Well-defined behavior and Safety). If this interface is additionally exposed to untrusted parties/content, then this is a strong sign that random API testing would be worthwhile here, also for security reasons. APIs can be anything from C++ layer code to APIs offered in the browser.

A good example for a fuzzing target here is the DOM (Document Object Model) and various other browser APIs. The browser exposes a variety of different APIs for working with documents, media, communication, storage, etc. with a growing complexity. Each of these APIs has potential bugs that can be uncovered with fuzzing. At Mozilla, we currently use domino (internal tool) for this purpose.

Random UI Interaction¶

A third way to test programs and in particular user interfaces is by directly interacting with the UI in a random way, typically in combination with other actions the program has to perform. Imagine for example an automated browser that surfs through the web and randomly performs actions such as scrolling, zooming and clicking links. The nice thing about this approach is that you likely find many issues that the end-user also experiences. However, this approach typically suffers from bad reproducibility (see also Reproducibility) and is therefore often of limited use.

An example for a fuzzing tool using this technique is Android Monkey. At Mozilla however, we currently don’t make much use of this approach.

Full Browser Fuzzing¶

The most obvious method of testing would be to test the full browser and doing so is required for certain features like the DOM and other APIs. The advantage here is that we have all the features of the browser available and testing happens closely to what we actually ship. The downside here though is that browser testing is by far the slowest of all testing methods. In addition, it has the most amount of non-determinism involved (resulting e.g. in intermittent testcases). Browser fuzzing at Mozilla is largely done with the Grizzly framework (meta bug) and one of the most successful fuzzers is the Domino tool (meta bug).

Summarizing, full browser fuzzing is the right technique to investigate if your feature really requires it. Consider using other methods (see below) if your code can be exercised in this way.

The Fuzzing Interface¶

Fuzzing Interface

The fuzzing interface is glue code living in mozilla-central in order to make it easier for developers and security researchers to test C/C++ code with either libFuzzer or afl-fuzz.

This interface offers a gtest (C++ unit test) level component based fuzzing approach and is suitable for anything that could also be tested/exercised using a gtest. This method is by far the fastest, but usually limited to testing isolated components that can be instantiated on this level. Utilizing this method requires you to write a fuzzing target similar to writing a gtest. This target will automatically be usable with libFuzzer and AFLFuzz. We offer a comprehensive manual that describes how to write and utilize your own target.

A simple example here is the SDP parser target, which tests the SipccSdpParser in our codebase.

Shell-based Fuzzing¶

Some of our fuzzing, e.g. JS Engine testing, happens in a separate shell program. For JS, this is the JS shell also used for most of the JS tests and development. In theory, xpcshell could also be used for testing but so far, there has not been a use case for this (most things that can be reached through xpcshell can also be tested on the gtest level).

Identifying the right level of fuzzing is the first step towards continuous fuzz testing of your code.

Defect Oracles¶

Fuzzing is only effective if you are able to know when a problem has been found. Crashes are typically problems if the unit being tested is safe for fuzzing (see Well-defined behavior and Safety). But there are many more problems that you would want to find, correctness issues, corruptions that don’t necessarily crash etc. For this, you need an oracle that tells you something is wrong.

The simplest defect oracle is the assertion (ex: MOZ_ASSERT). Assertions are a very powerful instrument because they can be used to determine if your program is performing correctly, even if the bug would not lead to any sort of crash. They can encode arbitrarily complex information about what is considered correct, information that might otherwise only exist in the developers’ minds.

External tools like the sanitizers (AddressSanitizer aka ASan, ThreadSanitizer aka TSan, MemorySanitizer aka MSan and UndefinedBehaviorSanitizer - UBSan) can also serve as oracles for sometimes severe issues that would not necessarily crash. Making sure that these tools can be used on your code is highly useful.

Examples for bugs found with sanitizers are bug 1419608, bug 1580288 and bug 922603, but since we started using sanitizers, we have found over 1000 bugs with these tools.

Another defect oracle can be a reference implementation. Comparing program behavior (typically output) between two programs or two modes of the same program that should produce the same outputs can find complex correctness issues. This method is often called differential testing.

One example where this is regularly used to find issues is the Mozilla JavaScript engine: Running random programs with and without JIT compilation enabled finds lots of problems with the JIT implementation. One example for such a bug is Bug 1404636.

Component Decoupling¶

Being able to test components in isolation can be an advantage for fuzzing (both for performance and reproducibility). Clear boundaries between different components and documentation that explains the contracts usually help with this goal. Sometimes it might be useful to mock a certain component that the target component is interacting with and that is much harder if the components are tightly coupled and their contracts unclear. Of course, this does not mean that one should only test components in isolation. Sometimes, testing the interaction between them is even desirable and does not hurt performance at all.

Avoiding external I/O¶

External I/O like network or file interactions are bad for performance and can introduce additional non-determinism. Providing interfaces to process data directly from memory instead is usually much more helpful.

Well-defined Behavior and Safety¶

This requirement mostly ties in where defect oracles ended and is one of the most important problems seen in the wild nowadays with fuzzing. If a part of your program’s behavior is unspecified, then this potentially leads to bad times if the behavior is considered a defect by fuzzing. For example, if your code has crashes that are not considered bugs, then your code might be unsuitable for fuzzing. Your component should be fuzzing safe, meaning that any defect oracle (e.g. assertion or crash) triggered by the fuzzer is considered a bug. This important aspect is often neglected. Be aware that any false positives cause both performance degradation and additional manual work for your fuzzing team. The Mozilla JS developers for example have implemented this concept in a “–fuzzing-safe” switch which disables harmful functions. Sometimes, crashes cannot be avoided for handling certain error conditions. In such situations, it is important to mark these crashes in a way the fuzzer can recognize and distinguish them from undesired crashes. However, keep in mind that crashes in general can be disruptive to the fuzzing process. Performance is an important aspect of fuzzing and frequent crashes can severely degrade performance.

Reproducibility¶

Being able to reproduce issues found with fuzzing is necessary for several reasons: First, you as the developer probably want a test that reproduces the issue so you can debug it better. Our feedback from most developers is that traces without a reproducible test can help to find a problem, but it makes the whole process very complicated. Some of these non-reproducible bugs never get fixed. Second, having a reproducible test also helps the triage process by allowing an automated bisection to find the responsible developer. Last but not least, the test can be added to a test suite, used for automated verification of fixes and even serve as a basis for more fuzzing.

Adding functionality to the program that improve reproducibility is therefore a good idea in case non-reproducible issues are found. Some examples are shown in the next section.

While many problems with reproducibility are specific for the project you are working on, there is one source of these problems that many programs have in common: Threading. While some bugs only occur in the first place due to concurrency, some other bugs would be perfectly reproducible without threads, but are intermittent and hard to with threading enabled. If the bug is indeed caused by a data race, then tools like ThreadSanitizer will help and we are currently working on making ThreadSanitizer usable on Firefox. For bugs that are not caused by threading, it sometimes makes sense to be able to disable threading or limit the amount of worker threads involved.

Supporting Code¶

Some possibilities of what support implementations for fuzzing can do have already been named in the previous sections: Additional defect oracles and functionality to improve reproducibility and safety. In fact, many features added specifically for fuzzing fit into one of these categories. However, there’s room for more: Often, there are ways to make it easier for fuzzers to exercise complex and hard to reach parts of your code. For example, if a certain optimization feature is only turned on under very specific conditions (that are not a requirement for the optimization), then it makes sense to add a functionality to force it on. Then, a fuzzer can hit the optimization code much more frequently, increasing the chance to find issues. Some examples from Firefox and SpiderMonkey:

• The FuzzingFunctions interface in the browser allows fuzzing tools to perform GC/CC, tune various settings related to garbage collection or enable features like accessibility mode. Being able to force a garbage collection at a specific time helped identifying lots of problems in the past.

• The –ion-eager and –baseline-eager flags for the JS shell force JIT compilation at various stages, rather than using the builtin heuristic to enable it only for hot functions.

• The –no-threads flag disables all threading (if possible) in the JS shell. This makes some bugs reproduce deterministically that would otherwise be intermittent and harder to find. However, some bugs that only occur with threading can’t be found with this option enabled.

Another important feature that must be turned off for fuzzing is checksums. Many file formats use checksums to validate a file before processing it. If a checksum feature is still enabled, fuzzers are likely never going to produce valid files. The same often holds for cryptographic signatures. Being able to turn off the validation of these features as part of a fuzzing switch is extremely helpful.

An example for such a checksum can be found in the FlacDemuxer.

Test Samples¶

Some fuzzing strategies make use of existing data that is mutated to produce the new random data. In fact, mutation-based strategies are typically superior to others if the original samples are of good quality because the originals carry a lot of semantics that the fuzzer does not have to know about or implement. However, success here really stands and falls with the quality of the samples. If the originals don’t cover certain parts of the implementation, then the fuzzer will also have to do more work to get there.

Fuzz Blockers¶

Fuzz blockers are issues that prevent fuzzers from being as effective as possible. Depending on the fuzzer and its scope a fuzz blocker in one area (or component) can impede performance in other areas and in some cases block the fuzzer all together. Some examples are:

• Frequent crashes - These can block code paths and waste compute resources due to the need to relaunch the fuzzing target and handle the results (regardless of whether it is ignored or reported). This can also include assertions that are mostly benign in many cases are but easily triggered by fuzzers.

• Frequent hangs / timeouts - This includes any issue that slows down or blocks execution of the fuzzer or the target.

• Hard to bucket - This includes crashes such as stack overflows or any issue that crashes in an inconsistent location. This also includes issues that corrupt logs/debugger output or provide a broken/invalid crash report.

• Broken builds - This is fairly straightforward, without up-to-date builds fuzzers are unable to run or verify fixes.

• Missing instrumentation - In some cases tools such as ASan are used as defect oracles and are required by the fuzzing tools to allow for proper automation. In other cases incomplete instrumentation can give a false sense of stability or make investigating issues much more time consuming. Although this is not necessarily blocking the fuzzers it should be prioritized appropriately.

Since these types of crashes harm the overall fuzzing progress, it is important for them to be addressed in a timely manner. Even if the bug itself might seem trivial and low priority for the product, it can still have devastating effects on fuzzing and hence prevent finding other critical issues.

Issues in Bugzilla are marked as fuzz blockers by adding “[fuzzblocker]” to the “Whiteboard” field. A list of open issues marked as fuzz blockers can be found on Bugzilla.

Documentation¶

It is important for the fuzzing team to know how your software, tests and designs work. Even obvious tasks, like how a test program is supposed to be invoked, which options are safe, etc. might be hard to figure out for the person doing the testing, just as you are reading this manual right now to find out what is important in fuzzing.

Contact Us¶

The fuzzing team can be reached at fuzzing@mozilla.com or on Matrix and will be happy to help you with any questions about fuzzing you might have. We can help you find the right method of fuzzing for your feature, collaborate on the implementation and provide the infrastructure to run it and process the results accordingly.