PPB: Pico Protobuf

PPB is a C11 lexer for binary protobuf data, built for decoding untrusted bytes under caller-controlled resource budgets. Decoding is allocation-free, in place, and fully iterative: numeric fields decode to 64-bit values, and everything else (strings, bytes, submessages, packed repeated fields) comes out as subslices of the read-only input buffer. Any byte sequence is safe to decode, regardless of the schema; malformed input results in a clean error, never undefined behavior. A header-only C++20 wrapper makes the C core easier to use, at the expense of some hidden (but bounded) recursion, and a lot of compile-time metaprogramming noise. A protoc schema generator for the C++ wrapper further makes it easy to synchronise the PPB schema with authoritative .proto files. Neither the C core nor the C++ wrapper uses non-local exits (longjmp) or exceptions.

The layered design goes from a hardened but annoying to use C core, to a friendlier C++ wrapper, to a protoc generator plugin for which we can meaningfully check conformance. The C core's memory safety, termination, and progress are statically verified with Frama-C's WP and Eva plugins, without any semantic requirement on configuration for safety (higher level postconditions do rely on the ppb_encoded_tag array making sense); the C and C++ APIs are also fuzzed under sanitizers, including with schemas emitted by the protoc plugin; finally, the result of decoding with generated schemas is validated against google's libprotobuf parser by differential fuzzers and the conformance suite.

Consider PPB when you parse protobuf from an untrusted producer (adversarial bytes are a tested input), when you want to explicitly issue every heap allocation yourself, with enough information to right-size containers before any field is dispatched, when message shapes are stable (declared directly in code, or generated from .proto files by the bundled protoc-gen-ppb plugin), or when "fast enough and predictable" beats "fastest on average."

The trade-off, compared with a full implementation (Google's libprotobuf) or a compact encoder and decoder (nanopb):

You give up encoding, groups, extensions, runtime descriptors, reflection, streaming input (the entire message must fit in one contiguous read-only buffer), and MSVC support (PPB requires gcc or clang).

You get a decoder that never allocates and never recurses (your code is in charge of allocation and recursion); exact occurrence counts and payload-byte totals before any value is dispatched, so every container can be sized upfront; a few hundred bytes of stack, and runtime proportional to the toplevel field count rather than payload bytes; a C core whose memory safety, termination, and progress guarantees are formally verified on top of unit tests and fuzzing; and decoded results validated against google's libprotobuf parser. Each of these claims has a re-verification recipe in AUDITING.md.

When the trade seems questionable, use nanopb or libprotobuf; PPB is deliberately a small lexer, not a full protobuf implementation.

PPB itself never calls malloc; it only writes through the caller's fields[] array. When your code needs to allocate output storage, ppb_prescan first aggregates per-field counts and payload-byte totals so you can size every container exactly once before any field-by-field dispatch runs. PPB never decides when to allocate, and hands you the information to allocate the right amount when you do. Minimizing allocations is up to your handlers: pre-reserve() std::vectors, prefer std::string_view over std::string, etc., and you'll minimize the impact of protobuf parsing on the heap.

PPB never writes through the input buffer: the encoded bytes may live in read-only memory (a PROT_READ mapping, a flashed section, a const array), and decoded bytes, string, submessage, and packed values are subslices of that same buffer, valid for as long as the buffer is. The decoder itself doesn't allocate any heap, uses a few hundred bytes of stack, and never descends into submessages on its own (the C++ wrapper recurses only within an explicit depth budget, zero by default), so there's no hidden stack utilisation.

The prescan statistics also let code check the input's order of magnitude before preallocating storage at the correct capacity.

Corrupt input is always handled safely and never silently reinterpreted; for example, truncated inputs, and corrupt length fields that exceed the input buffer's size all result in explicit errors codes. Except for niche gotchas, inputs acepted by both PPB and libprotobuf decode to identical values (checked with differential testing). The main differences are:

1. PPB rejects legacy groups
2. PPB rejects oversized tags that libprotobuf tolerates or truncates
3. PPB accepts overlong length varints that libprotobuf caps at five bytes
4. PPB leaves UTF-8 validation of string fields to the caller.

N.B., it's easy for corrupt protobuf bytes to still parse fine. To detect corruption rather than merely survive it, you must add your own checksum.

PPB makes two kinds of claims, each backed by different evidence.

When it comes to safety, PPB is safe (no memory corruption, no undefined behavior, no unbounded recursion or infinite loop) on any input. The evidence is a combination of static analysis for the C core, and fuzzing in combination with sanitizers for everything.

PPB also aims to accept all canonically encoded binary protobuf inputs, and to agree with google's libprotobuf for every input they both accept. This is tested with google's conformance testing, and with a fuzzer that compares the result of C++ PPB parsers with libprotobuf's.

The wire bytes are untrusted: any input buffer is safe to lex. The other inputs are trusted, with two tiers of preconditions. Correct output requires fields[] to be zero-initialized on allocation and tag arrays to be well formed (check once with ppb_validate_tags), but even invalid trusted inputs (an unsorted tag array, non-zero-initialized fields[], etc.) cause at worst surprising results or, in builds with assertions enabled, assertion failures; never memory unsafety, undefined behavior, non-termination, or successful return without progress (for lexn). SECURITY.md states the threat model and the full two-tier breakdown of trusted-input preconditions.

The evidence:

• Frama-C's WP discharges memory safety, termination, and progress for the C core, regardless of the contents of input buffers and arrays (trusted or otherwise); Eva additionally checks for runtime errors on a bounded harness. See the proof coverage and the admit ledger for the assumptions the provers accept without discharge.
• The C and C++ APIs are fuzzed under ASan, UBSan, MSan, and TySan, including a harness that feeds invalid trusted inputs (unvalidated tag arrays) to the public API.
• The C core never allocates, has no mutable global state, and needs < 400 bytes of stack, measured on gcc 12/x86-64.

The evidence, strongest first:

• Differential fuzzers decode the same bytes with PPB (through schemas emitted by the production generator) and with libprotobuf, then compare accept/reject decisions and decoded messages field by field.
• The conformance suite runs google's conformance_test_runner against a PPB-backed testee.
• Golden tests check the lexer against protoscope output, and the unit tests are hardened by mutation testing (mutants.py; surviving mutants are annotated in the source, with justifications).

Where the two parsers disagree on whether an input is acceptable, the divergence should be well understood and enumerated below; the differential fuzzers trap on any difference outside this list.

This table mirrors the whitelist in the header comment of differential/fuzz/fuzz_sink_bytes.cc, which is the machine-checked source of truth: that fuzzer feeds arbitrary bytes to both parsers and traps on any accept/reject or value difference outside these rows.

The full list is in Gotchas, decoding quirks, and footguns.

Of these rows, only the UTF-8 one shows up often. Run both parsers on realistic, lightly mutated inputs and almost every "PPB accepts, libprotobuf rejects" case is a proto3 string that fails UTF-8 validation. The rest need hand-built wire bytes (overlong tags, groups, 6-to-10-byte length varints) that random mutation rarely produces. To set the UTF-8 cases aside, retype every string as bytes in a shadow schema: libprotobuf stops enforcing UTF-8 on bytes, so the two parsers agree again and any remaining divergence is a non-UTF-8 one.

The guarantees are not uniform across the tree: assurance is strongest in the C core and weaker for the convenience C++ layer.

The last row is an important point to keep in mind: PPB doesn't do as much as other parsers, so there are more ways for user code to do the wrong thing.

PPB is an allocation-free non-recursive lexer for protobuf binary encoding (v2/v3, no groups). The PPB interface requires the entire serialized message to be in a contiguous read-only buffer, and decodes values to 64-bit values, or as subslices in that buffer. The lexer's wire-format edge cases are enumerated in Gotchas, decoding quirks, and footguns.

See README_CPP.md for a more convenient C++ wrapper; you should still read this document first to understand how the underlying library works. N.B., only the C side (ppb.h, ppb.c) is formally verified. The C++ side is merely unit tested and fuzzed (it's a lot of code, but almost all of it is consteval).

For .proto-driven workflows, protoc-gen-ppb is a protoc plugin that generates the C++ wrapper's ppb::schema headers straight from your .proto files (requires protoc and uv):

protoc --plugin=protoc-gen-ppb=generator/protoc_gen_ppb.py \
       --proto_path=protos --ppb_out=gen my.proto

This writes one gen/my.ppb.hpp header per input file, holding a compile-time schema, a field-key enum, and a recursion depth bound per message. README_CPP.md's "Generating schemas from .proto files" section shows how the generated names plug into ppb::reader; generator/README.md is the full reference (options, limitations, compile cost).

The generated decoding pipeline (generator, wrapper, C core) is validated against libprotobuf: Google's protobuf conformance suites run against a version-locked libprotobuf with zero unexpected failures (the proto3 suite has zero expected failures, too), and the differential harnesses in differential/ compare PPB's decodings with libprotobuf's parser on fixed, pseudorandom, and fuzzer-generated inputs. Deliberate divergences are documented in differential/GAPS.md.

For encoding, Google's protobuf libraries are the obvious choice. When the producer also cares about allocations and copies, ProtoZero (the one at https://github.com/google/perfetto/tree/main/include/perfetto/protozero, not to be confused with the mapbox project of the same name) pairs well with PPB.

The core pattern is: call ppb_validate_tags to confirm the array of tags to parse has a valid structure, call ppb_prescan once to collect field statistics for preallocation, then call ppb_lexn in a loop to walk the fields. While gathering statistics and validating the input, ppb_prescan also saves the last value associated with each tag, which implements exactly the last-write-wins semantics needed for non-repeated fields. We thus only need ppb_lexn for repeated fields, and only when ppb_prescan reports multiple occurrences of such a field (packed repeated fields, for example, usually appear at most once).

The ppb_lexn function is always safe to call, even without ppb_prescan. Both functions operate only on toplevel fields; call them recursively on nested submessages.

This usage pattern lets the calling program choose how to handle nesting and variable-length values. Both ppb_prescan and ppb_lexn's runtimes scale with the number of toplevel fields, not the total number of message bytes, so recursive processing time remains linear in the number of message bytes.

PPB targets C11 with GCC extensions; CI tests GCC and clang builds on 64-bit (x86-64, aarch64) and 32-bit (i686) little-endian platforms, and on big-endian 64-bit s390x (via QEMU).

MSVC is not supported (but patches are welcome). I'd also consider patches for strict C11 compliance, it just wouldn't be useful for me.

The C core and the C++ wrapper have different requirements. CI builds with the default GCC and clang provided by the CI image rather than specific fixed versions, so the practical floor is "a GCC or clang new enough to fully implement the standard each layer needs" (the wrapper's consteval-heavy metaprogramming is the binding constraint).

These rows are experimentally reproducible: rebuild the wrapper tests with EXTRA_FLAGS='-fno-exceptions -fno-rtti', toggle -mbmi2, and add -DNDEBUG to confirm the assertion-free build still passes. See AUDITING.md for the nm check that -DNDEBUG eliminates the assertion handler dependency.

Build the library with make to produce build/libppb.a and build/libppb.so. Link against build/libppb.a (or build/libppb.so) and include include/ppb/ppb.h. The library has no external dependencies and requires C11 (-std=c11).

struct ppb_buf: a read-only byte slice { const void *buf; size_t size; }. PPB never writes through the buf pointer. ppb_lexn updates the struct in place to advance through the buffer; ppb_prescan consumes a ppb_buf by value. Both construct subslices for length-prefixed payloads (variable-length values).

struct ppb_encoded_tag: a tag identity created with PPB_TAG(uint64_t field_number, enum ppb_wire_type). Callers assemble a const struct ppb_encoded_tag[] array sorted in ascending order and pass it to ppb_prescan / ppb_lexn; PPB never writes through this pointer. Field number -1 (i.e., UINT64_MAX cast to the field-number argument) is a catch-all that matches any unknown tag of a given wire type.

The tags array must be in strictly ascending order (by .bits); duplicates are rejected by ppb_validate_tags with PPB_ERROR_UNSORTED_FIELD_ARR, same as unsorted arrays.

struct ppb_field: mutable per-call state for one decoded field. After a call to ppb_prescan, field.m holds aggregate metadata (occurrence count, total/min/max value bytes for all wire types) and field.v holds the last decoded value. Use ppb_lexn to observe every field occurrence: it updates each field.v at most once per call. The caller owns the struct ppb_field[] array, which is passed alongside (but separately from) the tags array (field i matches tag i).

For atomic wire types (VARINT, I32, I64), field.m.lost_distinct_u64 reports whether ppb_prescan lost a distinct prior value to last-write-wins: it is set when two or more occurrences had different v.u64 bits, so callers can tell whether ppb_lexn is needed to recover the lost data. The flag is never set for LEN fields, since length-prefixed payloads aren't tracked through v.u64. Catch-all entries (PPB_TAG(-1, ...)) aggregate every unknown field of a given wire type into one bucket, so the flag fires whenever any two such fields had distinct v.u64 bits: e.g., one VARINT at field 12 and one VARINT at field 99 with different values will set it even though neither field number repeats. The flag is useless for catch-alls.

As a lexer, PPB exposes users to the subtle diversity in protobuf wire encoding; that's added flexibility when you want to do something specific, but can lead to surprise. The list below is meant to cover every decoding quirk in the C core (a missing entry is a doc bug); the C++ wrapper and the schema generator add their own quirks on top, listed in README_CPP.md.

1. Tags are only matched when encoded canonically (minimally) on the wire; a non-canonically encoded tag is treated as an unknown field. It's unknown if any protobuf encoder violates that assumption, but the assumption is deep in PPB's design. This is only an issue for tags; some encoders like to use overlong encodings for field lengths, and that's supported.

2. Value and length varints may span up to 10 bytes, redundant (overlong) encodings included. Like many protobuf implementations, PPB silently discards bits 1-6 of the 10th byte, so only bit 0 of byte 10 reaches bit 63 of the result. Varints longer than 10 bytes are rejected with PPB_ERROR_CORRUPT_VARINT. One divergence to be aware of: libprotobuf rejects tag and length varints longer than 5 encoded bytes, while PPB decodes them (an overlong tag as an unknown field, an overlong length as usual), so the two parsers disagree on accept/reject for such inputs.

3. Length prefixes are decoded as full 64-bit unsigned values and checked only against the remaining input, so payloads and messages larger than 2 GiB decode fine (libprotobuf, for one, caps messages at 2 GiB). The only size cap is the input buffer's PTRDIFF_MAX maximum.

4. Wire types 3 and 4 (legacy groups) and the reserved wire types 6 and 7 are rejected with PPB_ERROR_CORRUPT_TAG wherever they appear, including in unknown fields that would otherwise be skipped: a message that contains a group anywhere cannot be lexed.

5. Field number 0 is rejected with PPB_ERROR_CORRUPT_TAG in every encoding by ppb_prescan. ppb_lexn, on the other hand, rejects only the canonical single-byte encoding: an overlong field-0 tag lexes as an unknown field. The prescan pass also rejects (PPB_ERROR_CORRUPT_TAG) a tag whose value exceeds UINT32_MAX (field number above protobuf's 2**29 - 1 maximum) or whose varint encoding is longer than 5 bytes: a valid canonically encoded tag fits in a uint32_t / 5 bytes. This is a prescan-only check: a standalone ppb_lexn lexes such a tag as an unknown field instead. libprotobuf rejects tags longer than 5 bytes but truncates shorter tags to 32 bits, so it may accept (and misroute) a 5-byte tag whose value exceeds UINT32_MAX; PPB never truncates, and either decodes the tag correctly (lexn), or rejects it (prescan).

6. Remember to use ppb_zag32 when zigzag-decoding sint32 values: you must truncate the encoded integer to 32 bits before decoding. This only matters for non-canonical inputs and matches what Google's C++ parser does on such inputs.

7. The library does not validate that strings are encoded as utf-8. In fact, there's no difference between a bytes and a string field (or submessage or packed repeated) at the wire encoding level. If you care about that, consider a library like simdutf, but you might also want to pay attention to unicode normalization. Either way, that's out of scope for PPB. Validation also interacts with last-write-wins: when a singular string field appears several times on the wire, ppb_prescan retains only the last payload, but a conforming proto3 parser must reject the message when any occurrence is invalid UTF-8, even one overwritten by a later occurrence. If that's what you want, force a ppb_lexn walk (e.g., with singular field semantics in the C++ wrapper).

8. There's no explicit support for oneof, but you can make it happen by populating fields in wire order, and clearing the other types in a oneof when you populate a new one. That only works if you always use ppb_lexn (ppb::field_semantics::always_lexn semantics in C++) whenever you notice more than one member in a oneof is present: the prescan metadata may suffice to tell you what value each member took, but you can't recover the order.

9. "Last write wins" for submessages actually recurses in the submessage's constituent fields. That is, in order to match the way Google protobuf handles repeated values for a non-repeated submessage field, we have to keep parsing into (i.e., like MergeFrom) the submessage and mutate that submessage (and its submessages) in place when we encounter a new value for that message. This means proto3 semantics only work for the fields in a message at the toplevel. In the C++ wrapper, ppb::message<> already defaults to ppb::field_semantics::singular for this reason; give these submessages' direct fields regular non-proto3 last-write-wins semantics (repeated submessages count as toplevel). See README_CPP.md's "Embedded sub-messages" section for the merge-vs-replace details.

10. Packed and unpacked repeated fields have separate tags. It's a different wire type, so different entry. In practice, you may prefer to only support packed encoding when it's available, or at least support only the encoding you expect to see. If you want to support both, you'll have to force a lexn pass even when prescan has all the metadata, because, again, prescan doesn't tell you the order (well, it does, since you could look at the tag pointer, but that's complicated). In the C++ wrapper, that probably means setting ppb::field_semantics::always_lexn on the encoding you don't expect to see.

11. Unexpected encodings are treated like unknown tags (same issue as packed / unpacked repeated, except generalized to, e.g., receiving a LEN value instead of a varint). Other protobuf implementations tned to reject the message as ill-encoded.

12. Handling unknown tags is opt-in, with catch-all entries for each wire type... and decoding the actual tag for a catch-all entry is tricky. In order to decode the tag, you must take the tag ptr from the ppb_field_value struct, and construct a ppb_buf from that ptr to the end of the varint... but you can't know the varint's length without decoding the varint. Instead pad to the end of the original input buffer (we know the varint is valid and in bounds, otherwise prescan/lexn would have rejected it). And at last, you may pass that ppb_buf to ppb_decode_varint.

Complexity: Helper functions ppb_zag and ppb_decode_varint have a bounded runtime (asymptotically constant). The ppb_prescan and ppb_lexn function families run in constant space and Θ(n log m) time, where n is the number of toplevel fields consumed and m is num_fields. Runtime does not scale with payload sizes of length-prefixed fields; that's what makes recursive descent on nested submessages practical.

/*
 * All PPB public functions use the same error enum.  Error codes are
 * always strictly negative.
 */
enum ppb_error
{
    PPB_OK = 0,
    PPB_ERROR_UNSORTED_FIELD_ARR = -1,  /* tags[].bits not strictly ascending */
    PPB_ERROR_SENTINEL_FIELD_ARR =
        -2,  /* tags[].bits includes < 8 (field number 0 is forbidden by the protobuf spec) */
    PPB_ERROR_TRUNCATED_DATA = -3,  /* message cut short at the end of the `ppb_buf` */
    PPB_ERROR_CORRUPT_VARINT = -4,  /* invalid varint encoding (longer than 10 bytes) */
    PPB_ERROR_CORRUPT_TAG =
        -5,  /* invalid tag encoding (zero, longer than 8 bytes, or unsupported wire type) */
    PPB_ERROR_LIMIT_EXCEEDED = -6,  /* consumed bytes exceeded hard limit */
    PPB_ERROR_DEPTH_EXCEEDED = -7,  /* recursion depth budget exhausted (for ppb.hpp and client code) */

    /*
     * This error code is reserved for the convenience of client code
     * and never actually generated by PPB (C core or C++ wrapper).
     */
    PPB_ERROR_INVALID_UTF8 = -64  /* string field is not well-formed UTF-8 (not checked by PPB) */
};

/*
 * Traverses `buf` scanning all toplevel fields (up to `max_lexed_fields`).
 * Tags must be pre-validated with `ppb_validate_tags`; skipping validation
 * does not cause undefined behavior but may produce incorrect results or
 * trigger assertion failures when PPB is built with assertions.
 *
 * Returns the number of bytes traversed, or a negative `ppb_error`.
 * The count equals `buf.size` when the whole buffer was scanned; a
 * smaller count means `max_lexed_fields` stopped the scan early.
 */
static inline ptrdiff_t
ppb_prescan(struct ppb_buf buf, size_t num_fields, const struct ppb_encoded_tag *__restrict tags,
    struct ppb_field *__restrict fields, size_t max_lexed_fields)
{
    return ppb_prescan_impl(buf, num_fields, tags, fields, max_lexed_fields, SIZE_MAX, PPB_OK);
}

/*
 * Like `ppb_prescan`, but stops at the first field boundary where bytes
 * consumed >= `limit`.  If consumed bytes exceed `limit`, returns
 * `PPB_ERROR_LIMIT_EXCEEDED`.  Tags must be pre-validated with
 * `ppb_validate_tags`; skipping validation does not cause undefined
 * behavior but may produce incorrect results or trigger assertion
 * failures when PPB is built with assertions.
 */
static inline ptrdiff_t
ppb_prescan_with_hard_limit(struct ppb_buf buf, size_t limit, size_t num_fields,
    const struct ppb_encoded_tag *__restrict tags, struct ppb_field *__restrict fields,
    size_t max_lexed_fields)
{
    return ppb_prescan_impl(buf, num_fields, tags, fields, max_lexed_fields, limit, PPB_ERROR_LIMIT_EXCEEDED);
}

/*
 * Like `ppb_prescan`, but stops at the first field boundary where bytes
 * consumed >= `limit`.  Consuming more than `limit` bytes is not an error.
 * Tags must be pre-validated with `ppb_validate_tags`; skipping validation
 * does not cause undefined behavior but may produce incorrect results or
 * trigger assertion failures when PPB is built with assertions.
 */
static inline ptrdiff_t
ppb_prescan_with_soft_limit(struct ppb_buf buf, size_t limit, size_t num_fields,
    const struct ppb_encoded_tag *__restrict tags, struct ppb_field *__restrict fields,
    size_t max_lexed_fields)
{
    return ppb_prescan_impl(buf, num_fields, tags, fields, max_lexed_fields, limit, PPB_OK);
}

/*
 * Consumes from `buf` up to `max_lexed_fields` toplevel fields in
 * strictly ascending order.  Returns the decoded field range and status.
 * Tags must be pre-validated with `ppb_validate_tags`; skipping validation
 * does not cause undefined behavior but may produce incorrect results or
 * trigger assertion failures when PPB is built with assertions.
 */
static inline struct ppb_lexn_ret
ppb_lexn(struct ppb_buf *__restrict buf, size_t num_fields, const struct ppb_encoded_tag *__restrict tags,
    struct ppb_field *__restrict fields, size_t max_lexed_fields)
{
    return ppb_lexn_impl(buf, num_fields, tags, fields, max_lexed_fields, SIZE_MAX, PPB_OK);
}

/*
 * Like `ppb_lexn`, but stops at the first field boundary where bytes
 * consumed >= `limit`.  If consumed bytes exceed `limit`, the returned
 * `status` is `PPB_ERROR_LIMIT_EXCEEDED`.  Tags must be pre-validated with
 * `ppb_validate_tags`; skipping validation does not cause undefined behavior
 * but may produce incorrect results or trigger assertion failures when PPB
 * is built with assertions.
 */
static inline struct ppb_lexn_ret
ppb_lexn_with_hard_limit(struct ppb_buf *__restrict buf, size_t limit, size_t num_fields,
    const struct ppb_encoded_tag *__restrict tags, struct ppb_field *__restrict fields,
    size_t max_lexed_fields)
{
    return ppb_lexn_impl(buf, num_fields, tags, fields, max_lexed_fields, limit, PPB_ERROR_LIMIT_EXCEEDED);
}

/*
 * Like `ppb_lexn`, but stops at the first field boundary where bytes
 * consumed >= `limit`.  Consuming more than `limit` bytes is not an error.
 * Tags must be pre-validated with `ppb_validate_tags`; skipping validation
 * does not cause undefined behavior but may produce incorrect results or
 * trigger assertion failures when PPB is built with assertions.
 */
static inline struct ppb_lexn_ret
ppb_lexn_with_soft_limit(struct ppb_buf *__restrict buf, size_t limit, size_t num_fields,
    const struct ppb_encoded_tag *__restrict tags, struct ppb_field *__restrict fields,
    size_t max_lexed_fields)
{
    return ppb_lexn_impl(buf, num_fields, tags, fields, max_lexed_fields, limit, PPB_OK);
}

/*
 * Decodes zigzag-encoded sint32 and sint64 values.
 *
 * A 32-bit unsigned int value will decode to an `int32_t`,
 * but you want to use `ppb_zag32` for sint32.
 */
static inline int64_t
ppb_zag(uint64_t x)
{
    return (int64_t)((x >> 1) ^ -(x & 1));
}

/*
 * Decodes zigzag-encoded sint32 values.
 *
 * This is the same thing as `ppb_zag`, except the encoded value is
 * truncated to 32 bits before decoding, in order to match google's
 * C++ implementation (only relevant with non-canonical encoding).
 */
static inline int32_t
ppb_zag32(uint32_t x)
{
    return (int32_t)ppb_zag(x);
}

/*
 * Attempts to consume one varint from `buf`.
 *
 * Returns the decoded varint, or 0 on error.
 *
 * Confirm whether there was an error by looking at `error` (initially
 * zero): it will be strictly negative on error, and zero on success.
 *
 * `*error` is sticky: if `*error` is already non-zero on entry the
 * function still consumes input and returns the decoded varint, but
 * `*error` is left unchanged.  Most callers should ensure `error`
 * is zero-initialized on entry.
 */
uint64_t ppb_decode_varint(struct ppb_buf *__restrict buf, enum ppb_error *__restrict error);

/*
 * Validates a tag array: checks that all entries have `.bits > 7`
 * (field number 0 is forbidden in protobuf) and that the array is
 * strictly sorted ascending.
 *
 * N.B., the encoding sticks the wire type in the low bits of the tag,
 * so a list of encoded tags with strictly ascending, strict positive,
 * field numbers is valid.  When there are multiple `ppb_encoded_tag`s
 * with the same field number (e.g., for catch-alls with -1), the types
 * must follow the order in `enum ppb_wire_type`.
 *
 * Call once on any static tag array before passing it to `ppb_prescan`
 * or `ppb_lexn`.  Passing an unvalidated array to prescan or lexn does
 * not cause undefined behavior but may produce incorrect results or
 * trigger assertion failures when PPB is built with assertions.
 *
 * Returns `PPB_OK` on success, or the first error found.  The empty
 * tag array (`num_fields == 0`, `tags == NULL` allowed) returns
 * `PPB_OK`; the matching prescan / lexn calls then accept any
 * well-formed message and decode no fields, useful for
 * validate-only call sites.
 */
enum ppb_error ppb_validate_tags(size_t num_fields, const struct ppb_encoded_tag *tags);

#include "ppb/ppb.h"

enum { F_NAME = 0, F_ID = 1, F_DATA = 2, NUM_FIELDS };

/*
 * Tags are sorted ascending by .bits and never modified after init.
 * Call ppb_validate_tags once at program start (in main() or an
 * __attribute__((constructor)) function) to catch mis-sorted arrays
 * early: a bad tag array silently results in wrong output, not an error
 * (or maybe an assertion error if you built with assertions and get lucky).
 */
static const struct ppb_encoded_tag tags[NUM_FIELDS] = {
    [F_NAME] = PPB_TAG(1, PPB_WIRE_LEN),    /* string name = 1 */
    [F_ID]   = PPB_TAG(2, PPB_WIRE_VARINT), /* uint64 id   = 2 */
    [F_DATA] = PPB_TAG(5, PPB_WIRE_LEN),    /* bytes  data = 5 */
};
/* PPB_TAG preserves ordering by field and type: PPB_TAG(1,2) < PPB_TAG(2,0) < PPB_TAG(5,2). */

struct ppb_field fields[NUM_FIELDS] = { 0 };  /* mutable per-call state */

/* 0. Validate tags once at startup; bad arrays give wrong output, not errors. */
if (ppb_validate_tags(NUM_FIELDS, tags) != PPB_OK) abort();

/* 1. Validate and gather stats (for preallocation). */
struct ppb_buf msg = { .buf = wire_bytes, .size = wire_len };
ptrdiff_t scanned = ppb_prescan(msg, NUM_FIELDS, tags, fields, SIZE_MAX);
if ((size_t)scanned != msg.size) { /* error */ }

size_t id_count   = fields[F_ID].m.num_occurrences;
size_t name_bytes = fields[F_NAME].m.total_bytes;  /* for allocation */

/* 2. Lex fields. */
while (msg.size > 0)
{
    const char *old_buf = msg.buf;
    struct ppb_lexn_ret r = ppb_lexn(&msg, NUM_FIELDS, tags, fields, SIZE_MAX);
    if (r.status != PPB_OK) { /* error */ }

    size_t end = r.first_field + r.field_range;
    /*
     * If field_range == UINT32_MAX, the width is unknown; extend
     * to NUM_FIELDS.  This can only happen when NUM_FIELDS >=
     * UINT32_MAX, a pretty niche use case.
     */
    if (r.field_range == UINT32_MAX) end = NUM_FIELDS;
    for (size_t i = r.first_field; i < end; i++)
    {
        if ((const char *)fields[i].v.ptr < (const char *)old_buf ||
            (const char *)fields[i].v.ptr >= (const char *)msg.buf)
        {
            continue;
        }

        switch (i)
        {
        case F_NAME:
            use_string(fields[i].v.payload);
            break;
        case F_ID:
            use_id(fields[i].v.u64);
            break;
        case F_DATA:
            use_bytes(fields[i].v.payload);
            break;
        }
    }
}

Detecting decoded fields: in a ppb_lexn loop, save buf.buf before the call as old_buf; afterwards, old_buf <= field.v.ptr < buf.buf iff that field was decoded in this call. Why "iff": both ppb_prescan and ppb_lexn only write ptr when they match a tag, and ppb_lexn consumes input monotonically, so a ptr written by an earlier call lies strictly below old_buf and any ptr in [old_buf, buf.buf) was set in this call.

For a one-shot ppb_prescan on a freshly zero-initialized fields[], the simpler field.v.ptr != NULL suffices. field.v.ptr being NULL means we never decoded the field, or attempted to and failed. There's no way to tell the difference, so it's best to immediately bail when ppb_prescan* or ppb_lexn* return an error.

Nested submessages: when a PPB_WIRE_LEN field is a submessage, pass field.v.payload as the buf argument to a fresh ppb_prescan / ppb_lexn pair. PPB is iterative within one message level; recursion across levels is the caller's responsibility. Ideally, the nesting depth is capped in the schema itself. When that's impossible, remember to impose an arbitrary limit: unbounded recursion through nested messages is a recurring issue in the protobuf ecosystem (CVE-2024-7254, CVE-2026-0994).

int decode_msg(struct ppb_buf buf, size_t max_depth)
{
    if (max_depth == 0)
        return -1;

    /* ... prescan / lexn loop ... */
    {
        /* for each PPB_WIRE_LEN field that is a submessage: */
        if (decode_msg(fields[F_SUB].v.payload, max_depth - 1) < 0)
            return -1;
    }

    return 0;
}

Repeated fields: ppb_lexn stops as soon as a tag fails to be strictly greater than the last matched tag, so each call yields at most one occurrence of any given field. (Skipped unknown fields neither end the batch nor take part in that ordering check; install catch-alls when batch boundaries must reflect every field on the wire.) field.v holds the value decoded in this call; the outer loop naturally delivers each occurrence in the encoded order.

Catch-all entries: use PPB_TAG(-1, wire_type) to match any field of a given wire type that wasn't matched by a specific entry. Catch-all entries sort after all specific entries. ppb_lexn always stops after a catch-all match, so place catch-alls only where you can afford a potential overhead of one ppb_lexn call per field.

PPB matches tags by their full encoded form, so a wire-type mismatch on a known field number (e.g., the schema declares field 5 as PPB_WIRE_VARINT but the wire encoding is for a fixed64) is silently treated as an unknown field: the tag misses the specific entry, and is either routed to a catch-all of the wire type actually on the wire or skipped. Callers that need strict wire-type checking must enforce it themselves, probably through catch-alls.

A stricter caller can treat any catch-all hit as an error by installing one catch-all per wire type and rejecting messages that land on any of them.

#include "ppb/ppb.h"

enum {
    F_NAME,                 /* string name = 1 */
    F_ID,                   /* uint64 id   = 2 */
    F_DATA,                 /* bytes  data = 5 */
    F_CATCH_VARINT,         /* anything-else of wire type VARINT */
    F_CATCH_I64,            /* ... I64 */
    F_CATCH_LEN,            /* ... LEN */
    F_CATCH_I32,            /* ... I32 */
    NUM_FIELDS,
};

/*
 * Catch-alls sort after every specific tag, and within the catch-all
 * block the order follows enum ppb_wire_type (VARINT=0, I64=1, LEN=2,
 * I32=5).  The order below is what ppb_validate_tags accepts.
 */
static const struct ppb_encoded_tag tags[NUM_FIELDS] = {
    [F_NAME]         = PPB_TAG(1,  PPB_WIRE_LEN),
    [F_ID]           = PPB_TAG(2,  PPB_WIRE_VARINT),
    [F_DATA]         = PPB_TAG(5,  PPB_WIRE_LEN),
    [F_CATCH_VARINT] = PPB_TAG(-1, PPB_WIRE_VARINT),
    [F_CATCH_I64]    = PPB_TAG(-1, PPB_WIRE_I64),
    [F_CATCH_LEN]    = PPB_TAG(-1, PPB_WIRE_LEN),
    [F_CATCH_I32]    = PPB_TAG(-1, PPB_WIRE_I32),
};

int decode_strict(const void *wire_bytes, size_t wire_len)
{
    if (ppb_validate_tags(NUM_FIELDS, tags) != PPB_OK)
        return -1;

    struct ppb_field fields[NUM_FIELDS] = { 0 };
    struct ppb_buf msg = { .buf = wire_bytes, .size = wire_len };

    /*
     * Prescan validates the wire format and populates fields[].m.
     * Any unknown tag (or wire-type mismatch on a known field
     * number) lands on one of the F_CATCH_* slots.
     */
    if (ppb_prescan(msg, NUM_FIELDS, tags, fields, SIZE_MAX) < 0)
        return -1;

    for (size_t i = F_CATCH_VARINT; i < NUM_FIELDS; i++)
    {
        if (fields[i].m.num_occurrences != 0)
            return -1;  /* unknown field or wire-type mismatch */
    }

    /*
     * No catch-all hit: every wire-format tag matched a specific
     * entry with the right wire type.  Continue with the lexn loop
     * as in the earlier worked example, ignoring the F_CATCH_*
     * indices (they're guaranteed to be empty).
     */
    return 0;
}

Note that this rejects any unknown field, not just wire-type mismatches on known field numbers: catch-alls fire on both, and PPB has no way to distinguish the two cases. That gives up the usual protobuf forward-compatibility property (newer producers extending the schema will be rejected by older consumers), and is the trade-off baked into this style of strict checking.

A catch-all match also terminates the current ppb_lexn call, so the strict check can be made per-occurrence in the lexn loop rather than as a single wholesale check after prescan: any populated F_CATCH_* slot ends the parse early, and field.v.ptr points at the offending tag byte, so the caller can re-decode the field number and decide case by case whether to treat it as an unknown field (forward compatibility) or a wire-type mismatch on a known number (an actual schema violation).

The prescan hack above is simpler when we must look for unexpected wire types and giving up forward compatibility is acceptable.

Value fields in struct ppb_field_value:

• u64 / i64: full 64 bits, host byte order (the assembled value: a decoded varint, the bits of a fixed64, or a zero-extended fixed32).
• u32 / i32 / f: low 32-bit views (for fixed32 fields).
• d: double view of the full 64 bits.
• b[8]: raw bytes of u64 in host order, not the wire encoding.
• payload: struct ppb_buf pointing at the payload of a PPB_WIRE_LEN field.
• ptr: pointer to the field's tag byte in the original buffer (useful for catch-all).

The union works on either endian: on big-endian hosts the 32-bit views are laid out so they alias the low half of u64.

Zigzag: call ppb_zag(field.v.u64) to decode sint64 values, and ppb_zag32(field.v.u32) to decode sint32 values.

Byte length limiting: use ppb_prescan_with_hard_limit / ppb_lexn_with_hard_limit when the message length is known to be smaller than the remaining readable region: the hard limit avoids the end-of-buffer slow paths around the end of the message. Use the _with_soft_limit variants to implement chunked processing at approximate boundaries: pass a target chunk size as the limit and PPB will stop at the first clean field boundary at or past that size, so the chunk always ends just before another valid field. The soft limit can be exceeded by up to one field's worth: with a soft limit, the final field of the chunk is always parsed in full instead of returning a spurious error.

/* Chunked processing: stop at every ~CHUNK_SIZE bytes. */
struct ppb_buf cur = msg;
while (cur.size > 0)
{
    /* Zero `fields[]` so each chunk's stats stand alone. */
    memset(fields, 0, sizeof(fields));
    ptrdiff_t scanned = ppb_prescan_with_soft_limit(
        cur, /*limit=*/CHUNK_SIZE, NUM_FIELDS, tags, fields, SIZE_MAX);
    if (scanned < 0) { /* error */ break; }

    process_chunk(fields, cur, scanned);  /* copy `fields` if spawning threads */
    cur.buf = (const char *)cur.buf + scanned;
    cur.size -= (size_t)scanned;
}

Recovery after error: after a negative ppb_error from ppb_lexn or ppb_prescan, the message is unrecoverable. PPB does not support resuming a partial parse, and ppb_buf is not a streaming abstraction. On a ppb_lexn error, *buf points somewhere within the offending field (at its first tag or payload byte; the exact position is an implementation detail), but always after all successfully decoded fields. The contents of the fields[] array are unspecified. The array is safe to access, but we make no other guarantee.

examples/picoscope.c is a self-contained binary that disassembles a binary protobuf file to text, similarly to protoscope. Build it with make, then:

./build/picoscope file.pb
./build/picoscope -p file.pb   # protoscope-style output

The -p mode matches protoscope's output on the golden test corpus (that is what make test checks); on arbitrary inputs it lacks some of protoscope's rendering heuristics, such as hex and float value guesses, long-form:N annotations, and line wrapping.

The test suite runs unit tests and comparisons against protoscope:

make                # build/libppb.a, build/libppb.so, build/picoscope, build/ubench
make test           # run unit tests + golden tests
make unit           # unit tests only (build/test_ppb)
make unit_cpp       # C++ wrapper tests (see README_CPP.md)
make test EXTRA_FLAGS='-fsanitize=undefined,address'  # same with ubsan & asan
make fuzz           # quick libfuzzer-based run (requires clang)
make wp             # Frama-C WP verification (requires Docker)
make eva            # Frama-C EVA + WP (requires Docker)
make format         # clang-format-20 on all sources
./coverage.sh       # report code (line and branch) coverage for `make test`
make regen_test     # regenerate golden files (requires protoscope in PATH)
make generator_test # protoc-gen-ppb checks (requires protoc and uv; see generator/)
make differential-ci                     # libprotobuf differential suite, deterministic subset (see differential/)
make -C differential conformance-docker  # protobuf conformance suites in a pinned container image
make audit          # deterministic non-Docker evidence in one command (see AUDITING.md; analyze-gcc is memory-hungry)

Picoscope-based tests are high value for little effort. To add a test case for a valid protobuf message, generate protobuf bytes (e.g., encode a protoscope source file to binary) and convert to hex:

echo '1: 42  2: {"hello"}' | protoscope -s | xxd -p | tr -d '\n' \
    > testdata/my-test.hex
make regen_test   # writes testdata/my-test.expected

The test suite also automatically truncates every valid message by 1 to 32 bytes and checks that picoscope either errors out or produces a prefix of the expected output.

To add an invalid test case, place the .hex file in testdata/invalid/ and run make regen_test; it captures the expected error message into a .expected-error file.

If protoscope is not in PATH, pass it explicitly:

make regen_test PROTOSCOPE=/path/to/protoscope

The unit tests cover ppb_zag, varint decoding, ppb_prescan and ppb_lexn directly: field-array validation, metadata accumulation, max_lexed_fields early-exit, monotonic-run batching, unknown-field skipping, and error detection for malformed input. The golden tests exercise picoscope end-to-end across all wire types, multi-byte varint and tag encodings, repeated and nested fields. The invalid-input suite checks that corrupt or truncated data is rejected with the correct error.

Formal verification, mostly for memory safety and termination, is done with Frama-C, with make wp. Source formatting with make format uses clang-format-20.

The ACSL annotations include admitted properties; we try to confirm them with unit tests and annotations. The fuzz tests in particular dynamically test ACSL-proven postconditions, in addition to the usual property that UBSan and ASan must not flag undefined behavior or memory safety issues. While WP merely checks that LEN field.v.payload are readable subranges of the input buffer (via the buf_valid_range assertion in handle_field), the fuzz harness and picoscope additionally check that field.v.ptr and the payload slice fall within the call's consumed bytes and that payload.buf > ptr.

Validate the test suite with mutants.py: the script mutates the code and checks whether the mutation is detected by the test suite. Undetected mutations may point at gaps in the test suite (or maybe the mutant is expected or an equivalent formulation, and mutation testing should be disabled locally).