zlob.h

100% POSIX and glibc compatible globbing & file walking library for C, Zig, and Rust that is SIMD-first and platform-optimized, and supports all modern globbing formats.

zlob is a C library, zig library and a rust crate that makes globbing and file walking fast. Why? Because the available implementations are just either slow or uncomplete. Zlob is a feature-full file walker that supports all the platform specific nuances, gitignore, multicore file walking, 120% of available wildcard syntax and many more:

• 100% POSIX and glibc compatible with all the flags and features supported
• Faster than glibc up to 10x in specific cases and generally 1.2-1.7x faster. See benchmarks
• Faster than rust's, node's, bun's, python's implementation by far
• In addition to standard unix wildcard syntax supports ** recursive patterns, braces *.{c,h}, gitignore and bash extglob patterns
• Supports .gitignore out of the box
• Exports api for matching over paths (one or many) without involving file system
• Exposes path length in the output struct for seamless FFI
• Truly optimized for all platforms:
  • usage of getdents64 syscall for faster directory listing
  • usage of getattrslistbulk on macos when requesting metadata from the walker
  • uses ntdll directly on windows
  • windows paths & patterns are normalized at compile time (both "/" and "" accepted and treated the same in patterns)

Built for fff loved by many more amazing projects!

zlob is using SIMD first implementation. It is a primary reason it is written in zig to have a native portable SIMD support at a languages level, it significantly reduces certain bottlenecks. But the primary reason of speed is that zlob is firstly analyzes the pattern and then matches paths to this patterns making patterns like ./drivers/**/*.c parsed to [drivers] and *.c which makes it not spend the time on opening useless directories and making leaf matches like suffix for small extensions and other hot and common patterns to be faster because optimized for a hot branch invariant.

One of my favourite optimizations for this project is patterns like ./**/*.{c,rs,zig} this is usually the main reason glob is used and this pattern is the most optimized in the zlob implementation:

• recursive worker is using getdents64 syscall directly which dramatically improves directory listing
• gitignore implementation allows us to optionally skip large subdirectories out of the box
• and the actual *.{c,rs,zig} pattern is precompiled down to the SIMD bitmask matching that allows to match 3 extension at once

Checkout benchmarks yourself.

As much as I could I converted all the tests that I found from the glibc test suite, rust's glob crate and nodejs's fs.glob function. It passes 100% of test suite for both file system and string paths matching. As long as this we have a compatibility layer with direct libc and bash for compatibility, check or extend it here ./test/test_libc_comparison.sh

Any pattern you may think of should be already supported including gnu symbol classes, negation of character group, and event bash's extglob syntax.

Here are some examples:

zlob primarily support 3 public apis: C library, Zig library, and Rust crate

The easiest way is to look at include/zlob.h it exposes the same API as POSIX glob requires. Just change prefix glob -> zlob.

#include "zlob.h"

glob_t globbuf;
// recommended flag is default set of settings we recommended, see below
int ret = zlob("*.c", ZLOB_RECOMMENDED, NULL, &globbuf);

if (ret == 0) {
    for (size_t i = 0; i < globbuf.gl_pathc; i++) {
        printf("%s\n", globbuf.gl_pathv[i]);
    }
    zlobfree(&globbuf);
} else {
    fprintf(stderr, "glob error: %d\n", ret);
}

zlob exposes a native zig module. Add it to your build.zig.zon and import as @import("zlob").

const std = @import("std");
const zlob = @import("zlob");

pub fn main(init: std.process.Init) !void {
    const allocator = init.arena.allocator();

    if (try zlob.match(allocator, init.io, "**/*.{zig,zon}", .{
        .brace = true,
        .gitignore = true,
    })) |*result| {
        defer result.deinit();
        var it = result.iterator();
        while (it.next()) |path| {
            std.debug.print("{s}\n", .{path});
        }
    }

    // or match against an in-memory path list: no allocations, no io
    const paths = [_][]const u8{
        "src/lib.zig",
        "src/main.zig",
        "README.md",
    };
    var result = try zlob.matchPaths(allocator, "*.zig", &paths, .{});
    defer result.deinit();

    var it = result.iterator();
    while (it.next()) |p| {
        std.debug.print("{s}\n", .{p});
    }
}

zlob is also shared as an officially supported rust crate, find it on crates.io and read the rust docs

use zlob::{zlob, zlob_match_paths, ZlobFlags};

fn main() -> Result<(), zlob::ZlobError> {
    if let Some(result) = zlob("**/*.{rs,toml}", ZlobFlags::RECOMMENDED)? {
        for path in &result {
            println!("{}", path);
        }
    }

    let paths = ["src/lib.rs", "src/main.rs", "README.md"];
    if let Some(matches) = zlob_match_paths("*.rs", &paths, ZlobFlags::GITIGNORE)? {
        for path in &matches {
            println!("{}", path);
        }
    }

    Ok(())
}

Behavior is controlled using zlob flags. ZLOB_RECOMMENDED makes zlob behave like a modern glob implementation without sorting the output results, and enables all the modern features that you might need. Additional flags that can be used are:

• ZLOB_NOSORT - do not sort the output results, this is default behavior and it is recommended to use it for better performance
• ZLOB_BRACE - enable support for {a,b} patterns
• ZLOB_TILDE - expand ~ to the home directory
• ZLOB_TILDE_CHECK - checks the user exists when expanding ~user and returns error if it doesn't

• ZLOB_GITIGNORE - reads the .gitignore file in the scanning directory first and uses as a prefilter

• ZLOB_PERIOD - Allows to match hidden files using * and ? patterns, by default these patterns do not match hidden files

• ZLOB_EXTGLOB - enable support for bash extglob patterns like @(pattern-list), !(pattern-list), ?(pattern-list), *(pattern-list) and +(pattern-list)

• ZLOB_FOLLOW_SYMLINKS - follow symlinked directories when recursing with ** (see Symlinks and ** below)

• ZLOB_NOCHECK - if no matches found return the pattern itself as the only result

• ZLOB_NOMAGIC - if the pattern contains no special characters return the pattern itself as the only result

• ZLOB_NOESCAPE - disable backslash escaping

• ZLOB_MARK - append a slash to each directory match

• ZLOB_ONLYDIR - only match directories

• ZLOB_ERR - return error on unreadable directories

• ZLOB_APPEND - append results to the output buffer instead of overwriting it

• ZLOB_DOOFFS - reserve zlo_offs slots in the output buffer for custom use, these slots will be filled with NULL and the actual results will start from zlo_pathv[zlo_offs]

In addition to this zlob exposes zlob_at function that will open specific directory instead of requiring to manipulate CWD

zlob also exposes its traversal engine directly: a parallel recursive file walker designed to replace the Rust walkdir and ignore crates, available from Zig, Rust and C.

• Parallel: one directory = one task on a work stealing pool
• Bulk metadata: pass a mask of the attributes you need (size, mtime, inode, mode, ...) and only those are matched in a platform-specific optimized way
• Only required metadata: you likely don't need all the available metadata, zlob optimizes fstat to get only what you need.
• .gitignore first: Specifically optimized to handle file ignoring first P.S. Also let's use reuse assembled (incl nested) gitignore rules after the walk finished

1. Collect - efficient way to collect the results in to the memory if you need to store them
2. Run - call the callback per the matched entries (for grep style print on search and other workflows when you don't need to store the files list)

Traversal can also be narrowed with a one or many (z)glob patterns.

Here is how you can use it from different languages, starting with Rust:

use zlob::walk::{WalkBuilder, WalkFlags, WalkState, WalkMetadata};

// fastest: get all the results allocated
let results = WalkBuilder::new("/path/to/folder")
    // request metadata
    .metadata(WalkMetadata::SIZE | WalkMetadata::MTIME)
    .build()?;

for entry in results.iter() {
    println!("{} {:?}", entry.path().display(), entry.size());
}

// Raw walkdir/ignore-style traversal (skip gitignore files):
WalkBuilder::new(".").options(WalkFlags::GITIGNORE).run(|entry| {
    println!("{}", entry.path().display());

    WalkState::Continue
})?;

// Glob-scoped traversal: only src/ is ever descended into.
let rs_files = WalkBuilder::new(".").include("src/**/*.rs").build()?;

C (zlob_walk / zlob_walk_collect, see zlob.h):

zlob_walk_options_t opts = {0};
opts.flags = ZLOB_WALK_GITIGNORE;
opts.meta_mask = ZLOB_META_SIZE | ZLOB_META_MTIME;

zlob_walk_result_t res;
if (zlob_walk_collect(".", &opts, &res) == 0) {
    for (size_t i = 0; i < res.count; i++)
        printf("%s (%llu bytes)\n", res.entries[i].path,
               (unsigned long long)res.entries[i].size);
    zlob_walk_result_free(&res);
}

Zig (zlob.walk):

var results = try zlob.walk.collect(allocator, ".", .{
    .meta = .{ .size = true, .mtime = true },
});
defer results.deinit();
for (results.entries) |*e| std.debug.print("{s}\n", .{e.path});

Measured on a real 185k-entry repository (warm cache, i7-14700K, best of 5, cargo bench --bench walk_comparison -- "medium/"):

Entry counts agree with walkdir and ignore exactly on the same trees.

By default ** does not descend into symlinked directories. This matches bash globstar, zsh **, and walkdir — the dominant ** implementations. Symlinks are still matched by name like any other entry; only the recursion stops at them. POSIX glob(3) has no ** at all, so there is no "glibc default" to diverge from here.

Set ZLOB_FOLLOW_SYMLINKS to descend into symlinked directories. zlob keeps a (dev, ino) visited-set so it never loops and never emits the same physical file twice: each real directory is traversed at most once for the whole walk.

This is a deliberate difference from glob crate / nu-glob, which follow symlinks with no cycle tracking. On trees where one directory is reachable through several symlinks (e.g. a Bazel bazel-bin / bazel-out / bazel-<workspace> layout, all pointing into the same cache), those tools emit every aliased path — inflating their count several-fold and looping until PATH_MAX on a true cycle. zlob returns the de-duplicated set instead.

Match paths allowing you to submit a pointer of C strings or rust/zig-like slices and get the matches pointer back.

const char *paths[] = {
    "src/main.c", "src/utils.c", "src/tests/test.h", "readme.md", "src/lib.c",
};
const size_t path_count = sizeof(paths) / sizeof(paths[0]);

zlob_t pzlob;
int result = zlob_match_paths("**/*.c", paths, path_count, ZLOB_RECOMMENDED, &pzlob);

// Make sure that this does NOT free the input paths, the pointer to the paths are owned by the caller
// and have to be freed by the caller only after the zlob_t struct is freed to prevent dangling pointer
zlobfree(&pzlob);

This allows a very fast SIMD processing of the paths and supports NOT ALL the features of the standard FS globbing except ALTDIRFUNC which is not applicable because this mode is done to avoid using ALTDIRFUNC all along. Make sure that if you will use ZLOB_TILDE flag the paths input have to be absolute. Other flags like nomagic might not work as expected because they generally make very little sense.

Obviously to compile zlob as a C library you have to have installed zig toolchain (only 0.16.0) and then you can use standard make commands:

make
make test
make install <PREFIX>

I know it might be annoying to install zig but zig's linker is currently a decent way to cross compile any native code so I would definitely recommend trying it out.

Numbers below come from the criterion harness in rust/benches/glob_comparison.rs, comparing the zlob crate against the glob crate and the globset crate (paired with walkdir where it needs to walk the FS). The fixture is a Linux kernel checkout: 93,638 files / 6,157 directories / 36,685 .c files / 99 symlinks.

Here is a benchmark. Median wall time, lower is better.

Hardware / config: Linux x86_64 Intel(R) Core(TM) i7-14700K, 14 cores (28vcpu), ReleaseFast static build via zig 0.16.0 & cargo bench via criterion

Reproduce with:

# clones a shallow Linux kernel into /tmp/linux if no path is given
scripts/run-benchmarks.sh [/path/to/linux]

The script just sets REPO= and runs cargo bench --bench glob_comparison. The harness prints a match-count parity table at startup so you can see whether each library is doing the same amount of work.

This is my favourite part. zlob is not just zig zlob but also it means redneck in a various Eastern-European languages but in Polish it means "a manger", which I find very funny.

zlob is licensed under MIT license, see LICENSE file for more details.

P.S. No AI was used in the making of this README.md file thank you for reading it till the end.