libdog is a token-level revision control toolkit library. It contains various primitives, algorithms and data structures, including:

0. source code dogenizers for about 60 languages,
1. git compatibility utils: object and file formats, git's sync protocol,
2. CRDT weave diff/merge/blame (RGA/CausalTree like),
3. N.Fraser's diff cleanup,
4. B.Cohen's patience diff,
5. through its libabc dep, also Myers diff and other classic algos,
6. through libabc, various on-disk formats (hashmaps, sorted sets, etc).
7. many other things.

libdog is not "git reimplemented" like libgit2 or gitoxide. That was not the objective here as git's commands and formats are highly idiosyncratic legacy, to put that bluntly, and all of its diff/merge/blame logic is line-based. Meanwhile, git's object model is very much an Internet-standard-level thing. For that reason, libdog contains all the necessary and performance-critical revision control primitives that absolutely have to be implemented in a native lib. The size tells the story: gitoxide is about 270 KLoC of Rust. libdog's core is 14 KLoC of C, 24 counting its tests. The tokenizers are another 54 KLoC, but those are Ragel generated.

The revision control system per se uses all these primitives from a safer environment. In particular, libdog jab bindings form the foundation of Beagle the SCM system that is written in fully malleable JavaScript. This is similar to git's plumbing vs porcelain division, but deeper.

libdog is written in ABC, a very minimalistic zero-alloc C dialect.

Line-granular history is a historical accident of diff(1). Rename one variable and a line-based tool reports the whole line rewritten; two people editing opposite ends of the same line get a conflict that isn't one. libdog versions each language's native tokens. 60 languages are supported. For unknown grammars it falls back to the generic-text tokenizer.

Each tokenizer is a hand-written Ragel scanner that classifies source bytes into tagged tokens. The brilliance of Ragel is that there is no parse tree, no allocation, and no dependencies. For example, the C dogenizer can process 250MB/s on a laptop, which effectively means tokenization is free for any practical purpose. That was in fact the critical finding in applying CRDTs to the git data model: blobs can be parsed on the fly, so all the CausalTree logic can be applied transparently to very regular git blobs.

$ dogtok example.c

S	ok64
W	 
S	SumAll
P	(
S	u64s
W	 
S	a
P	)
W	 
P	{
W	\n    
S	u64
…
L	0
P	;
W	  
D	//
D	 
D	fold

A token is a tok32: a 5-bit tag, a display bit, a 2-bit diff side, a 24-bit end offset. THEME.h paints the tags.

A second pass (tok/DEF.h) enriches these with N for a defined name and C for a call site; F marks a filename column.

60 dogenizers, dispatched by extension or by filename, for files whose name is their type (Makefile, Dockerfile, .gitignore, .clang-format). Unknown extensions fall back to the plain-text lexer rather than failing.

Adding a language takes one .c.rl scanner, one keyword table, and one row in tok/TOK.c. To regenerate a lexer after editing its scanner:

ragel -C tok/CT.c.rl -o tok/CT.rl.c -L

Generated output is committed, so building libdog does not require Ragel.

Needs a C23 compiler, CMake 3.20+, and zlib.

git clone --recursive https://github.com/gritzko/libdog
cd libdog
cmake -B build -G Ninja
ninja -C build          # libdog.a and dogtok
ctest --test-dir build  # 23 test binaries

INDEX.md lists every module and function. Starting points:

• DOG.md the dog API, URI grammar, verb vocabulary, .be layout
• ULOG.md the event log format, on-disk and in-RAM
• tok/DEF.md, tok/BRCT.md definition marking, bracket matching
• abc/README.md the C dialect all of this is written in