SweetSyntax 🍭 A generic parser and AST explorer
for analyzing programming languages

nimble install sweetsyntax

API reference

• Fast, compiled and efficient (check benchmarks section)
• Generic parser & AST explorer with Syntactic validation
• Easy-to-use API for integration into various applications
• Zero-copy parsing using MemFiles
• Tree-sitter code folding
• JSON-based AST generator / indent-based dump tree
• ANSI, HTML and JSON-LD Renderers
• Built-in syntax support for: C, Crystal, D lang, Go, JavaScript, Nim, PHP, Python, Ruby, and Rust
• Context-aware error reporting while parsing
• Plugin System for creating custom syntaxes (soon 🔥)
• Written in Nim language

SweetSyntax is a powerful and flexible generic parser and AST explorer for analyzing programming languages! It is designed to be integrated into other applications, such as code editors, documentation generators, AI agents, linting tools, minifiers, and other sweet things!

Parse any language by defining its grammar in a YAML specification file: tokens, operators (prefix, infix, postfix, assignment), statement keywords, block delimiters, and feature flags (arrow functions, generators, async/await, template literals). The parser uses a Pratt parsing approach with a language-agnostic core and per-language statement handlers.

Key capabilities:

• Lexer: Generic tokenizer handling identifiers, literals (int, float, hex, octal, binary, bigint, string, regex), comments, and operators
• Parser: Pratt (precedence-climbing) parser with configurable operator precedence and associativity
• AST: Typed node tree with support for statements, expressions, infix/prefix/postfix operations, function declarations, and more
• YAML-driven: Language syntaxes are pure YAML, allowing for custom statement handlers
• Renderers: Terminal (ANSI), HTML, line-delimited JSON for editors/higher-level apps, and tree-sitter style code folding

SweetSyntax ships a family of token-level renderers that turn a lexer into a usable output stream. All renderers consume a SweetLexer directly, so they work with any YAML syntax spec.

The JSON renderer emits each token as its own NDJSON line, designed to be streamed to higher-level applications over websocket/udp so editors and IDEs can build syntax highlighting:

{"kind":"ident","scope":"storage.type","attr":["int"],"line":1,"col":1,"start":0,"stop":3,"value":"int"}
{"kind":"ident","scope":"variable","line":1,"col":5,"start":4,"stop":5,"value":"x"}
{"kind":"int","scope":"constant.numeric.integer","line":1,"col":9,"start":8,"stop":10,"value":"42"}

Every token exposes its kind, a derived TextMate-style scope (e.g. keyword.control, string.quoted.double, constant.numeric.integer, storage.type), user-defined attributes, 1-based line/column, byte offsets (start/stop), and the token value.

import sweetsyntax
import sweetsyntax/renderers/jsonrenderer

let syntax = getKnownSyntax(KnownSyntax.c)
var lx = initLexer(syntax.spec, "int x = 42;")
echo highlightJsonLd(lx)   # one JSON token per line

Code folding is fully optional and opt-in. computeFolds derives tree-sitter style fold regions from the token stream and emits them as their own NDJSON stream, so editors can subscribe to folds independently of token highlighting:

{"kind":"block","start":{"line":1,"col":12,"offset":11},"end":{"line":5,"col":1,"offset":38}}

Fold sources (controlled via FoldMode and flags):

• Braces (fmBraces, or fmAuto when the file contains {): {...} blocks spanning multiple lines
• Indentation (fmIndent): Python/Nim-style blocks, including nested blocks and else/elif branches
• Comments: multi-line /* ... */ and /** ... */ doc comments
• Preprocessor (preprocessorFolds = true): #if/#ifdef/#ifndef ... #endif and #region ... #endregion

import sweetsyntax
import sweetsyntax/renderers/foldrenderer

let syntax = getKnownSyntax(KnownSyntax.c)
var lx = initLexer(syntax.spec, "int main() {\n  return 0;\n}")
echo foldsToJsonLd(computeFolds(lx))   # fold regions as NDJSON

The sweetsyntax binary explores source files by extension:

sweetsyntax parse hello.nim   # validate: parse and report errors
sweetsyntax ast hello.nim     # print the full AST as JSON
sweetsyntax tree hello.nim    # print the AST as an indent-based tree

tree prints the AST Nim dumpTree-style (2 spaces per level, =value on leaf nodes). For echo "hi", 42:

nkCall
  nkIdent=echo
  nkLitString="hi"
  nkLitInt=42

The same rendering is available from the API via dumpTree (and treeRepr for a single line):

import sweetsyntax
import sweetsyntax/languages/nim

let ast = parseScript("hello.nim", nimHandlers, {})
echo dumpTree(ast)

Parse a C file into an AST:

import sweetsyntax
import sweetsyntax/languages/c
import pkg/openparser/json

let ast = parseScript("hello.c", cHandlers, {featLabeledStmt})
echo toJson(ast)

Parse a PHP file (namespaces, classes, typed signatures, arrow functions, match, and more):

import sweetsyntax
import sweetsyntax/languages/php
import pkg/openparser/json

let ast = parsePHP("hello.php")
echo toJson(ast)

PHP support includes namespaces, class/interface/trait/enum bodies with visibility modifiers and typed properties, nullable/union signatures, arrow functions (fn), match, goto/labels, declare, function-like keywords (new, isset, empty, unset, list, clone, exit/die, print), throw as an expression, PHP 8 attributes (#[...]), alternative control-flow syntax (if: ... endif;), and multi-block <?php ... ?> files.

Parse a Ruby script (def/class/module, blocks, modifiers, hashes, and more):

import sweetsyntax
import sweetsyntax/languages/ruby
import pkg/openparser/json

let ast = parseRuby("hello.rb")
echo toJson(ast)

Ruby support includes def/class/module (singleton methods, splat/keyword/block params, rescue/ensure clauses), if/unless/while/until/for/case/begin, do...end and {|...|} blocks, paren-less method calls (puts x, require 'json', attr_accessor :name), modifier forms (return nil unless ok), hash literals ({a: 1} and {a => 1}), instance/class/global variables (@x/@@x/$x), symbols (:sym), ranges, ternary, safe navigation (&.), lambdas (->), and # comments.

Highlight source as ANSI or HTML, or stream tokens/folds as NDJSON:

import sweetsyntax
import sweetsyntax/renderers/[asciirenderer, htmlrenderer, jsonrenderer, foldrenderer]

let syntax = getKnownSyntax(KnownSyntax.js)
var lx = initLexer(syntax.spec, "const x = 42;")

echo highlightAscii(lx)             # terminal output with colors
echo highlightJsonLd(lx)            # one JSON token per line
echo foldsToJsonLd(computeFolds(lx)) # fold regions as NDJSON

Interactive renderer examples live in example_renderers/.

SweetSyntax has built-in support for context-aware reporting. For example:

  return a == null || b == null ## NaN : a < b ? -1 : a > b ? 1 : a >= b ? 0 : NaN;
                                ^
Error (2:33) Unexpected prefix token: '#'

• Squeezy - A dead simple JavaScript and CSS validator, bundler and minifier
• ZenCode.app - A free and open-source cross-platform code editor built with Flutter, Nim and WebAPI

SweetSyntax is built for speed. Below is a hyperfine benchmark parsing and validating a full copy of d3.js (v7.9.0, ~20k lines, unminified, from cdnjs.com). The entire pipeline (lexing, parsing, and AST generation) completes in under 120ms on my 🔥 rastafarian Ryzen 5 with 6 cores/12 threads:

hyperfine --runs 4 './sweetsyntax_benc_d3'
Benchmark 1: ./sweetsyntax_benc_d3
  Time (mean ± σ):     116.5 ms ±   1.1 ms    [User: 107.5 ms, System: 7.1 ms]
  Range (min … max):   115.9 ms … 118.2 ms    4 runs

• 🐛 Found a bug? Create a new Issue
• 👋 Wanna help? Fork it!

	Switch to Open-Source LLMs via OpenCode GO, choosing from a variety of powerful models such as DeepSeek, Qwen, Kimi, GLM-5, MiniMax, MiMo. 🍕 Use our referral link to get started!

LGPLv3 license. Made by Humans from OpenPeeps.
Copyright OpenPeeps & Contributors — All rights reserved.