Skip to content

Generalize string escape#6

Open
hudson-ai wants to merge 37 commits into
guidance-ai:mainfrom
hudson-ai:generalize-string-escape
Open

Generalize string escape#6
hudson-ai wants to merge 37 commits into
guidance-ai:mainfrom
hudson-ai:generalize-string-escape

Conversation

@hudson-ai

@hudson-ai hudson-ai commented Mar 6, 2026

Copy link
Copy Markdown
Contributor

Generalize string_escape API

Generalizes the existing json_quote mechanism into a configurable string_escape API that supports arbitrary escape grammars (JSON, URL percent-encoding, XML, etc.).

Key changes

New types:

  • FallbackFormat — describes the hex fallback encoding (prefix + 2 hex digits + suffix) with valid_byte_ranges: Option<Vec<(u8, u8)>> to restrict which bytes are eligible. Inclusive ranges handle non-contiguous valid sets (e.g., XML 1.0 characters).
  • StringEscapeOptions — declarative escape grammar with escape_sequences, fallback: Option<FallbackFormat>, and must_escape byte list.

Constructors:

  • StringEscapeOptions::json()\u00XX fallback, standard control-char escapes
  • StringEscapeOptions::json_raw() — same but no \uXXXX fallback
  • StringEscapeOptions::url_percent_encoding()%HH for all non-unreserved bytes
  • StringEscapeOptions::xml() — named entities + &#xNN; fallback, only valid XML 1.0 characters

Bug fix:

  • quote_byteset now checks FallbackFormat::covers_byte() before generating fallback hex alternatives. Previously, bytes outside the valid range could get semantically incorrect fallback forms.

Legacy compatibility:

  • json_quote() still works — delegates to string_escape() via JsonQuoteOptions::to_string_escape_options().
  • FallbackFormat and StringEscapeOptions are exported from lib.rs.

hudson-ai and others added 21 commits February 25, 2026 13:39
Parameterize the quoting logic to support arbitrary quote characters
instead of hardcoded double-quote. This enables Python-style single-quoted
strings when used with quote_char='\''.

- Add quote_char field to JsonQuoteOptions (defaults to '"' for backward compat)
- Update all quoting logic to use quote_char instead of hardcoded '"'
- Per-quote-char caching to avoid cross-contamination
- Accept '\'' in allowed_escapes validation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce StringEscapeOptions, FallbackEscapeFormat, and QuoteEscapeMethod
types that declaratively describe any string literal escape grammar.

Add RegexBuilder::string_escape() method that uses these options to
transform a regex R into R' such that strings matching R' are valid
string literals whose deserialized content matches R.

Rewrite json_quote() as a thin wrapper that converts JsonQuoteOptions
into StringEscapeOptions and delegates to string_escape(). The
JsonQuoteOptions public API is unchanged from main.

New capabilities over the old JSON-only API:
- Configurable single-char escape mappings (e.g. \a, \v, \0)
- Configurable fallback escape format (\uXXXX or \xHH)
- Configurable quote character
- Configurable quote escape method (backslash or doubling)
- Configurable must-escape byte set

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The quote_char field is typed as char but the implementation operates on
of string_escape() to reject non-ASCII quote characters with a clear
error message instead of silently producing wrong behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In Doubling mode (e.g. YAML single-quoted strings), backslash is not an
escape prefix and should pass through as a literal character. Previously,
backslash was unconditionally added to must_escape_set and emitted as
\\\\ regardless of escape mode.

Now backslash is only treated as special when the grammar actually uses
it as an escape prefix (has single_char_escapes, has fallback format, or
uses Backslash quote escape method).

Add test verifying backslash is literal in Doubling mode.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The escape logic previously only handled bytes 0x00-0x1F in the inner
loop and hardcoded 0x7F separately. Bytes outside this range that
appeared in must_escape would be silently dropped.

Changes:
- Generalize the slow-path loop in quote_byteset to iterate all 256
  bytes in must_escape_set, not just 0..32
- Replace `bs_without_ctrl[0] = 0` (blanket clear of bits 0-31) with
  a targeted clear of only must_escape bytes
- Add a post-loop that handles must_escape bytes >= 0x20 that aren't
  covered by the control-char fast paths (quote_all_ctrl)
- This correctly handles 0x7F and any other non-control must_escape
  bytes via the same generalized path

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The UnicodeXXXX and HexHH branches were identical except for the prefix
string. Extract the prefix into a variable and share the hex digit regex
construction logic.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This mirrors the existing JsonQuote variant but uses the new
StringEscapeOptions. Users can now build a RegexAst tree containing
StringEscape nodes directly without going through RegexBuilder methods.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sort single_char_escapes and must_escape before comparing/hashing so
that two options with the same entries in different orders produce the
same cache key.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Avoid re-computing the conversion on every json_quote call by caching
the result in a HashMap keyed by JsonQuoteOptions (now derives Hash/Eq).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
UnicodeXXXX (\u00XX) can only correctly represent bytes 0x00-0x7F.
For bytes >= 0x80 the Unicode code point differs from the raw byte value,
causing a silent roundtrip mismatch. Now returns an error unless the byte
has an explicit single_char_escape mapping.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Avoids unconditionally flagging backslash as needing escape in Doubling
mode where backslash is a literal character.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Only include qc and backslash in the escape char byteset when they fall
in the control range (< 0x20). Since both are always >= 0x20 for typical
configurations, this removes spurious alternatives from the fast-path
regex that would accept escape sequences for non-control bytes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The slow path iterates all 256 bytes so the post-loop for bytes >= 0x20
was producing duplicate regex alternatives. Now only runs after the fast
path which only covers 0x00-0x1F.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace custom Hash/PartialEq impls with derived ones and add a
normalize() method that sorts and dedups the Vec fields. Called once
at string_escape entry before cache lookup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- must_escape byte 0x7F with HexHH (tests >= 0x20 escape path)
- UnicodeXXXX rejection for must_escape bytes >= 0x80
- Control-only regex (tests fast path correctness)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This reverts commit c7af1873e08b9c053520f4644db755e006e0623e.
Remove has_fallback parameter from quote_byteset (computed from options
internally). Apply rustfmt formatting across changed files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Expand docstrings on StringEscapeOptions, FallbackEscapeFormat,
  QuoteEscapeMethod, JsonQuoteOptions, and the RegexAst variants
- Document backslash handling semantics, normalization, error
  conditions, and the interaction between quote_escape and
  uses_backslash
- Clarify that \\ and quote_char in allowed_escapes are always
  applied during JsonQuoteOptions conversion (not gated)
- Remove dead branch (b'\\' < 0x20 is always false)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extracts the JSON-specific escape logic from json_quote into a configurable string_escape API, enabling support for various string literal escape grammars (JSON, Python, YAML, C-style) through a declarative StringEscapeOptions struct. The existing json_quote method is preserved as a thin wrapper that converts JsonQuoteOptions into StringEscapeOptions and delegates.

Changes:

  • Introduces StringEscapeOptions, FallbackEscapeFormat, and QuoteEscapeMethod types for configurable string escape grammars, with RegexBuilder::string_escape() as the core implementation.
  • Refactors json_quote() to delegate to string_escape() via JsonQuoteOptions::to_string_escape_options(), replacing the single expression cache with per-options caches.
  • Adds RegexAst::StringEscape variant and 8 new tests covering JSON equivalence, hex fallback, single-quote/doubling escapes, caching, high bytes, and error cases.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
src/regexbuilder.rs Core implementation: new types, string_escape() method, refactored json_quote(), cache changes, RegexAst::StringEscape variant
src/lib.rs Exports new public types: FallbackEscapeFormat, QuoteEscapeMethod, StringEscapeOptions
tests/basic.rs 8 new integration tests covering JSON equivalence, HexHH, single-quote, doubling, caching, high bytes, and error rejection

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/basic.rs Outdated
@hudson-ai hudson-ai changed the title Generalize string escape [DRAFT] Generalize string escape Mar 7, 2026
@hudson-ai

Copy link
Copy Markdown
Contributor Author

In retrospect, I'm marking as a draft. Backslash is a bit too hard coded -- it would be nice to be general enough to cover URL percent-encoding and XML.

hudson-ai and others added 2 commits March 6, 2026 21:01
- Add escape_prefix field to StringEscapeOptions (default b'\\')
- Add FallbackEscapeFormat::BareHH for bare hex after prefix (%HH)
- Add QuoteEscapeMethod::Normal for no special quote handling
- Add StringEscapeOptions::url_percent_encoding() constructor
- Generalize all internal logic from hardcoded backslash to ep
- Escape prefix is no longer implicitly doubled; must be listed
  in single_char_escapes for doubling (e.g., (b'\\', b'\\'))
- Add tests for percent encoding and Normal quote method

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace single_char_escapes, escape_prefix, quote_char, quote_escape,
FallbackEscapeFormat, and QuoteEscapeMethod with:
- escape_sequences: Vec<(u8, Vec<u8>)> for explicit per-byte escapes
- fallback_prefix: Option<Vec<u8>> + hex digits for remaining bytes
- max_fallback_byte: Option<u8> for validation
- must_escape: Vec<u8> (fully explicit, no implicit additions)

Remove raw_mode (delimiter wrapping moved to json_quote).
Remove FallbackEscapeFormat and QuoteEscapeMethod enums entirely.

Bytes with both an explicit escape and a fallback now accept either
form (e.g., JSON \b and \u0008 both valid for byte 0x08).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
json_quote delegates to string_escape, so the equivalence test was
tautological. The 4 pre-existing json_quote tests cover the legacy API.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/regexbuilder.rs Outdated
hudson-ai and others added 2 commits March 25, 2026 15:51
Redesign the string_escape API:

- Introduce FallbackFormat struct with prefix, suffix, and
  valid_byte_ranges fields. valid_byte_ranges uses Vec<(u8,u8)>
  inclusive ranges instead of a single max_byte threshold, enabling
  non-contiguous byte sets (e.g. XML 1.0 valid characters).

- Restructure StringEscapeOptions: replace fallback_prefix with
  fallback: Option<FallbackFormat>, and add must_escape field.

- Add constructors: json(), json_raw(), url_percent_encoding(), xml().

- Fix bug in quote_byteset: fallback hex was generated for bytes
  outside the valid range when they had explicit escape sequences.
  Now gated by FallbackFormat::covers_byte().

- Fix xml() constructor: must_escape only includes valid XML 1.0
  characters (0x09, 0x0A, 0x0D, markup chars), not forbidden
  control characters.

- Update json_quote to delegate through string_escape via
  JsonQuoteOptions::to_string_escape_options().

- Export FallbackFormat from lib.rs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/regexbuilder.rs
Comment thread tests/basic.rs Outdated
Comment thread src/regexbuilder.rs Outdated
Comment thread src/regexbuilder.rs Outdated
Comment thread src/regexbuilder.rs Outdated
hudson-ai and others added 2 commits March 25, 2026 16:31
- Fix doc inconsistency: document that string_escape errors when a
  must-escape byte is outside valid_byte_ranges without an explicit
  escape, rather than silently excluding it.
- Validate escape_sequences for conflicting duplicates in normalize().
- Optimize quote_byteset and needs-check to iterate must_escape slice
  instead of scanning all 256 bytes.
- Remove stray println! in test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/regexbuilder.rs
Comment thread src/regexbuilder.rs Outdated
Comment thread src/regexbuilder.rs
- Canonicalize valid_byte_ranges in normalize(): sort, validate
  (start <= end), and merge overlapping/adjacent ranges so
  semantically equivalent options hash identically for caching.
- Fix XML doc: 0x7F is valid in XML 1.0 (part of #x20-#xD7FF).
  Removed incorrect claim that 0x7F is forbidden.
- Document that to_string_escape_options() does not validate
  allowed_escapes (json_quote() handles that).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/regexbuilder.rs
xml() does not prevent forbidden control bytes from passing through
literally if they appear in the input regex. Document that callers
should constrain their input regex to valid XML characters.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/regexbuilder.rs Outdated
Comment thread src/regexbuilder.rs
Comment thread src/regexbuilder.rs
Comment thread src/regexbuilder.rs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/regexbuilder.rs Outdated
Comment thread src/regexbuilder.rs
Comment thread src/regexbuilder.rs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hudson-ai hudson-ai changed the title [DRAFT] Generalize string escape Generalize string escape Mar 26, 2026
@hudson-ai
hudson-ai requested a review from nking-1 March 26, 2026 21:38
@hudson-ai

Copy link
Copy Markdown
Contributor Author

@riedgar-ms @nopdive @nking-1 would appreciate a review, particularly of the pub api surface and whether it seems general / extensible enough for as much future proofing as possible...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants