Generalize string escape#6
Conversation
Parameterize the quoting logic to support arbitrary quote characters instead of hardcoded double-quote. This enables Python-style single-quoted strings when used with quote_char='\''. - Add quote_char field to JsonQuoteOptions (defaults to '"' for backward compat) - Update all quoting logic to use quote_char instead of hardcoded '"' - Per-quote-char caching to avoid cross-contamination - Accept '\'' in allowed_escapes validation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce StringEscapeOptions, FallbackEscapeFormat, and QuoteEscapeMethod types that declaratively describe any string literal escape grammar. Add RegexBuilder::string_escape() method that uses these options to transform a regex R into R' such that strings matching R' are valid string literals whose deserialized content matches R. Rewrite json_quote() as a thin wrapper that converts JsonQuoteOptions into StringEscapeOptions and delegates to string_escape(). The JsonQuoteOptions public API is unchanged from main. New capabilities over the old JSON-only API: - Configurable single-char escape mappings (e.g. \a, \v, \0) - Configurable fallback escape format (\uXXXX or \xHH) - Configurable quote character - Configurable quote escape method (backslash or doubling) - Configurable must-escape byte set Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The quote_char field is typed as char but the implementation operates on of string_escape() to reject non-ASCII quote characters with a clear error message instead of silently producing wrong behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In Doubling mode (e.g. YAML single-quoted strings), backslash is not an escape prefix and should pass through as a literal character. Previously, backslash was unconditionally added to must_escape_set and emitted as \\\\ regardless of escape mode. Now backslash is only treated as special when the grammar actually uses it as an escape prefix (has single_char_escapes, has fallback format, or uses Backslash quote escape method). Add test verifying backslash is literal in Doubling mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The escape logic previously only handled bytes 0x00-0x1F in the inner loop and hardcoded 0x7F separately. Bytes outside this range that appeared in must_escape would be silently dropped. Changes: - Generalize the slow-path loop in quote_byteset to iterate all 256 bytes in must_escape_set, not just 0..32 - Replace `bs_without_ctrl[0] = 0` (blanket clear of bits 0-31) with a targeted clear of only must_escape bytes - Add a post-loop that handles must_escape bytes >= 0x20 that aren't covered by the control-char fast paths (quote_all_ctrl) - This correctly handles 0x7F and any other non-control must_escape bytes via the same generalized path Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The UnicodeXXXX and HexHH branches were identical except for the prefix string. Extract the prefix into a variable and share the hex digit regex construction logic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This mirrors the existing JsonQuote variant but uses the new StringEscapeOptions. Users can now build a RegexAst tree containing StringEscape nodes directly without going through RegexBuilder methods. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sort single_char_escapes and must_escape before comparing/hashing so that two options with the same entries in different orders produce the same cache key. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Avoid re-computing the conversion on every json_quote call by caching the result in a HashMap keyed by JsonQuoteOptions (now derives Hash/Eq). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
UnicodeXXXX (\u00XX) can only correctly represent bytes 0x00-0x7F. For bytes >= 0x80 the Unicode code point differs from the raw byte value, causing a silent roundtrip mismatch. Now returns an error unless the byte has an explicit single_char_escape mapping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Avoids unconditionally flagging backslash as needing escape in Doubling mode where backslash is a literal character. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Only include qc and backslash in the escape char byteset when they fall in the control range (< 0x20). Since both are always >= 0x20 for typical configurations, this removes spurious alternatives from the fast-path regex that would accept escape sequences for non-control bytes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The slow path iterates all 256 bytes so the post-loop for bytes >= 0x20 was producing duplicate regex alternatives. Now only runs after the fast path which only covers 0x00-0x1F. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace custom Hash/PartialEq impls with derived ones and add a normalize() method that sorts and dedups the Vec fields. Called once at string_escape entry before cache lookup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- must_escape byte 0x7F with HexHH (tests >= 0x20 escape path) - UnicodeXXXX rejection for must_escape bytes >= 0x80 - Control-only regex (tests fast path correctness) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This reverts commit c7af1873e08b9c053520f4644db755e006e0623e.
Remove has_fallback parameter from quote_byteset (computed from options internally). Apply rustfmt formatting across changed files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Expand docstrings on StringEscapeOptions, FallbackEscapeFormat, QuoteEscapeMethod, JsonQuoteOptions, and the RegexAst variants - Document backslash handling semantics, normalization, error conditions, and the interaction between quote_escape and uses_backslash - Clarify that \\ and quote_char in allowed_escapes are always applied during JsonQuoteOptions conversion (not gated) - Remove dead branch (b'\\' < 0x20 is always false) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR extracts the JSON-specific escape logic from json_quote into a configurable string_escape API, enabling support for various string literal escape grammars (JSON, Python, YAML, C-style) through a declarative StringEscapeOptions struct. The existing json_quote method is preserved as a thin wrapper that converts JsonQuoteOptions into StringEscapeOptions and delegates.
Changes:
- Introduces
StringEscapeOptions,FallbackEscapeFormat, andQuoteEscapeMethodtypes for configurable string escape grammars, withRegexBuilder::string_escape()as the core implementation. - Refactors
json_quote()to delegate tostring_escape()viaJsonQuoteOptions::to_string_escape_options(), replacing the single expression cache with per-options caches. - Adds
RegexAst::StringEscapevariant and 8 new tests covering JSON equivalence, hex fallback, single-quote/doubling escapes, caching, high bytes, and error cases.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
src/regexbuilder.rs |
Core implementation: new types, string_escape() method, refactored json_quote(), cache changes, RegexAst::StringEscape variant |
src/lib.rs |
Exports new public types: FallbackEscapeFormat, QuoteEscapeMethod, StringEscapeOptions |
tests/basic.rs |
8 new integration tests covering JSON equivalence, HexHH, single-quote, doubling, caching, high bytes, and error rejection |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
In retrospect, I'm marking as a draft. Backslash is a bit too hard coded -- it would be nice to be general enough to cover URL percent-encoding and XML. |
- Add escape_prefix field to StringEscapeOptions (default b'\\') - Add FallbackEscapeFormat::BareHH for bare hex after prefix (%HH) - Add QuoteEscapeMethod::Normal for no special quote handling - Add StringEscapeOptions::url_percent_encoding() constructor - Generalize all internal logic from hardcoded backslash to ep - Escape prefix is no longer implicitly doubled; must be listed in single_char_escapes for doubling (e.g., (b'\\', b'\\')) - Add tests for percent encoding and Normal quote method Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace single_char_escapes, escape_prefix, quote_char, quote_escape, FallbackEscapeFormat, and QuoteEscapeMethod with: - escape_sequences: Vec<(u8, Vec<u8>)> for explicit per-byte escapes - fallback_prefix: Option<Vec<u8>> + hex digits for remaining bytes - max_fallback_byte: Option<u8> for validation - must_escape: Vec<u8> (fully explicit, no implicit additions) Remove raw_mode (delimiter wrapping moved to json_quote). Remove FallbackEscapeFormat and QuoteEscapeMethod enums entirely. Bytes with both an explicit escape and a fallback now accept either form (e.g., JSON \b and \u0008 both valid for byte 0x08). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
json_quote delegates to string_escape, so the equivalence test was tautological. The 4 pre-existing json_quote tests cover the legacy API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Redesign the string_escape API: - Introduce FallbackFormat struct with prefix, suffix, and valid_byte_ranges fields. valid_byte_ranges uses Vec<(u8,u8)> inclusive ranges instead of a single max_byte threshold, enabling non-contiguous byte sets (e.g. XML 1.0 valid characters). - Restructure StringEscapeOptions: replace fallback_prefix with fallback: Option<FallbackFormat>, and add must_escape field. - Add constructors: json(), json_raw(), url_percent_encoding(), xml(). - Fix bug in quote_byteset: fallback hex was generated for bytes outside the valid range when they had explicit escape sequences. Now gated by FallbackFormat::covers_byte(). - Fix xml() constructor: must_escape only includes valid XML 1.0 characters (0x09, 0x0A, 0x0D, markup chars), not forbidden control characters. - Update json_quote to delegate through string_escape via JsonQuoteOptions::to_string_escape_options(). - Export FallbackFormat from lib.rs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Fix doc inconsistency: document that string_escape errors when a must-escape byte is outside valid_byte_ranges without an explicit escape, rather than silently excluding it. - Validate escape_sequences for conflicting duplicates in normalize(). - Optimize quote_byteset and needs-check to iterate must_escape slice instead of scanning all 256 bytes. - Remove stray println! in test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Canonicalize valid_byte_ranges in normalize(): sort, validate (start <= end), and merge overlapping/adjacent ranges so semantically equivalent options hash identically for caching. - Fix XML doc: 0x7F is valid in XML 1.0 (part of #x20-#xD7FF). Removed incorrect claim that 0x7F is forbidden. - Document that to_string_escape_options() does not validate allowed_escapes (json_quote() handles that). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
xml() does not prevent forbidden control bytes from passing through literally if they appear in the input regex. Document that callers should constrain their input regex to valid XML characters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
@riedgar-ms @nopdive @nking-1 would appreciate a review, particularly of the pub api surface and whether it seems general / extensible enough for as much future proofing as possible... |
Generalize string_escape API
Generalizes the existing json_quote mechanism into a configurable string_escape API that supports arbitrary escape grammars (JSON, URL percent-encoding, XML, etc.).
Key changes
New types:
FallbackFormat— describes the hex fallback encoding (prefix + 2 hex digits + suffix) withvalid_byte_ranges: Option<Vec<(u8, u8)>>to restrict which bytes are eligible. Inclusive ranges handle non-contiguous valid sets (e.g., XML 1.0 characters).StringEscapeOptions— declarative escape grammar withescape_sequences,fallback: Option<FallbackFormat>, andmust_escapebyte list.Constructors:
StringEscapeOptions::json()—\u00XXfallback, standard control-char escapesStringEscapeOptions::json_raw()— same but no\uXXXXfallbackStringEscapeOptions::url_percent_encoding()—%HHfor all non-unreserved bytesStringEscapeOptions::xml()— named entities +&#xNN;fallback, only valid XML 1.0 charactersBug fix:
quote_bytesetnow checksFallbackFormat::covers_byte()before generating fallback hex alternatives. Previously, bytes outside the valid range could get semantically incorrect fallback forms.Legacy compatibility:
json_quote()still works — delegates tostring_escape()viaJsonQuoteOptions::to_string_escape_options().FallbackFormatandStringEscapeOptionsare exported fromlib.rs.