Skip to content

[Bug]: search-specs CLI crashes with UnicodeEncodeError on non-UTF-8 consoles #139

Description

@aniruddhaadak80

What happened?

forge/stage1_intake/search_specs.py prints query text and result snippets with ensure_ascii=False straight to print(). When stdout is not UTF-8 (Windows console with a non-UTF-8 code page, or any host with PYTHONIOENCODING set to a legacy codec), any non-ASCII query or snippet crashes the CLI with UnicodeEncodeError instead of returning results. I expected the CLI to emit UTF-8 output and exit 0 regardless of the console encoding, since every source file and cache in this pipeline is UTF-8.

Reproduction steps

  1. From the skill root, force a non-UTF-8 stdout encoding (portable on any OS):

    PYTHONIOENCODING=ascii python3 forge/stage1_intake/search_specs.py --collection cs2 rough --json

    Any query whose snippets contain non-ASCII text (for example the Vietnamese fixture terms) triggers it.

  2. Observe a traceback ending in UnicodeEncodeError and exit code 1, instead of the JSON payload.

The repo's own suite covers this path: test_human_output_is_nonempty_and_stable_on_cache_hits and test_untruncated_snippet_has_no_ellipses in forge/tests/test_search_specs.py search fixture corpora containing Vietnamese terms and an ellipsis snippet. Both pass on UTF-8 hosts and fail on Windows default consoles.

Reference image or input characteristics

Not image-related. Input: any search query plus an indexed corpus containing non-ASCII text (Vietnamese terms, ellipsis U+2026). Output: the JSON or human-readable result listing passed through print() with ensure_ascii=False.

Relevant output and evidence

Child-process stderr on a cp1252 Windows console:

File "forge/stage1_intake/search_specs.py", line 143, in _print_human
    print(f"Query: {payload['query']}")
UnicodeEncodeError: 'charmap' codec can't encode characters: character maps to <undefined>

The JSON path fails identically in _print_json (json.dumps(..., ensure_ascii=False) written to a cp1252 stream). The same print pattern exists across other forge CLIs, but this issue is scoped to search_specs.py; the rest can follow the same remedy once agreed.

Environment

  • img2threejs version or commit: upstream/main 6e60b5e
  • Python version: 3.14.3 (Windows; repo toolchain is python3, invoked here via the py launcher)
  • Host agent: OpenCode
  • Operating system: Windows 11 (cp1252 console); portable repro via PYTHONIOENCODING=ascii on any OS

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions