Skip to content

Decode UTF-16 files with a byte order mark - #440

Open
unknwon wants to merge 1 commit into
mainfrom
utf16-decode
Open

unknwon wants to merge 1 commit into
mainfrom
utf16-decode

Conversation

@unknwon

@unknwon unknwon commented Nov 15, 2026

Copy link
Copy Markdown
Member

Describe the pull request

Windows tools such as Notepad save "Unicode" files as UTF-16 with a byte order mark. skipBOM dropped the mark but parsed the remaining bytes as UTF-8, so every name and value held NUL bytes. f.Section("author") on testdata/UTF-16-LE-BOM.ini found nothing. The existing UTF-16 tests only checked that Load didn't fail, and the BE one was empty.

parse now decodes input that starts with FF FE or FE FF as UTF-16 LE or BE before lexing, using unicode/utf16, so no dependency is added. Neither prefix is valid UTF-8, so UTF-8 input can't be affected. Surrogate pairs decode, and a trailing odd byte becomes U+FFFD. Write still writes UTF-8, and the migration guide says so. UTF-16 without a byte order mark isn't detected.

Link to the issue: closes #66

Checklist

  • I agree to follow the Code of Conduct by submitting this pull request.
  • I have read and acknowledge the Contributing guide.
  • I have added test cases to cover the new code.

AI usage

Written by Amp (an AI coding agent) at the maintainer's request. Validation: go test ./..., go vet, and 40 s of FuzzWriteRoundTrip. The LE, BE and non-ASCII tests (CJK names, an emoji surrogate pair, CRLF line ends, an odd trailing byte) all fail on main.

A file starting with a UTF-16 LE or BE byte order mark was read as UTF-8
after the mark was dropped, so its names and values held NUL bytes. It
is now decoded as UTF-16 before parsing (#66).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Please add support for UTF16LE and UTF16BE

1 participant