You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
_FileSystemSpecBackend.open() ignores the utf-8 contract stated in Backend.open() #817
With no encoding=, fsspec falls back to locale.getpreferredencoding(). On a Windows machine with a non-UTF-8 locale that is cp1252, and any non-latin1 text fails to write:
# Windows 11, CPython 3.13.13, locale.getpreferredencoding() == 'cp1252'fs=fsspec.filesystem('file')
withfs.open(p, mode='w') asf:
f.write('abc统一码def')
# UnicodeEncodeError: 'charmap' codec can't encode characters in position 3-5open(p, 'rb').read()
# b'' - the file is created and left empty
The same write through epath.Path on the os backend produces the expected UTF-8 bytes, so the two backends disagree about a contract the abstract base class states explicitly.
The failure is not Windows-specific in principle. It reproduces on any platform whose preferred encoding is not UTF-8; ubuntu-latest defaults to UTF-8, which is why pytest_and_autopublish.yml has never surfaced it.
test_backend[_test_open] reports SUBFAILED[fsspec] with this UnicodeEncodeError, but the module has never run on Windows because it fails collection there on import grp.
I have no Linux or macOS machine, so the claim about non-UTF-8 POSIX locales is reasoned from locale.getpreferredencoding() semantics rather than measured.
Activity