Skip to content

_FileSystemSpecBackend.open() ignores the utf-8 contract stated in Backend.open() #817

Open

Description

@MohammedAlkindi

Backend.open() documents its own contract at backend.py:47:

"""`open`. Encoding should be utf-8."""

_OsPathBackend.open() honours it, passing encoding='utf-8' for text mode (backend.py:124-133). _FileSystemSpecBackend.open() does not:

def open(self, path: PathLike, mode: str) -> typing.IO[Union[str, bytes]]:
  return self.fs(path).open(path, mode=mode)

With no encoding=, fsspec falls back to locale.getpreferredencoding(). On a Windows machine with a non-UTF-8 locale that is cp1252, and any non-latin1 text fails to write:

# Windows 11, CPython 3.13.13, locale.getpreferredencoding() == 'cp1252'
fs = fsspec.filesystem('file')
with fs.open(p, mode='w') as f:
    f.write('abc统一码def')
# UnicodeEncodeError: 'charmap' codec can't encode characters in position 3-5

open(p, 'rb').read()
# b''  - the file is created and left empty

The same write through epath.Path on the os backend produces the expected UTF-8 bytes, so the two backends disagree about a contract the abstract base class states explicitly.

The failure is not Windows-specific in principle. It reproduces on any platform whose preferred encoding is not UTF-8; ubuntu-latest defaults to UTF-8, which is why pytest_and_autopublish.yml has never surfaced it.

test_backend[_test_open] reports SUBFAILED[fsspec] with this UnicodeEncodeError, but the module has never run on Windows because it fails collection there on import grp.

I have no Linux or macOS machine, so the claim about non-UTF-8 POSIX locales is reasoned from locale.getpreferredencoding() semantics rather than measured.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions