Skip to content

test_sqlite3 fails on non-UTF-8 locale #91922

Closed

Description

@serhiy-storchaka
$ LC_ALL=en_US.iso88591 ./python -m test -vuall test_sqlite3
...
======================================================================
ERROR: test_ctx_mgr_rollback_if_commit_failed (test.test_sqlite3.test_dbapi.MultiprocessTests.test_ctx_mgr_rollback_if_commit_failed)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_sqlite3/test_dbapi.py", line 1767, in test_ctx_mgr_rollback_if_commit_failed
    cx = sqlite.connect(TESTFN, timeout=self.CONNECTION_TIMEOUT)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe6 in position 16: unexpected end of data

======================================================================
ERROR: test_open_uri (test.test_sqlite3.test_dbapi.OpenTests.test_open_uri)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_sqlite3/test_dbapi.py", line 674, in test_open_uri
    with managed_connect(TESTFN) as cx:
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/test/test_sqlite3/test_dbapi.py", line 44, in managed_connect
    cx = sqlite.connect(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe6 in position 16: unexpected end of data

======================================================================
ERROR: test_open_with_path_like_object (test.test_sqlite3.test_dbapi.OpenTests.test_open_with_path_like_object)
Checks that we can successfully connect to a database using an object that
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_sqlite3/test_dbapi.py", line 670, in test_open_with_path_like_object
    with managed_connect(path) as cx:
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/test/test_sqlite3/test_dbapi.py", line 44, in managed_connect
    cx = sqlite.connect(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe6 in position 16: unexpected end of data

======================================================================
ERROR: test_trace_callback_content (test.test_sqlite3.test_hooks.TraceCallbackTests.test_trace_callback_content)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_sqlite3/test_hooks.py", line 279, in test_trace_callback_content
    con1 = sqlite.connect(TESTFN, isolation_level=None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe6 in position 16: unexpected end of data

----------------------------------------------------------------------

Activity

  1. serhiy-storchaka commented on Apr 26, 2022

    @serhiy-storchaka
    MemberAuthor

    It is not just tests. There is a bug in the code.

    It is due to this line:

    if (PySys_Audit("sqlite3.connect", "s", database) < 0) {

    database is a file path which can be not UTF-8. And "s" tries to decode it as UTF-8. There may be other similar bugs related to audition in other code. @tiran

    I afraid also that sqlite3.connect() does not work correctly with non-ASCII path on Windows. It could be more correct to use sqlite3_open16() on Windows, but it lacks the flags parameter.

    @erlend-aasland

  2. erlend-aasland commented on Apr 26, 2022

    @erlend-aasland
    Contributor

    I'd rather fix this by documenting that the database path must be UTF-8. I'm afraid that using both sqlite3_open16 and sqlite3_open_v2 will create too much complexity in the code; my initial reaction is that it is not worth the added complexity, but I will absolutely consider it.

  3. erlend-aasland commented on Apr 26, 2022

    @erlend-aasland
    Contributor

    Quoting the SQLite docs:

    Note to Windows users: The encoding used for the filename argument of sqlite3_open() and sqlite3_open_v2() must be UTF-8, not whatever codepage is currently defined. Filenames containing international characters must be converted to UTF-8 prior to passing them into sqlite3_open() or sqlite3_open_v2().

    We should add this information to the docs.

  4. erlend-aasland commented on Apr 27, 2022

    @erlend-aasland
    Contributor

    I'm troubled by this sentence in the SQLite docs:

    The default encoding will be UTF-8 for databases created using sqlite3_open() or sqlite3_open_v2(). The default encoding for databases created using sqlite3_open16() will be UTF-16 in the native byte order.

  5. erlend-aasland commented on May 18, 2022

    @erlend-aasland
    Contributor

    I propose to resolve this by improving the docs. Let me know if you agree.

  6. added a commit that references this issue on May 18, 2022
  7. serhiy-storchaka commented on May 18, 2022

    @serhiy-storchaka
    MemberAuthor

    #92926 resolves this issue. It also fixes support of non-decodable bytes paths. I am not sure about URIs, perhaps they need additional work.

  8. serhiy-storchaka commented on May 18, 2022

    @serhiy-storchaka
    MemberAuthor

    BTW, sqlite3_open_v2() works with non-UTF8 paths.

  9. added a commit that references this issue on May 20, 2022
  10. added 2 commits that reference this issue on May 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions