Bugfix: fix processing of UTF-8 files with BOM - #5506
Conversation
Signed-off-by: SSE4 <tomskside@gmail.com>
| tmp = handle.read() | ||
| if not binary: | ||
| import codecs | ||
| encodings = {codecs.BOM_UTF8: "utf_8_sig", |
There was a problem hiding this comment.
Wow, I can't believe this is necessary... So every python application out there that is reading text files should do something like this? I am not sure it makes sense, but most likely I am failing to understand the issue. Can't we just ask that conanfile.txt should have a standard ascii or utf8 encoding? Is this something that will be solved in Python3 and this is only for Python 2? Wdyt @lasote?
There was a problem hiding this comment.
Is it possible solving this using comments with encoding like # -*- coding: utf-8 -*- ?
There was a problem hiding this comment.
yes, seems so, I have found many such recommendations on stack overflow.
some text editors, mostly on Windows, defaults to save into UTF-8 with BOM, or UTF-16 with BOM, if they contain some non-ASCII byte sequences.
There was a problem hiding this comment.
from wiki:
Microsoft compilers[9] and interpreters, and many pieces of software on Microsoft Windows such as Notepad treat the BOM as a required magic number rather than use heuristics. These tools add a BOM when saving text as UTF-8, and cannot interpret UTF-8 unless the BOM is present or the file contains only ASCII. Google Docs also adds a BOM when converting a document to a plain text file for download.
There was a problem hiding this comment.
Wow, this is insane. Probably that code should be moved to decode_text function that it is already a bit insane.
There was a problem hiding this comment.
So, to confirm, please answer the question from @memsharded, Is this something that will be solved in Python3 and this is only for Python 2?
There was a problem hiding this comment.
no, it will not be solved by Python 3, I have exactly the same issue with Python 3 and UTF-8 with BOM files on Windows, which Notepad saves by default for me.
Signed-off-by: SSE4 <tomskside@gmail.com>
Changelog: Bugfix: fix processing of UTF-8 files with BOM
Docs: omit
@PYVERS: Macos@py27, Windows@py36, Linux@py27, py34
@tags: svn, slow
@revisions: 1
closes: #5504
developbranch, documenting this one.Note: By default this PR will skip the slower tests and will use a limited set of python versions. Check here how to increase the testing level by writing some tags in the current PR body text.