Closed Bug 669483 Opened 8 years ago Closed 8 years ago
.py log handling chokes on non-ASCII output
I wrote a test that dump()s "\xe9" as part of its output. On my local machine, this results in: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/logging/__init__.py", line 765, in emit self.stream.write(fs % msg.encode("UTF-8")) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 131: ordinal not in range(128) I still haven't figured out exactly what's going on, but I managed to come up with a patch that "fixes" this for me. Need to look into it more.
Assignee: nobody → gavin.sharp
Status: NEW → ASSIGNED
So from what I can gather the problem is this: - the logging code attempts to encode the passed-in string as UTF-8 (the stream's encoding in this case is UTF-8). The encode() method accepts unicode input, but otherwise falls back to assuming ASCII if the input is not of type "unicode" (which is the case for us). - our output is mostly already UTF-8 encoded, e.g. dump() produces bytes 0xC3 0xA9 for the JS string "\xe9" (though try server runs with an earlier patch did reveal that we also occasionally output invalid UTF-8 sequences, like 0xA0 for an NBSP in ISO-8559-1) So when the logging code attempts to interpret our UTF-8 as ASCII, it chokes. We can't just assume that it's all UTF-8, though, since we also produce ISO-8559-1. Thankfully jhammel points out that decode() in Python 2.3 can deal with this case by just ignoring the invalid sequences and leave them as-is, and since they're ISO-8859-1 they're already effectively "unicode".
This passed a all-unittests tryserver run without issues: http://tbpl.mozilla.org/?tree=Try&rev=3216385011d4
Comment on attachment 544272 [details] [diff] [review] patch Review of attachment 544272 [details] [diff] [review]: -----------------------------------------------------------------
Attachment #544272 - Flags: review?(ted.mielczarek) → review+
Target Milestone: --- → mozilla8
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.