Closed Bug 669483 Opened 13 years ago Closed 13 years ago

automation.py log handling chokes on non-ASCII output

Categories

(Testing :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla8

People

(Reporter: Gavin, Assigned: Gavin)

References

Details

Attachments

(1 file, 1 obsolete file)

I wrote a test that dump()s "\xe9" as part of its output. On my local machine, this results in:

Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/logging/__init__.py", line 765, in emit
    self.stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 131: ordinal not in range(128)

I still haven't figured out exactly what's going on, but I managed to come up with a patch that "fixes" this for me. Need to look into it more.
Attached patch potential patch (obsolete) — Splinter Review
Assignee: nobody → gavin.sharp
Status: NEW → ASSIGNED
So from what I can gather the problem is this:
- the logging code attempts to encode the passed-in string as UTF-8 (the stream's encoding in this case is UTF-8). The encode() method accepts unicode input, but otherwise falls back to assuming ASCII if the input is not of type "unicode" (which is the case for us).
- our output is mostly already UTF-8 encoded, e.g. dump() produces bytes 0xC3 0xA9 for the JS string "\xe9" (though try server runs with an earlier patch did reveal that we also occasionally output invalid UTF-8 sequences, like 0xA0 for an NBSP in ISO-8559-1)

So when the logging code attempts to interpret our UTF-8 as ASCII, it chokes. We can't just assume that it's all UTF-8, though, since we also produce ISO-8559-1. Thankfully jhammel points out that decode() in Python 2.3 can deal with this case by just ignoring the invalid sequences and leave them as-is, and since they're ISO-8859-1 they're already effectively "unicode".
Attached patch patchSplinter Review
Attachment #544075 - Attachment is obsolete: true
Attachment #544272 - Flags: review?(ted.mielczarek)
This passed a all-unittests tryserver run without issues:
http://tbpl.mozilla.org/?tree=Try&rev=3216385011d4
Comment on attachment 544272 [details] [diff] [review]
patch

Review of attachment 544272 [details] [diff] [review]:
-----------------------------------------------------------------
Attachment #544272 - Flags: review?(ted.mielczarek) → review+
http://hg.mozilla.org/mozilla-central/rev/8e54ffd82a58
Flags: in-testsuite-
Target Milestone: --- → mozilla8
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.