669483 - automation.py log handling chokes on non-ASCII output

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[:Gavin Sharp [email: [e-mail]]]

I wrote a test that dump()s "\xe9" as part of its output. On my local machine, this results in:

Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/logging/__init__.py", line 765, in emit
    self.stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 131: ordinal not in range(128)

I still haven't figured out exactly what's going on, but I managed to come up with a patch that "fixes" this for me. Need to look into it more.

Comment 1

[:Gavin Sharp [email: [e-mail]]]

Attachment: potential patch

Comment 2

[:Gavin Sharp [email: [e-mail]]]

So from what I can gather the problem is this:
- the logging code attempts to encode the passed-in string as UTF-8 (the stream's encoding in this case is UTF-8). The encode() method accepts unicode input, but otherwise falls back to assuming ASCII if the input is not of type "unicode" (which is the case for us).
- our output is mostly already UTF-8 encoded, e.g. dump() produces bytes 0xC3 0xA9 for the JS string "\xe9" (though try server runs with an earlier patch did reveal that we also occasionally output invalid UTF-8 sequences, like 0xA0 for an NBSP in ISO-8559-1)

So when the logging code attempts to interpret our UTF-8 as ASCII, it chokes. We can't just assume that it's all UTF-8, though, since we also produce ISO-8559-1. Thankfully jhammel points out that decode() in Python 2.3 can deal with this case by just ignoring the invalid sequences and leave them as-is, and since they're ISO-8859-1 they're already effectively "unicode".

Comment 3

[:Gavin Sharp [email: [e-mail]]]

Attachment: patch

Comment 4

[:Gavin Sharp [email: [e-mail]]]

This passed a all-unittests tryserver run without issues:
http://tbpl.mozilla.org/?tree=Try&rev=3216385011d4

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 544272 [details] [diff] [review]
patch

Review of attachment 544272 [details] [diff] [review]:
-----------------------------------------------------------------

Comment 6

[:Gavin Sharp [email: [e-mail]]]

http://hg.mozilla.org/integration/fx-team/rev/8e54ffd82a58

Comment 7

[:Gavin Sharp [email: [e-mail]]]

http://hg.mozilla.org/mozilla-central/rev/8e54ffd82a58