921694 - UnicodeDecodeError: 'ascii' codec can't decode byte 0x94 in position 230: ordinal not in range(128)

Status: RESOLVED DUPLICATE of bug 921388

(Testing :: XPCShell Harness, defect)

Description

[Justin Dolske [:Dolske]]

Attachment: Patch for bug 717490

Attached is a snapshot of a WIP patch for bug 717490.

When I build with it and run "mach xpcshell-test toolkit/components/passwordmgr/test/unit/test_storage_mozStorage_6.js", I get the following error. The test failure is mine, I believe (I'm updating some tests), but mach seems unhappy with it somehow.

...
 0:00.47 TEST-PASS | /Users/dolske/build/mozilla-central/obj/_tests/xpcshell/toolkit/components/passwordmgr/test/unit/test_storage_mozStorage_6.js | [run_test : 295] "newpass2" == "newpass2"
 0:00.47 
 0:00.47 TEST-PASS | /Users/dolske/build/mozilla-central/obj/_tests/xpcshell/toolkit/components/passwordmgr/test/unit/test_storage_mozStorage_6.js | [run_test : 296] 123 == 123
 0:00.47 
 0:00.47 TEST-PASS | /Users/dolske/build/mozilla-central/obj/_tests/xpcshell/toolkit/components/passwordmgr/test/unit/test_storage_mozStorage_6.js | [run_test : 297] 456 == 456
 0:00.47 
 0:00.47 TEST-PASS | /Users/dolske/build/mozilla-central/obj/_tests/xpcshell/toolkit/components/passwordmgr/test/unit/test_storage_mozStorage_6.js | [run_test : 298] true == true
 0:00.47 
 0:00.47 TEST-PASS | /Users/dolske/build/mozilla-central/obj/_tests/xpcshell/toolkit/components/passwordmgr/test/unit/test_storage_mozStorage_6.js | [run_test : 299] 8 == 8
 0:00.48 
 0:00.48 TEST-PASS | /Users/dolske/build/mozilla-central/obj/_tests/xpcshell/toolkit/components/passwordmgr/test/unit/test_storage_mozStorage_6.js | [run_test : 321] 1 == 1
 0:00.48 
 0:00.48 TEST-UNEXPECTED-FAIL | /Users/dolske/build/mozilla-central/obj/_tests/xpcshell/toolkit/components/passwordmgr/test/unit/test_storage_mozStorage_6.js | Process still running after test!
 0:00.48 >>>>>>>
 0:00.48 <<<<<<<
 0:00.48 INFO | Following exceptions were raised:
 0:00.48 Traceback (most recent call last):
  File "/Users/dolske/build/mozilla-central/testing/xpcshell/runxpcshelltests.py", line 145, in run
    self.run_test()
  File "/Users/dolske/build/mozilla-central/testing/xpcshell/runxpcshelltests.py", line 592, in run_test
    stdout, stderr = self.communicate(proc)
  File "/Users/dolske/build/mozilla-central/testing/xpcshell/runxpcshelltests.py", line 207, in communicate
    self.process_line(line)
  File "/Users/dolske/build/mozilla-central/testing/xpcshell/runxpcshelltests.py", line 471, in process_line
    self.report_message(line_string)
  File "/Users/dolske/build/mozilla-central/testing/xpcshell/runxpcshelltests.py", line 457, in report_message
    self.on_message(line, message)
  File "/Users/dolske/build/mozilla-central/testing/xpcshell/mach_commands.py", line 145, in <lambda>
    'on_message': (lambda obj, msg: xpcshell.log.info(msg)) \
  File "/Users/dolske/build/mozilla-central/testing/xpcshell/runxpcshelltests.py", line 735, in wrapped
    fn(*args, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 1132, in info
    self._log(INFO, msg, args, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 1250, in _log
    self.handle(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 1260, in handle
    self.callHandlers(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 1300, in callHandlers
    hdlr.handle(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 740, in handle
    rv = self.filter(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 604, in filter
    if not f.filter(record):
  File "/Users/dolske/build/mozilla-central/testing/xpcshell/mach_commands.py", line 37, in filter
    return not record.params['msg'].endswith("running test ...")
UnicodeDecodeError: 'ascii' codec can't decode byte 0x94 in position 230: ordinal not in range(128)

Error running mach:

    ['xpcshell-test', 'toolkit/components/passwordmgr/test/unit/test_storage_mozStorage_6.js']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

UnicodeDecodeError: 'ascii' codec can't decode byte 0x94 in position 230: ordinal not in range(128)

  File "/Users/dolske/build/mozilla-central/testing/xpcshell/mach_commands.py", line 237, in run_xpcshell_test
    return xpcshell.run_test(**params)
  File "/Users/dolske/build/mozilla-central/testing/xpcshell/mach_commands.py", line 105, in run_test
    return self._run_xpcshell_harness(**args)
  File "/Users/dolske/build/mozilla-central/testing/xpcshell/mach_commands.py", line 190, in _run_xpcshell_harness
    result = xpcshell.runTests(**filtered_args)
  File "/Users/dolske/build/mozilla-central/testing/xpcshell/runxpcshelltests.py", line 1447, in runTests
    raise exceptions[0]
Justins-MacBook-Pro:mozilla-central dolske$

Comment 1

[Justin Dolske [:Dolske]]

(This is with a current m-c pull, 1a2d9a04ffb2 + patch)

Comment 2

[Justin Dolske [:Dolske]]

I think this is an existing bug independent of my patch.

This is the line triggering the error:

toolkit/components/passwordmgr/test/unit/test_storage_mozStorage_6.js

...
// Check that values we specified were set, instead of defaults.
do_print("ok1");
do_check_eq("newpass2again\u0394\u00e8", tu2.password);
do_print("ok2");
...

I added the do_print() lines. "ok1" is logged, but mach asplodes before logging the "ok2".

Comment 3

[Gregory Szorc [:gps]]

This will happen if loggers attempt to pass Python's str type instead of unicode type into the logger and the underlying str contains non-ascii characters.

The xpcshell-test command should convert strings to unicode and munge invalid character sequences. e.g. value.decode('utf-8', 'replace') before passing values to the logger.

Comment 4

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

I've tried patch from Bug 921388, and it's working.
So I duplicate this bug.