1147183 - UnicodeEncodeError: 'ascii' codec can't encode characters in position 39-44: ordinal not in range(128) when running mach build python/check

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Mike Hommey [:glandium]]

0:00.77 TEST-PASS | /home/glandium/gecko/python/mozbuild/mozbuild/test/action/test_generate_browsersearch.py | test_invalid_unicode
 0:00.77 TEST-PASS | /home/glandium/gecko/python/mozbuild/mozbuild/test/action/test_generate_browsersearch.py | test_valid_unicode
 0:00.77 Read 2 engines: [u'\u767e\u5ea6', u'Google']
Exception in thread ProcessReader:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/glandium/gecko/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 893, in _read
    callback(line.rstrip())
  File "/home/glandium/gecko/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 813, in __call__
    e(*args, **kwargs)
  File "/home/glandium/gecko/python/mach/mach/mixin/process.py", line 86, in handleLine
    line_handler(line)
  File "/home/glandium/gecko/python/mozbuild/mozbuild/mach_commands.py", line 253, in on_line
    self.log(logging.INFO, 'build_output', {'line': line}, '{line}')
  File "/home/glandium/gecko/python/mach/mach/mixin/logging.py", line 54, in log
    extra={'action': action, 'params': params})
  File "/usr/lib/python2.7/logging/__init__.py", line 1224, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib/python2.7/logging/__init__.py", line 1279, in _log
    self.handle(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 1289, in handle
    self.callHandlers(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 1329, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 757, in handle
    self.emit(record)
  File "/home/glandium/gecko/python/mozbuild/mozbuild/mach_commands.py", line 102, in emit
    self.fh.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 39-44: ordinal not in range(128)


This comes from the test writing out:
Read 2 engines: [u'\u767e\u5ea6', u'Google']
Default engine is '百度'.

Comment 1

[Mike Hommey [:glandium]]

Attachment: Avoid test_generate_browsersearch.py printing out unicode

Comment 2

[(not currently active) Ted Mielczarek]

Ideally the test logging would not die on this, but maybe that's hard to fix.

Comment 3

[Gregory Szorc [:gps]]

I thought our test logging already did intelligent .decode('utf-8', 'replace') to handle unknown byte sequences :/

Comment 4

[Mike Hommey [:glandium]]

Maybe it should be improved (in a separate bug), but I do think that tests should not be outputting random stuff anyways.

Comment 5

[Mike Hommey [:glandium]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/28e7e3d470f1

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/28e7e3d470f1