1261235 - UnicodeDecodeError when dumping old-configure's config.log output

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Mike Hommey [:glandium]]

There are cases where old-configure's config.log can contain non ascii characters and that leads to:
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 5:
 ordinal not in range(128)

when we dump it, we fail.

The easy fix for the reading part is to use encoded_open instead of open, but that just moves the failure further later when printing out to stdout wants ascii too.

Comment 1

[Mike Hommey [:glandium]]

Attachment: MozReview Request: Bug 1261235 - Handle non-ascii in configure logging. r?ted

When reading config.log, with old-configure output, we may get non-ascii
strings, but that currently fails because we're using plain open() to
read it. So use encoded_open() instead (which does the same job for
other files in the same script).

Because the build system can be encapsulated in mach, python configure
can have a pipe as stdout/stderr, and in that case, sys.stdout/stderr
have an ascii encoding, failing to print out anything that doesn't
fit in ascii, consequently failing to print the things we've read from
config.log. So reopen stdout and stderr with the right encoding in
the configure output handler.

Review commit: https://reviewboard.mozilla.org/r/43655/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/43655/

Comment 2

[(not currently active) Ted Mielczarek]

Comment on attachment 8737019 [details]
MozReview Request: Bug 1261235 - Handle non-ascii in configure logging. r?ted

https://reviewboard.mozilla.org/r/43655/#review40229

Comment 3

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/667c295f3d3d

Comment 4

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/667c295f3d3d

Comment 5

[Samael Wang [:freesamael] (away for now)]

I'm encountering the following error on Mac OS X which looks related to this bug.

> 0:02.04 Traceback (most recent call last):
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/configure.py", line 94, in <module>
> 0:02.04     sys.exit(main(sys.argv))
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/configure.py", line 21, in main
> 0:02.04     sandbox = ConfigureSandbox(config, os.environ, argv)
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/__init__.py", line 143, in __init__
> 0:02.04     handler = ConfigureOutputHandler(stdout, stderr)
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/util.py", line 74, in __init__
> 0:02.04     self._stdout = fix_encoding(stdout)
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/util.py", line 69, in fix_encoding
> 0:02.04     return codecs.getwriter(locale.getpreferredencoding())(fh)
> 0:02.04   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 606, in getpreferredencoding
> 0:02.04     return getdefaultlocale()[1]
> 0:02.04   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 543, in getdefaultlocale
> 0:02.04     return _parse_localename(localename)
> 0:02.04   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 475, in _parse_localename
> 0:02.04     raise ValueError, 'unknown locale: %s' % localename
> 0:02.04 ValueError: unknown locale: UTF-8

The default locale on mac is as following
> $ locale
> LANG=
> LC_COLLATE="C"
> LC_CTYPE="UTF-8"
> LC_MESSAGES="C"
> LC_MONETARY="C"
> LC_NUMERIC="C"
> LC_TIME="C"
> LC_ALL=

I tried export LANG=en_US.UTF-8 / LC_ALL=en_US.UTF-8, it passes main configure but the another error occurs during configuring js/src:

> 0:06.29 js/src> Traceback (most recent call last):
> 0:06.29 js/src>   File "/Users/freesamael/Repos/gecko-dev/build/../configure.py", line 94, in <module>
> 0:06.29 js/src>     sys.exit(main(sys.argv))
> 0:06.29 js/src>   File "/Users/freesamael/Repos/gecko-dev/build/../configure.py", line 21, in main
> 0:06.30 js/src>     sandbox = ConfigureSandbox(config, os.environ, argv)
> 0:06.30 js/src>   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/__init__.py", line 143, in __init__
> 0:06.30 js/src>     handler = ConfigureOutputHandler(stdout, stderr)
> 0:06.30 js/src>   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/util.py", line 74, in __init__
> 0:06.30 js/src>     self._stdout = fix_encoding(stdout)
> 0:06.30 js/src>   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/util.py", line 69, in fix_encoding
> 0:06.30 js/src>     return codecs.getwriter(locale.getpreferredencoding())(fh)
> 0:06.30 js/src>   File "/Users/freesamael/Repos/gecko-build/obj-firefox/_virtualenv/lib/python2.7/codecs.py", line 1009, in getwriter
> 0:06.30 js/src>     return lookup(encoding).streamwriter
> 0:06.30 js/src> TypeError: must be string, not None

Any suggestion?

Comment 6

[Mike Hommey [:glandium]]

Please file a followup bug.