1263296 - Build fail on Mac OS X: "ValueError: unknown locale: UTF-8"

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Samael Wang [:freesamael] (away for now)]

Clone from bug 1261235 comment 5, when building on Mac OS X, the following error occurs:

> 0:02.04 Traceback (most recent call last):
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/configure.py", line 94, in <module>
> 0:02.04     sys.exit(main(sys.argv))
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/configure.py", line 21, in main
> 0:02.04     sandbox = ConfigureSandbox(config, os.environ, argv)
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/__init__.py", line 143, in __init__
> 0:02.04     handler = ConfigureOutputHandler(stdout, stderr)
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/util.py", line 74, in __init__
> 0:02.04     self._stdout = fix_encoding(stdout)
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/util.py", line 69, in fix_encoding
> 0:02.04     return codecs.getwriter(locale.getpreferredencoding())(fh)
> 0:02.04   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 606, in getpreferredencoding
> 0:02.04     return getdefaultlocale()[1]
> 0:02.04   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 543, in getdefaultlocale
> 0:02.04     return _parse_localename(localename)
> 0:02.04   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 475, in _parse_localename
> 0:02.04     raise ValueError, 'unknown locale: %s' % localename
> 0:02.04 ValueError: unknown locale: UTF-8

The default locale on mac is as following
> $ locale
> LANG=
> LC_COLLATE="C"
> LC_CTYPE="UTF-8"
> LC_MESSAGES="C"
> LC_MONETARY="C"
> LC_NUMERIC="C"
> LC_TIME="C"
> LC_ALL=

I tried export LANG=en_US.UTF-8 / LC_ALL=en_US.UTF-8, it passes main configure but then another error occurs during configuring js/src:

> 0:06.29 js/src> Traceback (most recent call last):
> 0:06.29 js/src>   File "/Users/freesamael/Repos/gecko-dev/build/../configure.py", line 94, in <module>
> 0:06.29 js/src>     sys.exit(main(sys.argv))
> 0:06.29 js/src>   File "/Users/freesamael/Repos/gecko-dev/build/../configure.py", line 21, in main
> 0:06.30 js/src>     sandbox = ConfigureSandbox(config, os.environ, argv)
> 0:06.30 js/src>   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/__init__.py", line 143, in __init__
> 0:06.30 js/src>     handler = ConfigureOutputHandler(stdout, stderr)
> 0:06.30 js/src>   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/util.py", line 74, in __init__
> 0:06.30 js/src>     self._stdout = fix_encoding(stdout)
> 0:06.30 js/src>   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/util.py", line 69, in fix_encoding
> 0:06.30 js/src>     return codecs.getwriter(locale.getpreferredencoding())(fh)
> 0:06.30 js/src>   File "/Users/freesamael/Repos/gecko-build/obj-firefox/_virtualenv/lib/python2.7/codecs.py", line 1009, in getwriter
> 0:06.30 js/src>     return lookup(encoding).streamwriter
> 0:06.30 js/src> TypeError: must be string, not None

Comment 1

[Samael Wang [:freesamael] (away for now)]

Related bug of python:

http://grokbase.com/t/python/python-bugs-list/1376yd84nq/issue18378-locale-getdefaultlocale-fails-on-mac-os-x-with-default-language-set-to-english

Comment 2

[Mike Hommey [:glandium]]

What does `python -c 'import locale; print locale.CODESET'` say?

Comment 3

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #2)
> What does `python -c 'import locale; print locale.CODESET'` say?

Forget it, the first traceback is enough information. Looks like we'll have to monkeypatch locale._parse_localename, or handle the ValueError in the caller, falling back to manually read the environment...

That said, the second traceback is weird. Does `LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 python -c 'import locale; print locale.getpreferredencoding()'` print None?

Comment 4

[Mike Hommey [:glandium]]

Ah, autoconf does stuff with LANG/LC_ALL too, which would impact configure in js/src...

Comment 5

[Mike Hommey [:glandium]]

So, interestingly, my mac has the same locales as you do, but my python is fine with it... where does your python come from? and what version is it?

Comment 6

[Samael Wang [:freesamael] (away for now)]

Yup after some trials it seems the issue only happens with homebrew's python. 
The built-in version of OS X 10.11 looks OK. Not sure how older OS X versions work though.

Should I invalidate this bug?

> $ /usr/local/bin/python -V
> Python 2.7.11
> $ /usr/local/bin/python -c 'import locale; print locale.getpreferredencoding()'
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 606, in getpreferredencoding
>     return getdefaultlocale()[1]
>   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 543, in getdefaultlocale
>     return _parse_localename(localename)
>   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 475, in _parse_localename
>     raise ValueError, 'unknown locale: %s' % localename
> ValueError: unknown locale: UTF-8
> 
> $ /usr/bin/python -V
> Python 2.7.10
> $ /usr/bin/python -c 'import locale; print locale.getpreferredencoding()'
> UTF-8

Comment 7

[Mike Hommey [:glandium]]

Attachment: MozReview Request: Bug 1263296 - Work around getpreferredencoding inconsistencies. r?gps

Whether it uses locale._parse_localename or nl_langinfo makes it have completely
different results in weird and/or widespread locale settings (LC_ALL=UTF-8 or
LC_ALL=C).

Review commit: https://reviewboard.mozilla.org/r/45319/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/45319/

Comment 8

[Gregory Szorc [:gps]]

Comment on attachment 8739605 [details]
MozReview Request: Bug 1263296 - Work around getpreferredencoding inconsistencies. r?gps

https://reviewboard.mozilla.org/r/45319/#review42423

::: python/mozbuild/mozbuild/configure/util.py:85
(Diff revision 1)
> +                        encoding = 'utf-8'
> +
> +                # locale._parse_localename makes locale.getpreferredencoding
> +                # return None when LC_ALL is C, instead of e.g. 'US-ASCII' or
> +                # 'ANSI_X3.4-1968' when it uses nl_langinfo.
> +                if encoding:

`encoding` may be undefined here if `locale.getpreferredencoding()` raises and LC_ALL != 'UTF-8'.

Add a `encoding = None` above the `try` to fix.

Comment 9

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/9f469891dd2c

Comment 10

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/9f469891dd2c