Build fail on Mac OS X: "ValueError: unknown locale: UTF-8"

RESOLVED FIXED in Firefox 48

Status

defect
RESOLVED FIXED
3 years ago
a year ago

People

(Reporter: freesamael, Unassigned)

Tracking

unspecified
mozilla48
Dependency tree / graph

Firefox Tracking Flags

(firefox48 fixed)

Details

Attachments

(1 attachment)

Clone from bug 1261235 comment 5, when building on Mac OS X, the following error occurs:

> 0:02.04 Traceback (most recent call last):
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/configure.py", line 94, in <module>
> 0:02.04     sys.exit(main(sys.argv))
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/configure.py", line 21, in main
> 0:02.04     sandbox = ConfigureSandbox(config, os.environ, argv)
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/__init__.py", line 143, in __init__
> 0:02.04     handler = ConfigureOutputHandler(stdout, stderr)
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/util.py", line 74, in __init__
> 0:02.04     self._stdout = fix_encoding(stdout)
> 0:02.04   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/util.py", line 69, in fix_encoding
> 0:02.04     return codecs.getwriter(locale.getpreferredencoding())(fh)
> 0:02.04   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 606, in getpreferredencoding
> 0:02.04     return getdefaultlocale()[1]
> 0:02.04   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 543, in getdefaultlocale
> 0:02.04     return _parse_localename(localename)
> 0:02.04   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 475, in _parse_localename
> 0:02.04     raise ValueError, 'unknown locale: %s' % localename
> 0:02.04 ValueError: unknown locale: UTF-8

The default locale on mac is as following
> $ locale
> LANG=
> LC_COLLATE="C"
> LC_CTYPE="UTF-8"
> LC_MESSAGES="C"
> LC_MONETARY="C"
> LC_NUMERIC="C"
> LC_TIME="C"
> LC_ALL=

I tried export LANG=en_US.UTF-8 / LC_ALL=en_US.UTF-8, it passes main configure but then another error occurs during configuring js/src:

> 0:06.29 js/src> Traceback (most recent call last):
> 0:06.29 js/src>   File "/Users/freesamael/Repos/gecko-dev/build/../configure.py", line 94, in <module>
> 0:06.29 js/src>     sys.exit(main(sys.argv))
> 0:06.29 js/src>   File "/Users/freesamael/Repos/gecko-dev/build/../configure.py", line 21, in main
> 0:06.30 js/src>     sandbox = ConfigureSandbox(config, os.environ, argv)
> 0:06.30 js/src>   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/__init__.py", line 143, in __init__
> 0:06.30 js/src>     handler = ConfigureOutputHandler(stdout, stderr)
> 0:06.30 js/src>   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/util.py", line 74, in __init__
> 0:06.30 js/src>     self._stdout = fix_encoding(stdout)
> 0:06.30 js/src>   File "/Users/freesamael/Repos/gecko-dev/python/mozbuild/mozbuild/configure/util.py", line 69, in fix_encoding
> 0:06.30 js/src>     return codecs.getwriter(locale.getpreferredencoding())(fh)
> 0:06.30 js/src>   File "/Users/freesamael/Repos/gecko-build/obj-firefox/_virtualenv/lib/python2.7/codecs.py", line 1009, in getwriter
> 0:06.30 js/src>     return lookup(encoding).streamwriter
> 0:06.30 js/src> TypeError: must be string, not None
Summary: Build fail on Mac OS X → Build fail on Mac OS X: "ValueError: unknown locale: UTF-8"
What does `python -c 'import locale; print locale.CODESET'` say?
(In reply to Mike Hommey [:glandium] from comment #2)
> What does `python -c 'import locale; print locale.CODESET'` say?

Forget it, the first traceback is enough information. Looks like we'll have to monkeypatch locale._parse_localename, or handle the ValueError in the caller, falling back to manually read the environment...

That said, the second traceback is weird. Does `LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 python -c 'import locale; print locale.getpreferredencoding()'` print None?
Ah, autoconf does stuff with LANG/LC_ALL too, which would impact configure in js/src...
So, interestingly, my mac has the same locales as you do, but my python is fine with it... where does your python come from? and what version is it?
Yup after some trials it seems the issue only happens with homebrew's python. 
The built-in version of OS X 10.11 looks OK. Not sure how older OS X versions work though.

Should I invalidate this bug?

> $ /usr/local/bin/python -V
> Python 2.7.11
> $ /usr/local/bin/python -c 'import locale; print locale.getpreferredencoding()'
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 606, in getpreferredencoding
>     return getdefaultlocale()[1]
>   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 543, in getdefaultlocale
>     return _parse_localename(localename)
>   File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 475, in _parse_localename
>     raise ValueError, 'unknown locale: %s' % localename
> ValueError: unknown locale: UTF-8
> 
> $ /usr/bin/python -V
> Python 2.7.10
> $ /usr/bin/python -c 'import locale; print locale.getpreferredencoding()'
> UTF-8
Whether it uses locale._parse_localename or nl_langinfo makes it have completely
different results in weird and/or widespread locale settings (LC_ALL=UTF-8 or
LC_ALL=C).

Review commit: https://reviewboard.mozilla.org/r/45319/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/45319/
Attachment #8739605 - Flags: review?(gps)
Attachment #8739605 - Flags: review?(gps) → review+
Comment on attachment 8739605 [details]
MozReview Request: Bug 1263296 - Work around getpreferredencoding inconsistencies. r?gps

https://reviewboard.mozilla.org/r/45319/#review42423

::: python/mozbuild/mozbuild/configure/util.py:85
(Diff revision 1)
> +                        encoding = 'utf-8'
> +
> +                # locale._parse_localename makes locale.getpreferredencoding
> +                # return None when LC_ALL is C, instead of e.g. 'US-ASCII' or
> +                # 'ANSI_X3.4-1968' when it uses nl_langinfo.
> +                if encoding:

`encoding` may be undefined here if `locale.getpreferredencoding()` raises and LC_ALL != 'UTF-8'.

Add a `encoding = None` above the `try` to fix.

Comment 10

3 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/9f469891dd2c
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48

Updated

a year ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.