1298565 - configure fails if preprocessor emits non-ASCII characters

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Akihiko Odaki]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0
Build ID: 20160823070410

Steps to reproduce:

Run configure with preprocessors which emits non-ASCII characters (GCC with LANG=ja_JP.utf-8)


Actual results:

The configuration fails showing the following log:
 0:03.04 DEBUG: <truncated - see config.log for full output>
 0:03.04 DEBUG: | %KERNEL DragonFly
 0:03.04 DEBUG: | #elif __OpenBSD__
 0:03.04 DEBUG: | %KERNEL OpenBSD
 0:03.04 DEBUG: | #elif __FreeBSD__
 0:03.04 DEBUG: | %KERNEL FreeBSD
 0:03.04 DEBUG: | #elif __linux__
 0:03.04 DEBUG: | %KERNEL Linux
 0:03.04 DEBUG: | #elif _WIN32 || __CYGWIN__
 0:03.04 DEBUG: | %KERNEL WINNT
 0:03.04 DEBUG: | #elif __NetBSD__
 0:03.04 DEBUG: | %KERNEL NetBSD
 0:03.04 DEBUG: | #elif __APPLE__
 0:03.04 DEBUG: | %KERNEL Darwin
 0:03.05 DEBUG: | #endif
 0:03.05 DEBUG: | #if _MSC_VER || __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
 0:03.05 DEBUG: | %ENDIANNESS little
 0:03.05 DEBUG: | #elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
 0:03.05 DEBUG: | %ENDIANNESS big
 0:03.05 DEBUG: | #endif
 0:03.05 DEBUG: Executing: `/usr/bin/gcc -E /tmp/conftest.lQ9Dn6.c`
 0:03.05 Traceback (most recent call last):
 0:03.05   File "/home/root3/mozilla-central/configure.py", line 107, in <module>
 0:03.05     sys.exit(main(sys.argv))
 0:03.05   File "/home/root3/mozilla-central/configure.py", line 26, in main
 0:03.05     sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
 0:03.05   File "/home/root3/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 260, in run
 0:03.05     func(*args)
 0:03.06   File "/home/root3/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 302, in _value_for
 0:03.06     return self._value_for_depends(obj)
 0:03.06   File "/home/root3/mozilla-central/python/mozbuild/mozbuild/util.py", line 924, in method_call
 0:03.06     cache[args] = self.func(instance, *args)
 0:03.06   File "/home/root3/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 329, in _value_for_depends
 0:03.06     return func(*resolved_args)
 0:03.06   File "/home/root3/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 773, in wrapped
 0:03.06     return new_func(*args, **kwargs)
 0:03.06   File "/home/root3/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 518, in wrapper
 0:03.06     ret = template(*args, **kwargs)
 0:03.06   File "/home/root3/mozilla-central/build/moz.configure/checks.configure", line 53, in wrapped
 0:03.06     ret = func(*args, **kwargs)
 0:03.06   File "/home/root3/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 773, in wrapped
 0:03.06     return new_func(*args, **kwargs)
 0:03.06   File "/home/root3/mozilla-central/build/moz.configure/toolchain.configure", line 640, in valid_compiler
 0:03.06     host_or_target)
 0:03.06   File "/home/root3/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 773, in wrapped
 0:03.06     return new_func(*args, **kwargs)
 0:03.06   File "/home/root3/mozilla-central/build/moz.configure/toolchain.configure", line 319, in check_compiler
 0:03.07     info = get_compiler_info(compiler, language)
 0:03.07   File "/home/root3/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 773, in wrapped
 0:03.07     return new_func(*args, **kwargs)
 0:03.07   File "/home/root3/mozilla-central/build/moz.configure/toolchain.configure", line 279, in get_compiler_info
 0:03.07     if line.startswith('%'):
0:03.07 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe7 in position 6: ordinal not in range(128)


Expected results:

The configuration succeeds.

Comment 1

[Akihiko Odaki]

Attachment: Bug 1298565 - Do not decode preprocessor output as ASCII

Review commit: https://reviewboard.mozilla.org/r/74692/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/74692/

Comment 2

[Akihiko Odaki]

Bug 1298565 - Do not decode preprocessor output as ASCII r?glandium | Review Request | Review Board
https://reviewboard.mozilla.org/r/74692/

This change fixes this issue by regarding the output as bytearray.

Comment 3

[Mike Hommey [:glandium]]

Comment on attachment 8785523 [details]
Bug 1298565 - Do not decode preprocessor output as ASCII

https://reviewboard.mozilla.org/r/74692/#review73076

::: build/moz.configure/toolchain.configure:273
(Diff revision 1)
>          #endif
>      ''')
>  
>      result = try_preprocess(compiler, language, check)
>  
> +    log.debug(result)

Please don't add this debug, that's meant to be output by the loop you're modifying.

::: build/moz.configure/toolchain.configure:279
(Diff revision 1)
> +
>      if not result:
>          raise FatalCheckError(
>              'Unknown compiler or compiler not supported.')
>  
> +    # Metadata emitted by preprocessors such as GCC with LANG=ja_JP.utf-8 may

Fun fact: that's also my locale, and GCC doesn't emit non ascii characters... aha! it does if I install gcc-6-locales.

::: build/moz.configure/toolchain.configure:282
(Diff revision 1)
> -    for line in result.splitlines():
> -        if line.startswith('%'):
> -            k, _, v = line.partition(' ')
> -            k = k.lstrip('%')
> -            data[k] = v.replace(' ', '')
> +    for line in result.split(b'\n'):
> +        if line.startswith(b'%'):
> +            k, _, v = line.partition(b' ')
> +            k = k.lstrip(b'%')
> +            data[k] = v.replace(b' ', b'')

The whole thing makes absolutely no sense. result is a proper unicode string, with the japanese output from gcc properly handled. Once splitlines is called, each line is also a proper unicode string (as per log.warning("%r", line)). Then line.startswith('%') fails for whatever reason, and it makes no sense. Interestingly, changing to line.starswith(b'%') works. None of the other changes are required. And it still doesn't make sense. Especially when doing u'# 1 "<\u7d44\u307f\u8fbc\u307f>"'.startswith(u'%') (which is what this is all supposed to be doing) works in a python shell.

I was starting to think it's the sandbox, but even unlocking __builtins__ doesn't solve the problem.

Comment 4

[Mike Hommey [:glandium]]

Ahhh log.* functions are playing tricks autoconverting str arguments to unicode. At least that makes sense now.

I'd say to just change the line.startswith. The others shouldn't be necessary.

Comment 5

[Akihiko Odaki]

Comment on attachment 8785523 [details]
Bug 1298565 - Do not decode preprocessor output as ASCII

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/74692/diff/1-2/

Comment 6

[Mike Hommey [:glandium]]

Comment on attachment 8785523 [details]
Bug 1298565 - Do not decode preprocessor output as ASCII

https://reviewboard.mozilla.org/r/74692/#review73452

Comment 7

[Hiroyuki Ikezoe (:hiro)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8a03b6b1552a
Pushed to a try.

Hello Akihiko, could you please "fix" two open issues (just push "Fix" button or some sort of buttons) on the review so that I can land your patch on the review board.  It seems that the buttons is invisible/unable to other people.

https://reviewboard.mozilla.org/r/74692/#comment94768

Comment 8

[Akihiko Odaki]

I meant to change the state on the review board. I pushed those buttons accordingly.

Comment 9

[Hiroyuki Ikezoe (:hiro)]

Great! Thanks!

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6fe90a668d04
Do not decode preprocessor output as ASCII r=glandium

Comment 11

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/6fe90a668d04