962391 - "mach build" produces a UnicodeEncodeError

Status: NEW

(Firefox Build System :: General, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

Attachment: Python backtrace

Running a clobber "mach build" of m-c on a Linux box inside a screen session gives me an error that looks like the attached dump. The last line (which indicates the position and ordinal) may vary from one run to the next.

I modified mach_commands.py and stuck in a |print(repr(msg))| just above the self.fh.write call that borks, so as to get the actual msg that triggers the error. In this example, the message was:

u'\x1b[34m 1:13.57\x1b[m\x0f /home/kats/zspace/mozilla-git-2/media/mtransport/third_party/nrappkit/src/log/r_log.c: In function \ufffd\ufffd\ufffdr_log_register\ufffd\ufffd\ufffd:'

Presumably the compiler is injecting those \ufffd [1] replacement characters into its warning message (or whatever it's about to spit out) and mach/python doesn't deal with those nicely?

[1] http://www.fileformat.info/info/unicode/char/0fffd/index.htm

Comment 1

[Gregory Szorc [:gps]]

Boo. I thought we fixed this a long, long time ago. You sure this is on a recent changeset?

Comment 2

[Kartikaya Gupta (email:[e-mail])]

Yeah this was recent m-c.

Comment 3

[Henry Addo]

I think I'm hitting similar issue at the moment rendering me not to be able to make a build for Fennec.

Issuing `mach build` results in 

[hg:globe_with_mag-1079182]☿ mach build
 0:00.34 /usr/bin/gmake -f client.mk -s
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u263f' in position 10: ordinal not in range(128)

  File "~/mozilla-central/python/mozbuild/mozbuild/mach_commands.py", line 387, in build
    silent=not verbose)
  File "~/mozilla-central/python/mozbuild/mozbuild/base.py", line 552, in _run_make
    return fn(**params)
  File "~/mozilla-central/python/mozbuild/mozbuild/base.py", line 595, in _run_command_in_srcdir
    return self.run_process(cwd=self.topsrcdir, **args)
  File "~/mozilla-central/python/mach/mach/mixin/process.py", line 136, in run_process
    p.run()
  File "~/mozilla-central/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 727, in run
    self.proc = self.Process([self.cmd] + self.args, **args)
  File "~/mozilla-central/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 114, in __init__
    universal_newlines, startupinfo, creationflags)
  File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception

Comment 4

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Greg, do you know what's going on here?

Comment 5

[Henry Addo]

To add more info, 
I'm on Fedora 22 beta. This started happening after recent update to my system. Issuing 

wget -q https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py && python bootstrap.py

I selected option 2 to build Firefox for Android it ended up with this error.

Package autoconf213-2.13-34.fc22.noarch is already installed, skipping.
Package mercurial-3.3.3-1.fc22.x86_64 is already installed, skipping.
Dependencies resolved.
Nothing to do.
Complete!
Traceback (most recent call last):
  File "bootstrap.py", line 163, in <module>
    sys.exit(main(sys.argv))
  File "bootstrap.py", line 154, in main
    dasboot.bootstrap()
  File "/tmp/tmpVaien9/mozboot/bootstrap.py", line 134, in bootstrap
    'content. Like --repo, you should not need to set this.')
  File "/tmp/tmpVaien9/mozboot/base.py", line 125, in install_mobile_android_packages
NotImplementedError: Cannot bootstrap Firefox for Android: mozboot.base does not yet implement install_mobile_android_packages()

Comment 6

[Gregory Szorc [:gps]]

Comment #3 and comment #5 are separate bugs. Comment #5 should be filed as a separate bug and assigned to nalexander.

Just to confirm, what revision of mozilla-central is encountering this error? `hg log -r .`.

Comment 7

[Henry Addo]

(In reply to Gregory Szorc [:gps] from comment #6)
> Comment #3 and comment #5 are separate bugs. Comment #5 should be filed as a
> separate bug and assigned to nalexander.

Moved to a new Bug 1158443 

> 
> Just to confirm, what revision of mozilla-central is encountering this
> error? `hg log -r .`.

Status of `hg log -r .` is

changeset:   241031:d94e535e171c                                                       │                                                                                       
tag:         fx-team                                                                   │                                                                                       
user:        Jordan Santell <[e-mail]>                                       │                                                                                       
date:        Mon Apr 13 18:00:02 2015 -0700                                            │                                                                                       
summary:     Bug 1147945 - Let the profiler's buffer size and sample rate be configurab│                                                                                       
le via prefs. r=vp  

Not sure. Even fresh clone of central gives the error.

Comment 8

[Marcos Caceres [:marcosc]]

Same issue here - seem that ohmyzsh themes that use any unicode symbols or emojis will cause this problem.   

mach build
 0:00.36 /usr/bin/make -f client.mk MOZ_PARALLEL_BUILD=7 -s
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u25cf' in position 42: ordinal not in range(128)

  File "/Users/marcos/gecko/python/mozbuild/mozbuild/mach_commands.py", line 387, in build
    silent=not verbose)
  File "/Users/marcos/gecko/python/mozbuild/mozbuild/base.py", line 552, in _run_make
    return fn(**params)
  File "/Users/marcos/gecko/python/mozbuild/mozbuild/base.py", line 595, in _run_command_in_srcdir
    return self.run_process(cwd=self.topsrcdir, **args)
  File "/Users/marcos/gecko/python/mach/mach/mixin/process.py", line 136, in run_process
    p.run()
  File "/Users/marcos/gecko/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 727, in run
    self.proc = self.Process([self.cmd] + self.args, **args)
  File "/Users/marcos/gecko/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 114, in __init__
    universal_newlines, startupinfo, creationflags)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
Process Process-1:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    func(*targs, **kargs)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function
    p.join()
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 145, in join
    self.run()
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/marcos/gecko/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 56, in _collect
    res = self._popen.wait(timeout)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/forking.py", line 154, in wait
    while not pipe.poll(sleep_interval):
    return self.poll(0)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/forking.py", line 135, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function
    p.join()
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 145, in join
    res = self._popen.wait(timeout)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/forking.py", line 154, in wait
    return self.poll(0)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/forking.py", line 135, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt

Comment 9

[Gregory Szorc [:gps]]

STR are needed here.

How exactly is Unicode being introduced to the build environment via ohmyzsh? Through an environment variable? If so, which one?

Comment 10

[Marcos Caceres [:marcosc]]

(In reply to Gregory Szorc [:gps] from comment #9)
> STR are needed here.
> 
> How exactly is Unicode being introduced to the build environment via
> ohmyzsh? Through an environment variable? If so, which one?

Not sure, tbh :( I can't see much wrong with line 387 of mach_commands.py... but then again, I know next to nothing about mach or how that code works.

Comment 11

[Henry Addo]

I'm on zshell as well with custom .zshrc file. I'm not using ohmyzsh but I'm manipulating the right prompt which ohmyzsh does also. Commenting out this line [1] in my .zshrc file passes this error. I'm not exactly sure why though.

[1]: https://github.com/eyedol/tools/blob/master/zshrc#L112

Comment 12

[Gregory Szorc [:gps]]

My guess is your PS1 variable contains non-ascii.

But... so does mine. Somehow a Python unicode type is getting passed down into process execution land. subprocess.py is attempting to coerce that unicode into a str using the default conversion encoding of "ascii" and failing. So, the question is: where is the unicode type coming from.

Try this:

1) Open python/mozbuild/mozbuild/base.py
2) In _run_make() immediately before "return fn(**params)", insert a new line: "import pdb; pdb.set_trace()"
3) Run mach and hit the breakpoint
4) Type "pp params"
5) Find string values that have the |u'| prefix on them. That denotes a Unicode type and will likely point to our culprit.

Comment 13

[Marcos Caceres [:marcosc]]

So, I think you are right... this is what I get for the following:

echo $PS1:
%{%f%b%k%}$(build_left_prompt)

And then... note the "", which I can't copy/paste because iTerm is using a custom font (souce code powerline)... but there are at least 4 of them: 
echo $(build_left_prompt)
%{%K{black}%}%{%F{011}%} %(!.%{%F{yellow}%}.)marcos@%m%{%K{blue}%F{black}%}%{%F{black}%} %~%{%K{yellow}%F{blue}%}%{%F{black}%} %F{black}%f %F{black}%F{black}splashscreen%f %F{black}●%f %F{black}⍟ 9%f%f %{%k%F{yellow}%}%{%f%}

Comment 14

[Marcos Caceres [:marcosc]]

Attachment: Fancy shell is fancy.

Added screenshot of theme "powerlevel9k". Repo: https://github.com/bhilburn/powerlevel9k

Comment 15

[Henry Addo]

Yup!.

This is what I have with the breakpoint.

-> return fn(**params)
(Pdb) pp params
{u'append_env': {'MACH': u'1'},
 u'args': [u'/usr/bin/gmake', u'-f', u'client.mk', u'-s'],
 u'ensure_exit_code': False,
 u'explicit_env': None,
 u'ignore_children': True,
 u'line_handler': <bound method BuildOutputManager.on_line of <mach.commands.05100728ed2911e4b3248c705a82b138.BuildOutputManager object at 0x7fcc5e6980d0>>,
 u'log_level': 20,
 u'pass_thru': False,
 u'require_unix_environment': False}

Comment 16

[Henry Addo]

Mine I supposed is this ☿; My echo $PS1 [%{%}hg%{%}:!^%{%}]☿

Comment 17

[Yurii Kolesnykov]

I can confirm this on my Arch Linux box.

Comment 18

[Frédéric Wang (:fredw)]

Attachment: Workaround patch

So IIUC, before calling a subprocess we convert all the environment key/value from "bytes" data to system / unicode strings. This is possibly submitting non-ASCII strings.

This patch removes the decoding and seems to workaround the problem for me. However, it's possible that environment values will then not be passed correctly to subprocess, so I'm not sure this patch is quite correct.

BTW in my case, the culprit was CHANGE_LOG_NAME = "Frédéric Wang".

Comment 19

[Frédéric Wang (:fredw)]

(In reply to Frédéric Wang (:fredw) from comment #18)
> BTW in my case, the culprit was CHANGE_LOG_NAME = "Frédéric Wang".

(...used for WebKit contribution https://www.webkit.org/coding/contributing.html)

Comment 20

[a1984993]

(In reply to Frédéric Wang (:fredw) from comment #18)

I had this problem on openSUSE and the workaround patch helped. thanks!

Comment 21

[Mike Hoye [:mhoye]]

Just had a new contributor report this. STR is to set their PS1 prompt to the Euro symbol instead of the dollar sign.

Comment 22

[Andrew Halberstadt [:ahal]]

The comments from 11 months ago are a regression from bug 1050561, and fixing it is tracked by bug 1152428. Unfortunately the fix breaks comm-central builds apparently.

I won't dupe this bug, as I'm unclear whether the initial problem filed 2 years ago is still relevant or not.