1629159 - running mach with python 3.7.3: TypeError: a bytes-like object is required, not 'str'

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Kai Engert (:KaiE:)]

When building m-c (as part of a thunderbird build), on a debian buster system, which uses python 3.7.3, I get the following build error, actually, configure error. And not use one error, I get it many times, it seems like every single action inside mach triggers it. I'll paste the first two.

 0:00.42 /usr/bin/make -f client.mk -s configure
 0:00.43 cd /home/user/moz/commcent/obj-thunder-opt
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.7/logging/__init__.py", line 1037, in emit
    stream.write(msg + self.terminator)
TypeError: a bytes-like object is required, not 'str'
Call stack:
  File "../mach", line 258, in <module>
    main(sys.argv[1:])
  File "../mach", line 254, in main
    sys.exit(mach.run(args))
  File "/home/user/moz/commcent/mozilla/python/mach/mach/main.py", line 360, in run
    return self._run(argv)
  File "/home/user/moz/commcent/mozilla/python/mach/mach/main.py", line 464, in _run
    **vars(args.command_args))
  File "/home/user/moz/commcent/mozilla/python/mach/mach/registrar.py", line 109, in _run_command_handler
    result = fn(**kwargs)
  File "/home/user/moz/commcent/mozilla/python/mozbuild/mozbuild/build_commands.py", line 138, in build
    append_env=append_env)
  File "/home/user/moz/commcent/mozilla/python/mozbuild/mozbuild/controller/building.py", line 1079, in build
    append_env=append_env)
  File "/home/user/moz/commcent/mozilla/python/mozbuild/mozbuild/controller/building.py", line 1378, in configure
    append_env=append_env)
  File "/home/user/moz/commcent/mozilla/python/mozbuild/mozbuild/controller/building.py", line 1460, in _run_client_mk
    if self._check_clobber(mozconfig, os.environ):
  File "/home/user/moz/commcent/mozilla/python/mozbuild/mozbuild/controller/building.py", line 1551, in _check_clobber
    {'msg': line.rstrip()}, '{msg}')
  File "/home/user/moz/commcent/mozilla/python/mach/mach/mixin/logging.py", line 54, in log
    extra={'action': action, 'params': params})
Message: '{msg}'
Arguments: ()
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.7/logging/__init__.py", line 1037, in emit
    stream.write(msg + self.terminator)
TypeError: a bytes-like object is required, not 'str'
Call stack:
  File "../mach", line 258, in <module>
    main(sys.argv[1:])
  File "../mach", line 254, in main
    sys.exit(mach.run(args))
  File "/home/user/moz/commcent/mozilla/python/mach/mach/main.py", line 360, in run
    return self._run(argv)
  File "/home/user/moz/commcent/mozilla/python/mach/mach/main.py", line 464, in _run
    **vars(args.command_args))
  File "/home/user/moz/commcent/mozilla/python/mach/mach/registrar.py", line 109, in _run_command_handler
    result = fn(**kwargs)
  File "/home/user/moz/commcent/mozilla/python/mozbuild/mozbuild/build_commands.py", line 138, in build
    append_env=append_env)
  File "/home/user/moz/commcent/mozilla/python/mozbuild/mozbuild/controller/building.py", line 1079, in build
    append_env=append_env)
  File "/home/user/moz/commcent/mozilla/python/mozbuild/mozbuild/controller/building.py", line 1378, in configure
    append_env=append_env)
  File "/home/user/moz/commcent/mozilla/python/mozbuild/mozbuild/controller/building.py", line 1514, in _run_client_mk
    }, 'Adding make options from {path}\n    {content}')
  File "/home/user/moz/commcent/mozilla/python/mach/mach/mixin/logging.py", line 54, in log
    extra={'action': action, 'params': params})
Message: 'Adding make options from {path}\n    {content}'
Arguments: ()
--- Logging error ---

I backed out https://hg.mozilla.org/mozilla-central/rev/351492756d098c92864edd686c6feea68ef1a6e8 from bug 1617748 which allows my build to proceed.

Comment 1

[Ricky Stewart]

Attachment: patch

Comment 2

[Ricky Stewart]

Hmm, never seen anything like this before. Evidently your stdout/stderr is a binary stream and doesn't want text output, which is highly unusual. That tells me that either 1) your system is configured in a very unusual way unlike any system I've ever heard of, or 2) you've stumbled across some road-less-traveled codepath I'm currently unaware of where stdout/stderr gets monkey-patched in a weird, broken way. (2) is more likely IMO, although I'm not enough of a Python expert to say confidently that it's not (1). :)

Since I can't reproduce I'll need slightly more information. Can you apply this patch locally (un-backout that revision, please, at least for this test) and tell me what the output is? Any additional information you can provide would be helpful. In particular if you could copy-paste your env that would be helpful as well, because that would make it way easier for me to tell how Python is interpreting your locale.

Comment 3

[Ricky Stewart]

I'm also interested in seeing if something like revision D70429 would fix your issue. That is, I don't think this patch will fix the problem as-is, but understanding what's wrong in your case can let us augment this patch to be more complete and correct.

Comment 4

[Kai Engert (:KaiE:)]

I'm using my own wrapper script around the actual build command, which measures build time and saves a build log, and automatically sets the MOZCONFIG environment variable.

This is how I execute mach:

/usr/bin/time --output=time.log --append nice -10 ../mach --log-file build.log $@

Comment 5

[Kai Engert (:KaiE:)]

../mach because I do cd mozilla/comm for building thunderbird

Comment 6

[Kai Engert (:KaiE:)]

(In reply to Ricky Stewart from comment #2)

Since I can't reproduce I'll need slightly more information. Can you apply this patch locally (un-backout that revision, please, at least for this test) and tell me what the output is?

sys.stdout = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
sys.getdefaultencoding() = utf-8
sys.stdout.encoding = UTF-8
dir(sys.stdout.encoding) = ['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'capitalize', 'casefold', 'center', 'count', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'format_map', 'index', 'isalnum', 'isalpha', 'isascii', 'isdecimal', 'isdigit', 'isidentifier', 'islower', 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']

Comment 7

[Magnus Melin [:mkmelin]]

FWIW, I don't get this on my machine with Python 3.7.5.
Did you try just ./mach from the mozilla directory? There have been bugs at least in the past that running it from the "wrong" directory doesn't work properly for all cases.

Comment 8

[Kai Engert (:KaiE:)]

I've experimented more, and identified the cause: It's the --log-file parameter for mach!

works: ../mach -v build

fails: ../mach --log-file /tmp/build.log -v build

Comment 9

[Ricky Stewart]

Attachment: Bug 1629159 - Open the mach log-file with the correct mode.

Comment 10

[Ricky Stewart]

Genius! Thanks for the repro :) Sending the fix out for review now.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6eb3134baecc
Open the mach log-file with the correct mode. r=dmajor

Comment 12

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/6eb3134baecc