Closed Bug 1577603 Opened 5 months ago Closed 5 months ago

mach - enable python3 for clobber

Categories

(Firefox Build System :: General, task)

task
Not set

Tracking

(firefox70 fixed)

RESOLVED FIXED
mozilla70
Tracking Status
firefox70 --- fixed

People

(Reporter: egao, Assigned: egao)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

Out of the py2 'blacklist' of mach commands, clobber is one of them, and a prime target to exercise the process of module migration.

By default, when ./mach clobber is run within a python3.5 virtualenv and clobber command removed from the py2 blacklist:

(python35_development) egao@egao-44538:~/src/mozilla-central$ ./mach clobber
Error running mach:

    ['clobber']

The error occurred in mach itself. This is likely a bug in mach itself or a
fundamental problem with a loaded module.

You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

SystemError: <built-in method __getitem__ of ReadOnlyDict object at 0x105f66458> returned a result with an error set

  File "/Users/egao/src/mozilla-central/python/mach/mach/main.py", line 360, in run
    return self._run(argv)
  File "/Users/egao/src/mozilla-central/python/mach/mach/main.py", line 404, in _run
    Registrar.register_conditional_names(context)
  File "/Users/egao/src/mozilla-central/python/mach/mach/registrar.py", line 63, in register_conditional_names
    instance = MachRegistrar._instance(handler, context)
  File "/Users/egao/src/mozilla-central/python/mach/mach/registrar.py", line 101, in _instance
    instance = cls(context)
  File "/Users/egao/src/mozilla-central/python/mozbuild/mozbuild/base.py", line 895, in __init__
    self.mozconfig
  File "/Users/egao/src/mozilla-central/python/mozbuild/mozbuild/base.py", line 324, in mozconfig
    return self.mozconfig_and_target[0]
  File "/Users/egao/src/mozilla-central/python/mozbuild/mozbuild/base.py", line 316, in mozconfig_and_target
    self.topsrcdir, self._mozconfig, os.environ.get('MOZCONFIG'))
  File "/Users/egao/src/mozilla-central/python/mozbuild/mozbuild/util.py", line 954, in __call__
    self[args] = self.func(*args)
  File "/Users/egao/src/mozilla-central/python/mozbuild/mozbuild/base.py", line 302, in get_mozconfig_and_target
    sandbox.include_file(os.path.join(base_dir, 'init.configure'))
  File "/Users/egao/src/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 436, in include_file
    exec_(code, self)
  File "/Users/egao/src/mozilla-central/python/mozbuild/mozbuild/util.py", line 55, in exec_
    exec(object, globals, locals)
  File "/Users/egao/src/mozilla-central/build/moz.configure/init.configure", line 7, in <module>
    include('util.configure')
  File "/Users/egao/src/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 762, in include_impl
    self.include_file(what)
  File "/Users/egao/src/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 436, in include_file
    exec_(code, self)
  File "/Users/egao/src/mozilla-central/python/mozbuild/mozbuild/util.py", line 55, in exec_
    exec(object, globals, locals)
  File "/Users/egao/src/mozilla-central/build/moz.configure/util.configure", line 414, in <module>
    always = dependable(True)
  File "/Users/egao/src/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 804, in wrapper
    ret = template(*args, **kwargs)
  File "/Users/egao/src/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 1149, in wrapped
    return new_func(*args, **kwargs)
  File "/Users/egao/src/mozilla-central/build/moz.configure/util.configure", line 407, in dependable
    if isinstance(obj, SandboxDependsFunction):
(python35_development) egao@egao-44538:~/src/mozilla-central$

:ahal - the last portion of the stack trace File "/Users/egao/src/mozilla-central/build/moz.configure/util.configure", line 407, in dependable if isinstance(obj, SandboxDependsFunction): seems to happen for every command I try in my python3.5 environment with mozilla-central revision 8edbf8fe48bf.

Interestingly it doesn't happen with my python3.7 environment.

My understanding is that we are to use python3.5 since that's what the build system uses.

Flags: needinfo?(ahal)

I can't seem to reproduce. If I run:
python3.5 mach clobber

I get:

TypeError: Can't mix strings and bytes in path components

  File "/home/ahal/dev/mozilla-central/python/mozbuild/mozbuild/mach_commands.py", line 210, in clobber
    Clobberer(self.topsrcdir, self.topobjdir, self.substs).remove_objdir(full)
  File "/home/ahal/dev/mozilla-central/python/mozbuild/mozbuild/controller/clobber.py", line 163, in remove_objdir
    self.delete_dirs(self.topobjdir, paths)
  File "/home/ahal/dev/mozilla-central/python/mozbuild/mozbuild/controller/clobber.py", line 122, in delete_dirs
    path = os.path.join(root, p)
  File "/home/ahal/.pyenv/versions/3.5.7/lib/python3.5/posixpath.py", line 89, in join
    genericpath._check_arg_types('join', a, *p)
  File "/home/ahal/.pyenv/versions/3.5.7/lib/python3.5/genericpath.py", line 145, in _check_arg_types
    raise TypeError("Can't mix strings and bytes in path components") from None

which looks like a normal Python 3 incompatibility to me. For things that the build doesn't depend on, we can target 3.6.. but since clobber is related to the build workflow, yes we should try and keep it at 3.5.

Flags: needinfo?(ahal)

Issue was due to the use of Python 3.5.0 that was either installed on macOS mojave, or installed via homebrew.

Once upgraded to 3.5.7 using pyenv install 3.5.7 and regenerating the virtualenv using the new binary, the cryptic error is replaced with the one :ahal saw in comment 3.

Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8ebdab6f0a35
convert mach clobber to run with python 3.5 r=nalexander
Status: NEW → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla70
Assignee: nobody → egao
Regressions: 1578198
You need to log in before you can comment on or make changes to this bug.