1576155 - mach try is busted

Status: RESOLVED WONTFIX

(Developer Infrastructure :: Try, defect, P1)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Hi. Which trying to do a bisection on ubuntu 18 I encountered the following error. This happens on inbound, central, autoland, beta, as far as I tried.

alexandru.ionescu@P6201:~/workspace/mozilla-central$ ./mach try empty
Error running mach:

['try', 'empty']

The error occurred in mach itself. This is likely a bug in mach itself or a
fundamental problem with a loaded module.

You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ImportError: cannot import name import_sibling_modules

File "/home/alexandru.ionescu/workspace/mozilla-central/python/mach/mach/main.py", line 359, in run
return self._run(argv)
File "/home/alexandru.ionescu/workspace/mozilla-central/python/mach/mach/main.py", line 416, in _run
args = parser.parse_args(argv)
File "/usr/lib/python2.7/argparse.py", line 1705, in parse_args
args, argv = self.parse_known_args(args, namespace)
File "/usr/lib/python2.7/argparse.py", line 1737, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/usr/lib/python2.7/argparse.py", line 1946, in _parse_known_args
stop_index = consume_positionals(start_index)
File "/usr/lib/python2.7/argparse.py", line 1902, in consume_positionals
take_action(action, args)
File "/usr/lib/python2.7/argparse.py", line 1811, in take_action
action(self, namespace, argument_values, option_string)
File "/home/alexandru.ionescu/workspace/mozilla-central/python/mach/mach/dispatcher.py", line 173, in call
if handler.parser:
File "/home/alexandru.ionescu/workspace/mozilla-central/python/mach/mach/decorators.py", line 82, in parser
self._parser = self._parser()
File "/home/alexandru.ionescu/workspace/mozilla-central/tools/tryselect/mach_commands.py", line 37, in call
mod = importlib.import_module('tryselect.selectors.{}'.format(self.selector))
File "/usr/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "/home/alexandru.ionescu/workspace/mozilla-central/build/mach_bootstrap.py", line 403, in call
module = self._original_import(name, globals, locals, fromlist, level)
File "/home/alexandru.ionescu/workspace/mozilla-central/tools/tryselect/selectors/empty.py", line 7, in <module>
from ..cli import BaseTryParser
File "/home/alexandru.ionescu/workspace/mozilla-central/build/mach_bootstrap.py", line 403, in call
module = self._original_import(name, globals, locals, fromlist, level)
File "/home/alexandru.ionescu/workspace/mozilla-central/tools/tryselect/cli.py", line 12, in <module>
from .templates import all_templates
File "/home/alexandru.ionescu/workspace/mozilla-central/build/mach_bootstrap.py", line 403, in call
module = self._original_import(name, globals, locals, fromlist, level)
File "/home/alexandru.ionescu/workspace/mozilla-central/tools/tryselect/templates.py", line 20, in <module>
from .tasks import resolve_tests_by_suite
File "/home/alexandru.ionescu/workspace/mozilla-central/build/mach_bootstrap.py", line 403, in call
module = self._original_import(name, globals, locals, fromlist, level)
File "/home/alexandru.ionescu/workspace/mozilla-central/tools/tryselect/tasks.py", line 21, in <module>
from taskgraph.generator import TaskGraphGenerator
File "/home/alexandru.ionescu/workspace/mozilla-central/build/mach_bootstrap.py", line 403, in call
module = self._original_import(name, globals, locals, fromlist, level)
File "/home/alexandru.ionescu/workspace/mozilla-central/taskcluster/taskgraph/generator.py", line 15, in <module>
from .optimize import optimize_task_graph
File "/home/alexandru.ionescu/workspace/mozilla-central/build/mach_bootstrap.py", line 403, in call
module = self._original_import(name, globals, locals, fromlist, level)
File "/home/alexandru.ionescu/workspace/mozilla-central/taskcluster/taskgraph/optimize/init.py", line 24, in <module>

Comment 1

[Andrew Halberstadt [:ahal]]

This is happening from .pyc files being left lying around across a file rename. You can work around the issue by running ./mach clobber python after updating. Unfortunately there's not much we can do about it. There were previous bugs filed with this error that I wanted to dupe to, but I can't find them anymore.

Comment 2

[Alexandru Ionescu (needinfo me) [:alexandrui]]

I did that... it worked a push but after hg update to some history revisions I got the same issue.

Comment 3

[Andrew Halberstadt [:ahal]]

Yes, unfortunately you will hit this error every time you run ./mach try on a commit after https://hg.mozilla.org/mozilla-central/rev/856b78de0b75 , and then update to a commit before that and run ./mach try a second time. So if your bisection range has you bouncing around before/after that commit many times, you'll need to run ./mach clobber python again and again.

Comment 4

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Thank you, ahal. It is good to know that there is a way for doing it.

Comment 5

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Can I ask what particulary this ./mach clobber python does?

Comment 6

[Andrew Halberstadt [:ahal]]

It removes all the .pyc files left lying around in the tree. In this instance, you hit the problem because taskcluster/taskgraph/optimize.py was renamed to taskcluster/taskgraph/optimize/__init__.py. So when generating the taskgraph on a new revision and then updating to an older one, you'll have both:
taskcluster/taskgraph/optimize.py
taskcluster/taskgraph/optimize/__init__.pyc

in your tree. Guess which one Python uses? In hindsight I should have crafted my commits more carefully to prevent this from happening.. but it wasn't something I thought about at the time.

Comment 7

[Tom Prince [:tomprince]]

If you add PYTHONDONTWRITEBYTECODE=1 to your environment (i.e. export PYTHONDONTWRITEBYTECODE=1), then .pyc files won't be generated.

Comment 8

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Tom Prince [:tomprince] from comment #7)

If you add PYTHONDONTWRITEBYTECODE=1 to your environment (i.e. export PYTHONDONTWRITEBYTECODE=1), then .pyc files won't be generated.

But I assume this has a drawback... is it worth?

Comment 9

[Tom Prince [:tomprince]]

I've had this variable set in my environment forever, and haven't ever run into issues. There is probably a small cost to compiling python files to bytecode, but I suspect that actually running the code typically vastly outweights the cost of doing that.