1198153 - Moving python files can lead to stray .pyc files that break other python tools (e.g. test harnesses)

Status: NEW

(Firefox Build System :: Mach Core, enhancement)

Description

[James Graham [:jgraham]]

Occasionally moving files around in the source tree will leave an old .pyc file in a location further up the import path list than the new .py file. This means that code expecting to import the new module may get the old version instead, leading to difficult to understand and reproduce bugs.

We should have some mechanism to clobber the .pyc files after a change that moves them around, and a hg hook to enforce that the relevant action is taken when a change is committed to inbound that actually moves a python file.

I imagine that such a clobber mechanism might have to live in mach.

(this is the non-build-specific version of bug 685957).

Comment 1

[Mike Hommey [:glandium]]

If you use the python virtualenv, bug 1188224 should have fixed this.

Comment 2

[James Graham [:jgraham]]

So you need to do self.activate_virtualenv() in the command? Can we make that declarative and possibly opt-out i.e.

@Command("foo", virtualenv=False)
def foo():
    pass

Comment 3

[Andrew Halberstadt [:ahal]]

Also bug 1184696 implemented the clobber mechanism already:

    $ mach clobber python

I guess if you wanted to leave this open for implementing a hook, you could.

Comment 4

[Gregory Szorc [:gps]]

The custom import hook from bug 1188224 is installed when running `mach` or any Python command that executes in the build system's virtualenv. That means .pyc files with unattached .py files should be ignored in most circumstances. I'm kinda surprised you are still seeing this after bug 1188224?

We paved over the problem with the import hook. Not writing .pyc files in the source directory is something we should do. But it is a PITA, as you need to replace the built-in import mechanism. This is very difficult in Python 2.7 and slightly easier in Python 3, as Python 3.3 made import code fully implemented in Python and thus modifiable.

Regarding comment #2, mach hacks up sys.path. See build/mach_bootstrap.py. In addition, the virtualenv has a supetset of those packages/paths. self.activate_virtualenv() from a mach command will pull in those additional packages - and allow you to install additional packages from pip. It's all very convoluted and sub-optimal :/