685957 - We shouldn't litter our tree with .pyc files

Status: NEW

(Firefox Build System :: General, defect)

Description

[Jeff Hammel]

When you import a file in python, by default a .pyc file will be created in the same directory.  These require manual wiping sometimes and otherwise litter the tree.  Using e.g. python -B should fix this problem, though other solutions are also worth considering

Comment 1

[Benjamin Smedberg]

It sucks that python doesn't have a way to specify where .pyc files should go instead of next to the source file, but I really don't think that we want to disable .pyc files by default, since that could significantly affect build times. If it really bugs you, can you just specify PYTHONDONTWRITEBYTECODE=x in your environment?

Comment 2

[Jeff Hammel]

It less bothers me than leads to either genuine or imagined errors and the recommendation of wiping .pyc files before rebuilding.  Maybe `make -f client.mk` or similar should do find . -iname '*.pyc' -delete instead?

Comment 3

[Ryan VanderMeulen [:RyanVM]]

I've had build failures that required deleting a bad .pyc file to get going again.

Comment 4

[Benjamin Smedberg]

I must admit I've never experienced that form of error. Do we know what caused it? Python is normally pretty good at checking the timestamps of .pyc files and rebuilding them on any mismatch.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Unfortunately, it happened to me a few days ago. I asked on IRC and Ms2ger suggested deleting the .pyc file it was complaining about, and it worked. Is there a way to find old posts from a certain user on pastebin.m.o?

Comment 6

[Jeff Hammel]

I was given the same advice. finding and deleteing the bad .pyc files on make takes almost no time on my system.  IMHO, its better doing this than chancing on yet another way things can break

Comment 7

[Jeff Hammel]

Maybe the thing to do is to make `make clobber` delete the .pyc files and maybe check-sync-dirs too

Comment 8

[(not currently active) Ted Mielczarek]

The former is feasible, the latter is not. We run check-sync-dirs for every build, so we don't want to clobber the .pyc files.

Comment 9

[Matt Brubeck (:mbrubeck)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #4)
> I must admit I've never experienced that form of error. Do we know what
> caused it? Python is normally pretty good at checking the timestamps of .pyc
> files and rebuilding them on any mismatch.

In the last case I saw, it happened because a source file changed from a plain .py file to a preprocessed .py.in file, but the old .pyc file was still in the srcdir, and so (as far as I can tell) it was used rather than the .py file which was now in the objdir.

Maybe that could be avoided if objdir directories were always before srcdir directories in the python path?

Comment 10

[Gregory Szorc [:gps]]

Erik Rose and I were talking Python packaging and I was lamenting over the state of pyc files and lack of control. He informed me of a clever solution employed by Debian: symlinks.

Essentially you create a directory tree that mirrors the source tree. e.g.

topsrcdir
  mozbuild
    backend
      make.py

topobjdir
  mozbuild
    backend

Then you create symlinks back to the source .py files:

topobjdir
  mozbuild
    backend
      make.py -> topsrcdir/mozbuild/backend/make.py

You then point sys.path at topobjdir. Python doesn't follow the symlinks and it creates the .pyc files in the same directory as the symlink:

topobjdir
  mozbuild
    backend
      make.py -> topsrcdir/mozbuild/backend/make.py
      make.pyc

This could work for us on platforms where symlinks are supported. Windows might be a little funky. A downside is we'd need to walk mapped directories to reconcile the destination file list, probably on every build. But we do have code in mozpack that should make this relatively pain free. If we can pull it off, I think I like this more than the custom importer solution because the custom importer will be extremely hacky, especially on 2.7.