We shouldn't litter our tree with .pyc files

NEW
Unassigned

Status

Firefox Build System
General
7 years ago
4 months ago

People

(Reporter: Jeff Hammel, Unassigned)

Tracking

(Depends on: 1 bug, Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [mozbase])

(Reporter)

Description

7 years ago
When you import a file in python, by default a .pyc file will be created in the same directory.  These require manual wiping sometimes and otherwise litter the tree.  Using e.g. python -B should fix this problem, though other solutions are also worth considering

Comment 1

7 years ago
It sucks that python doesn't have a way to specify where .pyc files should go instead of next to the source file, but I really don't think that we want to disable .pyc files by default, since that could significantly affect build times. If it really bugs you, can you just specify PYTHONDONTWRITEBYTECODE=x in your environment?
(Reporter)

Comment 2

7 years ago
It less bothers me than leads to either genuine or imagined errors and the recommendation of wiping .pyc files before rebuilding.  Maybe `make -f client.mk` or similar should do find . -iname '*.pyc' -delete instead?
I've had build failures that required deleting a bad .pyc file to get going again.

Comment 4

7 years ago
I must admit I've never experienced that form of error. Do we know what caused it? Python is normally pretty good at checking the timestamps of .pyc files and rebuilding them on any mismatch.
Unfortunately, it happened to me a few days ago. I asked on IRC and Ms2ger suggested deleting the .pyc file it was complaining about, and it worked. Is there a way to find old posts from a certain user on pastebin.m.o?
(Reporter)

Comment 6

7 years ago
I was given the same advice. finding and deleteing the bad .pyc files on make takes almost no time on my system.  IMHO, its better doing this than chancing on yet another way things can break
(Reporter)

Updated

7 years ago
See Also: → bug 686233
(Reporter)

Comment 7

7 years ago
Maybe the thing to do is to make `make clobber` delete the .pyc files and maybe check-sync-dirs too
The former is feasible, the latter is not. We run check-sync-dirs for every build, so we don't want to clobber the .pyc files.

Updated

6 years ago
Depends on: 795995
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #4)
> I must admit I've never experienced that form of error. Do we know what
> caused it? Python is normally pretty good at checking the timestamps of .pyc
> files and rebuilding them on any mismatch.

In the last case I saw, it happened because a source file changed from a plain .py file to a preprocessed .py.in file, but the old .pyc file was still in the srcdir, and so (as far as I can tell) it was used rather than the .py file which was now in the objdir.

Maybe that could be avoided if objdir directories were always before srcdir directories in the python path?
Erik Rose and I were talking Python packaging and I was lamenting over the state of pyc files and lack of control. He informed me of a clever solution employed by Debian: symlinks.

Essentially you create a directory tree that mirrors the source tree. e.g.

topsrcdir
  mozbuild
    backend
      make.py

topobjdir
  mozbuild
    backend

Then you create symlinks back to the source .py files:

topobjdir
  mozbuild
    backend
      make.py -> topsrcdir/mozbuild/backend/make.py

You then point sys.path at topobjdir. Python doesn't follow the symlinks and it creates the .pyc files in the same directory as the symlink:

topobjdir
  mozbuild
    backend
      make.py -> topsrcdir/mozbuild/backend/make.py
      make.pyc

This could work for us on platforms where symlinks are supported. Windows might be a little funky. A downside is we'd need to walk mapped directories to reconcile the destination file list, probably on every build. But we do have code in mozpack that should make this relatively pain free. If we can pull it off, I think I like this more than the custom importer solution because the custom importer will be extremely hacky, especially on 2.7.

Updated

11 months ago
Blocks: 1380405

Updated

4 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.