951736 - pymake only updates sys.path from PYCOMMANDPATH at import time

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

In bug 949875, we were hitting weird build failures on pymake when switching some imports to function scope so they could be delay loaded. I tracked down the issue to how pymake munges sys.path when executing native commands.

The logic for executing native pymake commands live at:

https://hg.mozilla.org/users/bsmedberg_mozilla.com/pymake/file/42627616700c/pymake/process.py#l378

Of interest is the sys.path munging that occurs in load_module_recursive:

https://hg.mozilla.org/users/bsmedberg_mozilla.com/pymake/file/42627616700c/pymake/process.py#l349

Essentially, the current behavior munges sys.path, imports the requested module, then unmunges sys.path before executing the command. This, of course, means sys.path is not adjusted during execution of the native command, which breaks lazy imports.

I'm surprised this has worked for as long as it has.

I'm planning to rewrite the sys.path munging. I plan to always have sys.path convey PYTHONCOMMANDPATH while the command is executing and restore it afterwards. I will also not make things conditional on whether the requested module is already in sys.path.

Comment 1

[(not currently active) Ted Mielczarek]

In hindsight that looks pretty silly. I don't know why I implemented it that way. PythonJob.run already has a try/finally for tweaking os.environ, it seems like a good place to munge sys.path as well.

Comment 2

[Gregory Szorc [:gps]]

Attachment: Adjust sys.path while native commands are executed

Patch with test.

Comment 3

[Gregory Szorc [:gps]]

https://hg.mozilla.org/users/bsmedberg_mozilla.com/pymake/rev/8fa921cbf2ff
https://hg.mozilla.org/integration/mozilla-inbound/rev/0f982fab4b85

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/0f982fab4b85