1217575 - |mach python| and |mach python-test| don't forward sys.path to their python subprocesses

Status: RESOLVED WONTFIX

(Firefox Build System :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

When running |mach python-test|, the SEARCH_PATHS don't get forwarded to the testing subprocess, making it necessary to install any packages that otherwise would have been automatically found.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: MozReview Request: Bug 1217575 - Forward sys.path to subprocesses in |mach python| and |mach python-test| so in-tree modules are automatically found, r=gps

Bug 1217575 - Forward sys.path to subprocess in |mach python-test| so in-tree modules are automatically found, r=gps

Comment 2

[Andrew Halberstadt [:ahal]]

Comment on attachment 8677671 [details]
MozReview Request: Bug 1217575 - Forward sys.path to subprocesses in |mach python| and |mach python-test| so in-tree modules are automatically found, r=gps

Bug 1217575 - Forward sys.path to subprocesses in |mach python| and |mach python-test| so in-tree modules are automatically found, r=gps

Comment 3

[Mike Hommey [:glandium]]

This shouldn't be needed at all. If things are missing from the python path, then they should either be added to sys.path at runtime (which mach_test_package_bootstrap partially does), or added to the virtualenv (in build/virtualenv_packages.txt).

Comment 4

[Andrew Halberstadt [:ahal]]

Currently the items in SEARCH_PATHS [1] aren't added to the python subprocess. I wasn't aware of build/virtualenv_packages.txt, I guess we could add everything from SEARCH_PATHS to there, but I would prefer not to maintain two separate lists of packages that should be identical anyway.

[1] https://dxr.mozilla.org/mozilla-central/source/build/mach_bootstrap.py#67

Comment 5

[Andrew Halberstadt [:ahal]]

Also, I'm not talking about 'mach_test_package_bootstrap' here. I'm talking about the packages added at runtime in 'mach_bootstrap' that don't get forwarded to the python subprocesses [1] spawned by |mach python| and |mach python-test|.

[1] https://dxr.mozilla.org/mozilla-central/source/python/mach_commands.py#78

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 8677671 [details]
MozReview Request: Bug 1217575 - Forward sys.path to subprocesses in |mach python| and |mach python-test| so in-tree modules are automatically found, r=gps

https://reviewboard.mozilla.org/r/22993/#review21881

I'm not thrilled by this change.

The Python in the objdir's virtualenv is intentionally separate and more well-defined than the Python environment that `mach` sets up. All the sys.path hacking we do in mach_bootstrap.py is meant only as a stop-gap to enable mach commands to run. It is a horrible hack that I wish didn't exist. But fixing it is a lot of work.

If the limitations of the virtualenv-based Python are getting in the way, the proper thing to do is modify mozilla-central/build/virtualenv_packages.txt so necessary paths/packages are available to the virtualenv.

MozReview doesn't allow you to r-. But this would be a r- review.

Comment 7

[Andrew Halberstadt [:ahal]]

Many (if not most) of our python unittests are standalone and don't need an objdir to begin with. So I guess I don't understand why |mach python-test| needs to be tied to the objdir. But at least it'll still create the _virtualenv even if there is no build, so I guess that works.