1543149 - Use `mach python` to run mozharness build script and remove PYTHONPATH hack

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Andrew Halberstadt [:ahal]]

For many years mozharness has had out-of-date mozbase modules copied in its main directory. I recently removed these copies in bug 1195299 by making sure we run all our mozharness scripts with mach python instead of the system python. This meant that things like mozbase would be available to the mozharness scripts.

Unfortunately, using mach python somehow caused the Windows ccov builds to hang somewhere in-between test packaging and calling browser/installer/Makefile, I have no idea why. See bug 1542242 for more info.

In order to avoid backing out the mozbase duplication removal (which has been really annoying for years), I decided to land a PYTHONPATH hack en lieu of using mach python.

This bug is the follow-up to remove that hack and figure out why exactly the ccov builds were hanging.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: The Fix (which breaks Win ccov builds)

STR

1. Apply this patch
2. Run:

$ ./mach try fuzzy --full -q "^build-win64-ccov"

Results

The build job hangs after packaging tests. Here's the relevant part of a log:

14:05:49 INFO - package-tests> Wrote 72285 files in 54373432 bytes to target.web-platform.tests.tar.gz in 39.02s
14:05:49 INFO - package-tests> mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox'
[taskcluster 2019-04-07T14:18:02.388Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/fHyuCZfgQpe6BkY5crxuJg/0/on/us-west-2/i-02a455cadb66f0fc1/until/1554647882.465
[taskcluster 2019-04-07T14:35:03.170Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/fHyuCZfgQpe6BkY5crxuJg/0/on/us-west-2/i-02a455cadb66f0fc1/until/1554648903.22
[taskcluster 2019-04-07T14:52:03.886Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/fHyuCZfgQpe6BkY5crxuJg/0/on/us-west-2/i-02a455cadb66f0fc1/until/1554649923.912
[taskcluster 2019-04-07T15:09:04.598Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/fHyuCZfgQpe6BkY5crxuJg/0/on/us-west-2/i-02a455cadb66f0fc1/until/1554650944.584
[taskcluster:error] Aborting task...
[taskcluster 2019-04-07T15:10:01.465Z] ERROR: The process "2556" not found.
[taskcluster 2019-04-07T15:10:01.465Z]
...
[taskcluster 2019-04-07T15:10:01.723Z] === Task Finished ===

Link to full log

Expected

Here is what should happen after test packaging:

14:23:16 INFO - package-tests> Wrote 72285 files in 54373432 bytes to target.web-platform.tests.tar.gz in 59.50s
14:23:16 INFO - package-tests> mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox'
14:26:27 INFO - package> mozmake.EXE[5]: Entering directory 'z:/build/build/src/obj-firefox/browser/installer'
14:26:27 INFO - package> NO_PKG_FILES="core bsdecho js js-config jscpucfg nsinstall viewer TestGtkEmbed elf-dynstr-gc mangle* maptsv* mfc* msdump* msmap* nm2tsv* nsinstall* res/samples res/throbber shlibsign* certutil* pk12util* BadCertServer* OCSPStaplingServer* SymantecSanctionsServer* GenerateOCSPResponse* chrome/chrome.rdf chrome/app-chrome.manifest chrome/overlayinfo components/compreg.dat components/xpti.dat content_unit_tests necko_unit_tests *.dSYM SmokeDMD"
14:26:27 INFO - package> z:/build/build/src/obj-firefox/_virtualenvs/init/Scripts/python.exe z:/build/build/src/toolkit/mozapps/installer/packager.py

Comment 2

[Michael Shal [:mshal]]

I saw a similar hang when trying to land bug 1537574. Without the PYTHONPATH hack, Windows build jobs would hang and eventually hit the task timeout. I think simply trying a win32-opt build should be sufficient to trigger the behavior.

Comment 3

[Mike Hommey [:glandium]]

Attachment: Bug 1543149 - Use `mach python` to run mozharness build script and remove PYTHONPATH hack.

Also fix the gecko path on extra-config-path flags on mac tasks
(currently not causing problems because unused).

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/da17b47771c6
Use `mach python` to run mozharness build script and remove PYTHONPATH hack. r=releng-reviewers,aki

Comment 5

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/da17b47771c6