1314574 - funsize-balrog-submitter.py not picking up .pth for extra library paths

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Simon Fraser [:sfraser] ⌚️GMT]

When making changes for bug 1312433, funsize-balrog-submitter.py could not import the new balrogclient module, despite it being present. 

On investigation, this is because https://dxr.mozilla.org/mozilla-central/source/release/docker/funsize-balrog-submitter/scripts/funsize-balrog-submitter.py#13

  sys.path.insert(0, os.path.join(
      os.path.dirname(__file__), "/home/worker/tools/lib/python"))

is just adding a regular path, not a site directory to sys.path, and .pth files are not processed outside a site directory.

It works later on because of https://dxr.mozilla.org/mozilla-central/source/release/docker/funsize-balrog-submitter/scripts/funsize-balrog-submitter.py#17 and the util module calls 'site.addsitedir(os.path.join(os.path.dirname(__file__), ".."))' which triggers the processing of the .pth file.

The shortest code change to make it work would be putting the `util` import before the others, but that leaves the mechanism as feeling slightly magic when reading the script. 

Code review coming in shortly for a change from sys.path.insert to a site.addsitedir() call, to make it explicitly clear what's happening, and so that the .pth file processing is not magic when reading the script.

The `os.path.dirname(__file__)` component doesn't appear to do anything useful, so will be removed. The absolute path as the second component means the first argument to os.path.join is ignored.

Comment 1

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: bug 1314574 replace sys.path with site.addsitedir so that .pth files are processed.

Review commit: https://reviewboard.mozilla.org/r/90046/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/90046/

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 8806670 [details]
bug 1314574 replace sys.path with site.addsitedir so that .pth files are processed.

https://reviewboard.mozilla.org/r/90046/#review89606

I tested the patch using the following scenario:

* apply the current patch to mozilla-inbound
* modified Dockerfile to clone the tools repo by revision: `RUN hg clone -r c41157cfe717 https://hg.mozilla.org/build/tools /home/worker/tools`
* built the image
* entered the image and tried to run the script. It didn't complain about `ImportError` \o/

We need to land this patch on all branches to make sure our release automation is ready for the main patch, then regenerate the nightly funsize docker image, and only after these steps land the tools patch.

I can land this patch with a=release.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e1caaca1ab3f
replace sys.path with site.addsitedir so that .pth files are processed.  r=rail a=release DONTBUILD

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 8806670 [details]
bug 1314574 replace sys.path with site.addsitedir so that .pth files are processed.

https://hg.mozilla.org/integration/mozilla-inbound/rev/e1caaca1ab3fd1f835f997039079b00bfef3d181
https://hg.mozilla.org/releases/mozilla-aurora/rev/d5b46ae42336bfeddaea1bd58a07045e4c31d890
https://hg.mozilla.org/releases/mozilla-release/rev/fb97135d05dbb0c88c7313f3920a8b902a5967fa
https://hg.mozilla.org/releases/mozilla-esr45/rev/46b2558ca46916b58bdf6a91787cb946592dfe70

No need to land to beta, because it will be replaced by aurora in a couple of weeks and we are not going to build releases off of beta.

Comment 5

[Rail Aliiev [:rail]]

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #2)

> regenerate the nightly funsize docker image, and only after these steps land the tools patch.

Err, the other way around. Land the tools patch, then regenerate.

Comment 6

[Rail Aliiev [:rail]]

Attachment: Bug 1314574 - Bump funsize version

Review commit: https://reviewboard.mozilla.org/r/90058/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/90058/

Comment 7

[Simon Fraser [:sfraser] ⌚️GMT]

Comment on attachment 8806691 [details]
Bug 1314574 - Bump funsize version

https://reviewboard.mozilla.org/r/90058/#review89616

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 8806691 [details]
Bug 1314574 - Bump funsize version

remote:   https://hg.mozilla.org/build/puppet/rev/9642b77dda4f6a30f972c82b4706c1e6c9d106f0
remote:   https://hg.mozilla.org/build/puppet/rev/48a2567031d9b0d670f50349a9aba39c5027341d

Comment 9

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/e1caaca1ab3f