1347483 - blessings package should be downloaded from internal pypi and not from external one

Status: RESOLVED FIXED

Description

[Carsten Book [:Tomcat]]

seems there was a problems today loading blessing from https://pypi.python.org/ and so caused test/builds failures.

This should be downloaded from the internal mirror maybe.

example log https://treeherder.mozilla.org/logviewer.html#?job_id=83918720&repo=autoland&lineNumber=1208

Comment 1

[Pete Moore [:pmoore][:pete]]

I guess we already have mechanics in place to mirror content, perhaps we are just not enforcing it? It raises the broader question about how we avoid external content being downloaded during build process, for the risk it poses to security (unvetted content being introduced in builds) and also being able to run builds and releases also when external services are unavailable.

I'm guessing we've already tackled these problems in the past, maybe just a question of enforcement and/or monitoring? In the past did we block access to the canonical pypi for this reason? If we don't block, maybe we should at least monitor outbound access from workers, so we can build a picture of which external services we depend on (even if we don't immediately tackle security concern)?

Comment 2

[Pete Moore [:pmoore][:pete]]

Ed, I remember this was a favourite topic of yours in the past - do you have context on the history of this topic?

Comment 3

[Ed Morley [:emorley]]

Ah yeah you may be thinking of bug 617414, albeit that was more about tests themselves (which absolutely shouldn't rely on external resources) rather than package installation (eg at one point Talos jobs were inadvertently timing the pageloads of assets on real sites!).

Regarding Python package installation specifically, there are a few related but distinct issues:

1) How to improve python package determinism (in terms of both reproducibility and security).
2) To what extent CI setup should depend on services not under Mozilla's control (considerations: determinism/uptime/security/speed of download/convenience). 
3) Finding places where we don't do sensible caching (eg local pip cache) and so hammer PyPI or Mozilla's mirror of it, causing unnecessary load and increasing job end to end times.

IMO the correct way to resolve #1 is by using pip's --require-hashes feature (pip v8+), which:
* ensures all packages are pinned to a particular version
* ensures all sub-dependencies are specified (normally these just get silently pulled in unpinned)
* uses hashes to ensure consistency/security

With that fixed, then #2 is more about uptime/speed of download/convenience, and not security.

With #3 fixed, speed of download and uptime is less of a big deal, and convenience (ie: not having to constantly update packages on the internal PyPI mirror by hand) might win out. 

That said, with #1 fixed, perhaps it's safe to just automate the population of the internal PyPI mirror (and turn it into more of a transparent cache of PyPI), and thus reduce the pain points of using it?

Comment 4

[Pete Moore [:pmoore][:pete]]

Tomcat has created a separate bug (bug 1347488) to address the larger issue raised in comments 1-3. Thanks Tomcat!

This bug then can remain about the specifics of mirroring the blessings package internally, and the other bug is a good place for discussing the a solution to the general problem. Sorry for getting sidetracked in this bug! :)

Comment 5

[Michael Shal [:mshal]]

We already have the blessings package in tree under python/blessings, which appears to be v1.3 (the minimum version required by mozlog). I bisected the tree and it seems we started to download this during the virtualenv setup because of some changes in bug 1335873.

ahal, is there a way we can use the in-tree blessings package for mozlog so that we don't need to rely on pypi for it?

Comment 6

[Ted Mielczarek [:ted.mielczarek]]

Historically the policy has been that the build must not hit the network. (We just don't have very good enforcement of that.) It's OK for uncommon mach commands to hit the network, or for CI builds to fetch things that aren't in the tree from Mozilla mirrors.

Comment 7

[Michael Shal [:mshal]]

ahal is out so I'll try to help see if we can use the in-tree blessings for mozlog. ni?ing myself so I don't forget on Monday.

Comment 8

[OrangeFactor Robot]

13 failures in 777 pushes (0.017 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 13

Platform breakdown:
* windows2012-64: 4
* windows2012-32: 4
* linux64: 2
* osx-10-7: 1
* linux64-stylo: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1347483&startday=2017-03-13&endday=2017-03-19&tree=all

Comment 9

[Michael Shal [:mshal]]

I tried changing the mozlog entry in packages.txt back to a pth instead of setup.py, which is what it was before bug  1335873. This seems to fix the problem - a 'mach build' no longer downloads a blessings package. The marionette test still downloads blessings from our internal pypi as before. I don't notice any difference in the output, but I might be missing something.

:maja_zf, since you reviewed the patch in bug 1335873, do you have any insight here? Or are there other things we should check to see if the setup.py version of packages.txt is really necessary for something?

Here's my try push for reference:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c17b3f0c4c35af14c1a5d96df4ccdc82803fdfc3

Comment 10

[Maja Frydrychowicz (:maja_zf)]

The call to setup.py picks up pytest-mozlog, which is a pytest plugin used by the marionette-harness-tests to output logs that result in useful failure summaries and other info on Treeherder. (This is the py(mnh) job on TH, only scheduled if certain files change) 

See https://bugzilla.mozilla.org/show_bug.cgi?id=1335873#c9 for fuller explanation, as well as [1].

In any case, I figured out a different way to register the pytest-mozlog plugin so that we don't need to call setup.py anymore. Patch coming up. It works locally. Let's see what try has to say:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5b74291013a8a27b407357c62033013a8e1bb939


[1] http://docs.pytest.org/en/latest/writing_plugins.html#setuptools-entry-points

Comment 11

[Maja Frydrychowicz (:maja_zf)]

Created attachment 8850132 [details]
Bug 1347483 - blessings package should not be downloaded during mach virtualenv setup;

This fixes a regression introduced in Bug 1335873, which changes the mozbase
packages.txt to call mozlog's setup.py. Calling setup.py registers the
pytest_mozlog plugin for marionette-harness tests.

Instead, we can register the pytest-mozlog plugin via command-line arguments
to pytest, which are set in pytest.ini for the marionette-harness tests.
As a result, we can revert the mozbase packages.txt to not refer to mozlog's
setup.py

I'm leaving the pytest entry-point in mozlog's setup.py so that external
consumers don't have to register the pytest_mozlog plugin manually.

Review commit: https://reviewboard.mozilla.org/r/122824/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/122824/

Comment 12

[Michael Shal [:mshal]]

Comment on attachment 8850132 [details]
Bug 1347483 - blessings package should not be downloaded during mach virtualenv setup;

https://reviewboard.mozilla.org/r/122824/#review125110

I can confirm this stops downloading blessings during regular mach commands (like 'mach build') so this looks good to me!

::: commit-message-8c89d:1
(Diff revision 1)
> +Bug 1347483 - blessings package should be downloaded from internal pypi and not from external one; r?whimboo,mshal

IMO this line should be changed to something like "blessings package should not be downloaded during mach virtualenv setup", since we aren't changing it from an external to an internal pypi, but rather just using the in-tree version.

I realize that this comes from the bug title, but I think this more accurately describes what's actually being changed.

Comment 13

[Maja Frydrychowicz (:maja_zf)]

Comment on attachment 8850132 [details]
Bug 1347483 - blessings package should not be downloaded during mach virtualenv setup;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/122824/diff/1-2/

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0cf4ce9588ca
blessings package should not be downloaded during mach virtualenv setup; r=mshal

Comment 15

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/0cf4ce9588ca

Comment 16

[Maja Frydrychowicz (:maja_zf)]

Please uplift to 54. Test-only uplift. Uplift Bug 1345109 first.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/47b5215fc8ad