Closed Bug 1347483 Opened 3 years ago Closed 3 years ago

blessings package should be downloaded from internal pypi and not from external one


(Infrastructure & Operations :: CIDuty, task)

Not set


(firefox54 fixed, firefox55 fixed)

Tracking Status
firefox54 --- fixed
firefox55 --- fixed


(Reporter: cbook, Assigned: maja_zf)




(1 file)

seems there was a problems today loading blessing from and so caused test/builds failures.

This should be downloaded from the internal mirror maybe.

example log
I guess we already have mechanics in place to mirror content, perhaps we are just not enforcing it? It raises the broader question about how we avoid external content being downloaded during build process, for the risk it poses to security (unvetted content being introduced in builds) and also being able to run builds and releases also when external services are unavailable.

I'm guessing we've already tackled these problems in the past, maybe just a question of enforcement and/or monitoring? In the past did we block access to the canonical pypi for this reason? If we don't block, maybe we should at least monitor outbound access from workers, so we can build a picture of which external services we depend on (even if we don't immediately tackle security concern)?
Ed, I remember this was a favourite topic of yours in the past - do you have context on the history of this topic?
Flags: needinfo?(emorley)
Ah yeah you may be thinking of bug 617414, albeit that was more about tests themselves (which absolutely shouldn't rely on external resources) rather than package installation (eg at one point Talos jobs were inadvertently timing the pageloads of assets on real sites!).

Regarding Python package installation specifically, there are a few related but distinct issues:

1) How to improve python package determinism (in terms of both reproducibility and security).
2) To what extent CI setup should depend on services not under Mozilla's control (considerations: determinism/uptime/security/speed of download/convenience). 
3) Finding places where we don't do sensible caching (eg local pip cache) and so hammer PyPI or Mozilla's mirror of it, causing unnecessary load and increasing job end to end times.

IMO the correct way to resolve #1 is by using pip's --require-hashes feature (pip v8+), which:
* ensures all packages are pinned to a particular version
* ensures all sub-dependencies are specified (normally these just get silently pulled in unpinned)
* uses hashes to ensure consistency/security

With that fixed, then #2 is more about uptime/speed of download/convenience, and not security.

With #3 fixed, speed of download and uptime is less of a big deal, and convenience (ie: not having to constantly update packages on the internal PyPI mirror by hand) might win out. 

That said, with #1 fixed, perhaps it's safe to just automate the population of the internal PyPI mirror (and turn it into more of a transparent cache of PyPI), and thus reduce the pain points of using it?
Flags: needinfo?(emorley)
See Also: → 1347488
Tomcat has created a separate bug (bug 1347488) to address the larger issue raised in comments 1-3. Thanks Tomcat!

This bug then can remain about the specifics of mirroring the blessings package internally, and the other bug is a good place for discussing the a solution to the general problem. Sorry for getting sidetracked in this bug! :)
Component: General Automation → Buildduty
QA Contact: catlee → bugspam.Callek
Assignee: nobody → aobreja
We already have the blessings package in tree under python/blessings, which appears to be v1.3 (the minimum version required by mozlog). I bisected the tree and it seems we started to download this during the virtualenv setup because of some changes in bug 1335873.

ahal, is there a way we can use the in-tree blessings package for mozlog so that we don't need to rely on pypi for it?
Depends on: 1335873
Flags: needinfo?(ahalberstadt)
Historically the policy has been that the build must not hit the network. (We just don't have very good enforcement of that.) It's OK for uncommon mach commands to hit the network, or for CI builds to fetch things that aren't in the tree from Mozilla mirrors.
ahal is out so I'll try to help see if we can use the in-tree blessings for mozlog. ni?ing myself so I don't forget on Monday.
Flags: needinfo?(ahalberstadt) → needinfo?(mshal)
I tried changing the mozlog entry in packages.txt back to a pth instead of, which is what it was before bug  1335873. This seems to fix the problem - a 'mach build' no longer downloads a blessings package. The marionette test still downloads blessings from our internal pypi as before. I don't notice any difference in the output, but I might be missing something.

:maja_zf, since you reviewed the patch in bug 1335873, do you have any insight here? Or are there other things we should check to see if the version of packages.txt is really necessary for something?

Here's my try push for reference:
Flags: needinfo?(mshal) → needinfo?(mjzffr)
The call to picks up pytest-mozlog, which is a pytest plugin used by the marionette-harness-tests to output logs that result in useful failure summaries and other info on Treeherder. (This is the py(mnh) job on TH, only scheduled if certain files change) 

See for fuller explanation, as well as [1].

In any case, I figured out a different way to register the pytest-mozlog plugin so that we don't need to call anymore. Patch coming up. It works locally. Let's see what try has to say:

Flags: needinfo?(mjzffr)
Assignee: aobreja → mjzffr
Comment on attachment 8850132 [details]
Bug 1347483 - blessings package should not be downloaded during mach virtualenv setup;

I can confirm this stops downloading blessings during regular mach commands (like 'mach build') so this looks good to me!

::: commit-message-8c89d:1
(Diff revision 1)
> +Bug 1347483 - blessings package should be downloaded from internal pypi and not from external one; r?whimboo,mshal

IMO this line should be changed to something like "blessings package should not be downloaded during mach virtualenv setup", since we aren't changing it from an external to an internal pypi, but rather just using the in-tree version.

I realize that this comes from the bug title, but I think this more accurately describes what's actually being changed.
Attachment #8850132 - Flags: review?(mshal) → review+
Pushed by
blessings package should not be downloaded during mach virtualenv setup; r=mshal
Closed: 3 years ago
Resolution: --- → FIXED
Please uplift to 54. Test-only uplift. Uplift Bug 1345109 first.
Whiteboard: [checkin-needed-aurora]
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.