1477021 - figure out how to regularly update python and dependencies for in-tree docker images

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[[e-mail] (:bhearsum)]

We recently did a one-off update of our in-tree Docker images to get Python and the dependencies we use up-to-date. We need to make sure they stay up to date, which is difficult for a couple of reasons:
1) The images only rebuild when the Dockerfile (or something it references) changes. This means that when a new Python version becomes available, we won't pick it up until the Dockerfile changes for some other reason.
2) Dependencies are version pinned (on purpose), so we need something that bumps those pinned versions periodically.

I'm wondering if the periodic update script (or something like it) might be able to help us here? Maybe force a rebuild with a comment change once per week, and have it use the pyup script to bump dependencies?

Simon, you've worked a lot with periodic file updates - does that seem viable to you?

Comment 1

[[e-mail] (:bhearsum)]

Whoops, wrong person needinfo'ed!

Comment 2

[Simon Fraser [:sfraser] ⌚️GMT]

`./mach python-safety` should be the thing to use here. I need to adjust it slightly to produce output compatible with treeherder, and add then we can set up a cron task to check all the files.

Also on the todo list is to make managing the results of the task easier. I think this will be a case of having a mapping of requirements file to owner, in a similar manner to histograms' json file.

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Simon Fraser [:sfraser] ⌚️GMT from comment #2)
> `./mach python-safety` should be the thing to use here. I need to adjust it
> slightly to produce output compatible with treeherder, and add then we can
> set up a cron task to check all the files.
> 
> Also on the todo list is to make managing the results of the task easier. I
> think this will be a case of having a mapping of requirements file to owner,
> in a similar manner to histograms' json file.

This looks like it will get us part of the way there, but I don't think safety will care about outdated dependencies that don't have vulnerabilities. Maybe we can have a similar command that runs pyup, and run that from a cronjob?

Comment 4

[Simon Fraser [:sfraser] ⌚️GMT]

> This looks like it will get us part of the way there, but I don't think
> safety will care about outdated dependencies that don't have
> vulnerabilities. Maybe we can have a similar command that runs pyup, and run
> that from a cronjob?

*grumble* 'pip3 search pyup' didn't return 'pyupio'. Yeah, we should add the regular check, too, although I don't know if it should be the same level of error as the others?

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Simon Fraser [:sfraser] ⌚️GMT from comment #4)
> > This looks like it will get us part of the way there, but I don't think
> > safety will care about outdated dependencies that don't have
> > vulnerabilities. Maybe we can have a similar command that runs pyup, and run
> > that from a cronjob?
> 
> *grumble* 'pip3 search pyup' didn't return 'pyupio'. Yeah, we should add the
> regular check, too, although I don't know if it should be the same level of
> error as the others?

Hmmm, that's a good question. I've been thinking of it more like the periodic file update jobs than a test (because our primary goal here is to make sure that dependencies are regularly bumped for the in-tree requirements files). We could do both, though. I agree that as a test, the error level is lower than a safety. Perhaps the same level as the WARNINGs that python-safety generates about unpinned dependencies?

Comment 6

[[e-mail] (:bhearsum)]

And argh, I just realized that pyup.io doesn't actually work unless you give it a github or gitlab repot operate on -- you can't feed it a requirements file and have it spit out an updated requirements file. So, that's going to make dependency updates tougher. We might need to look at a different strategy for doing them (eg: Pipfile/Pipfile.lock, or some other method to get the completed pinned set of dependencies from a short list of unpinned ones).

Comment 7

[Simon Fraser [:sfraser] ⌚️GMT]

The current 'safety' just scans a requirements.txt file that you give it, hopefully the 'pyup' cli works the same way and we can wrap it inside a command and FileFinder, and so on

Comment 8

[[e-mail] (:bhearsum)]

(In reply to Simon Fraser [:sfraser] ⌚️GMT from comment #7)
> The current 'safety' just scans a requirements.txt file that you give it,
> hopefully the 'pyup' cli works the same way and we can wrap it inside a
> command and FileFinder, and so on

Unfortunately, 'pyup' doesn't know how to operate on local files - it only operates on repos. We might be able to use some of its internals to make something that operates locally, though...

Comment 9

[[e-mail] (:bhearsum)]

Simon and I spoke about this today, here's the path I'm going to pursue for dependencies:
- Switch our in-tree requirements files to Pipfile/Pipfile.lock. The pipfile will specify unpinned first level dependencies.
- Write a new job, similar to the current periodic file update jobs, that runs "pipenv update" and attachs patches to phabricator.

I still don't have a better idea for ensuring we have up-to-date Python other than regularly forcing rebuilds of affected images.

Comment 10

[[e-mail] (:bhearsum)]

Attachment: switch funsize-update-generator to a Pipfile

This is prep for setting up automatic dependency updates for it. I've put what I believe are the first level dependencies into the Pipfile, and generated a Pipfile.lock with pipenv. I purposely downgraded aiohttp to an older version, which will help verify that the automatic updates work when they land.

Comment 11

[[e-mail] (:bhearsum)]

Attachment: create a docker image that can update Pipfile.lock, and attach diffs to phabricator

This is part 2 of the work for automatic dependency update, which sets up a Docker image that is capable of updating in-tree Pipfile.locks, and attaching patches for them to Phabricator. It's largely based on the periodic file update work, with unnecessary features removed. You can see it in action at https://tools.taskcluster.net/groups/PI8R8DqBS7ePULstalFY_Q/tasks/PI8R8DqBS7ePULstalFY_Q/details.

Comment 12

[[e-mail] (:bhearsum)]

Attachment: create pipfile-update task for funsize-update-generator

This creates a task that actual does pipfile updates for funsize-update-generator. Like the periodic file updates, it does not run on check-in. I intend for it to run once a week through cron eventually (for now, I'd just like to get the code checked in so it's easier to test).

Comment 13

[Phabricator Automation]

Comment on attachment 8994991 [details]
switch funsize-update-generator to a Pipfile

Simon Fraser [:sfraser] ⌚️GMT has approved the revision.

https://phabricator.services.mozilla.com/D2373

Comment 14

[Phabricator Automation]

Comment on attachment 8994992 [details]
create a docker image that can update Pipfile.lock, and attach diffs to phabricator

Simon Fraser [:sfraser] ⌚️GMT has approved the revision.

https://phabricator.services.mozilla.com/D2374

Comment 15

[Phabricator Automation]

Comment on attachment 8994994 [details]
create pipfile-update task for funsize-update-generator

Simon Fraser [:sfraser] ⌚️GMT has approved the revision.

https://phabricator.services.mozilla.com/D2375

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d937887ddcb1
switch funsize-update-generator to a Pipfile. r=sfraser
https://hg.mozilla.org/integration/mozilla-inbound/rev/1ac1ba81c59c
create a docker image that can update Pipfile.lock, and attach diffs to phabricator. r=sfraser
https://hg.mozilla.org/integration/mozilla-inbound/rev/235b296cbd77
create pipfile-update task for funsize-update-generator. r=sfraser

Comment 17

[[e-mail] (:bhearsum)]

Attachment: run funsize with "pipenv run"

I tried out the new funsize image by tweaking an existing job and discovered that it was failing to find aiohttp. We need to run it through "pipenv run" now, otherwise none of the dependencies are available.

Comment 18

[Phabricator Automation]

Comment on attachment 8995180 [details]
run funsize with "pipenv run"

Simon Fraser [:sfraser] ⌚️GMT has approved the revision.

https://phabricator.services.mozilla.com/D2426

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a21fe3bd833d
run funsize with 'pipenv run'. r=sfraser

Comment 20

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/d937887ddcb1
https://hg.mozilla.org/mozilla-central/rev/1ac1ba81c59c
https://hg.mozilla.org/mozilla-central/rev/235b296cbd77
https://hg.mozilla.org/mozilla-central/rev/a21fe3bd833d

Comment 21

[[e-mail] (:bhearsum)]

These patches landed successfully, but I still need to enable cron jobs to actually update Pipfile.lock.

Comment 22

[[e-mail] (:bhearsum)]

Attachment: Schedule pipfile updates to run once per week.

Comment 23

[Phabricator Automation]

Comment on attachment 8995524 [details]
Schedule pipfile updates to run once per week.

Simon Fraser [:sfraser] ⌚️GMT has approved the revision.

https://phabricator.services.mozilla.com/D2467

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/10dea25a944b
Schedule pipfile updates to run once per week. r=sfraser

Comment 25

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/10dea25a944b

Comment 26

[[e-mail] (:bhearsum)]

Looks like this didn't quite work this week. It looks like it's because I didn't enable the taskcluster proxy, so the arc secret couldn't be accessed. Should be an easy fix.

Comment 27

[[e-mail] (:bhearsum)]

Attachment: Add taskcluster proxy to pipfile update, so arc secret can be accessed.

Comment 28

[Phabricator Automation]

Comment on attachment 8995964 [details]
Add taskcluster proxy to pipfile update, so arc secret can be accessed.

Johan Lorenzo [:jlorenzo] has approved the revision.

https://phabricator.services.mozilla.com/D2509

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/25b58eff68a8
Add taskcluster proxy to pipfile update, so arc secret can be accessed. r=jlorenzo

Comment 30

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/25b58eff68a8

Comment 31

[[e-mail] (:bhearsum)]

Everything went well with the new Pipfile job this time - we're done here!

Comment 32

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/1141d88bd144
https://hg.mozilla.org/releases/mozilla-esr60/rev/863c9f000faa
https://hg.mozilla.org/releases/mozilla-esr60/rev/14632c6e5fa2
https://hg.mozilla.org/releases/mozilla-esr60/rev/079bd02d070b
https://hg.mozilla.org/releases/mozilla-esr60/rev/cd5df0ec57cb