1478023 - [emr-spark-bootstrap] Configure pyup-bot for security-only updates

Status: RESOLVED WONTFIX

(Data Platform and Tools :: General, enhancement, P2)

Description

[Jeff Klukas [:klukas] (UTC-4)]

We currently get multiple PRs coming in to emr-spark-bootstrap per week to update dependencies, and we have a policy of ignoring them since we've found the cost:benefit ratio to be unappealing.

We'd like to reduce the frequency of PRs by configuring the bot to only suggest updates when they fix security vulnerabilities.

Instructions on how to do this below from an email from Gregory Guthe <[e-mail]>

---


We'd like to make sure every application stays up to date with the latest version of packages, particularly when they fix security vulnerabilities. NodeJS developers have long adopted Greenkeeper.io to automatically update their dependencies, and now Python developers can do the same with Pyup.io.

Pyup has a free tier that limits updates to once a month, so we bought an organization subscription to get security updates for outdated python packages sooner. We'd like to encourage everyone to make use of it.

To enable Pyup for your python repos, do the following:

   1. Add a pyup config to your repo (example config: https://github.com/mozilla-services/antenna/blob/master/.pyup.yml)

   2. Add https://github.com/mozsvcpyup as a collaborator to your repo

   3. Notify [e-mail] to enable the integration in pyup

This will periodically create pull requests for security updates.

Pyup can also be run in CI if you don't want it to create pull requests automatically. If you prefer to use this approach, contact us and we'll get you setup.

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/emr-bootstrap-spark/pull/427

Comment 2

[Rob Hudson [:robhudson]]

https://github.com/mozilla/emr-bootstrap-spark/commit/405298aa

Comment 3

[Jannis Leidel [:jezdez]]

I don't agree with limiting these PRs to security only updates given my experience in Python projects at Mozilla that it compounds technical debt. Without visibility by showing up as tickets/issues or pull requests dependency updates are never surfaced for work estimation and work planning and effectively ignored until it's too late.

Of course I get that constant PR updates can be annoying and GitHub has trouble limiting the notifications easily. But I would consider this a configuration issue of PyUP (e.g. making the update bi-weekly instead) or the GitHub team that is subscribed to the repo.

There is also no need for many people to receive PR notifications for this repo so I would suggest to create a new team with the lowest number of people, and asks other to "watch" the repo if they want to continue receiving notifications.

Comment 4

[Jeff Klukas [:klukas] (UTC-4)]

The deployment model for emr-spark-bootstrap makes testing and releasing updates an expensive and potentially disruptive process.

We can spin up a dev ATMO cluster with pending bootstrap changes, but then the testing process is manual to ensure that sample jupyter notebooks run successfully, etc.

Of the two changes I've deployed so far to emr-spark-bootstrap, one of them had to be rolled back due to breaking workflows.

I agree that putting off library updates means accruing technical debt, but it also seems unrealistic at this point to do even a once-every-2-weeks library update due to lack of automated testing.

We may want to close this issue and spawn a separate one to look at the larger issue of how to better automate testing for emr-spark-bootstrap updates or how to version usage so that we can make more frequent changes.

Comment 5

[Mark Reid [:mreid]]

(In reply to Jeff Klukas [:klukas] (UTC-4) from comment #4)
> We may want to close this issue and spawn a separate one to look at the
> larger issue of how to better automate testing for emr-spark-bootstrap
> updates or how to version usage so that we can make more frequent changes.

Good idea. I've filed bug 1509869 for the broader concern of improving testability.