1501494 - roll out python 3.7.1 rpm to releng centos puppet infra

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation, defect)

Description

[Aki Sasaki (not active)]

Once we build an RPM of python 3.7.1, we need to test and deploy it. To do that,
we need to:

- write a puppet patchset to roll it out to our CentOS hosts. I already have
  a patchset going here [1].

- test deploying that patchset to at least one test host, using a puppet environment [2]
  I generally use signing-linux-dev1.srv.releng.use1.mozilla.com, but any
  scriptworker that isn't in active use can do. If needed, we can quarantine
  scriptworkers to take them out of the active pools here [3].

  - once the package installs successfully, we should probably also try a quick
    script test, either by running the scriptworker script manually, or by
    reenabling it in the pool and monitoring its tasks.

- once the tests look good and the puppet patchset is r+ed, we need to deploy.
  This is complicated by the ~2min/module virtualenv rebuild time, which means
  signing scriptworkers can be out of commission for >70min at a time.
  This means we either need to:

  - debug and fix the slow virtualenv recreation. ~2min per module may mean
    slow network, slow ssl, inefficient installs, ?, or

  - roll out in waves. For python 3.6.5-2, I:

    1. set up a puppet environment that contained my patchset and a staging patch
       that pinned all scriptworkers to my environment. I kept this environment
       rebased against master to avoid any divergence other than the python
       version.

    2. quarantined some number of scriptworkers here [3]

    3. ssh'ed to those instances and puppetized

    4. unquarantined and watched them go green

    5. repeated steps 2-4 until all scriptworkers were updated. You can also
       leave a handful of un-updated scriptworkers, as long as they're
       quarantined before the next step.

    6. landed the puppet patch and watched it roll out in papertrail

    7. unpinned all of the pinned scriptworkers in my environment. I
       basically updated my puppet environment to master, which means they'll
       all self-unpin on the next run.

[1] https://github.com/escapewindow/build-puppet/commits/py37
[2] https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_user_environment
[3] https://tools.taskcluster.net/provisioners/scriptworker-prov-v1/worker-types

Comment 1

[Aki Sasaki (not active)]

-> docker