1241811 - [16/01/22] Update all mozmill-ci machines for latest OS updates and virtualenv 14.0.1

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We should do another round of system updates for security reasons and also update virtualenv to the latest release 14.0.1 which includes pip 8.0.0:

http://virtualenv.readthedocs.org/en/latest/changes.html#id1

Reason is that on Treeherder our logs spit out the following lines for each job, which is annoying especially for failing jobs:

> You are using pip version 7.1.2, however version 8.0.0 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
> Return code: 0 

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=Firefox%20UI&filter-tier=2&filter-tier=3&selectedJob=3132753

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

On OS X we have to get the system update, virtualenv, and Java installed. Whereby the latter aborts due to a not approved installer. Given that Java 8 Update 71 has been released yesterday, it may not have been approved yet? I will try it later again.

Something I wonder is an upgrade of Python to version 2.7.11. I will also have to check that.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Bumping virtualenv to the latest release will introduce pip 8.0. With this installed the current rev of mozharness will cause bustage on mozmill-ci because --download-cache is no longer available. Therefore I pushed the fix for https://github.com/mozilla/mozmill-ci/issues/737 to staging and production. With that we are fine now to get all machines updated.

Also thinking more about a Python upgrade I will dismiss that for now. We are fine with 2.7.9 and we should keep it until we move with the appropriate platforms to Treeherder.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Upgrading virtualenv to 14.0.1 caused another bugstage on staging OS X machine:

https://treeherder.allizom.org/#/jobs?repo=mozilla-aurora&filter-searchStr=Firefox%20UI&exclusion_profile=false&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=1613492

Could not install python package: /Users/mozauto/jenkins/workspace/mozilla-aurora_functional/build/venv/bin/pip install --download-cache /Users/mozauto/jenkins/workspace/mozilla-aurora_functional/build/venv/cache --timeout 120 -r /Users/mozauto/jenkins/workspace/mozilla-aurora_functional/build/tests/requirements.txt --no-index --find-links

This should have been actually fixed in mozharness by bug 1240993. Not sure why this is still happening with the latest mozharness as upgraded earlier today via https://github.com/mozilla/mozmill-ci/pull/739.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Looks like those issues were an overlap between tests got triggered while I disabled shutdown mode in Jenkins to run the mozharness job. So all fine on those OS X hosts with latest virtualenv.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

FWIW you can disable the PIP version check by setting PIP_DISABLE_PIP_VERSION_CHECK=1

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks Dave. I will remember that env variable. But this time we really want to upgrade also regarding the new possibility of integrated peep features.

In regards of Ubuntu i suggest we go away from the python-virtualenv package and install virtualenv via pip instead.

> sudo apt-get remove --purge python-virtualenv
> sudo -E pip install virtualenv

This way we can ensure to have the latest version available. 14.04 is a lot behind and won't even get an update.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

All staging machines have been updated now. So far no regressions visible. Given that the release churn is over now I will get started with production machines soon.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Actually new beta builds are coming up. So I have to wait until the update testing has been finished.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I have installed updates on nearly all OS X machines. Now I have seen that Oracle released another version of Java. Given that we would have a fragmentation between different hosts.

I think that I will no longer update Java as long as it is not really necessary. Our hosts are behind a firewall which should be safe. So I don't see a reason to spend extra time on that, and it always needs manual work. It's better to concentrate on TC and get all stuff moved over there.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

All machines have been updated. See you again in about 3 months hopefully. :)