1408197 - 57.0b8 beetmover failures - 'error in cryptography setup command: Invalid environment marker: python_version < '3''

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P1)

Description

[Nick Thomas [:nthomas] (UTC+12)]

* mar depends on cryptography
* in the last 24 hours that had 2.1 and 2.1.1 releases
* the changelog (https://cryptography.io/en/latest/changelog/#v2-1) has 
   'Installing cryptography now requires pip 6 or newer'
* we have 
  'pip 1.5.6 from /mozharness/build/venv/local/lib/python2.7/site-packages (python 2.7)'

We end up with this error in any job that uses beetmover:

22:28:38     INFO -  Downloading/unpacking cryptography (from mar)
22:28:38     INFO -    Running setup.py (path:/tmp/pip-build-L3Fp9t/cryptography/setup.py) egg_info for package cryptography
22:28:38     INFO -      error in cryptography setup command: Invalid environment marker: python_version < '3'
22:28:38     INFO -      Complete output from command python setup.py egg_info:
22:28:38     INFO -      error in cryptography setup command: Invalid environment marker: python_version < '3'

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Bug 1408197 - fix beetmover by using a modern pip & setuptools, a=release

Review commit: https://reviewboard.mozilla.org/r/188946/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/188946/

Comment 2

[Chris AtLee [:catlee]]

Comment on attachment 8918052 [details]
Bug 1408197 - fix beetmover by using a modern pip & setuptools,  a=release

https://reviewboard.mozilla.org/r/188946/#review194254

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

https://hg.mozilla.org/releases/mozilla-beta/rev/c50a31492fe7fe38a75c8b4722f24685ba7dac03
https://hg.mozilla.org/releases/mozilla-release/rev/436438126ec502b53123a3f40c57593c11043fad
https://hg.mozilla.org/releases/mozilla-esr52/rev/efa03deffad75f19fc598a757239e99b22a8bceb

mozreview will handle autoland when that reopens.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e006f8a22781
fix beetmover by using a modern pip & setuptools, r=catlee a=release

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Turns out that upgrading pip and setuptools does not get us green beetmover tasks. We can process cryptography 2.1.1's setup.py now, but hit an error installing backports.lzma:

02:19:30     INFO -      x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/mozharness/build/venv/include -I/root/include -I/opt/local/include -I/usr/local/include -I/usr/include/python2.7 -c backports/lzma/_lzmamodule.c -o build/temp.linux-x86_64-2.7/backports/lzma/_lzmamodule.o
02:19:30     INFO -      backports/lzma/_lzmamodule.c:115:18: fatal error: lzma.h: No such file or directory
02:19:30     INFO -       #include <lzma.h>
02:19:30     INFO -                        ^
02:19:30     INFO -      compilation terminated.

It turns out we have hit this in 58.0b7 (and presumably earlier) but it's not fatal. After the initial failure to 'pip install mar' we retry, and pip says 'Requirement already satisfied'. This smells like a bug in early versions of pip - ie not rolling back unsuccessful installs, and/or not checking for all dependencies on subsequent install requests.

I contemplated providing lzma.h by adding the liblzma-dev package to taskcluster/docker/beet-mover/Dockerfile but there may be other bugs we hit after that. Instead I've reverted the pip/setuptools upgrade and pinned cryptography to 2.0.3 in https://hg.mozilla.org/releases/mozilla-beta/rev/d96198d960f3cbc4d17eb9990c801c1903d454fd

This should leave us in the same state as previous releases - mar gets installed but without all the requirements, but we get away with it because we never use them in beet_mover. It only wants BZ2MarFile at https://hg.mozilla.org/releases/mozilla-beta/file/56e25919ea0c/testing/mozharness/external_tools/extract_and_run_command.py#l41.

TODO: graft d96198d960f on other branches if it works

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/e006f8a22781

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4fd5250dedfc
beetmover fix round 2, r=bustage a=release

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Landed the cryptography==2.0.3 pinning on the other branches
https://hg.mozilla.org/releases/mozilla-release/rev/03ac2dcd9553a8cd6bc87197bb375f4803e86498
https://hg.mozilla.org/releases/mozilla-esr52/rev/8a8e95e992bd4bf2ff051954d9d502e31dbad601
https://hg.mozilla.org/integration/mozilla-inbound/rev/4fd5250dedfcfb133c244992db3019a432cec203

So we're in a working, if not optimal, state everywhere.

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/4fd5250dedfc

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

This happened again today with 57.0b12 build1, as we somehow have pip 8.1.2 via bug 1297515 now. We're going to use bug 1410960 to add the missing header to resolve it.