1449350 - distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('six')

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Particularly on esr52, in the first fifteen minutes of the hour. eg: https://treeherder.mozilla.org/logviewer.html#?job_id=170619437&repo=mozilla-esr52&lineNumber=2093

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [gecko] Move six earlier in websocketprocessbridge_requirements.txt

Comment 2

[Aki Sasaki (not active)]

Comment on attachment 8962908 [details] [diff] [review]
[gecko] Move six earlier in websocketprocessbridge_requirements.txt

We could even move this to the top, above twisted; six doesn't require twisted, but an unpinned twisted dep may require six at some point.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [gecko] Move six to first in websocketprocessbridge_requirements.txt

Updated patch, carrying r+

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b09e7b25bcbe
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('six'), r=aki

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

https://hg.mozilla.org/releases/mozilla-esr52/rev/06407f04400950678a1a262d34de63dba568e2c5

We're not seeing bustage on these branches, but more completeness:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b09e7b25bcbeb5ece5ee280d2037983f98267d75
https://hg.mozilla.org/releases/mozilla-beta/rev/cf9457f7129ac3b711e0a268c976cdaa39133e01

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

The backstory here is
* we saw a lot of red windows tests jobs on https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr52&revision=57bd35fa86186b240d889fa8d8fad97033d4ddbf
* these all run on older hardware, t-w732, with python not able to talk TLSv1.2
* if we use pip to install a package containing a setup.py which specifies requirements, then setuptools doesn't respect the args we've given to pip to install from our pypi servers, so it talks to pypi.python.org
* pypi.python.org is migrating to a new CDN provider, and having brownouts on TLSv1.0 and v1.1 for the first 15 minutes of the hour

So jobs fail when the starts align. The (attempted) fix is get pip to grab six before txWS needs it, so we setuptools doesn't need to reach out for anything. I presume we don't have test failures on newer branches because they can still talk to external pypi, but they shouldn't be doing so.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [gecko] Add six to initial virtualenv setup

We're still hitting errors with attachment 8962910 [details] [diff] [review], eg https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr52&filter-searchStr=windows&filter-tier=1&selectedJob=170695247.

Here's your suggestion to install six immediately after setting up the virtualenv.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8962970 [details] [diff] [review]
[gecko] Add six to initial virtualenv setup

Backout earlier patch:
https://hg.mozilla.org/releases/mozilla-esr52/rev/ebb75ac4798c5e17be6233ad82d2ad8e64ed4585

Land this one with version pinned:
https://hg.mozilla.org/releases/mozilla-esr52/rev/1d384e482912ef42efadfd09bfb87de6755a66fa

Will follow up on inbound/beta once we have some data about effectiveness.

Comment 9

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/b09e7b25bcbe

Comment 10

[Intermittent Failures Robot]

1 failures in 718 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1449350&startday=2018-03-26&endday=2018-04-01&tree=trunk

Comment 11

[Aki Sasaki (not active)]

https://hg.mozilla.org/releases/mozilla-esr52/rev/c6c34cb89dfdaa77ae8a7349a092cac3e0e8a635

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Note to self - come back here and figure out what we want to do with syncing branches.

Comment 13

[Intermittent Failures Robot]

43 failures in 47 pushes (0.915 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-esr52: 42
* autoland: 1

Platform breakdown:
* windows7-32-vm: 42
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1449350&startday=2018-04-28&endday=2018-04-28&tree=trunk

Comment 14

[Intermittent Failures Robot]

45 failures in 780 pushes (0.058 failures/push) were associated with this bug in the last 7 days. 

This is the #32 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-esr52: 42
* mozilla-inbound: 1
* mozilla-central: 1
* autoland: 1

Platform breakdown:
* windows7-32-vm: 42
* osx-10-10: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1449350&startday=2018-04-23&endday=2018-04-29&tree=trunk

Comment 15

[Intermittent Failures Robot]

11 failures in 551 pushes (0.02 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 5
* mozilla-esr60: 3
* mozilla-inbound: 1
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* osx-10-10: 6
* macosx64-nightly: 4
* macosx64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1449350&startday=2018-04-30&endday=2018-05-06&tree=trunk

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #8)
> Backout earlier patch:
> https://hg.mozilla.org/releases/mozilla-esr52/rev/
> ebb75ac4798c5e17be6233ad82d2ad8e64ed4585

Relanded as https://hg.mozilla.org/releases/mozilla-esr52/rev/187838f9aadb406508259dc934cd99f66da7ce6b because something broke between dc927ef3e774 on May 26 and restoring buidlbot jobs today (bug 1465288). Possibly 1463592.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Still hitting errors on esr52 so I've transplanted bug 1459391 there too, https://hg.mozilla.org/releases/mozilla-esr52/rev/a7ced1813d421901f213e238f62e92d31be50037.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

I think we're done here.