1415655 - Intermittent Could not install python package: /Users/cltbld/tasks/task_1510167958/build/venv/bin/pip install --timeout 120 --no-index --find-links http://pypi.pvt.build.mozilla.org/pub --find-links http://pypi.pub.build.mozilla.org/pub

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P1)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de

https://treeherder.mozilla.org/logviewer.html#?job_id=143104982&repo=autoland

https://queue.taskcluster.net/v1/task/Xa_i9eGnRf2-bFplk9qpSQ/runs/0/artifacts/public/logs/live_backing.log

Could not install python package: /Users/cltbld/tasks/task_1510167958/build/venv/bin/pip install --timeout 120 --no-index --find-links http://pypi.pvt.build.mozilla.org/pub --find-links http://pypi.pub.build.mozilla.org/pub --trusted-host pypi.pub.build.mozilla.org --trusted-host pypi.pvt.build.mozilla.org jsonschema==2.5.1 failed after 5 tries!

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Closed the trees for this because of its frequency.

Comment 2

[Chris AtLee [:catlee]]

We think this is due to the recent mdc1 move train.

It's likely that machines in mdc1 can't access pypi.pvt properly.

Comment 3

[:dhouse]

```
11:06:41     INFO -  Ignoring indexes: https://pypi.python.org/simple
11:06:41     INFO -  Collecting psutil>=3.1.1
11:06:43     INFO -    HTTP error 403 while getting http://pypi.pvt.build.mozilla.org/pub/psutil-3.1.1.tar.gz (from http://pypi.pvt.build.mozilla.org/pub/)
11:06:43     INFO -    Could not install requirement psutil>=3.1.1 from http://pypi.pvt.build.mozilla.org/pub/psutil-3.1.1.tar.gz because of error 403 Client Error: Forbidden for url: http://pypi.pvt.build.mozilla.org/pub/psutil-3.1.1.tar.gz
11:06:43     INFO -  Could not install requirement psutil>=3.1.1 from http://pypi.pvt.build.mozilla.org/pub/psutil-3.1.1.tar.gz because of HTTP error 403 Client Error: Forbidden for url: http://pypi.pvt.build.mozilla.org/pub/psutil-3.1.1.tar.gz for URL http://pypi.pvt.build.mozilla.org/pub/psutil-3.1.1.tar.gz (from http://pypi.pvt.build.mozilla.org/pub/)
11:06:43  WARNING - Return code: 1
```

Comment 4

[:dhouse]

In bug 1412965, similar failures to pypi.pvt were reported and found to be intermittent from the moonshots: "So far this seems to be limited to linux moonshot machines 285-295 and it's not consistent.  403 returned when contacting pypi:"

Comment 5

[Chris AtLee [:catlee]]

garndt has disabled the machines that were most recently moved to mdc1.

Comment 6

[Chris AtLee [:catlee]]

Adding these two lines to generated/allow-build-net-zlb.conf fixes the issue we think

    RewriteCond %{HTTP:X-Cluster-Client-Ip} !^10\.49\.56\.\d+$
    RewriteCond %{HTTP:X-Cluster-Client-Ip} !^10\.49\.58\.\d+$

Comment 7

[Kendall Libby [:fubar] (he/him)]

Added those two subnets to releng::init in IT puppet, which in turns makes its way into the apache configs. 

To ssh://git-internal.mozilla.org/sysadmins/puppet.git
   33f8974c3e..ac4f3644b2  master -> master

Comment 8

[Intermittent Failures Robot]

87 failures in 849 pushes (0.102 failures/push) were associated with this bug in the last 7 days. 

This is the #15 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* mozilla-inbound: 61
* autoland: 26

Platform breakdown:
* osx-10-10: 60
* macosx64-stylo-disabled: 26
* windows10-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1415655&startday=2017-11-06&endday=2017-11-12&tree=all

Comment 9

[Alin Selagea [:aselagea]]

All the occurrences of this issues happened on Nov 8, hasn't resurfaced since then.