911750 - Because of missing or bad proxy settings, mozdownload on mozmill-ci nodes is not able to download Firefox builds from https://ftp.mozilla.org via the proxy

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Not sure what's going on now but as it seems like the proxy environment settings are all gone from the OS X nodes. As result we are not able to download builds of Firefox:

[mozilla-aurora_endurance] $ mozmill-env/run mozdownload --type=$BUILD_TYPE --branch=mozilla-aurora --platform=$PLATFORM --locale=$LOCALE --build-id=$BUILD_ID --date=$DATE --retry-attempts=10 --retry-delay=30 --directory=builds
Retrieving list of builds from https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/09
Build not found: 'HTTPSConnectionPool(host='ftp.mozilla.org', port=443): Request timed out. (timeout=60.0)'
Will retry in 30.0 seconds...
Retrying... (attempt 1)
Retrieving list of builds from https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/09
Build not found: 'HTTPSConnectionPool(host='ftp.mozilla.org', port=443): Request timed out. (timeout=60.0)'
Will retry in 30.0 seconds...
Retrying... (attempt 2)

I'm not sure why this started to happen today across all os x boxes without any restart. But I will investigate that now.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This widens up now to happen across all platforms. Michael, could something which has been done by bug 880709 caused this problem?

Here an example executed via a normal terminal session on OS X:

$ curl https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/09
curl: (7) couldn't connect to host

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For the Linux nodes I can see:

requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I sorted this out. It's a regression we have in mozdownload. So moving this back into the Mozilla QA infrastructure component to investigate further.

Comment 4

[Michael Henry [:tinfoil]]

Please give me some more information:

A) When did they stop having access?
B) Can they access other sites via HTTPS
C) How about HTTP?
D) How about FTP?

Try the following sites:

http://yahoo.com/
https://google.com/
ftp://ftp.mit.edu/

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Actually we were on 1.7.2 before. So it could also have been the 1.8 release, which I would really blame. Anything related to mozdownload is tracked on that issue now:

https://github.com/mozilla/mozdownload/issues/153

Once it is fixed we can upgrade our mozmill environment to 1.5.22 again.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Michael, please see comment 3. It's failing on our side. The network routes are still perfectly setup.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

At least on Linux the wrong proxy settings where present for our new Ubuntu 13.04 VMs. Instead of 'http://proxy.dmz.scl3.mozilla.com:3128' they pointed to 'https://proxy.dmz.scl3.mozilla.com:3128'. The necessary work will be tracked on bug 911795.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I unblocked ourselves here by updating all of our existing VMs and Mac minis today to the correct proxy settings. I wish that we will have that documented and fixed across all the templates too soonish.