All users were logged out of Bugzilla on October 13th, 2018

Because of missing or bad proxy settings, mozdownload on mozmill-ci nodes is not able to download Firefox builds from https://ftp.mozilla.org via the proxy

RESOLVED FIXED

Status

--
major
RESOLVED FIXED
5 years ago
4 months ago

People

(Reporter: whimboo, Assigned: whimboo)

Tracking

Details

(Assignee)

Description

5 years ago
Not sure what's going on now but as it seems like the proxy environment settings are all gone from the OS X nodes. As result we are not able to download builds of Firefox:

[mozilla-aurora_endurance] $ mozmill-env/run mozdownload --type=$BUILD_TYPE --branch=mozilla-aurora --platform=$PLATFORM --locale=$LOCALE --build-id=$BUILD_ID --date=$DATE --retry-attempts=10 --retry-delay=30 --directory=builds
Retrieving list of builds from https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/09
Build not found: 'HTTPSConnectionPool(host='ftp.mozilla.org', port=443): Request timed out. (timeout=60.0)'
Will retry in 30.0 seconds...
Retrying... (attempt 1)
Retrieving list of builds from https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/09
Build not found: 'HTTPSConnectionPool(host='ftp.mozilla.org', port=443): Request timed out. (timeout=60.0)'
Will retry in 30.0 seconds...
Retrying... (attempt 2)

I'm not sure why this started to happen today across all os x boxes without any restart. But I will investigate that now.
(Assignee)

Comment 1

5 years ago
This widens up now to happen across all platforms. Michael, could something which has been done by bug 880709 caused this problem?

Here an example executed via a normal terminal session on OS X:

$ curl https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/09
curl: (7) couldn't connect to host
Component: Infrastructure → Netops: DC ACL Request
OS: Mac OS X → All
Product: Mozilla QA → Infrastructure & Operations
QA Contact: adam
Hardware: x86_64 → All
Version: unspecified → other
(Assignee)

Updated

5 years ago
Summary: All OS X nodes cannot connect to https://ftp.mozilla.org to download builds of Firefox → All Mozmill CI nodes in qa.scl3.mozilla.com cannot connect to https://ftp.mozilla.org to download builds of Firefox
(Assignee)

Updated

5 years ago
Flags: needinfo?(mhenry)
(Assignee)

Comment 2

5 years ago
For the Linux nodes I can see:

requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
(Assignee)

Comment 3

5 years ago
I sorted this out. It's a regression we have in mozdownload. So moving this back into the Mozilla QA infrastructure component to investigate further.
Severity: critical → major
Component: Netops: DC ACL Request → Infrastructure
Flags: needinfo?(mhenry)
Product: Infrastructure & Operations → Mozilla QA
QA Contact: adam
Version: other → unspecified
(Assignee)

Updated

5 years ago
Summary: All Mozmill CI nodes in qa.scl3.mozilla.com cannot connect to https://ftp.mozilla.org to download builds of Firefox → Regression in mozdownload 1.9 doesn't let all Mozmill CI nodes not connect to https://ftp.mozilla.org via the proxy
Please give me some more information:

A) When did they stop having access?
B) Can they access other sites via HTTPS
C) How about HTTP?
D) How about FTP?

Try the following sites:

http://yahoo.com/
https://google.com/
ftp://ftp.mit.edu/
(Assignee)

Comment 5

5 years ago
Actually we were on 1.7.2 before. So it could also have been the 1.8 release, which I would really blame. Anything related to mozdownload is tracked on that issue now:

https://github.com/mozilla/mozdownload/issues/153

Once it is fixed we can upgrade our mozmill environment to 1.5.22 again.
Summary: Regression in mozdownload 1.9 doesn't let all Mozmill CI nodes not connect to https://ftp.mozilla.org via the proxy → Regression in mozdownload 1.8/1.9 doesn't let all Mozmill CI nodes not connect to https://ftp.mozilla.org via the proxy
(Assignee)

Comment 6

5 years ago
Michael, please see comment 3. It's failing on our side. The network routes are still perfectly setup.
(Assignee)

Comment 7

5 years ago
At least on Linux the wrong proxy settings where present for our new Ubuntu 13.04 VMs. Instead of 'http://proxy.dmz.scl3.mozilla.com:3128' they pointed to 'https://proxy.dmz.scl3.mozilla.com:3128'. The necessary work will be tracked on bug 911795.
Depends on: 911795
(Assignee)

Updated

5 years ago
Depends on: 911816
(Assignee)

Comment 8

5 years ago
I unblocked ourselves here by updating all of our existing VMs and Mac minis today to the correct proxy settings. I wish that we will have that documented and fixed across all the templates too soonish.
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Summary: Regression in mozdownload 1.8/1.9 doesn't let all Mozmill CI nodes not connect to https://ftp.mozilla.org via the proxy → Because of missing or bad proxy settings, mozdownload on mozmill-ci nodes is not able to download Firefox builds from https://ftp.mozilla.org via the proxy
Whiteboard: [automation-blocked]

Updated

4 months ago
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.