898574 - Nodes in *.qa.scl3.mozilla.com cannot reach external machines via the proxy

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[u279076]

Trying to run the betatest update tests for Firefox 23.0b9 today all tests are failing.

mozdownload is returning an error:
> urllib2.URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>

You can see a sample report here:
> http://mm-ci-master.qa.scl3.mozilla.com:8080/job/ondemand_update/16090/console

The config file we are using is here:
> https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx23b9#Betatest

This is a blocker to signing off Firefox 23.0b9 today.

Comment 1

[u279076]

Actually, after testing updates manually I can see there are no updates on betatest. Would this affect mozdownload?

https://aus3.mozilla.org/update/3/Firefox/23.0/20130722172257/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/betatest/Darwin%2010.8.0/default/default/update.xml?force=1

Comment 2

[Aki Sasaki (not active)]

We haven't received an "Updates available on betatest" for 23.0b9 yet.
I know at least one win32 l10n repack job is still running.

Comment 3

[u279076]

I filed bug 898587 to investigate the missing updates.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Well, the problem here is that the proxy has been stopped working. We cannot connect to any external host because the proxy is down. Here an example via wget:

$ wget https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/23.0b6/linux-x86_64/en-US/firefox-23.0b6.tar.bz2
--2013-07-26 14:05:02--  https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/23.0b6/linux-x86_64/en-US/firefox-23.0b6.tar.bz2
Resolving proxy.dmz.scl3.mozilla.com (proxy.dmz.scl3.mozilla.com)... 10.22.74.42
Connecting to proxy.dmz.scl3.mozilla.com (proxy.dmz.scl3.mozilla.com)|10.22.74.42|:3128... connected.
OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Unable to establish SSL connection.

As Anthony said, please get this fixed because it's blocking the QA signoff for the beta we want to release today.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Also a reference for a download with curl and verbose output

$ curl https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/23.0b6/linux-x86_64/en-US/firefox-23.0b6.tar.bz2 -vvv >a
* About to connect() to proxy proxy.dmz.scl3.mozilla.com port 3128 (#0)
*   Trying 10.8.74.76...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* connected
* Connected to proxy.dmz.scl3.mozilla.com (10.8.74.76) port 3128 (#0)
* Establish HTTP proxy tunnel to ftp.mozilla.org:443
> CONNECT ftp.mozilla.org:443 HTTP/1.1
> Host: ftp.mozilla.org:443
> User-Agent: curl/7.27.0
> Proxy-Connection: Keep-Alive
> 
* Easy mode waiting response from proxy CONNECT
< HTTP/1.0 403 Forbidden
< Server: squid/3.1.10
< Mime-Version: 1.0
< Date: Fri, 26 Jul 2013 21:26:51 GMT
< Content-Type: text/html
< Content-Length: 3168
< X-Squid-Error: ERR_ACCESS_DENIED 0
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from proxy1.dmz.phx1.mozilla.com
< X-Cache-Lookup: NONE from proxy1.dmz.phx1.mozilla.com:3128
< Via: 1.0 proxy1.dmz.phx1.mozilla.com (squid/3.1.10)
< Connection: keep-alive
< 
* Received HTTP code 403 from proxy after CONNECT
* Closing connection #0
curl: (56) Received HTTP code 403 from proxy after CONNECT

Comment 6

[Michael Henry [:tinfoil]]

Been working with :ashughes in private.  The issue appears resolved, but waiting on :ashughes final input (in progress)

Comment 7

[u279076]

All our testruns now successfully download and stage builds. We still fail because there are no updates available but that's a separate bug. I'm calling this fixed.

Thanks for your help Michael.

Comment 8

[Michael Henry [:tinfoil]]

Thanks, I'll update with a postmortum shortly.

Comment 9

[u279076]

(In reply to Michael Henry [:tinfoil] from comment #8)
> Thanks, I'll update with a postmortum shortly.

Michael, Release Engineering was asking me today for the post-mortem notes on this so it doesn't happen again. Unfortunately I wasn't in a position to give any good answers. Can you follow up with an email to [e-mail] with the details soonest?

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Michael, this would be really an important information for us. We want to raise a bug to get a Nagios check implemented, and would need the information. Please let us know ASAP. Thanks.

Comment 11

[Michael Henry [:tinfoil]]

Honestly, I cannot answer.  I was using the extra time to try to figure it out, but cannot.
It seemed to be an issue with how the zeus load balancers interacted with the squid proxies. Unfortunately, it didn't generate any errors to help isolate the issue.

The issue is likely not to occur again as I took the zeus load balancers out of the mix for further testing before they are re-added.

The responsibility of the proxies is currently being transitioned to IT, and I'm creating a bug to furrther investigate Zeus/Squid interaction before they can be re-introduced.  There will also be a testing and change controll process before they can be re-added.

I've also ensured that the proxies are being monitored for traffic issues with nagios (they were), and have taken the time to document a proper response procedure should an outtage occur.  The drafts are being reviewed and will be put into mana later today.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Michael Henry [:tinfoil] from comment #11)
> The responsibility of the proxies is currently being transitioned to IT, and
> I'm creating a bug to furrther investigate Zeus/Squid interaction before
> they can be re-introduced.  There will also be a testing and change controll
> process before they can be re-added.

Can you please CC me on that bug given my strong interest in getting this not happening again? Thanks.

> I've also ensured that the proxies are being monitored for traffic issues
> with nagios (they were), and have taken the time to document a proper
> response procedure should an outtage occur.  The drafts are being reviewed
> and will be put into mana later today.

It would be great if you can supply an URL once the mana page has been updated.