1317987 - Intermittent Caught exception: <urlopen error [Errno 1] _ssl.c:504: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Treeherder Bug Filer]

Filed by: tomcat [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=6717517&repo=autoland

https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-macosx64-debug/1479298173/autoland_yosemite_r7-debug_test-mochitest-gl-e10s-1-bm106-tests1-macosx-build176.txt.gz

Comment 1

[Carsten Book [:Tomcat]]

seems this hit autoland and other trees

Comment 2

[Carsten Book [:Tomcat]]

closed integration trees now

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

who is working on this?  if the trees are closed for this bug I would expect to see a link to a bug, or a ni? or a comment about a specific person working on something.

Comment 4

[Carsten Book [:Tomcat]]

pinged people on irc but no response so far, so trying here

Comment 5

[Chris Cooper [:coop] (he/him)]

This is affecting Mac testers and all three flavors of Windows testers.

Comment 6

[Amy Rich [:arr] [:arich]]

The actual error appears to be coming from contacting queue.taskcluster.net similar to in bug 1260584.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Bug 1260548 was about 5xx errors from cloudfront

Comment 8

[Greg Arndt [:garndt]]

I have opened up a support ticket with Heroku to find out if they have changed anything with our SSL endpoint that we were not aware of that could break this.

It appears to be an issue with hosts using sslv3 to connect. 

9:14 AM <@daveio> and as expected openssl s_client -ssl3 -connect queue.taskcluster.net:443 fails
9:16 AM <@daveio> the build job needs to be changed to use at least tlsv1, ideally tlsv1.2

Nothing has changed with these jobs recently, so it points to something on the heroku end.

Comment 9

[April King [:April]]

There is no SSLv3 on this system; don't know if there was before:

1    ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2               ECDH,P-256,256bits prime256v1
2    ECDHE-RSA-AES128-SHA256     TLSv1.2               ECDH,P-256,256bits prime256v1
3    ECDHE-RSA-AES128-SHA        TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1
4    ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2               ECDH,P-256,256bits prime256v1
5    ECDHE-RSA-AES256-SHA384     TLSv1.2               ECDH,P-256,256bits prime256v1
6    ECDHE-RSA-AES256-SHA        TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1
7    AES128-GCM-SHA256           TLSv1.2               None
8    AES128-SHA256               TLSv1.2               None
9    AES128-SHA                  TLSv1,TLSv1.1,TLSv1.2 None
10   AES256-GCM-SHA384           TLSv1.2               None
11   AES256-SHA256               TLSv1.2               None
12   AES256-SHA                  TLSv1,TLSv1.1,TLSv1.2 None
13   DES-CBC3-SHA                TLSv1,TLSv1.1,TLSv1.2 None

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Retriggered jobs are green.

Joel was able to successfully make a request on a 10.10 loaner using urllib.

I don't think this is SSL-related.  I think it was a transient network issue.

Comment 11

[Gregory Szorc [:gps]]

I /think/ that "SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure" message is a bit misleading. As far as protocol selection is concerned in OpenSSL, there are constants with "SSLv23" or "SSL23" in them that actually mean "use the newest version SSL+TLS that both ends support."

If we're actually using SSL (and not TLS) in automation, that's a security issue and any services still using SSL deserve to break.

Comment 12

[Greg Arndt [:garndt]]

Heroku has investigated and confirmed that nothing has changed with our SSL endpoint recently.  Also, they confirmed that SSLv3 is indeed disabled just to avoid confusion in the future.  There analysis points to a network issue.  If we have any additional logs to provide them, please let me know.

These failures seem concentrated 100% to windows and mac running out of scl3 or route their traffic through scl3 (like the win7vm instances).

Comment 13

[Greg Arndt [:garndt]]

Also, trees have been reopened as of 2016-11-16T18:05:57+00:00

Comment 14

[Intermittent Failures Robot]

468 failures in 119 pushes (3.933 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* autoland: 466
* mozilla-central: 2

Platform breakdown:
* osx-10-10: 244
* windows8-64: 87
* windows7-32-vm: 87
* windows7-32: 39
* linux32: 5
* android-4-3-armv7-api15: 4
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317987&startday=2016-11-16&endday=2016-11-16&tree=all

Comment 17

[Intermittent Failures Robot]

479 failures in 715 pushes (0.67 failures/push) were associated with this bug in the last 7 days. 

This is the #1 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* autoland: 466
* mozilla-central: 8
* mozilla-aurora: 5

Platform breakdown:
* osx-10-10: 251
* windows7-32-vm: 88
* windows8-64: 87
* windows7-32: 43
* linux32: 5
* android-4-3-armv7-api15: 3
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317987&startday=2016-11-14&endday=2016-11-20&tree=all

Comment 18

[Alin Selagea [:aselagea]]

No other occurrence since Nov 16, so it likely was a temporary network issue. Will mark as resolved for now, please feel free to reopen if needed.