1230476 - Download failures (CERTIFICATE_VERIFY_FAILED) for HTTPS connections for some Windows nodes in qa.scl3.mozilla.com

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: log output

We see some kind of bustage in downloading the Firefox installer for the last beta candidate builds of Firefox:

> caught exception: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

This is only for Windows but no other platform and also only for beta and maybe also release candidate builds. I haven't seen this yet for Nightly or Aurora builds. I wonder if there is something special with the candidates folder compared to nightlies.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So this looks a lot like:
http://stackoverflow.com/questions/27804710/python-urllib2-ssl-error/27826829#27826829.

It started since we no longer use mozdownload (requests library) to download the builds but mozharness which is using urllib2.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Here the results of the latest Firefox 43.0b9 build 1 candidate:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=730297e45977&filter-searchStr=Firefox%20UI&exclusion_profile=false

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So this is clearly a problem in Python and with urllib2. Running curl as external command or using urllib is working just fine:

>>> import urllib2
>>> urllib2.urlopen('https://archive.mozilla.org/pub/mozilla.org/firefox/candida
tes/43.0b9-candidates/build2/win64/ar/Firefox%20Setup%2043.0b9.exe')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\Python27\lib\urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "c:\Python27\lib\urllib2.py", line 431, in open
    response = self._open(req, data)
  File "c:\Python27\lib\urllib2.py", line 449, in _open
    '_open', req)
  File "c:\Python27\lib\urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "c:\Python27\lib\urllib2.py", line 1240, in https_open
    context=self._context)
  File "c:\Python27\lib\urllib2.py", line 1197, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate ve
rify failed (_ssl.c:581)>
>>> import urllib
>>> urllib.urlopen('https://archive.mozilla.org/pub/mozilla.org/firefox/candidat
es/43.0b9-candidates/build2/win64/ar/Firefox%20Setup%2043.0b9.exe')
<addinfourl at 41691696 whose fp = <socket._fileobject object at 0x0231BBF0>>
>>>

Python version installed on the nodes is:

Python 2.7.9 (default, Dec 10 2014, 12:24:55) [MSC v.1500 32 bit (Intel)] on win
32

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Looks like for nightly builds we don't use a certificate?

$ curl -vvv https://archive.mozilla.org/pub/mozilla.org/firefox/candidates/43.0
b9-candidates/build2/win64/ar/Firefox%20Setup%2043.0b9.exe >setup.exe -I
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*
  Trying 10.22.74.73...
* Connected to proxy.dmz.scl3.mozilla.com (10.22.74.73) port 3128 (#0)
* Establish HTTP proxy tunnel to archive.mozilla.org:443
> CONNECT archive.mozilla.org:443 HTTP/1.1
> Host: archive.mozilla.org:443
> User-Agent: curl/7.41.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.0 200 Connection established
<
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
*   CAfile: C:\Program Files (x86)\Git\bin\curl-ca-bundle.crt
  CApath: none
* TLSv1.0, TLS handshake, Client hello (1):
} [95 bytes data]
* TLSv1.0, TLS handshake, Server hello (2):
{ [53 bytes data]
* TLSv1.0, TLS handshake, CERT (11):
{ [2830 bytes data]
* TLSv1.0, TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.0, TLS handshake, Client key exchange (16):
} [262 bytes data]
* TLSv1.0, TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.0, TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.0, TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.0, TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.0 / AES256-SHA
* Server certificate:
*        subject: C=US; ST=California; L=Mountain View; O=Mozilla Corporation; C
N=ftp.mozilla.org
*        start date: 2015-07-29 00:00:00 GMT
*        expire date: 2016-12-31 12:00:00 GMT
*        subjectAltName: archive.mozilla.org matched
*        issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*        SSL certificate verify ok.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0>
HEAD /pub/mozilla.org/firefox/candidates/43.0b9-candidates/build2/win64/ar/Firef
ox%20Setup%2043.0b9.exe HTTP/1.1
> User-Agent: curl/7.41.0
> Host: archive.mozilla.org
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/x-msdownload
< Content-Length: 47503920
< Connection: keep-alive
< Accept-Ranges: bytes
< Date: Fri, 04 Dec 2015 05:11:41 GMT
< ETag: "dddd3b976623a8a8b9c9057a03016619"
< Last-Modified: Fri, 04 Dec 2015 04:28:42 GMT
< x-amz-replication-status: COMPLETED
< x-amz-version-id: lW5T_7e0cAMnxmomP73QP37luWe.lAUo
< Age: 17028
< X-Cache: Hit from cloudfront
< Via: 1.1 5f661f96cafca5f46332cba7329de62c.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: L2GRouNrW31qcVMjLExlSiMCUn7cCL_AOB2dk0aNFYY2S7OFwqzixQ==
<
  0 45.3M    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host proxy.dmz.scl3.mozilla.com left intact


mozauto@MM-WIN-7-64-3 /C/jenkins/workspace/release-mozilla-beta_functional
$ curl -vvv -I http://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora
/firefox-44.0a2.en-US.win32.installer.exe
*   Trying 10.22.74.73...
* Connected to proxy.dmz.scl3.mozilla.com (10.22.74.73) port 3128 (#0)
> HEAD http://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora/firefox-
44.0a2.en-US.win32.installer.exe HTTP/1.1
> User-Agent: curl/7.41.0
> Host: ftp.mozilla.org
> Accept: */*
> Proxy-Connection: Keep-Alive
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
HTTP/1.0 200 OK
< Content-Type: application/x-msdownload
Content-Type: application/x-msdownload
< Content-Length: 43527840
Content-Length: 43527840
< Date: Fri, 04 Dec 2015 10:03:32 GMT
Date: Fri, 04 Dec 2015 10:03:32 GMT
< x-amz-replication-status: COMPLETED
x-amz-replication-status: COMPLETED
< x-amz-version-id: IRAtVcu1FkQJoCkuv5CJUrikBQmpVToL
x-amz-version-id: IRAtVcu1FkQJoCkuv5CJUrikBQmpVToL
< Cache-Control: max-age=3600
Cache-Control: max-age=3600
< Last-Modified: Thu, 03 Dec 2015 12:09:16 GMT
Last-Modified: Thu, 03 Dec 2015 12:09:16 GMT
< ETag: "c507e8f8fe81db322748f1a63436da54"
ETag: "c507e8f8fe81db322748f1a63436da54"
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Server: AmazonS3
Server: AmazonS3
< X-Cache: Miss from cloudfront
X-Cache: Miss from cloudfront
< X-Amz-Cf-Id: 5UcNg6Q6UbtJkvCwMtoIM6y-4oqfo24MUiPeOvOFWGu86XYY_5fT3g==
X-Amz-Cf-Id: 5UcNg6Q6UbtJkvCwMtoIM6y-4oqfo24MUiPeOvOFWGu86XYY_5fT3g==
< X-Cache: MISS from proxy1.dmz.scl3.mozilla.com
X-Cache: MISS from proxy1.dmz.scl3.mozilla.com
< X-Cache-Lookup: MISS from proxy1.dmz.scl3.mozilla.com:3128
X-Cache-Lookup: MISS from proxy1.dmz.scl3.mozilla.com:3128
< Via: 1.1 0b23a780329adf733747676dfab6cc26.cloudfront.net (CloudFront), 1.0 pro
xy1.dmz.scl3.mozilla.com (squid/3.1.23)
Via: 1.1 0b23a780329adf733747676dfab6cc26.cloudfront.net (CloudFront), 1.0 proxy
1.dmz.scl3.mozilla.com (squid/3.1.23)
* HTTP/1.0 connection set to keep alive!
< Connection: keep-alive
Connection: keep-alive

<
* Connection #0 to host proxy.dmz.scl3.mozilla.com left intact

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So it's actually not happening for all Windows hosts! Only some:

* mm-win-7-64-1.qa.scl3.mozilla.com 
* mm-win-7-64-3.qa.scl3.mozilla.com 
* mm-win-81-32-2.qa.scl3.mozilla.com 
* mm-win-81-32-3.qa.scl3.mozilla.com 
* mm-win-81-64-2.qa.scl3.mozilla.com 

I checked the installed root certificates via the certificate manager and it looks like some invalid certs are installed. I will attach two screenshots in a minute.

So moving to Mozilla QA / Infrastructure to get this fixed. Mozharness seem to work fine.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: screenshot_invalid.png

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: screenshot_valid.png

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Removing the COMODO certificate didn't help but the DigiCert Global Root CA is missing on those boxes. I installed it on one of those and everything works.

https://www.digicert.com/CACerts/DigiCertGlobalRootCA.crt

I wonder why it is not part of all Windows installations. All machines have the same patch level. Maybe a former system update was ignored?

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Here some more information in how to get it fixed:

http://answers.microsoft.com/en-us/windows/forum/all/windows-7-64-bit-not-being-offered-kb931125-roots/aae0741d-df61-4a28-aaa3-d77c80929af1?auth=1
https://community.spiceworks.com/topic/584022-missing-or-invalid-root-certificates-on-windows-7-64bit

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hm, this tool installs about 360 certificates on the machine. Most of those are only intended for server use. So I think for now I will simply manually install the missing DigiCert Global Root CA on all the machines. We should keep an eye out for other certificate issues in the future. It may really be a Windows related update problem, maybe caused that we update our machines after longer intervals.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I have updated all machines for the DigiCert Global Root certificate now and verified that I can open HTTPS locations with urllib2.urlopen().

Dustin, have you ever noticed such a thing that certificates are not getting updated on a Windows host? Maybe you have a better reference for us to get the underlying issue fixed.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Python 2.7.9 also can't use SNI, if I remember correctly.  I think we use puppet to install those certificates, but I'm not sure.

Comment 13

[Chris AtLee [:catlee]]

Python 2.7.9+ does support SNI, according to http://docs.python-requests.org/en/latest/community/faq/#what-are-hostname-doesn-t-match-errors

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Just to clarify it... this is not a problem with Python but a missing root certificate on Windows slave nodes. Usually those certs are getting updated and deleted by Windows updates, but that doesn't work for us with manual system updates.

Comment 15

[Mark Cornmesser [:markco]]

(In reply to Dustin J. Mitchell [:dustin] from comment #12)
> Python 2.7.9 also can't use SNI, if I remember correctly.  I think we use
> puppet to install those certificates, but I'm not sure.

Puppet for builders, GPO for testers.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, looks like getting an answer why Windows update is misisng to install those root certificates automatically is hard.

I will close this bug as fixed for now. If other issues are popping up for certificate failures, I will open a new bug. Thanks for your feedback.