1013631 - Blobber is spamming logs with requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=40044268&tree=Fx-Team#error9 
https://tbpl.mozilla.org/php/getParsedLog.php?id=40044299&tree=Mozilla-Aurora


On the logs for failed runs, they're showing this error:

15:15:47  WARNING - The mochitest suite: mochitest-devtools-chrome-1 ran with return status: WARNING
15:15:47     INFO - Running post-action listener: _resource_record_post_action
15:15:47     INFO - Running post-run listener: _resource_record_post_run
15:15:48     INFO - Total resource usage - Wall time: 1847s; CPU: 15.0%; Read bytes: 81262592; Write bytes: 529007616; Read time: 756914; Write time: 1741755
15:15:48     INFO - pull - Wall time: 13s; CPU: 11.0%; Read bytes: 4440064; Write bytes: 56477184; Read time: 180541; Write time: 868129
15:15:48     INFO - install - Wall time: 2s; CPU: 10.0%; Read bytes: 155648; Write bytes: 86657024; Read time: 2872; Write time: 89721
15:15:48     INFO - run-tests - Wall time: 1833s; CPU: 15.0%; Read bytes: 73904640; Write bytes: 364579328; Read time: 531225; Write time: 494423
15:15:48     INFO - Running post-run listener: _upload_blobber_files
15:15:48     INFO - Blob upload gear active.
15:15:48     INFO - Preparing to upload files from C:\slave\test\build\blobber_upload_dir.
15:15:48     INFO - Files from C:\slave\test\build\blobber_upload_dir are to be uploaded with <fx-team> branch at the following location(s): https://blobupload.elasticbeanstalk.com
15:15:48     INFO - Running command: ['C:\\slave\\test\\build\\venv\\Scripts\\python', 'C:\\slave\\test\\build\\venv\\Scripts\\blobberc.py', '-u', 'https://blobupload.elasticbeanstalk.com', '-a', 'C:\\slave\\test\\oauth.txt', '-b', 'fx-team', '-d', 'C:\\slave\\test\\build\\blobber_upload_dir']
15:15:48     INFO - Copy/paste: C:\slave\test\build\venv\Scripts\python C:\slave\test\build\venv\Scripts\blobberc.py -u https://blobupload.elasticbeanstalk.com -a C:\slave\test\oauth.txt -b fx-team -d C:\slave\test\build\blobber_upload_dir
15:15:49     INFO -  Traceback (most recent call last):
15:15:49     INFO -    File "C:\slave\test\build\venv\Scripts\blobberc.py", line 241, in <module>
15:15:49     INFO -      main()
15:15:49     INFO -    File "C:\slave\test\build\venv\Scripts\blobberc.py", line 227, in main
15:15:49     INFO -      filetype_whitelist = get_server_whitelist(args['--url'])
15:15:49     INFO -    File "C:\slave\test\build\venv\Scripts\blobberc.py", line 65, in get_server_whitelist
15:15:49     INFO -      response = requests.get(url)
15:15:49     INFO -    File "C:\slave\test\build\venv\lib\site-packages\requests\api.py", line 55, in get
15:15:49     INFO -      return request('get', url, **kwargs)
15:15:49     INFO -    File "C:\slave\test\build\venv\lib\site-packages\requests\api.py", line 44, in request
15:15:49     INFO -      return session.request(method=method, url=url, **kwargs)
15:15:49     INFO -    File "C:\slave\test\build\venv\lib\site-packages\requests\sessions.py", line 335, in request
15:15:49     INFO -      resp = self.send(prep, **send_kwargs)
15:15:49     INFO -    File "C:\slave\test\build\venv\lib\site-packages\requests\sessions.py", line 438, in send
15:15:49     INFO -      r = adapter.send(request, **kwargs)
15:15:49     INFO -    File "C:\slave\test\build\venv\lib\site-packages\requests\adapters.py", line 331, in send
15:15:49     INFO -      raise SSLError(e)
15:15:49     INFO -  requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
15:15:49    ERROR - Return code: 1

Comment 2

[Pete Moore [:pmoore][:pete]]

Seems the certificate is self-signed: https://blobupload.elasticbeanstalk.com/

So it seems *not* to be because of a certificate expiring, rather that it is self signed.

Possible reasons I see for this happening:

1) The certificate has changed on the server
2) The home made Certificate Authority has disappeared off the slaves (puppet change?) or is now not getting picked up for some reason
3) Strict certificate checking has been enabled, which wasn't previously there

It may also be something else, these are just the thoughts that come into my mind! :)

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

I'm pretty sure this is a regression from the last deploy of blobber code. There were server and client side changes. The interesting bit is that not all platforms are failing, as there are still lots of 202 responses in the httpd log on the instance serving blobupload.elasticbeanstalk.com.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

On the client side we have
  https://github.com/MihaiTabara/blobuploader/compare/be45ae4481a8...81f8e9424955

The error is in the new get_server_whitelist() function in blobberc.py, the requests.get call doesn't have a verify=cert.where() argument (like we do in post_file).

>>> import requests
>>> from blobuploader import cert

>>> requests.get('https://blobupload.elasticbeanstalk.com/blobs/whitelist')
Traceback (most recent call last):
...
requests.exceptions.SSLError: [Errno 1] _ssl.c:507: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

>>> requests.get('https://blobupload.elasticbeanstalk.com/blobs/whitelist', verify=cert.where())
<Response [200]>

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bump blobuploader version to 1.1.6

Auch, that's my bad. Should've thought of this, sorry.

Fixed the change in blobuploader here: https://github.com/MihaiTabara/blobuploader/commit/c9f32fe0d11beb22b5cc97b9c40f5d0b6351533c

Released new version 1.1.6 for blobuploader - lies as usual here - https://pypi.python.org/pypi/blobuploader

Mozharness patch to bump version here.

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 8426830 [details] [diff] [review]
Bump blobuploader version to 1.1.6

transplanted to production
remote:   https://hg.mozilla.org/build/mozharness/rev/67a6092d975d
remote:   https://hg.mozilla.org/build/mozharness/rev/6bcc9b0f3383

Comment 7

[Justin Wood (:Callek)]

Something here went live today

Comment 8

[(not currently active) Ted Mielczarek]

Seems fixed.