1314925 - Nightlies show error Failed to log stats. Exception = [Errno 185090050] _ssl.c:340: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Carsten Book [:Tomcat]]

06:09:24 INFO - SSLError: [Errno 185090050] _ssl.c:340: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib

https://treeherder.mozilla.org/logviewer.html#?job_id=5479437&repo=mozilla-central

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

The Windows Nightly was also affected by this: https://treeherder.mozilla.org/logviewer.html#?job_id=5482419&repo=mozilla-central

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Normal seems awfully low-priority for total nightly bustage.

Comment 3

[Rail Aliiev [:rail]]

Attachment: Bug 1314925 - Add certifi to vendor libs

Review commit: https://reviewboard.mozilla.org/r/90502/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/90502/

Comment 4

[[e-mail] (:bhearsum)]

Comment on attachment 8807285 [details]
Bug 1314925 - Add certifi to vendor libs

https://reviewboard.mozilla.org/r/90502/#review90220

For posterity, the problem here is that the old balrog submitter code used a specific CA bundle to specify the SSL certificates. When we switched to balrogclient, it started using "verify=True", which appears to tell requests to use certifi instead.

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8807285 [details]
Bug 1314925 - Add certifi to vendor libs

https://hg.mozilla.org/build/tools/rev/f701da5c119e2e4f3eb2ba556ee1d16861805569

Comment 6

[Rail Aliiev [:rail]]

I believe the issue is https://hg.mozilla.org/build/tools/file/tip/lib/python/vendor/requests-2.7.0/requests/certs.py#l22. It points nowhere if certifi is not available.

Comment 7

[Intermittent Failures Robot]

78 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 41
* mozilla-aurora: 37

Platform breakdown:
* osx-10-10: 25
* linux64: 14
* linux32: 14
* windows8-64: 8
* windows8-32: 5
* osx-10-7: 3
* android-4-2-x86: 3
* android-4-0-armv7-api15: 3
* windowsxp: 2
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1314925&startday=2016-11-03&endday=2016-11-03&tree=all

Comment 8

[Rail Aliiev [:rail]]

I see green nightlies and l10n repacks on m-c \o/

Comment 9

[Intermittent Failures Robot]

80 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 41
* mozilla-aurora: 38
* mozilla-inbound: 1

Platform breakdown:
* osx-10-10: 25
* linux32: 15
* linux64: 14
* windows8-64: 8
* windows8-32: 5
* windowsxp: 3
* osx-10-7: 3
* android-4-2-x86: 3
* android-4-0-armv7-api15: 3
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1314925&startday=2016-10-31&endday=2016-11-06&tree=all

Comment 10

[Bill Gianopoulos [:WG9s]]

Android aurora nightlies failed with this error this morning.

Comment 11

[Rail Aliiev [:rail]]

Can you point to he corresponding logs please?

I looked at https://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-08-00-40-19-mozilla-aurora/mozilla-aurora-linux64-nightly-bm74-build1-build8.txt.gz and it shows something different:

02:50:51     INFO -  ConnectionError: ('Connection aborted.', BadStatusLine("''",))

The following log is harmless and will be removed on next uplift by bug 1315104:

00:40:49  WARNING - Failed to log stats. Exception = [Errno 185090050] _ssl.c:340: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib

Comment 12

[Rail Aliiev [:rail]]

looks like balrog-admin is not happy, I get Secure Connection Failed in Firefox.

Comment 13

[Rail Aliiev [:rail]]

Closing this bug, opened bug 1315947.