Permafailing Windows build signing jobs with ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

RESOLVED FIXED

Status

Release Engineering
Buildduty
RESOLVED FIXED
7 months ago
7 months ago

People

(Reporter: KWierso, Assigned: catlee)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [stockwell infra])

(Assignee)

Comment 1

7 months ago
Looks like the SSL certs for the signing servers expired.
Assignee: nobody → catlee

Comment 2

7 months ago
54 failures in 211 pushes (0.256 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 33
* mozilla-inbound: 21

Platform breakdown:
* windows2012-64: 29
* windows2012-32: 24
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1387278&startday=2017-08-03&endday=2017-08-03&tree=all
Autoland and inbound closed 2017-08-03T23:31+00:00 for this. There are some errors on try and beta too because it's a global problem.
(Assignee)

Comment 4

7 months ago
signing4,5,6.build.mozilla.org are updated now, and seem to be taking jobs.

These handle all signing except for OSX DMG signing.
I've reopened autoland and inbound based on green Bs results there.
(Assignee)

Comment 6

7 months ago
mac-v2-signing{1,2,3,4,6,7} have been updated.

I think we're done for now. Will file bugs tomorrow for adding checks for SSL certificate expiry, and a few adjustments to the documentation / tooling.
Status: NEW → RESOLVED
Last Resolved: 7 months ago
Resolution: --- → FIXED

Updated

7 months ago
Whiteboard: [stockwell infra]

Comment 7

7 months ago
69 failures in 888 pushes (0.078 failures/push) were associated with this bug in the last 7 days. 

This is the #17 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 39
* mozilla-inbound: 26
* mozilla-central: 4

Platform breakdown:
* windows2012-64: 36
* windows2012-32: 32
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1387278&startday=2017-07-31&endday=2017-08-06&tree=all
You need to log in before you can comment on or make changes to this bug.