Spidermonkey builds broken with abort: could not find web.cacerts: /etc/mercurial/cacert/cacert.pem

RESOLVED FIXED

Status

--
blocker
RESOLVED FIXED
2 years ago
5 months ago

People

(Reporter: cbook, Unassigned)

Tracking

({intermittent-failure})

Details

(URL)

(Reporter)

Description

2 years ago
abort: could not find web.cacerts: /etc/mercurial/cacert/cacert.pem 

https://treeherder.mozilla.org/logviewer.html#?job_id=5087366&repo=mozilla-central
(Reporter)

Comment 1

2 years ago
closed integration trees for this problem :(
Severity: normal → blocker
Depends on: 1303766
Landed a fix for this (see https://bugzilla.mozilla.org/show_bug.cgi?id=1303766#c25).

Started regenerating all Linux AMIs, will probably take 1.5-2 hours to finish.
Is this related to the android mozilla-central and android nightly fails?
Status update - an additional fix was required in the puppet code; new golden AMIs were started for Windows, but not for Linux. markco is verifying the y-2008 AMI is good, b-2008 is still going, linux AMIs will be started soon.
b-2008 and y-2008 instances are regenerated, verified, and we've removed all instances running on older AMI. Linux AMI generating has been running about 20 minutes (looks like the runtime is about 40 minutes normally).
bld-linux64 golden is done, terminated old instances, new jobs are working for desktop and spidermonkey.
try-linux64 golden seems wedged, investigating.
We have a try-linux64 image in use1, but got the previous image copied to usw2 (bug 1152624, as ami-6b03dc0b). Manually recovered from that, and terminated all old-ami try-linux-spot instances. Almost there.
try-linux64 are running jobs, which is good because they haven't been generated August 18 (bug 1305564 for alerts on that).

At this point I think all the buildbot infra is working OK.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
21 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 11
* mozilla-aurora: 6
* mozilla-inbound: 1
* mozilla-central: 1
* fx-team: 1
* autoland: 1

Platform breakdown:
* windows8-64: 10
* windowsxp: 7
* linux32: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1305406&startday=2016-09-26&endday=2016-09-26&tree=all
status-firefox50: --- → fixed
status-firefox51: --- → fixed
status-firefox52: --- → fixed
21 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 11
* mozilla-aurora: 6
* mozilla-inbound: 1
* mozilla-central: 1
* fx-team: 1
* autoland: 1

Platform breakdown:
* windows8-64: 10
* windowsxp: 7
* linux32: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1305406&startday=2016-09-26&endday=2016-10-02&tree=all

Updated

5 months ago
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.