Closed Bug 1025842 Opened 9 years ago Closed 9 years ago

mock 'archives' are fragile on spot instances


(Release Engineering :: General, defect)

Not set


(Not tracked)



(Reporter: nthomas, Unassigned)



Bug 1024962 was try spot in us-west-2. Today we have bld spot in us-west-2 showing similar symptoms - eg bld-linux64-spot-443, 459, 477, 320, 473, 466 all failing to unpack the root archive mock caches in what look newly launched spot instances. The failure is quick, and the next build is successful.

I had assumed it was bug 1023477 but it seems we have backed out the fs changes there.
seems this hit more and more slaves
Severity: normal → critical
and closed inbound now due to a higher failure rate
(In reply to Carsten Book [:Tomcat] from comment #2)
> and closed inbound now due to a higher failure rate

update 6:20 - all integration trees are closed since this problem is spreading 

affected builders as example
slave: bld-linux64-spot-1002 -

slave: bld-linux64-spot-431 -

slave: bld-linux64-spot-1001 -

slave: bld-linux64-spot-075 -

slave: bld-linux64-spot-175 -
Severity: critical → blocker
I again suspect changes in bug 1023477...
Blocks: 1023477
I'm still seeing failures:

Are these just from old instances?
yes, just checked bld-linux64-spot-092, it was based on one of the "bad" AMIs.
Have we purged all the instances bad on the bad AMI(s) ?
(In reply to Nick Thomas [:nthomas] from comment #8)
> Have we purged all the instances bad on the bad AMI(s) ?
bld-linux64-spot-494 looks to have failed 4 times in a similar manner as ones from

also, might be related, bld-linux64-spot-136 had issues with zip/objcopy step in 'make buildsymbols'.

How can I get the list of 'bad' AMI's so I can verify either of these?

rail - I'm guessing you have ran that script against all known bad ami's so these spot hostnames might be good now?
I think this may be affecting more than just spot instances.

Single locale android repacks - at least the ones that run on ix machines - are failing because they never run "mock install". Eg:

mock install was never run (presumably because this condition failed:, and autoconf213 was never installed.
No longer blocks: 1026480
Blocks: 1025801
Seems to be working better now.
Closed: 9 years ago
Resolution: --- → WORKSFORME
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.