Closed Bug 1217863 Opened 9 years ago Closed 9 years ago

tst-emulator64 instances are running out of disk space

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: kmoir, Unassigned)

References

Details

Attachments

(1 file)

[puppet] Use cleanup task to handle emulator detritus
542 bytes, patch
kmoir
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
http://buildbot-master68.bb.releng.usw2.mozilla.com:8201/buildslaves/tst-emulator64-spot-329 I think this is kmoir: 10:36 AM <Tomcat|sheriffduty> Callek_cloud: tst-emulator64-spot-329 out of space kmoir: "10:40 AM <Tomcat|sheriffduty> Callek_cloud: and tst-emulator64-spot-025 joined that pary" kmoir I think the problem is that we are getting m3.xlarge (1x32GB) for those instance types and we really want c3.xlarge (2x40) kmoir the configs bid on either instance type kmoir and perhaps the new debug tests or gtests consume all the space kmoir I can't tell because the instances have been terminated now So we either 1) be more aggressive on cleanup 2) restrict instance type to c3.xlarge
Summary: emulator64 instance types are running out of disk space → tst-emulator64 instances are running out of disk space
Also, problem could be that a test is consuming more disk space that it used to.
Other theory could be is that since I enabled so many new Android debug tests earlier this week, the additional load is causing instances to live longer which may reveal a problem since looking at the runner logs it appears purge builds only runs on startup. For instance on tst-emulator64-spot-957, purge builds lastran at 2015-10-22 22:12:46, uptime at 08:33:07 is 10:21,
There's plenty of these now! I've had to close trees for it.
Is there anywhere to check uptime before a job starts and force it to do a purge_builds if uptime is greater than some value (three or four hours, maybe)? That would hopefully confirm or deny comment 2's theory without having to revert whatever changes were made last week.
Flags: needinfo?(kmoir)
Managed to catch tst-emulator64-spot-371 in the act, as it were. * of the 20G on / only 507M was free * about 10G was used in /tmp/android-cltbld * about 50 files of the style /tmp/android-cltbld/emulator-yHgXIL were present, each 220M, going back to 'Oct 26 04:48 PDT' * uptime reported that it had been up continuously since then 'Oct 26 04:42 PDT' * the most recent was in use, and a 'Android armv7 API 9 try opt test jsreftest-2' job was running at the time So, a runner task that does find /tmp/android-cltbld -mmin +30 -delete before buildbot starts would fix this up.
I think this should be OK. The -f arg will prevent errors when that directory is missing, say on non-emulator instances.
Attachment #8679875 - Flags: review?(kmoir)
FTR, I'm not pushing that out urgently because today's troubles are likely due to huge load spike we resulted in the whole emulator pool working flat out for several hours. With that all dealt with, and a bit of a lull for the tree closure, a lot of those instances will recycle themselves and come back with empty /tmp. The trees reopened a few minutes ago.
Flags: needinfo?(kmoir)
Attachment #8679875 - Flags: review?(kmoir) → review+
Attachment #8679875 - Flags: checked-in+
started the process to regenerate the golden ami for this instance type too
new golden ami is available
Seems like the new ami does not have the change and we had failures again today. Looks like there was a puppet error when generating the ami
from #buildduty arr maybe something weird happened in the copy phase arr so things in usw2 won't have the change, but things in use1 will arr (new instances) kmoir that makes sense arr well, I'm not sure how the heck the copy got munged like that, though arr kmoir: when you ran the script by hand, did you include the --copy-to-region bits? kmoir arr: I didn't run it by hand. I just modified the crontab to run it at the time arr ah arr something strange happened, because there are TWO amis with the name spot-tst-emulator64-2015-10-28-09-25 in usw2 arr even though they show separate creation dates arr and there is none for the earlier one arr wonders how the heck that state came about arr kmoir: I've initiated a copy of the good ami from use1 to usw2 kmoir arr: okay thanks
Looks like there are a many instances who have been up for > 1 day and haven't run check_ami for over a day. Thus they don't have the patch and are running out of disk space. Going to open a bug against runner to run check_ami more frequently.
I ended up terminating the 54 old instances still running, as they were continuing to cause grief to sheriffs. Probably we're done with this particular bug, and follow up with bug 1219763 to figure out why the newer ami wasn't being picked up more quickly.
thanks Nick, I should have used the terminate by ami script to delete them. Looking at the instances, it appears runner doesn't run again after buildbot has started which means check_ami doesn't run either for instances that have been up for several days
Looks like amis are being recycled regularly according to the console now that bug 1219763 is fixed. All the current instances for this type are from today.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: