Closed Bug 751070 Opened 13 years ago Closed 13 years ago

retire sjc1 VMs

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: standard8, Assigned: bhearsum)

References

Details

Attachments

(2 files)

drop VMs from buildbot-configs
7.21 KB, patch
nthomas
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
remove from puppet-manifests
11.10 KB, patch
nthomas
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
Noticed on the Thunderbird try builds, various Linux builders failing with: Error: unable to free 7.00 GB of space. Free space only 6.45 GB try-linux64-slave03 https://tbpl.mozilla.org/php/getParsedLog.php?id=11381746&tree=Thunderbird-Try try-linux64-slave04 https://tbpl.mozilla.org/php/getParsedLog.php?id=11382816&tree=Thunderbird-Try try-linux64-slave07 https://tbpl.mozilla.org/php/getParsedLog.php?id=11381723&tree=Thunderbird-Try
These old VMs only have 15G of space on /builds, so they're suffering from bumping CCACHE to 10G (bug 703348).
Blocks: 703348
I think we should just retire these VMs. We have 20 ix boxes in scl which we don't normally max out, plus Hal has more VMs to come online in scl3 real soon now. --> try-linux64-slave01 through to try-linux64-slave10 disabled in slavealloc
Agreed. I'll do the deed.
Assignee: nobody → bhearsum
Apparently we're losing all of the sjc1 VMs today and tomorrow. Let's track all of the RelEng work involved there in one place. bug 744452 is the related IT bug.
Status: NEW → ASSIGNED
Summary: Various linux builders can't free enough space, "Error: unable to free 7.00 GB of space. Free space only 6.45 GB" → retire sjc1 VMs
Technically, this leaves us without staging slaves for Thunderbird preproduction/staging linux try. Those should get re-added when the scl3 VMs are added. In the meantime, people can add them locally if they need to.
Attachment #621624 - Flags: review?(nrthomas)
We have to graceful shutdown these VMs first thing tomorrow morning EDT at the very latest.
Component: Release Engineering → Release Engineering: Machine Management
QA Contact: release → armenzg
Attachment #621647 - Flags: review?(nrthomas)
Attachment #621624 - Flags: review?(nrthomas) → review+
Attachment #621647 - Flags: review?(nrthomas) → review+
Comment on attachment 621624 [details] [diff] [review] drop VMs from buildbot-configs Landed on default.
Attachment #621624 - Flags: checked-in+
Comment on attachment 621647 [details] [diff] [review] remove from puppet-manifests Landed, and updated the puppet masters.
Attachment #621647 - Flags: checked-in+
I removed a bunch of sjc1 machines from slavealloc.
made it to production today.
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: