566026 - increase disk space on moz2-linux64-slaveNN VMs

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task, P2)

Description

[[e-mail] (:bhearsum)]

These machines keep running out of disk space when doing builds. Their /builds drive is only 15G, we should bump it up to at least 30G.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Is there any reason we didn't punt this over to Server Ops yet ?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

NB, the staging slave moz2-linux64-slave07 already has a 39G partition at /builds.

Comment 4

[Rail Aliiev [:rail]]

We are going to use these slaves for the upcoming 3.7a5 release (build should be started tomorrow, June 9).

Moving to IT (somehow still assigned to releng) and increasing the importance to blocker.

Known build slaves:

moz2-linux64-slave01
moz2-linux64-slave02
moz2-linux64-slave03
moz2-linux64-slave04
moz2-linux64-slave05
moz2-linux64-slave06
moz2-linux64-slave08
moz2-linux64-slave09
moz2-linux64-slave10
moz2-linux64-slave11
moz2-linux64-slave12

Comment 5

[Phong Tran [:phong]]

Can I just blow away the the current 15 GB drive and add a 40 GB drive?

Comment 6

[Rail Aliiev [:rail]]

Yes, blow them away. Could you leave one of the slaves (let's say moz2-linux64-slave12) not touched. We'll process it after we test the rest slaves.

Comment 7

[Phong Tran [:phong]]

These are done:

moz2-linux64-slave01
moz2-linux64-slave02
moz2-linux64-slave03
moz2-linux64-slave04
moz2-linux64-slave05
moz2-linux64-slave06

Comment 8

[Phong Tran [:phong]]

the rest are done except for slave12 per comment 6.

Comment 9

[Rail Aliiev [:rail]]

Thanks a lot for the prompt action!

All of slaves (except 09, which complains about puppet certificate) are up and running.

Could you do the same thing with slave12? It was idle and I shut it down gracefully from production-master03.build.mozilla.org:8010.

Comment 10

[Rail Aliiev [:rail]]

Notes:

I had to pull the slaves out of production before to say go.

To make the slaves up I did the following using cssh.

# fdisk /dev/sdb (allocate all space for a primary partition #1)
# mkfs.ext3 /dev/sdb1
# mount -a
# cd /builds
# mkdir slave logs && chown cltbld: slave logs
# rsync -av moz2-linux64-slave12:/builds/slave/buildbot.tac slave/
# sed -i "s/moz2-linux64-slave12/`hostname -s`/g" slave/buildbot.tac
# reboot

Comment 11

[Phong Tran [:phong]]

slave12 is also done.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Copying the tac file from slave12 pushed all these slaves onto pm03. I've moved 
* moved 1,2,3,4,6 to pm for m-c/m-1.9.2 (and 5 was already there)
* moved 8 to pm02 for 1.9.1
* left 9,10,11 on pm03 for addonsmgr,places,electrolysis

12 should probably go to pm03 when set up again.

Comment 13

[Rail Aliiev [:rail]]

(In reply to comment #12)
> Copying the tac file from slave12 pushed all these slaves onto pm03. I've moved 
> * moved 1,2,3,4,6 to pm for m-c/m-1.9.2 (and 5 was already there)
> * moved 8 to pm02 for 1.9.1
> * left 9,10,11 on pm03 for addonsmgr,places,electrolysis

Thanks a lot for doing this.

> 12 should probably go to pm03 when set up again.

Set up and attached to pm03.