572578 - Expand disk space on khan

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Wil Clouser [:clouserw]]

We talked a bit in IRC and I said I'd file this bug.  It sounds like we could add an array to khan or mrz wanted to try SAS via NFS.  I don't know what the conclusion was, but it sounded like we should do something. :)

Comment 1

[matthew zeier [:mrz]]

Phong - price out an SB40c with 6x300GB SAS drives and with 6x146GB.  Wil's concerned about NFS performance so not sure if that's the best way to go.

Comment 2

[Wil Clouser [:clouserw]]

I'm open minded, I've never used SAS+NFS.  I think I'd be more comfortable if we had a reliable way to monitor what was slow.  Benchmarking current disk vs NFS would be good.  Are the NFS mounts in /data SAS?

Comment 3

[matthew zeier [:mrz]]

No, all off SATA backed storage.

Aravind might be able to allocate some faster NFS space for you get benchmark.

Comment 4

[Phong Tran [:phong]]

Did we decide that NFS will work for this or should I order the storage blade?

Comment 5

[Shyam Mani [:fox2mike]]

mrz says storage blade, punting back to phong.

Comment 6

[Phong Tran [:phong]]

sb40 with 6x300 GB drives on order.

Comment 7

[Phong Tran [:phong]]

the storage has arrived.  we will need to schedule a downtime to move this blade to a different chassis that has space for the storage blade.

Comment 8

[matthew zeier [:mrz]]

guys, can we get this scheduled?

Comment 9

[Wil Clouser [:clouserw]]

The main concern of the webdev team is just getting notice - we're flexible on the actual time it happens.  If you want a time, I'll throw out Thursday @ 6pm, but we're easy.

Comment 10

[Phong Tran [:phong]]

This requires us to physically move khan to a different blade chassis that has space for the storage blade to be right next to it.  The other alternative is to get shaver's okay for downtime for sm-valgrind01 and move that blade instead.

Comment 11

[Justin Dow [:jabba]]

I can move the blade to the other chassis tonight at 6pm. Once that is done, the rest should be able to be done remotely. If we need to move /home to the new storage blade, we might need an additional downtime to make that move, but should be short and trivial.

Comment 12

[matthew zeier [:mrz]]

=> ctrl slot=3 show config

Smart Array P400 in Slot 3    (sn: PAFGL0T9SYP03N)

   array A (SAS, Unused Space: 0 MB)

      logicaldrive 1 (1.4 TB, RAID 5, OK)

      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 300 GB, OK)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 300 GB, OK)
      physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SAS, 300 GB, OK)
      physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SAS, 300 GB, OK)
      physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SAS, 300 GB, OK)
      physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SAS, 300 GB, OK)

[root@khan ~]# fdisk /dev/cciss/c1d0 

Unable to read /dev/cciss/c1d0
[root@khan ~]# fdisk /dev/cciss/c1d0

/var/log/messages shows kernel messages that make me think something broke in the driver.  

When can khan go down for a quick reboot?

Comment 13

[Wil Clouser [:clouserw]]

6pm tonight?

Comment 14

[matthew zeier [:mrz]]

Missed the reboot and folks are logged in.  If everyone can jump offline I can do it over the weekend.

Comment 15

[matthew zeier [:mrz]]

That reboot worked.  Working on copying /home to this new drive.  

Hopefully everyone can refrain from logging back in until this is done.

Comment 16

[matthew zeier [:mrz]]

Ran the large rsync yesterday.  Just need a window to move /home over.  Should be < 30 mins.

Comment 17

[Wil Clouser [:clouserw]]

monday @ 6pm?  or 7:30am?

Comment 18

[Justin Dow [:jabba]]

I can do 7:30am if it's not too late to make the announcement. Let me know.

Comment 19

[Justin Dow [:jabba]]

Didn't hear anything, so let's do this at 6pm tonight.

Comment 20

[Justin Dow [:jabba]]

This is done now. The old /home is still around under /home.old, but I want to remove that as soon as I get an "ok" from webdev. I did rsync -a on it, so everything should be there.

Comment 21

[Wil Clouser [:clouserw]]

Just sent to webdev@

> Hey,
> 
> IT finished expanding /home on khan today.  The old directory is in /home.old.
> It should be the same, but double check if you're missing anything.  If you 
> don't say anything by the end of the week, they're going to whack it.
> 
> Wil

Comment 22

[Justin Dow [:jabba]]

/home.old has been nuked and there is now plenty of disk space to go around.