837705 - More disk needed for Socorro dev/stage database systems

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Sheeri Cabral [:sheeri]]

+++ This bug was initially created as a clone of Bug #790690 +++

dev and stage need more disks, too. Production (tp-master01-socorro01 and tp-master02-socorro02) has been updated.

Comment 1

[Laura Thomson :laura]

Moving this to IT so that we actually order these.  Causing problems/pages again.

Comment 2

[Sheeri Cabral [:sheeri]]

Shyam, can we order disks for dev/stage socorro so we can increase the disk space for those environments, since we've upgraded production already.

Comment 3

[Sheeri Cabral [:sheeri]]

dev and stage are 2 different machines...let's get them to the right capacity...given the stats below, Laura what do you think the right capacity is for dev? Stage is the same as production, 1 instance, so we can just order the same capacity as production, but dev has a few things there....so I'm not sure about those.

Right now production looks like this:
/dev/mapper/vg_dbsoc-lv_wal
                       50G   15G   36G  29% /wal
/dev/sdb1             2.5T  724G  1.8T  29% /pgdata

stage looks like this:
/dev/mapper/vg_dbsoc-lv_wal
                       50G   33M   50G   1% /wal
/dev/cciss/c0d0p1     1.1T  665G  453G  60% /pgdata


dev looks like this:
/dev/mapper/vg_dbsoc-lv_wal
                       50G   33M   50G   1% /wal
/dev/cciss/c0d0p1     1.1T  869G  249G  78% /pgdata
10.8.70.226:/vol/pio_symbols
                      2.9T  2.6T  239G  92% /mnt/socorro/symbols

[[e-mail]1 pgdata]# du -sh * | grep G
63G     crash-stats-dev
58G     devdb
688G    pgslave
61G     tmp

Comment 4

[Shyam Mani [:fox2mike]]

Corey can grab this.

Comment 5

[Jake Maul [:jakem]]

Clarifying summary, because it confused me.. we just did prod. :)

Comment 6

[Corey Shields [:cshields]]

what are the dev and stage hostnames so I can find their SNs in inventory?

Comment 7

[Selena Deckelmann :selenamarie :selena]

(In reply to Corey Shields [:cshields] from comment #6)
> what are the dev and stage hostnames so I can find their SNs in inventory?

Here you go: 

socorro1.stage.db.phx1.mozilla.com
socorro1.dev.db.phx1.mozilla.com

Comment 8

[Corey Shields [:cshields]]

Adding Rich to this.

Rich, need to upgrade disks in 2 SB40c's (sn SGI1250012 and SGI125000Y).  Can we hit 3TB in each of those?

Comment 9

[Rich Pomper]

Max capacity for drives is  6 x 1TB either(limited by 2.5 inch form factor)
1TB 6G SAS 7.2K rpm SFF DP Midline HDD
or
HP 1TB 3G SATA 7.2K 2.5in MDL HDD

new Storage Bldes D2200 can do 12 drives of 1 TB SATA or 900GB SAS

Comment 10

[Corey Shields [:cshields]]

Hate SATA, let's go with SAS..   But we can go with 6 easily, do we have to get 1TB drives?  Do you have a 500 or 600GB we can go x6?

Comment 11

[Rich Pomper]

sorry didn't realize you meant 3 TB total (thought it was individual drives).  yes we can get 6 of these for SB4C storage blades
600 GB - hot-swap - 2.5" SFF - SAS-2 - 10000 rpm
Drives are $435 each
Lead time is about 10 days

Comment 12

[Corey Shields [:cshields]]

(In reply to Rich Pomper from comment #11)
> sorry didn't realize you meant 3 TB total (thought it was individual
> drives).  yes we can get 6 of these for SB4C storage blades
> 600 GB - hot-swap - 2.5" SFF - SAS-2 - 10000 rpm
> Drives are $435 each
> Lead time is about 10 days

Great!  Email me a quote for 12 please.  Thanks!

Comment 13

[Sheeri Cabral [:sheeri]]

Ship date is now 4/15. FYI.

Comment 14

[Sheeri Cabral [:sheeri]]

Rich, is there a tracking # for these?

Comment 15

[Sheeri Cabral [:sheeri]]

We should have received these in phoenix already.

Comment 16

[Sheeri Cabral [:sheeri]]

Can this be scheduled into a phoenix DC trip?

1st order
12 drives (6 per box) shipped via the following tracking information
Shipped Via - FEDEX GROUND delivered last Friday
Tracking#(s) - 075570099379942 , 075570099379959



2nd order
Shipped Via - FEDEX GROUND 166454776296887 delivered on Monday

We will be returning one order.

Comment 17

[Derek Moore [:dmoore]]

(In reply to Sheeri Cabral [:sheeri] from comment #16)
> Can this be scheduled into a phoenix DC trip?
> 

:sheeri, we'll have people onsite on 4/25 and 4/29

Comment 18

[Derek Moore [:dmoore]]

4/29 is the next trip.

Comment 19

[Van Le [:van]]

I am on site with the drives. Please ping me in #dcops to coordinate for the upgrade as I am unable to find any DBAs.

Comment 20

[Matt Pressman [:mpressman]]

notified #breakpad that we are starting on stage first

Comment 21

[Van Le [:van]]

I swapped out all 6 drives at once on socorro1.stage.db.phx1.mozilla.com which had unexpected results.

mpressman wants to swap 1 drive at a time on socorro1.dev.db.phx1.mozilla.com to avoid downtime. Informed him that this won't be completed this trip and he was okay with continuing and completing the upgrade next trip.

Comment 22

[Van Le [:van]]

fyi, next scheduled trip is 5/14.

Comment 23

[Van Le [:van]]

These drives are taking a few hours to rebuild after every drive replacement. I was able to swap out 2 drives yesterday and will be swapping out the final 2 today.

Comment 24

[Van Le [:van]]

Completed - 6x 600gb drives swapped into storage array.


[[e-mail]1 ~]$ sudo hpacucli ctrl all show config

Smart Array P400 in Slot 3                (sn: PAFGL0T9S0U00O)


      logicaldrive 1 (1.1 TB, RAID 6 (ADG), OK)
   array A (SAS, Unused Space: 1717342  MB)


      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 600 GB, OK)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 600 GB, OK)
      physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SAS, 600 GB, OK)
      physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SAS, 600 GB, OK)
      physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SAS, 600 GB, OK)
      physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SAS, 600 GB, OK)


Smart Array P410i in Slot 0 (Embedded)    (sn: 5001438006B81890)


      logicaldrive 1 (136.7 GB, RAID 1, OK)
   array A (SAS, Unused Space: 0  MB)


      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)