1234143 - increase /data volume size

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Amy Rich [:arr] [:arich]]

The usw2 and use1 puppetmasters ran out of disk space on Friday, and the scl3 ones are close. We need to increase the disk size of each to accommodate more data.

Comment 1

[Amy Rich [:arr] [:arich]]

I shut down usw2 shortly after 0900 UTC to snapshot the /data volume it's still only  at 56%. I waited till about 1230 UTC to shut down use1 so that the golden AMIs could finish. It's just started creating the snapshot now.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I suspect this was rail syncing the new Ubuntu repositories

Comment 3

[Rail Aliiev [:rail]]

ouch... I tried to avoid this by syncing the repo in https://bugzilla.mozilla.org/show_bug.cgi?id=1220658#c47 first to my laptop, checking its size, comparing to releng-puppet2.srv.releng.scl3.mozilla.com, and only after these steps mirroring data to releng-puppet2.srv.releng.scl3.mozilla.com... For some reason I thought, that all puppet masters have identical disk sizes. :/

Sorry about that.

Comment 4

[Amy Rich [:arr] [:arich]]

releng-puppet1.srv.releng.usw2.mozilla.com finally finished it's snapshot after 6.5 hours.

usw-2a
instance id: i-42c0274d
ebs id: vol-b5a370a7
attachment point: /dev/xvdc 
snapshot id: snap-767c972f
start time: 0908 UTC
completion time: 1525 UTC
New volume id: vol-abf5e76a
/dev/mapper/cloud_root-data    411686128 330870620  59896420  85% /data


releng-puppet1.srv.releng.use1.mozilla.com is at 28%

Comment 5

[Amy Rich [:arr] [:arich]]

az: use-1a
instance id: i-81266c6e
ebs id: vol-2852826f
attachment point: 
snapshot id: snap-dc3ac9dd
start time: 1242 UTC
completion time: (more than 7 hours)
New volume id: vol-c3eaa23e

/dev/mapper/cloud_root-data       503835 327483    150754  69% /data

It's syncing up now.