1266272 - Add a mountpoint on sumotools for /data/backup-drop

Status: RESOLVED FIXED

(Infrastructure & Operations :: Virtualization, task)

Description

[Pythian Team73]

I notice that sumotools disk space is flapping. As sumotools is a virtual machine, we can add a mountpoint and move either MySQL to it (40G) or /data/backup-drop to it (currently 29G), or if we want, move both to it.

I'd recommend moving /data/backup-drop to a new mountpoint as:
1) mountpoints can be added online/hot, so no disruption is needed
2) The existing /data/backup-drop files can be moved to a new directory, and a symlink can be put in place, with no interruption in service.

Comment 1

[Chris Knowles [:cknowles]]

So, normally we've been adding a /data mountpoint ... but obviously, this would disrupt all the things... so some questions:

1) please confirm the full hostname - is this "sumotools1.webapp.phx1.mozilla.com"?
2) Please confirm the size that you'd like.  You mention 40G and 29G, and I'm not sure if that's additive

What I can do is add a disk, format it, and get it ready to add - and then give it back to you so that you can mount it up wherever and do you migration tasks.  (and I'm happy to assist wherever it makes sense to do so) Let me know if that plan makes sense to you.

Comment 2

[Greg Cox [:gcox]]

This box is weird, assuming sumotools1.webapp.phx1.mozilla.com is correct.

1) / is 180G and is only half full as I look at it now.  What's ballooning and causing the flapping, what was done for cleanup, and why do we need to act?  (i.e. got an old bug on this flapping?)

2) /data is not mysql datafiles (which are back in /var/lib/mysql like old-school-usual), but rather dump data.  So whether we make /data or /data/backup-drop is kinda irrelevant to mysql processing, it's just relevant to these data dumps.

Also irrelevant, is the timing:
[[e-mail]1 /]# find /data -mmin -180 ; date
Thu Apr 21 05:41:45 PDT 2016

That's 3 hours of nothing happening in /data.  Plenty of time to change any part of /data that warrants it.

3) If we fork off a new area, be it for /data or /var/lib/mysql or whatever, I'm going to want to claw back some of that root vol space.  Which means downtime.  So, let's figure out all of what this box should look like, and go from there.

Comment 3

[Pythian Team73]

Hi,

Please add 200GB space as /data partition.
Please let us know when done.

Thanks,
Rohit Kumar

Comment 4

[Greg Cox [:gcox]]

Assuming you meant sumotools1.webapp.phx1, and assuming you're not going to answer anything about why you want this when it's shrinking so quickly, done.  Copied all the data over, added to fstab, mounted, removed the old data from the root partition.

Comment 5

[Pythian Team73]

Hi Greg,

Thanks for update.
Apologies for less details in last post.
We need the new mount to keep the backups, as when backups file count increase during the month, we had to remove few to keep the space in control as MySQL database was also on same mount.

I checked the new mounts and seems good.

Thanks,
Rohit Kumar