684245 - Clear up space on dm-symbolpush01

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Ashish Vijayaram [:ashish]]

dm-symbolpush01 is out of space on /mnt/netapp/breakpad:

07:47:57 < nagios-sjc1> [24] dm-symbolpush01:disk - /mnt/netapp/breakpad is WARNING: DISK WARNING - free space: /mnt/netapp/breakpad 215104 MB (10% inode=94%):

Comment 1

[(not currently active) Ted Mielczarek]

I didn't get a chance to really do anything here today, and I'm probably not going to get anything done this weekend. If it keeps getting worse, can we allocate some more space in the interim? I know this keeps growing, but we keep making more builds...

Comment 2

[Ashish Vijayaram [:ashish]]

I doubt we would be able to add any space especially over the (long) weekend. However, I'm cc'ing jakem and phong who'd know better about increasing Netapp storage.

10.253.0.11:/vol/socorro/symbols is mounted on /mnt/netapp/breakpad

Comment 3

[Jake Maul [:jakem]]

Unfortunately growing this isn't as clean as some scenarios.

The problem is this volume is already at the maximum size for deduplication. We can go bigger, but new data will not be deduplicated anymore. That means you're future space usage will be 25% faster than it is right now... that's how much deduplication is saving us at the moment, 581GB.

Doubly unfortunate, when a volume is grown beyond this size limitation, deduplication can never be re-enabled, even if the volume is subsequently shrunk back down under 2TB. So I can't raise it today to get through the weekend, and then go back to normal next week after it's been cleaned up.

I'd prefer to follow a "wait and see" approach for the weekend. If it becomes absolutely critical we can grow the volume, but I'd rather not take the efficiency hit if we can help it.

This is something that's likely to get better later this year when we move into the new datacenter: the newer filers have higher deduplication limits, and I believe we'll have more disk space to begin with.

Comment 4

[(not currently active) Ted Mielczarek]

Ok, thanks for the info. It's going to take me a little bit to figure out what's going on here, it might just be that disk usage has been creeping up, and the spate of releases this week pushed it over the top. I've also filed bug 684251 for getting a better long-term solution for this.

Comment 5

[(not currently active) Ted Mielczarek]

Just for reference:

Filesystem            Size  Used Avail Use% Mounted on

10.253.0.11:/vol/socorro/symbols
                      1.3T  852G  401G  69% /mnt/netapp/breakpad


[tmielczarek@dm-symbolpush01 symbols_ffx]$ du -sh .
631G    .

Comment 6

[Jake Maul [:jakem]]

Thanks!

Comment 7

[(not currently active) Ted Mielczarek]

I didn't actually do anything, I was just trying to measure what's there. Did anyone else fix this, or did we just have a lot of stuff that got cleaned up over the weekend and got us back down to acceptable levels?

Comment 8

[Jake Maul [:jakem]]

You're right, this isn't fixed. I don't know when you took your "df -h" snapshot, but it's nothing like that now:

[root@dm-symbolpush01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       18G  5.2G   12G  32% /
/dev/sda1              99M   26M   69M  27% /boot
tmpfs                 252M     0  252M   0% /dev/shm
10.253.0.11:/vol/socorro/symbols
                      2.0T  1.8T  225G  89% /mnt/netapp/breakpad

Comment 9

[(not currently active) Ted Mielczarek]

*sigh* Really old scrollback in a screen session, apparently! Sorry for the confusion.

Comment 10

[(not currently active) Ted Mielczarek]

[tmielczarek@dm-symbolpush01 symbols_tbrd]$ du -sh
308G    .

Comment 11

[(not currently active) Ted Mielczarek]

Just to note, last time I was looking into this:

[tmielczarek@dm-symbolpush01 symbols_ffx]$ du -sh .
230G    .
[tmielczarek@dm-symbolpush01 symbols_tbrd]$ du -sh .
112G    .

(from that same scrollback). So we've over doubled the size of both Firefox and Thunderbird symbol directories. :-/

Comment 12

[(not currently active) Ted Mielczarek]

[tmielczarek@dm-symbolpush01 symbols_mob]$ du -sh
405G    .


405GB of mobile symbols. I think I know where to start looking.

Comment 13

[(not currently active) Ted Mielczarek]

So it looks like most of the space here is from the rapid release change. We stranded a bunch of old version numbers (like Firefox 4.2). Currently the script will clean those up after 180 days, but they're only 4-5 months old right now, so that would take a while. I pushed a change to bump that number down to 90 days, which should result in a fair amount of cleanup happening:
http://hg.mozilla.org/build/tools/rev/0e576701f123

-> jakem to update the script on dm-symbolpush01

Comment 14

[Jake Maul [:jakem]]

I found where the hg repo lives on dm-symbolpush01 and updated it, then copied this script to where cron expects it to be. It's running now.


Abbreviated update output (irrelevant parts snipped):

[root@dm-symbolpush01 /]# cd ./opt/symbol_tools/build_tools/
[root@dm-symbolpush01 build_tools]# cd buildfarm/
[root@dm-symbolpush01 buildfarm]# cd breakpad/

[root@dm-symbolpush01 breakpad]# hg pull
pulling from http://hg.mozilla.org/build/tools
searching for changes
adding changesets
adding manifests
adding file changes
added 377 changesets with 679 changes to 142 files
(run 'hg update' to get a working copy)

[root@dm-symbolpush01 breakpad]# hg update
140 files updated, 0 files merged, 4 files removed, 0 files unresolved

Comment 15

[Jake Maul [:jakem]]

There are 2 symbols_ directories that are not currently being cleaned up. I want to verify if this is on purpose or by accident:

symbols_mob - 405GB
symbols_os - 20GB

Our current cron job specifically enumerates directories to clean, like so:

DIRECTORIES="camino fedora ffx mod opensuse penelope sbrd sea solaris tbrd ubuntu xr"

Note that 'mod' doesn't exist (typo perhaps?), and 'mob' and 'os' are not in the list.

Would it be safe to transform this into a simple "find" command that selects all symbols_* directories instead? That seems more future-proof to me.

Comment 16

[(not currently active) Ted Mielczarek]

We do not want to cleanup symbols_os, those are OS-level symbols that we'll want to persist. We do want to cleanup symbols_mob, it sounds like "mod" is probably a typo.

Comment 17

[Jake Maul [:jakem]]

Sounds good... I commented our cron job about the symbols_os exclusion, and fixed the apparent typo for mod/mob. Thanks!

I'm rerunning now, but this bug doesn't need to hang around waiting for that. Closing it out again.

Comment 18

[(not currently active) Ted Mielczarek]

Looks like we've already made a dent:
10.253.0.11:/vol/socorro/symbols
                      2.0T  1.6T  384G  81% /mnt/netapp/breakpad


I suspect cleaning up symbols_mob will help a lot too.

Comment 19

[Jake Maul [:jakem]]

Ted, would you be willing to revert some of the changes in your update? Specifically the stdout->stderr changes. This is now emailing us daily, with no actionable information. The cron job throws away stdout output, so it's safe to dump anything there. We could throw away stderr output as well, but I'd really rather be notified if something goes wrong.

Comment 20

[(not currently active) Ted Mielczarek]

I pushed a fix. Sorry about that, I didn't think that through very well.

Comment 21

[Jake Maul [:jakem]]

Excellent, thanks. I've pulled the new revision out of the repo and put it into place.

Thanks for the help on this issue. We've made a pretty big dent in the usage:


Filesystem            Size  Used Avail Use% Mounted on
10.253.0.11:/vol/socorro/symbols
                      2.0T  1.4T  585G  71% /mnt/netapp/breakpad