684387 - clean up stuff in surf:/mnt/netapp/stage/archive.mozilla.org/pub/firefox

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect, P3)

Description

[Justin Dow [:jabba]]

15:59 < nagios-sjc1> [39] surf:disk - /mnt/netapp/stage/archive.mozilla.org/pub/firefox is WARNING: DISK 
                     WARNING - free space: /mnt/netapp/stage/archive.mozilla.org/pub/firefox 144826 MB (4%):

Comment 1

[Jake Maul [:jakem]]

FYI: I've grown this volume slightly on the NetApp. There is very little room to grow it further, but it should help a little. I wouldn't consider it a "fix", but it seems like we're perpetually fighting disk space on some of these volumes, so hopefully this will help some.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

We hit this again today:
* jakem allocated another 100GB on the netapp to the partition to make nagios shush up
* I'll move some old candidates directories from firefox/nightly/ to ~cltdbld/old-candidates/dirs to create more space
* bug 692621 is the plan to move the old nightlies files off Netapp onto the big-disk but non-HA-head which is cm-ixstore01

Comment 3

[Jake Maul [:jakem]]

[root@surf firefox]# nice ionice -c3 -n99 du -hcx --max-depth=1
2.4G    ./bundles
2.4T    ./nightly
219G    ./try-builds
1.1T    ./releases
20K     ./tinderbox-builds
3.7T    .
3.7T    total

I'm working on a breakdown of 'nightly' now. I can say though that it has directories 2004-2011, plus a number of others that aren't dates (<version>-candidates, latest-<product> being the 2 common patterns).

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] from comment #2)
> * I'll move some old candidates directories from firefox/nightly/ to
> ~cltdbld/old-candidates/dirs to create more space

I moved 3.6.12 through to 21 last week, and deleted some 3.5.x (from before the terminal release of .19). There is a further a move in progress for 4.0* and 5.0*, which is another 225G.

(In reply to Jake Maul [:jakem] from comment #3)
> I'm working on a breakdown of 'nightly' now. I can say though that it has
> directories 2004-2011, plus a number of others that aren't dates
> (<version>-candidates, latest-<product> being the 2 common patterns).

I looked at that too (sizes in MB):
13297	2004
25311	2005
101991	2006
78301	2007
110015	2008
164078	2009
177494	2010
746424	2011

And 2011 breaks down by month as:
14597	1
15000	2
21311	3
43954	4
63154	5
67644	6
73548	7
53653	8
139319	9
254250	10

We have quite a few cron jobs in place for cleaning up old content (eg ffxbld), and the growth in monthly size is a function of the number of branches we're doing nightlies for now, plus keeping logs on ftp.m.o. There was nothing obviously taking up a whole lot of unexpected space, but it's lots of dirs and files so would have been easy to miss something.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Remove references to 5.0b4-candidates in update verify for mozilla-beta

Update verify for 8.0b3 failed when it came to do the full check for 5.0b4 de/en-US/ru, because of these leftover references to the candidates directories. Turns out we never shipped 5.0b4, and there are < 100 blocklist pings per day, so I think it's safe to remove them here. Let me know if you think we should remove them from the patcher config too.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 566714 [details] [diff] [review]
[tools] Remove references to 5.0b4-candidates in update verify for mozilla-beta

http://hg.mozilla.org/build/tools/rev/a3273080adcc

Comment 7

[Jake Maul [:jakem]]

The full list is a bit bigger, but here's everything in 'nightly' over 1GB. Obviously most of the space usage is in the top few directories, but it would be good to take a quick look at the others too, and maybe catch some low-hanging fruit. :)

2.0T    .
715G    ./2011
174G    ./2010
161G    ./2009
108G    ./2008
100G    ./2006
77G     ./2007
72G     ./contrib
47G     ./6.0.2-candidates
39G     ./8.0b2-candidates
35G     ./6.0-candidates
33G     ./7.0b4-candidates
30G     ./7.0-candidates
30G     ./7.0.1-candidates
25G     ./6.0.1-candidates
25G     ./2005
22G     ./8.0b1-candidates
21G     ./7.0b1-candidates
21G     ./6.0b1-candidates
19G     ./8.0b3-candidates
19G     ./7.0b6-candidates
19G     ./7.0b5-candidates
19G     ./7.0b3-candidates
19G     ./7.0b2-candidates
19G     ./6.0b3-candidates
19G     ./3.6.22-candidates
18G     ./latest-mozilla-central-l10n
18G     ./latest-mozilla-aurora-l10n
18G     ./6.0b5-candidates
18G     ./6.0b4-candidates
18G     ./6.0b2-candidates
17G     ./3.5.19-candidates
13G     ./2004
9.2G    ./3.6.23-candidates
4.6G    ./3.0.19-candidates
1.1G    ./latest-jaegermonkey

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

This got superseded by bug 715840.