659844 - Trim mozilla-releases rsync module - May 2011 edition

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Motivation:
 surf:mozilla-releases rsync size is CRITICAL: mozilla-releases rsync is 125 GB.

The limit is 110G.

nthomas@surf:/mnt/netapp/stage/releases.mozilla.org/pub/mozilla.org$ du -scm *
11023	addons
1	bouncer
9069	calendar
2855	camino
50044	firefox
2442	mobile
24535	seamonkey
25596	thunderbird
2416	xulrunner
1	zz

nthomas@surf:/mnt/netapp/stage/releases.mozilla.org/pub/mozilla.org$ du -scm {firefox,seamonkey,thunderbird,xulrunner}/releases/*
329	firefox/releases/2.0.0.20
704	firefox/releases/3.0.19-real-real
6202	firefox/releases/3.5.19
6681	firefox/releases/3.6.17
12447	firefox/releases/4.0
13080	firefox/releases/4.0.1
10604	firefox/releases/5.0b2
0	firefox/releases/latest
0	firefox/releases/latest-3.5
0	firefox/releases/latest-3.6
0	firefox/releases/latest-4.0
1	firefox/releases/README
2877	seamonkey/releases/2.0.11
2867	seamonkey/releases/2.0.12
2844	seamonkey/releases/2.0.13
2878	seamonkey/releases/2.0.14
225	seamonkey/releases/2.1a3
1192	seamonkey/releases/2.1b1
2471	seamonkey/releases/2.1b2
2506	seamonkey/releases/2.1b3-bad
3333	seamonkey/releases/2.1b3-real
3346	seamonkey/releases/2.1rc1
1	seamonkey/releases/README
331	thunderbird/releases/2.0.0.24
4582	thunderbird/releases/3.0.11
4752	thunderbird/releases/3.1.10
4738	thunderbird/releases/3.1.7
4725	thunderbird/releases/3.1.8
4661	thunderbird/releases/3.1.9
524	thunderbird/releases/3.3a1
649	thunderbird/releases/3.3a2
628	thunderbird/releases/3.3a3
0	thunderbird/releases/latest
0	thunderbird/releases/latest-3.0
0	thunderbird/releases/latest-3.1
1	thunderbird/releases/README
177	xulrunner/releases/1.9.0.11
310	xulrunner/releases/1.9.1.19
345	xulrunner/releases/1.9.2.17
955	xulrunner/releases/2.0
632	xulrunner/releases/2.0rc1

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

And

nthomas@surf:/mnt/netapp/stage/releases.mozilla.org/pub/mozilla.org$ du -scm calendar/sunbird/releases/*
2580	calendar/sunbird/releases/0.8
3206	calendar/sunbird/releases/0.9
2765	calendar/sunbird/releases/1.0b1
8550	total


Callek, could we pull SeaMonkey 2.0.11, 2.0.12, 2.0.13, 2.1a3, 2.1b1, 2.1b2, 2.1b3-bad, and 2.1b3-real ?

Gozer/jhopkins: can we pull 3.1.7, 3.1.8, 3.1.9, 3.3a1, 3.3a2 ? Do you know anything about the Sunbird releases ?

I'll figure out what's going on with Firefox.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Ok, so Firefox 5.0b2 isn't really on the mirrors. To get the size in the MOTD (and hence the nagios check) we're syncing the mozilla-releases module, which uses rsyncd-mozilla-prereleases.exclude rather than rsyncd-mozilla.exclude. The check is meant to make sure we're not asking mirrors to provide unbounded amounts of space.

Dave, can I go ahead and add a new module to surf:/etc/rsynd.confg that uses rsyncd-mozilla.exclude, and update /root/bin/sync-stage1-releases to use that module ?

Comment 3

[Justin Wood (:Callek)]

(In reply to comment #1)
> Callek, could we pull SeaMonkey 2.0.11, 2.0.12, 2.0.13, 2.1a3, 2.1b1, 2.1b2,
> 2.1b3-bad, and 2.1b3-real ?

Yes, that was my thinking/list todo anyway, I'll get a SeaMonkey website bug on file in the next few minutes to track that side.

> And
> 
> nthomas@surf:/mnt/netapp/stage/releases.mozilla.org/pub/mozilla.org$ du -scm
> calendar/sunbird/releases/*
> 2580	calendar/sunbird/releases/0.8
> 3206	calendar/sunbird/releases/0.9
> 2765	calendar/sunbird/releases/1.0b1
> 8550	total
> 
> Gozer/jhopkins: can we pull 3.1.7, 3.1.8, 3.1.9, 3.3a1, 3.3a2 ? Do you know
> anything about the Sunbird releases ?

From what I know there are to be no more sunbird releases, as its been EOL'ed long ago, so I bet that dropping those from releases.m.o would be fine, CC-ing Fallen as well for input there though.

Comment 4

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Do I understand correctly, that trimming Sunbird will keep the releases available somewhere (I dunno, ftp.mozilla.org?), but just keep it from rsyncing to the mirrors? In that case its fine with me.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

That's right, ftp://ftp.mozilla.org/, and links to download.m.o will still work OK for really light traffic.

Comment 6

[John Hopkins (:jhopkins)]

Nick: it's ok to remove Thunderbird 3.1.7 3.1.8 3.1.9 3.3a1 3.3a2 from the mirrors.

By the way, is this something we can do ourselves?  What's the procedure?  Thanks.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #6)
> By the way, is this something we can do ourselves?  What's the procedure? 
> Thanks.

I think we can do that, but perhaps we'll block on getting the control files tracked in version control somewhere. Then we'll all know who to bla^H^H^H ask if something goes wrong.

Comment 8

[Dave Miller [:justdave]]

(In reply to comment #2)
> Dave, can I go ahead and add a new module to surf:/etc/rsynd.confg that uses
> rsyncd-mozilla.exclude, and update /root/bin/sync-stage1-releases to use
> that module ?

Sure, go for it.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

I've added the SeaMonkey, Thunderbird, and Sunbird releases mentioned in earlier comments. Fixed Firefox by setting up a mozilla-releases-mirrors module in /etc/rsyncd.conf and using that in /root/bin/sync-stage1-releases.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Down to 77G. Thanks everyone.