672572 - please allow releng-mirror01 to sync mozilla-prerelease module from stage-rsync.mozilla.org

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[[e-mail] (:bhearsum)]

I'm testing against mozilla-current for now, but before I can declare bug 671075 I'll need to start syncing this module.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Did you mean rsync or firewall config here ?

I suspect you have to hit stage::mozilla-releases if you want the module using rsyncd-mozilla-prereleases.exclude, and that requires an exception in stage:/etc/rsyncd.conf. AIUI releases-rsync.m.o::mozilla-release uses rsyncd-mozilla-releases.exclude

Comment 2

[[e-mail] (:bhearsum)]

(In reply to comment #1)
> Did you mean rsync or firewall config here ?

There's no firewall rules in my way (I'm currently using mozilla-current as a test), so definitely rsync config.

> I suspect you have to hit stage::mozilla-releases if you want the module
> using rsyncd-mozilla-prereleases.exclude, and that requires an exception in
> stage:/etc/rsyncd.conf. AIUI releases-rsync.m.o::mozilla-release uses
> rsyncd-mozilla-releases.exclude

Ah, I thought prereleases was served from releases-rsync, too. I'll give this a try tomorrow.

Comment 3

[Dave Miller [:justdave]]

You have to rsync from stage-rsync.mozilla.org to get mozilla-prerelease

Firewall should be open, but you'll need to be whitelisted in the rsyncd.conf.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to comment #3)
> You have to rsync from stage-rsync.mozilla.org to get mozilla-prerelease
> 
> Firewall should be open, but you'll need to be whitelisted in the
> rsyncd.conf.

Thanks, updating the summary.

Comment 5

[Amy Rich [:arr] [:arich]]

I've added 10.12.51.243 to /etc/rsyncd.conf on surf.  Let me know if it's working for you now.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to comment #5)
> I've added 10.12.51.243 to /etc/rsyncd.conf on surf.  Let me know if it's
> working for you now.

Apologies for the delay in testing. I'm still getting permission denied, though:
[root@releng-mirror01 html]# rsync -n -v -a --delete --delete-before stage-rsync.mozilla.org::mozilla-prereleases/ /var/www/html/mozilla-prereleases


---------------------------------------------------------------------------
            R E L E A S E S - R S Y N C . M O Z I L L A . O R G
---------------------------------------------------------------------------

                   Current module sizes (updated 2011-07-26)

                      mozilla-releases: 103GB
                       mozilla-current: 27GB

         Questions/comments/complaints: [e-mail]

---------------------------------------------------------------------------



@ERROR: access denied to mozilla-prereleases from unknown (63.245.222.66)

Comment 7

[[e-mail] (:bhearsum)]

Doesn't work from stage.m.o either, fwiw (but the module doesn't exist there anyways...)
[root@releng-mirror01 ~]# rsync -n -v -a --delete --delete-before stage.mozilla.org::mozilla-prereleases/ /var/www/html/mozilla-prereleases


---------------------------------------------------------------------------
                    S T A G E . M O Z I L L A . O R G
---------------------------------------------------------------------------

                   Current module sizes (updated 2011-07-26)

                           mozilla-all: 8.4TB
                      mozilla-releases: 103GB
                       mozilla-current: 27GB

         Questions/comments/complaints: [e-mail]

---------------------------------------------------------------------------



@ERROR: Unknown module 'mozilla-prereleases'

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

You'd want mozilla-releases from stage.m.o.

Comment 9

[Amy Rich [:arr] [:arich]]

Okay, I'll hand this over to justdave to get stage-rsync.mozilla.org configured.

Comment 10

[Dave Miller [:justdave]]

(In reply to comment #5)
> I've added 10.12.51.243 to /etc/rsyncd.conf on surf.  Let me know if it's
> working for you now.

surf != stage-rsync :)  That's why it wasn't working.

63.245.222.66 has been added to the whitelist on pv-mirror02.  Give it a shot now.

Comment 11

[Amy Rich [:arr] [:arich]]

Dave: yeah, I was going off of Nick's comment that stage::mozilla-releases should work, too.  That's obviously not the case, which is why I tossed it over to you to change stage-rsync.  Thanks!  :}

Comment 12

[Amy Rich [:arr] [:arich]]

So I allocated 150G for the vm (about 25G more than you asked for for storage), and it just filled it up completely.

du -sk /var/www/html/mozilla-prereleases
144609304	/var/www/html/mozilla-prereleases

Comment 13

[[e-mail] (:bhearsum)]

(In reply to comment #12)
> So I allocated 150G for the vm (about 25G more than you asked for for
> storage), and it just filled it up completely.
> 
> du -sk /var/www/html/mozilla-prereleases
> 144609304	/var/www/html/mozilla-prereleases

Looks like mozilla-prereleases is extra big. We'll have to start keeping a closer an eye on that. I just removed 3.6.18, 5.0, 6.0b1, and 6.0b2 from it (and also 3.6.18/5.0 from the releases module, to keep them as in sync as possible), which should bring the size down dramatically.

I also filed bug 674504 to get this module's size monitored.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

I don't think we can remove 3.6.18 or 5.0 when there are still lots of windows 3.6 users looping on 3.6.18 updates. I think we'll end up killing dm-download02, which will be all that's left carrying those files.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

That wasn't the clearest, but hopefully you get the drift. We could make it so the mirrors have 3.6.18/update/ and 5.0/update/ for the traffic, and save on the space for the installers (which mozilla.com no longer link to).

Comment 16

[[e-mail] (:bhearsum)]

(In reply to comment #15)
> That wasn't the clearest, but hopefully you get the drift. We could make it
> so the mirrors have 3.6.18/update/ and 5.0/update/ for the traffic, and save
> on the space for the installers (which mozilla.com no longer link to).

I ended up adding 5.0 and 3.6.18 back to releases wholesale.

Comment 17

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #15)
> That wasn't the clearest, but hopefully you get the drift. We could make it
> so the mirrors have 3.6.18/update/ and 5.0/update/ for the traffic, and save
> on the space for the installers (which mozilla.com no longer link to).

I ended up doing something similar to this after discussion with Nick. I added all of these to the prereleases and releases excludes:
3.6.18:
* linux-i686/
* mac/
* source/
* update/mac/
* win32/

3.6.19:
* update/linux-i686/
* update/win32/

5.0:
* linux-i686/
* linux-x86_64/
* mac/
* source/
* update/mac/
* win32/

5.0.1:
* update/linux-i686/
* update/linux-x86_64/
* update/win32/

So, for 3.6.18 & 5.0 we still have Linux and Windows MARs on the mirrors. For 3.6.19 & 5.0.1, we've still got Mac MARs. Next week after we ship 3.6.20 and 6.0, we can drop these releases entirely, I believe. But that's certainly not something we need to track in this bug.