1186202 - archive.mozilla.org/ftp.mozilla.org changes

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Description

[Jeremy Orem [:oremj]]

* fix archive.mozilla.org SSL certificate
* remove landing page from http://ftp.mozilla.org/
* remove /pub/(.*) -> /pub/mozilla.org/$1 redirect
* GET / should just return an apache directory index showing /pub/ and whatever else is in that directory, assuming there is nothing private

Comment 1

[Jake Maul [:jakem]]

> * fix archive.mozilla.org SSL certificate
no problem

> * remove landing page from http://ftp.mozilla.org/
don't know why this exists, but seems fine to me- justdave might have some idea why it exists

> * remove /pub/(.*) -> /pub/mozilla.org/$1 redirect
This is slightly scary- I'm fairly sure mozilla.org is just a symlink back to . (pub/ ... yes, it means it's circular). That in itself is fine, but we probably should check logs and/or bouncer configs to make sure we won't be breaking firefox downloads by doing this. Bouncer sends 302's to CDNs, which use the FTP cluster as the origin, and being a symlink it affects everything on the FTP cluster regardless of VirtualHost.

> * GET / should just return an apache directory index showing /pub/ and whatever else is in that directory, assuming there is nothing private
This seems fine, I think the only thing there is /pub/. We should probably double-check first.

Comment 2

[Dave Miller [:justdave]]

I was asked about why we had an index.html page on the ftp server...

IIRC its original purpose was to try to inform people that they should use releases.mozilla.org rather than ftp.mozilla.org when spreading around links to release files in order to avoid DoSing us.  At some point after that, we got directory-specific redirects in place that sent you there anyway.  And now it's all on a CDN anyhow.  The original message appears to no longer be there.

Comment 3

[Dave Miller [:justdave]]

(In reply to Jeremy Orem [:oremj] from comment #0)
> * remove /pub/(.*) -> /pub/mozilla.org/$1 redirect

The history behind this one:

Originally everything was in /pub/
It all got moved to /pub/mozilla.org/ when we started getting volunteer mirrors to mirror our stuff so they could put it alongside other things they mirrored on the same ftp server.

The weird symlinking was so that old links would continue to work.  It's been 16 years since those old style links were officially used for anything.  Probably nothing of value would be hurt by breaking them at this point.

Comment 4

[Jeremy Orem [:oremj]]

To add, if we remove that redirect, it should means that both:

/pub/firefox/.../firefox.exe and /pub/mozilla.org/firefox/.../firefox.exe should be 200. As it is now the former would be 302 /pub/mozilla.org/firefox/.../firefox.exe

Comment 5

[Jake Maul [:jakem]]

I found a few things in config files that are expecting mozilla.org/ to be in the path, that will be (potentially) broken by removing the redirect that ensures it is. That is, access/redirect rules may not be applied because the contents would be accessible at a URL path other than what is currently expected.

I would guess that most of it is stuff we probably don't care about anymore, in which case it should be removed. If we do care, they'll need to be generalized to work with or without "mozilla.org/" in the path.

# cat labs/jetpack/.htaccess
Redirect 307 /pub/mozilla.org/labs/jetpack/addon-sdk-latest.zip https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-1.17.zip
Redirect 307 /pub/mozilla.org/labs/jetpack/addon-sdk-latest.tar.gz https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-1.17.tar.gz
Redirect 307 /pub/mozilla.org/labs/jetpack/jetpack-sdk-latest.zip https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-1.17.zip
Redirect 307 /pub/mozilla.org/labs/jetpack/jetpack-sdk-latest.tar.gz https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-1.17.tar.gz


    # FTP should not be able to reach funnelcake12 or 14 builds
    <Directory /mnt/netapp/stage/archive.mozilla.org/pub/mozilla.org/metrics/13.0.1-funnelcake12>
        AllowOverride None
        Order Allow,Deny
        Deny from all
    </Directory>
    <Directory /mnt/netapp/stage/archive.mozilla.org/pub/metrics/13.0.1-funnelcake12>
        AllowOverride None
        Order Allow,Deny
        Deny from all
    </Directory>
    <Directory /mnt/netapp/stage/archive.mozilla.org/pub/mozilla.org/metrics/14.0.1-funnelcake14>
        AllowOverride None
        Order Allow,Deny
        Deny from all
    </Directory>
    <Directory /mnt/netapp/stage/archive.mozilla.org/pub/metrics/14.0.1-funnelcake14>
        AllowOverride None
        Order Allow,Deny
        Deny from all
    </Directory>

    RewriteRule ^/pub/mozilla\.org/firefox/nightly/3\.0b1-candidates/ http://developer.mozilla.org/devnews/index.php/2007/11/07/were-happy-that-you-digg-us-but/ [R=302,L]
    RewriteRule ^/pub/mozilla\.org/firefox/nightly/3\.0b3-candidates/ http://developer.mozilla.org/devnews/index.php/2008/02/07/firefox-3-beta-3-expected-early-next-week/ [R=302,L]
    RewriteRule ^/pub/mozilla\.org/firefox/nightly/3\.0b4-candidates/ http://wiki.mozilla.org/Releases/Firefox_3.0b4 [R=302,L]
    RewriteRule ^/pub/mozilla\.org/firefox/nightly/3\.0b5-candidates/ http://wiki.mozilla.org/Releases/Firefox_3.0b5 [R=302,L]
    RewriteRule ^/pub/mozilla\.org/firefox/nightly/3\.0rc1-candidates/ http://wiki.mozilla.org/Releases/Firefox_3.0rc1 [R=302,L]
    RewriteRule ^/pub/mozilla\.org/firefox/nightly/3\.0rc2-candidates/ http://wiki.mozilla.org/Releases/Firefox_3.0rc2 [R=302,L]

    RewriteCond %{REMOTE_ADDR} !^10\.
    RewriteRule ^/pub/mozilla\.org/firefox/nightly/4\.0b(.*)-candidates/ http://wiki.mozilla.org/Releases/Firefox_4.0b$1 [R=302,L]

    # Adding rule per bug 407825
    RewriteCond %{REQUEST_URI} !.*\.mar$
    RewriteRule ^/pub/mozilla\.org/firefox/nightly/3\.0b2-candidates/ http://developer.mozilla.org/devnews/index.php/2007/12/10/firefox-3-beta-2-expected-in-late-december/ [R=302,L]

    <Directory /mnt/netapp/stage/archive.mozilla.org/pub/mozilla.org/firefox/releases/4.0>
        # something guaranteed to be invalid just so index.html doesn't get used
        DirectoryIndex jkfldklsaghfjkl.html
    </Directory>


    RewriteRule ^/pub/mozilla\.org.* - [L]

Comment 6

[Jake Maul [:jakem]]

(In reply to Jeremy Orem [:oremj] from comment #0)

> * remove landing page from http://ftp.mozilla.org/
Done.

> * GET / should just return an apache directory index showing /pub/ and
> whatever else is in that directory, assuming there is nothing private
Done.


TODO:

> * fix archive.mozilla.org SSL certificate
ftp.m.o is already a SAN cert with a couple other names on it... we can easily add archive.m.o to the list.

> * remove /pub/(.*) -> /pub/mozilla.org/$1 redirect
comment 5 is my only concern here at this point- to summarize, some configs that rely on "mozilla.org/" being in the path may be bypassable with this redirect removed. Most of it seems old, and maybe we don't care. It's worth noting that they already were bypassable, in a different way: since mozilla.org is a symlink to ., it can be chained as many times as you want (ex: https://ftp.mozilla.org/pub/mozilla.org/mozilla.org/).

I think we can probably safely just ignore this concern and remove the redirect. Everything I see is old, and none of the rules are for (still valid) security purposes.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Looks like the bouncer side of product delivery doesn't use .../mozilla.org/... already. The mirror definitions all end in /pub, and locations start at the likes of /firefox, so we get 

$ curl -IL "https://download.mozilla.org/?product=firefox-latest&os=win&lang=en-US"
HTTP/1.1 302 Found
Server: Apache
...
Location: http://download-installer.cdn.mozilla.net/pub/firefox/releases/39.0/win32/en-US/Firefox%20Setup%2039.0.exe

Comment 8

[Jake Maul [:jakem]]

SSL cert is done, and the redirect is removed.