536808 - preferred links to old releases are dead (archive.mozilla.org only shows current/latest)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Description

[Felix Miata]

e.g. : http://releases.mozilla.org/pub/mozilla.org/firefox/releases/ only shows current/latest version dirs which is probably a newly established behavior. However, http://archive.mozilla.org/pub/mozilla.org/firefox/releases/ shows exactly the same thing, while http://stage.mozilla.org/pub/mozilla.org/firefox/releases/ apparently shows everything http://releases.mozilla.org/pub/mozilla.org/firefox/releases/ used to show. The current behavior results in a massive number of dead links on http://www.mozilla.org/ports/os2/ and probably a lot of other places. It also means the only way to offer access to old builds seems to be via dissemination of the existence of http://stage.mozilla.org/ to people that maybe shouldn't know about it.

Comment 1

[Dave Miller [:justdave]]

It's been that way for a few years actually.  The only possible change might be to the list of stuff being excluded from getting sent to the releases pool.

Changing the protocol from http to ftp should still get around the redirects.

It'd probably be possible to set up a RedirectMap script which would read the excludes file from the rsync module and not redirect anything that's on the exclude list.  That'd be more of a project to pull of, but probably the correct long-term solution.

Comment 2

[Dave Miller [:justdave]]

(In reply to comment #1)
> It'd probably be possible to set up a RedirectMap script which would read the
> excludes file from the rsync module and not redirect anything that's on the
> exclude list.  That'd be more of a project to pull of, but probably the correct
> long-term solution.

Nick and I discussed this on IRC the other day and decided we'd like to go ahead and do this.  It'll be better for performance this way than pointing people at another server that's reading the same filesystem over NFS from the FTP server.  In reality it's probably not going to be that hard to pull off, but will probably take a couple hours of walking the tree fixing up the exclude list to be more accurate so we can ditch all the funky bind mounts.

I'll try to tackle this myself after I get a few other things caught up, but Nick's welcome to attempt it himself if he's feeling bored before I get to it. :)

Comment 3

[Dave Miller [:justdave]]

Stage 1 of this is done.  The bind mounts on surf and dm-download02 are gone, the mozilla-releases module is now using the full archive.m.o docroot with an exclude file.  The exclude file in use is located at zz/rsync-mozilla-releases.exclude within the ftp tree.  Note to anyone that edits this, the specific excludes need to happen before the includes, and the global catch-all exclude needs to be last, after the includes.

Comment 4

[matthew zeier [:mrz]]

[need apache rewrite map]

Comment 5

[Chris Turra [:cturra]]

just triaging some old bugs and this appears to have been resolved already. each of the links make their way back to the ftp cluster.


$ curl -IL http://releases.mozilla.org/pub/mozilla.org/firefox/releases/
HTTP/1.1 301 Moved Permanently
Server: Apache
X-Backend-Server: pp-web01
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1
Date: Mon, 26 May 2014 21:33:20 GMT
Location: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/
Transfer-Encoding: chunked
Connection: Keep-Alive
X-Cache-Info: cached

HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Cache-Control: max-age=300
Content-Type: text/html;charset=UTF-8
Date: Mon, 26 May 2014 21:33:41 GMT
Expires: Mon, 26 May 2014 21:38:41 GMT
Last-Modified: Mon, 26 May 2014 21:29:54 GMT
Server: ECAcc (sjc/4EC2)
X-Backend-Server: ftp4.dmz.scl3.mozilla.com
X-Cache: HIT
X-Cache-Info: caching
Content-Length: 92503


$ curl -I http://archive.mozilla.org/pub/mozilla.org/firefox/releases/
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: ftp1.dmz.scl3.mozilla.com
Cache-Control: max-age=300
Content-Type: text/html;charset=UTF-8
Date: Mon, 26 May 2014 21:33:30 GMT
Expires: Mon, 26 May 2014 21:38:30 GMT
Transfer-Encoding: chunked
Access-Control-Allow-Origin: *
Connection: Keep-Alive
X-Cache-Info: caching



$ curl -I http://stage.mozilla.org/pub/mozilla.org/firefox/releases/
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: ftp2.dmz.scl3.mozilla.com
Cache-Control: max-age=300
Content-Type: text/html;charset=UTF-8
Date: Mon, 26 May 2014 21:33:59 GMT
Expires: Mon, 26 May 2014 21:38:59 GMT
Transfer-Encoding: chunked
Access-Control-Allow-Origin: *
Connection: Keep-Alive