590821 - [amo] Set up amo file server.

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations: Projects, task)

Description

[Jeff Balogh (:jbalogh)]

For add-on downloads we serve experimental files locally instead of using the mirrors.  That sounds like a decision that needs to be reexamined, but it's what we have.  We also need to serve files locally between the time they're approved and the time we think they'll be on the mirrors.

Since we have the bad worker config and need all the Pythons we can get, I want to dump the file handling on Apache using X-Sendfile.

http://modules.apache.org/search.php?id=1143
http://tn123.ath.cx/mod_xsendfile/
http://john.guen.in/past/2007/4/17/send_files_faster_with_xsendfile/

Comment 1

[Wil Clouser [:clouserw]]

The last time this came up:


  14:59 <jbalogh> oremj: do we have xsendfile for the amo servers?
  15:26 <oremj> jbalogh: nope, we don't have it
  15:42 <jbalogh> oremj: could we add mod_xsendfile?
  16:02 <oremj> jbalogh: what will it be used for?
  16:02 <oremj> I remember evaluating it for remora 2.0, but I don't remember the conclusion :-(
  16:03 <jbalogh> the new release has an page that writes a bunch of the boilerplate for starting an addon
  16:03 <jbalogh> and then we need to deliver that as a zip file
  16:04 <oremj> ah
  16:04 <jbalogh> and I've heard in multiple places that it's better to use xsendfile than to tie up the application
  16:04 <oremj> so probably super low traffic?
  16:04 <jbalogh> presumably
  16:04 <jbalogh> there's not that many people anxious to make addons
  16:09 <oremj> jbalogh: it's a better way to send a file. In this case I'd rather not since it is low traffic and would add a new apache module that needs to be recompiled for httpd updates.
  16:09 <oremj> and it looks somewhat unmaintained now
  16:09 <oremj> or maybe it just hasn't needed an update...
  16:10 <jbalogh> they wrote it perfect the first time ;)
  16:10 <jbalogh> alright, thanks for looking
  16:11 <oremj> I think it'd probably be worth it if it was going to see a lot of traffic

Comment 2

[Jeff Balogh (:jbalogh)]

The new use case is addons.mozilla.org/downloads, the place where all xpis come from.  It's going to be slightly higher traffic.

Comment 3

[Jeremy Orem [:oremj]]

Are these files coming from the netapp share?

Comment 4

[Jeff Balogh (:jbalogh)]

Yes: REPO_PATH = '/mnt/netapp_amo/addons.mozilla.org-remora/files'

Comment 5

[Jeremy Orem [:oremj]]

It would be easy to host these somewhere else completely. Is it easy to set the base URL?

Comment 6

[Jeff Balogh (:jbalogh)]

I can set up zamboni to point wherever we want.  We'd still need the netapp for remora.

fligtar/clouserw/jorge: Is there any reason not to have an open relay for disabled or unreviewed files?  I can already give you a link to any unreviewed file on AMO by tacking on ?confirmed.

Comment 7

[Jeremy Orem [:oremj]]

Oh, yeah I was thinking about serving them not through the app. If we need to serve them through the app this won't work.

Comment 8

[Justin Scott [:fligtar]]

(In reply to comment #6)
> fligtar/clouserw/jorge: Is there any reason not to have an open relay for
> disabled or unreviewed files?  I can already give you a link to any unreviewed
> file on AMO by tacking on ?confirmed.

Only public files should be on the mirrors.

Comment 9

[Jeff Balogh (:jbalogh)]

(In reply to comment #8)
> (In reply to comment #6)
> > fligtar/clouserw/jorge: Is there any reason not to have an open relay for
> > disabled or unreviewed files?  I can already give you a link to any unreviewed
> > file on AMO by tacking on ?confirmed.
> 
> Only public files should be on the mirrors.

We're not talking about mirrors.  We're talking about a local file server that's not blocked by the app.  We would still pass requests through the app, but then redirect to a publicly available file.

And why should only public files be on the mirrors?

Comment 10

[Justin Scott [:fligtar]]

(In reply to comment #9)
> And why should only public files be on the mirrors?

Because we want to control access to unreviewed files, even if it means just adding ?confirmed. Lots of things crawl the mirrors.

Comment 11

[Jeremy Orem [:oremj]]

Do unreviewed and reviewed files look the same on the filesystem i.e., same names/directories?

Comment 12

[Jeff Balogh (:jbalogh)]

fligtar is alright with hosting these on a separate public file server as long as we disallow robots and don't have public listings.

Comment 13

[Jeremy Orem [:oremj]]

Excellent. So we just need to host anything under /mnt/netapp_amo/addons.mozilla.org-remora/files?

Comment 14

[Jeff Balogh (:jbalogh)]

(In reply to comment #13)
> Excellent. So we just need to host anything under
> /mnt/netapp_amo/addons.mozilla.org-remora/files?

AFAIK, yes.  That's the only place I see us slurping up add-ons.

Comment 15

[Jeremy Orem [:oremj]]

Matthew, do we have 2 spare blades that I could use for this? Alternatively, I could pull 1 or 2 servers out of the amo cluster.

Jeff, I'm thinking we can use httpd/mod_worker + our CDN cache for this and httpd will use sendfile without any custom mods (http://httpd.apache.org/docs/current/mod/core.html#enablesendfile). We are using NFS, so any sort of sendfile might not be an option.

Comment 16

[Jeff Balogh (:jbalogh)]

Sounds good to me.  The regular mirrors don't do https so I don't think we need that here either.

Comment 17

[matthew zeier [:mrz]]

We have decommissioned Weave hardware in sjc and I'm sure we have spare hardware in phx (needs to be setup in both locations right?).

Comment 18

[Wil Clouser [:clouserw]]

I was just talking to oremj and we were discussing making this box the new static.addons.mozilla.org.  That would mean it needs SSL and Django, but wouldn't need PHP at all, so it could run mod_worker or whatever.

This would then serve all user avatars, add-on images, *add-on videos*, and sandboxed add-ons.  We can add the same rewrite rules we have in remora that say "If file exists on disk, serve it, otherwise send request to app" to keep things fast.

Thoughts?

Comment 19

[Wil Clouser [:clouserw]]

What's the status of this?  I think we can skip the StAMO stuff in comment 18 since that server is up and running already.

Comment 20

[Jeremy Orem [:oremj]]

Is this important right now since videos have been delayed? Am I mistaken that they've been delayed?

Comment 21

[Wil Clouser [:clouserw]]

I mentioned in comment 19 that we can skip making this StAMO, so this bug is just about serving add-on files (not videos).

Hosting videos has been delayed, yes (but has no effect on this bug).

Comment 22

[Jeff Balogh (:jbalogh)]

(In reply to comment #13)
> Excellent. So we just need to host anything under
> /mnt/netapp_amo/addons.mozilla.org-remora/files?

This is all we need to close this bug.  The stuff in comment 18 is scope creep that can be done later.

Comment 23

[Jeremy Orem [:oremj]]

We are just going with this alias for now:

Alias /files /mnt/netapp_amo/addons.mozilla.org-preview/files

It is already on NAMO/PAMO.