811760 - Need Origin Server for Secure Air Mozilla Archiving

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P4)

Description

[Richard A Milewski[:richard]]

We need a web server on which to temporarily place files for processing by Vid.ly.  These are videos of restricted events (MoCo and Mozillians).  

The web server should be configured for basic auth, and we will need ssh access to the password file so that we can rotate passwords for the Vid.ly user.

This machine does not need large amounts of storage as files will be removed shortly after processing.

Comment 1

[Jake Maul [:jakem]]

Need some more info on this... confused as to how this would be used, so I'm not sure what sort of setup is appropriate.

What web app will this be running?

How will content be uploaded to it?

Wondering on the SSH access... if rotating a password is the only use case, we might want to come up with something different. If there's other reasons too, that might affect what we want to do.


We could probably just give you they keys to a one-off VM and let you do whatever you like, but that's not very good for us if we ever need to have a hand in maintaining it. Without knowing more about the usage, it's hard to know what's appropriate.

Comment 2

[Jake Maul [:jakem]]

Need some more info on this... confused as to how this would be used, so I'm not sure what sort of setup is appropriate.

What web app will this be running?

How will content be uploaded to it?

Wondering on the SSH access... if rotating a password is the only use case, we might want to come up with something different. If there's other reasons too, that might affect what we want to do.


We could probably just give you they keys to a one-off VM and let you do whatever you like, but that's not very good for us if we ever need to have a hand in maintaining it. Without knowing more about the usage, it's hard to know what's appropriate.

Comment 3

[Zandr Milewski [:zandr]]

I can flesh out the use case.

Vid.ly handles all of our transcoding and CDN for videos on air.m.o. They ingest, transcode, and host based on an API call that takes a source URL as an argument. In normal use, that's videos.m.o, because this is all public. They do support private videos (with a nice one-time-token scheme), but we still need to host the video somewhere for them to ingest it.

They support basic auth, so we'd need a password-protected area. I think the right answer is probably another directory on the existing videos-origin with basic auth configured on the http side and we'll rotate that password every so often.

It occurs to me that if the apache config on videos origin will support it, this as simple as a directory in the existing airmozilla section with an .htaccess file. I'll try that this afternoon.

Comment 4

[Zandr Milewski [:zandr]]

Well, I don't seem to be able to create dotfiles over a webdav mount.

So, given that, we could:

a) create a new videos project in LDAP+puppet and an LDAP account for vidly to use
b) mount the storage from some host somewhere to install .htaccess and .htpasswd files

Opinions?

Comment 5

[Brandon Burton [:solarce]]

(In reply to Zandr Milewski [:zandr] from comment #4)
> Well, I don't seem to be able to create dotfiles over a webdav mount.
> 
> So, given that, we could:
> 
> a) create a new videos project in LDAP+puppet and an LDAP account for vidly
> to use
> b) mount the storage from some host somewhere to install .htaccess and
> .htpasswd files
> 
> Opinions?

How much space do you need?

Comment 6

[Brandon Burton [:solarce]]

For reference, we have 45G free on the server that currently hosts our video origins

Comment 7

[Richard A Milewski[:richard]]

Brandon,

A quick check shows we've added about 37G to videos origin so far in Q4.  That indicates we'll be 100% full sometime in Q1 next year.

Comment 8

[Richard A Milewski[:richard]]

(In reply to Brandon Burton [:solarce] from comment #5)
> 
> How much space do you need?

It's hard to tell because I don't know the volume of video that will be coming out of Engagement for Mozillians, but I'm guessing One TB should be sufficient for more than a year.

Comment 9

[Richard A Milewski[:richard]]

Ping!

Comment 10

[Richard A Milewski[:richard]]

Ping!

Comment 11

[Jake Maul [:jakem]]

Let's go with comment 4. The username and password will be in LDAP, so there's no need for a .htpasswd file... the config block to enforce this user could be directly in the Apache config, which I suspect may be easier to manage than an actual .htaccess file.

As for storage space, we should be able to grow the current volume easily... I don't know how big, but certainly a couple hundred GB should be no problem, and that'd get us through Q1 and probably Q2. I suspect a lot more would be possible, too.

Comment 12

[Jason Crowe [:jd]]

I think everything is done for this. I will close it out now but please reopen with details of work remaining if there is any.

Thanks