1079739 - Please deploy videur on Loop-Server Stage

Status: RESOLVED WONTFIX

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Alexis Metaireau (:alexis)]

We need to deploy videur on loop-stage, this means:

- Add in the nginx configuration the videur scripts so they're activated when loop starts (they're at https://github.com/mozilla/videur/tree/master/lib), and this should be installed via make install if needed;
- Remove the proxy configuration (instead of pointing to the nodejs it uses the spec file to do the routing);

We already have openresty deployed, so no need to deploy it again :)

Here is an example of nginx conf: https://github.com/mozilla/videur#usage

Comment 1

[James Bonacci [:jbonacci]]

Do you want this to travel with loop-server 0.12.4?
Or on the next release?
Or indepenently?

Comment 2

[Alexis Metaireau (:alexis)]

this is independent of 0.12.4, the next big release (0.13) should have it, though, so we can start to do some QOS.

Comment 3

[Benson Wong [:mostlygeek]]

Does loop-server currently have specs for videur ready? If so, where are they located? Is it an HTTP endpoint or a static file with the application?

Comment 4

[Alexis Metaireau (:alexis)]

Thanks for taking this Benson,

I don't actually find anything in the loop repo, but I know we have thought about what to do already and which API would need to have videur rules.

Not sure if we have a videur specfile yet or not. Tarek,might now that :)

Comment 5

[Alexis Metaireau (:alexis)]

Benson, this is now in loop-server, we have the videur spec in there, see https://github.com/mozilla-services/loop-server/commit/029006c5ff4900766e3558b1ca9c22f8ebb65f8b

Comment 6

[James Bonacci [:jbonacci]]

Is this for Benson or Dean? What is the priority for this update?
We still have not finished 0.12.5.

Either way, would like to test this before pushing to Prod, so I will open a separate bug for that...

Comment 7

[Benson Wong [:mostlygeek]]

The plan is to *not* disrupt the current loop dev/qa/ops cycle with another new thing. So this is what I'm going to do: 

- deploy a new separate staging stack with videur lua and loop-server videur code
  - this will use the same stage redis instance
  - it'll have its own loop-videur.stage.mozaws.net endpoint
- QA will load test it w/ :tarek's new QA code (to be merged) 
  - verifies that current functionality is unaffectred
  - verifies that videur is not introducing new and unwanted side effects
  - verifies that rate limiting is working, see: 
    https://github.com/mozilla-services/loop-server/pull/231
- create an ops plan to merge the videur changes into main puppet-config code
- deploy to real stage

Comment 8

[James Bonacci [:jbonacci]]

Not sure what this is:
QA will load test it w/ :tarek's new QA code (to be merged)

So, I guess default to "old" Dev IAM (where I think loop-server Stage still lives)

This will still need to be officially scheduled - my guess is early next week.

Comment 9

[James Bonacci [:jbonacci]]

So, 0.12.5 + videur changes to Stage in old Dev IAM please
Need the videur load test changes in a branch of that I guess

I will test it as I have time

Comment 10

[James Bonacci [:jbonacci]]

We can probably get some help on QA-ing this (security side of things) from Yvan's team...
Added a getinfo

Comment 11

[James Bonacci [:jbonacci]]

What's the schedule for this testing? 
0.12.5 is finally out the door so we can put some focus here before we move on to 0.12.6 or 0.13.0

Comment 12

[Benson Wong [:mostlygeek]]

Going to need to see the rate limiting code (https://github.com/mozilla-services/loop-server/pull/231/files) back ported into the 12.x branch, hopefully a version 0.12.6 before we start implementing a demo deploy of this.

Comment 13

[Bob Micheletto [:bobm]]

Here are the next steps, as I understand them:

1.) Write an RPM build script for Videur.
   a.) Requires the lua-sec RPM available from our yum repository.
   b.) Requires the following luarocks not available from our yum repository: lua-resty, lua-cjson, lrexlib-posix, date.
   c.) Write build scripts, or otherwise include RPMs for items in step b.
2.) Write a Puppet module for Videur.
3.) Modify loop-server Puppet module, and loop-server Cloudformation document to:
   a.) Arbitrarily enable/disable Videur based on a Cloudformation parameter.
   b.) Override api-specs.json options included with loop-server application.
4.) Verify IP rate limiting works by running load test against Videur enabled loop-server stack.
5.) Verify throughput isn't unacceptably impacted by running a load test against a Videur enabled loop-server stack with upper limits set sufficiently high enough not to trigger rate limiting.

Comment 14

[Richard Pappalardo [:rpapa][:rpappalardo]]

FYI:  Current plan is:

next release (by this Wed.)
loop-server 0.15.5 - which will have one fix for Bug 1125788 (Move to a bigger redis instance for loop)

afterwards:
loop-server 0.15.6 - will go out with one fix for Videur only so we can properly test

Comment 15

[Tarek Ziadé (:tarek)]

"Override api-specs.json options included with loop-server application."

Right now the loop-server app points to that file to serve it as a static document on the /api-specs URL

This URL can be anything as long as you properly configure Nginx

Comment 16

[Richard Pappalardo [:rpapa][:rpappalardo]]

(In reply to Bob Micheletto [:bobm] from comment #13)
> Here are the next steps, as I understand them:
> 
> 1.) Write an RPM build script for Videur.
>    a.) Requires the lua-sec RPM available from our yum repository.
>    b.) Requires the following luarocks not available from our yum
> repository: lua-resty, lua-cjson, lrexlib-posix, date.
>    c.) Write build scripts, or otherwise include RPMs for items in step b.
> 2.) Write a Puppet module for Videur.
> 3.) Modify loop-server Puppet module, and loop-server Cloudformation
> document to:
>    a.) Arbitrarily enable/disable Videur based on a Cloudformation parameter.
>    b.) Override api-specs.json options included with loop-server application.
> 4.) Verify IP rate limiting works by running load test against Videur
> enabled loop-server stack.
> 5.) Verify throughput isn't unacceptably impacted by running a load test
> against a Videur enabled loop-server stack with upper limits set
> sufficiently high enough not to trigger rate limiting.

Hi Tarek -
since it only makes sense to test videur within the context of a specific product, do we need a separate staging env for this or could we simply add a loadtest to loop-server that would exercise videur?  I'm not sure then how we might separate videur test from loop-server test (or even if that would be necessary).

Comment 17

[Rémy Hubscher (:natim)]

Hello Bob,

We have a specific LoadTest that can verify videur configuration. https://github.com/mozilla-services/loop-server/pull/232

Comment 18

[Alexis Metaireau (:alexis)]

Bob, we can add this to the current staging configuration and see if the loadtest (actually, any loadtest should fail after a certain amount of requests, I believe, depending how videur is configured).

Comment 19

[Alexis Metaireau (:alexis)]

Adding Bob as a NI on this, I think we can move forward.

Comment 20

[Richard Pappalardo [:rpapa][:rpappalardo]]

bobm noted some anomalous 307s, 404s, malformed reqs, etc. last Fri.  
Noting here for reference so we can test for similar after videur is in place:
5:04 PM <bobm> Weird anomalous spike in 307 requests for loop-server traffic.
5:05 PM <bobm> Starting at about 4:37 PST
5:05 PM <bobm> 00:37 UTC
5:05 PM <bobm> er.. 307 status
5:06 PM <benbangert> bobm: I will assume loop-push is fine.... :)
5:06 PM <bobm> Also 404s.
5:06 PM <bobm> I'll assume the same.
5:07 PM → msander joined ([e-mail])
5:07 PM <bobm> Looks like some malformed requests.
5:08 PM <bobm> A lot of GETs on bad looking version strings.
5:08 PM <bobm> Getting some samples.
5:09 PM <bobm> GET /v0/v2.2/XXXXXXXXXXXXXXX HTTP/1.1

Comment 21

[Benson Wong [:mostlygeek]]

Reassigning this to :bobm. Not sure if it is still applicable.

Comment 22

[Rémy Hubscher (:natim)]

It is still something we want for loop. We can try to work on it after the 0.17.4 deploy in production.