Please deploy videur on Loop-Server Stage

RESOLVED WONTFIX

Status

RESOLVED WONTFIX
4 years ago
9 months ago

People

(Reporter: alexis+bugs, Assigned: bobm)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [qa+])

(Reporter)

Description

4 years ago
We need to deploy videur on loop-stage, this means:

- Add in the nginx configuration the videur scripts so they're activated when loop starts (they're at https://github.com/mozilla/videur/tree/master/lib), and this should be installed via make install if needed;
- Remove the proxy configuration (instead of pointing to the nodejs it uses the spec file to do the routing);

We already have openresty deployed, so no need to deploy it again :)

Here is an example of nginx conf: https://github.com/mozilla/videur#usage
(Reporter)

Updated

4 years ago
Assignee: nobody → dwilson
Do you want this to travel with loop-server 0.12.4?
Or on the next release?
Or indepenently?
Status: NEW → ASSIGNED
Flags: needinfo?(alexis+bugs)
Whiteboard: [qa+]
(Reporter)

Comment 2

4 years ago
this is independent of 0.12.4, the next big release (0.13) should have it, though, so we can start to do some QOS.
Flags: needinfo?(alexis+bugs)
Assignee: dwilson → bwong
Does loop-server currently have specs for videur ready? If so, where are they located? Is it an HTTP endpoint or a static file with the application?
Flags: needinfo?(alexis+bugs)
(Reporter)

Comment 4

4 years ago
Thanks for taking this Benson,

I don't actually find anything in the loop repo, but I know we have thought about what to do already and which API would need to have videur rules.

Not sure if we have a videur specfile yet or not. Tarek,might now that :)
Flags: needinfo?(alexis+bugs) → needinfo?(tarek)
(Reporter)

Comment 5

4 years ago
Benson, this is now in loop-server, we have the videur spec in there, see https://github.com/mozilla-services/loop-server/commit/029006c5ff4900766e3558b1ca9c22f8ebb65f8b
Flags: needinfo?(tarek)
Is this for Benson or Dean? What is the priority for this update?
We still have not finished 0.12.5.

Either way, would like to test this before pushing to Prod, so I will open a separate bug for that...
Blocks: 1087846
Summary: Deploy videur on loop-stage → Please deploy videur on Loop-Server Stage
The plan is to *not* disrupt the current loop dev/qa/ops cycle with another new thing. So this is what I'm going to do: 

- deploy a new separate staging stack with videur lua and loop-server videur code
  - this will use the same stage redis instance
  - it'll have its own loop-videur.stage.mozaws.net endpoint
- QA will load test it w/ :tarek's new QA code (to be merged) 
  - verifies that current functionality is unaffectred
  - verifies that videur is not introducing new and unwanted side effects
  - verifies that rate limiting is working, see: 
    https://github.com/mozilla-services/loop-server/pull/231
- create an ops plan to merge the videur changes into main puppet-config code
- deploy to real stage
Not sure what this is:
QA will load test it w/ :tarek's new QA code (to be merged)

So, I guess default to "old" Dev IAM (where I think loop-server Stage still lives)

This will still need to be officially scheduled - my guess is early next week.
So, 0.12.5 + videur changes to Stage in old Dev IAM please
Need the videur load test changes in a branch of that I guess

I will test it as I have time
We can probably get some help on QA-ing this (security side of things) from Yvan's team...
Added a getinfo
Flags: needinfo?(yboily)
What's the schedule for this testing? 
0.12.5 is finally out the door so we can put some focus here before we move on to 0.12.6 or 0.13.0
Flags: needinfo?(yboily) → needinfo?(bwong)
Flags: needinfo?(yboily)
Going to need to see the rate limiting code (https://github.com/mozilla-services/loop-server/pull/231/files) back ported into the 12.x branch, hopefully a version 0.12.6 before we start implementing a demo deploy of this.
Flags: needinfo?(bwong)
(Assignee)

Updated

4 years ago
Blocks: 1126611
(Assignee)

Comment 13

4 years ago
Here are the next steps, as I understand them:

1.) Write an RPM build script for Videur.
   a.) Requires the lua-sec RPM available from our yum repository.
   b.) Requires the following luarocks not available from our yum repository: lua-resty, lua-cjson, lrexlib-posix, date.
   c.) Write build scripts, or otherwise include RPMs for items in step b.
2.) Write a Puppet module for Videur.
3.) Modify loop-server Puppet module, and loop-server Cloudformation document to:
   a.) Arbitrarily enable/disable Videur based on a Cloudformation parameter.
   b.) Override api-specs.json options included with loop-server application.
4.) Verify IP rate limiting works by running load test against Videur enabled loop-server stack.
5.) Verify throughput isn't unacceptably impacted by running a load test against a Videur enabled loop-server stack with upper limits set sufficiently high enough not to trigger rate limiting.
FYI:  Current plan is:

next release (by this Wed.)
loop-server 0.15.5 - which will have one fix for Bug 1125788 (Move to a bigger redis instance for loop)

afterwards:
loop-server 0.15.6 - will go out with one fix for Videur only so we can properly test
Flags: needinfo?(yboily)
QA Contact: rpappalardo
"Override api-specs.json options included with loop-server application."

Right now the loop-server app points to that file to serve it as a static document on the /api-specs URL

This URL can be anything as long as you properly configure Nginx
(In reply to Bob Micheletto [:bobm] from comment #13)
> Here are the next steps, as I understand them:
> 
> 1.) Write an RPM build script for Videur.
>    a.) Requires the lua-sec RPM available from our yum repository.
>    b.) Requires the following luarocks not available from our yum
> repository: lua-resty, lua-cjson, lrexlib-posix, date.
>    c.) Write build scripts, or otherwise include RPMs for items in step b.
> 2.) Write a Puppet module for Videur.
> 3.) Modify loop-server Puppet module, and loop-server Cloudformation
> document to:
>    a.) Arbitrarily enable/disable Videur based on a Cloudformation parameter.
>    b.) Override api-specs.json options included with loop-server application.
> 4.) Verify IP rate limiting works by running load test against Videur
> enabled loop-server stack.
> 5.) Verify throughput isn't unacceptably impacted by running a load test
> against a Videur enabled loop-server stack with upper limits set
> sufficiently high enough not to trigger rate limiting.

Hi Tarek -
since it only makes sense to test videur within the context of a specific product, do we need a separate staging env for this or could we simply add a loadtest to loop-server that would exercise videur?  I'm not sure then how we might separate videur test from loop-server test (or even if that would be necessary).
Flags: needinfo?(tarek)
Hello Bob,

We have a specific LoadTest that can verify videur configuration. https://github.com/mozilla-services/loop-server/pull/232
(Reporter)

Comment 18

4 years ago
Bob, we can add this to the current staging configuration and see if the loadtest (actually, any loadtest should fail after a certain amount of requests, I believe, depending how videur is configured).
Flags: needinfo?(tarek)
(Reporter)

Comment 19

4 years ago
Adding Bob as a NI on this, I think we can move forward.
Flags: needinfo?(bobm)
bobm noted some anomalous 307s, 404s, malformed reqs, etc. last Fri.  
Noting here for reference so we can test for similar after videur is in place:
5:04 PM <bobm> Weird anomalous spike in 307 requests for loop-server traffic.
5:05 PM <bobm> Starting at about 4:37 PST
5:05 PM <bobm> 00:37 UTC
5:05 PM <bobm> er.. 307 status
5:06 PM <benbangert> bobm: I will assume loop-push is fine.... :)
5:06 PM <bobm> Also 404s.
5:06 PM <bobm> I'll assume the same.
5:07 PM → msander joined (msander@moz-gjt6gc.jjrr.25u9.1010.2600.IP)
5:07 PM <bobm> Looks like some malformed requests.
5:08 PM <bobm> A lot of GETs on bad looking version strings.
5:08 PM <bobm> Getting some samples.
5:09 PM <bobm> GET /v0/v2.2/XXXXXXXXXXXXXXX HTTP/1.1
Reassigning this to :bobm. Not sure if it is still applicable.
Assignee: bwong → bobm
It is still something we want for loop. We can try to work on it after the 0.17.4 deploy in production.

Updated

3 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
(Assignee)

Updated

9 months ago
Flags: needinfo?(bobm)
You need to log in before you can comment on or make changes to this bug.