We need to deploy videur on loop-stage, this means: - Add in the nginx configuration the videur scripts so they're activated when loop starts (they're at https://github.com/mozilla/videur/tree/master/lib), and this should be installed via make install if needed; - Remove the proxy configuration (instead of pointing to the nodejs it uses the spec file to do the routing); We already have openresty deployed, so no need to deploy it again :) Here is an example of nginx conf: https://github.com/mozilla/videur#usage
Do you want this to travel with loop-server 0.12.4? Or on the next release? Or indepenently?
Status: NEW → ASSIGNED
this is independent of 0.12.4, the next big release (0.13) should have it, though, so we can start to do some QOS.
Does loop-server currently have specs for videur ready? If so, where are they located? Is it an HTTP endpoint or a static file with the application?
Thanks for taking this Benson, I don't actually find anything in the loop repo, but I know we have thought about what to do already and which API would need to have videur rules. Not sure if we have a videur specfile yet or not. Tarek,might now that :)
Flags: needinfo?(alexis+bugs) → needinfo?(tarek)
Benson, this is now in loop-server, we have the videur spec in there, see https://github.com/mozilla-services/loop-server/commit/029006c5ff4900766e3558b1ca9c22f8ebb65f8b
Is this for Benson or Dean? What is the priority for this update? We still have not finished 0.12.5. Either way, would like to test this before pushing to Prod, so I will open a separate bug for that...
Summary: Deploy videur on loop-stage → Please deploy videur on Loop-Server Stage
The plan is to *not* disrupt the current loop dev/qa/ops cycle with another new thing. So this is what I'm going to do: - deploy a new separate staging stack with videur lua and loop-server videur code - this will use the same stage redis instance - it'll have its own loop-videur.stage.mozaws.net endpoint - QA will load test it w/ :tarek's new QA code (to be merged) - verifies that current functionality is unaffectred - verifies that videur is not introducing new and unwanted side effects - verifies that rate limiting is working, see: https://github.com/mozilla-services/loop-server/pull/231 - create an ops plan to merge the videur changes into main puppet-config code - deploy to real stage
Not sure what this is: QA will load test it w/ :tarek's new QA code (to be merged) So, I guess default to "old" Dev IAM (where I think loop-server Stage still lives) This will still need to be officially scheduled - my guess is early next week.
So, 0.12.5 + videur changes to Stage in old Dev IAM please Need the videur load test changes in a branch of that I guess I will test it as I have time
We can probably get some help on QA-ing this (security side of things) from Yvan's team... Added a getinfo
What's the schedule for this testing? 0.12.5 is finally out the door so we can put some focus here before we move on to 0.12.6 or 0.13.0
Flags: needinfo?(yboily) → needinfo?(bwong)
Going to need to see the rate limiting code (https://github.com/mozilla-services/loop-server/pull/231/files) back ported into the 12.x branch, hopefully a version 0.12.6 before we start implementing a demo deploy of this.
Here are the next steps, as I understand them: 1.) Write an RPM build script for Videur. a.) Requires the lua-sec RPM available from our yum repository. b.) Requires the following luarocks not available from our yum repository: lua-resty, lua-cjson, lrexlib-posix, date. c.) Write build scripts, or otherwise include RPMs for items in step b. 2.) Write a Puppet module for Videur. 3.) Modify loop-server Puppet module, and loop-server Cloudformation document to: a.) Arbitrarily enable/disable Videur based on a Cloudformation parameter. b.) Override api-specs.json options included with loop-server application. 4.) Verify IP rate limiting works by running load test against Videur enabled loop-server stack. 5.) Verify throughput isn't unacceptably impacted by running a load test against a Videur enabled loop-server stack with upper limits set sufficiently high enough not to trigger rate limiting.
FYI: Current plan is: next release (by this Wed.) loop-server 0.15.5 - which will have one fix for Bug 1125788 (Move to a bigger redis instance for loop) afterwards: loop-server 0.15.6 - will go out with one fix for Videur only so we can properly test
QA Contact: rpappalardo
"Override api-specs.json options included with loop-server application." Right now the loop-server app points to that file to serve it as a static document on the /api-specs URL This URL can be anything as long as you properly configure Nginx
(In reply to Bob Micheletto [:bobm] from comment #13) > Here are the next steps, as I understand them: > > 1.) Write an RPM build script for Videur. > a.) Requires the lua-sec RPM available from our yum repository. > b.) Requires the following luarocks not available from our yum > repository: lua-resty, lua-cjson, lrexlib-posix, date. > c.) Write build scripts, or otherwise include RPMs for items in step b. > 2.) Write a Puppet module for Videur. > 3.) Modify loop-server Puppet module, and loop-server Cloudformation > document to: > a.) Arbitrarily enable/disable Videur based on a Cloudformation parameter. > b.) Override api-specs.json options included with loop-server application. > 4.) Verify IP rate limiting works by running load test against Videur > enabled loop-server stack. > 5.) Verify throughput isn't unacceptably impacted by running a load test > against a Videur enabled loop-server stack with upper limits set > sufficiently high enough not to trigger rate limiting. Hi Tarek - since it only makes sense to test videur within the context of a specific product, do we need a separate staging env for this or could we simply add a loadtest to loop-server that would exercise videur? I'm not sure then how we might separate videur test from loop-server test (or even if that would be necessary).
Hello Bob, We have a specific LoadTest that can verify videur configuration. https://github.com/mozilla-services/loop-server/pull/232
Bob, we can add this to the current staging configuration and see if the loadtest (actually, any loadtest should fail after a certain amount of requests, I believe, depending how videur is configured).
Adding Bob as a NI on this, I think we can move forward.
bobm noted some anomalous 307s, 404s, malformed reqs, etc. last Fri. Noting here for reference so we can test for similar after videur is in place: 5:04 PM <bobm> Weird anomalous spike in 307 requests for loop-server traffic. 5:05 PM <bobm> Starting at about 4:37 PST 5:05 PM <bobm> 00:37 UTC 5:05 PM <bobm> er.. 307 status 5:06 PM <benbangert> bobm: I will assume loop-push is fine.... :) 5:06 PM <bobm> Also 404s. 5:06 PM <bobm> I'll assume the same. 5:07 PM → msander joined (firstname.lastname@example.org.IP) 5:07 PM <bobm> Looks like some malformed requests. 5:08 PM <bobm> A lot of GETs on bad looking version strings. 5:08 PM <bobm> Getting some samples. 5:09 PM <bobm> GET /v0/v2.2/XXXXXXXXXXXXXXX HTTP/1.1
Reassigning this to :bobm. Not sure if it is still applicable.
Assignee: bwong → bobm
It is still something we want for loop. We can try to work on it after the 0.17.4 deploy in production.
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.