Closed
Bug 1079739
Opened 10 years ago
Closed 9 years ago
Please deploy videur on Loop-Server Stage
Categories
(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)
Cloud Services
Operations: Deployment Requests - DEPRECATED
x86_64
Linux
Tracking
(Not tracked)
RESOLVED
WONTFIX
People
(Reporter: alexis+bugs, Assigned: bobm)
References
Details
(Whiteboard: [qa+])
We need to deploy videur on loop-stage, this means: - Add in the nginx configuration the videur scripts so they're activated when loop starts (they're at https://github.com/mozilla/videur/tree/master/lib), and this should be installed via make install if needed; - Remove the proxy configuration (instead of pointing to the nodejs it uses the spec file to do the routing); We already have openresty deployed, so no need to deploy it again :) Here is an example of nginx conf: https://github.com/mozilla/videur#usage
Reporter | ||
Updated•10 years ago
|
Assignee: nobody → dwilson
Comment 1•10 years ago
|
||
Do you want this to travel with loop-server 0.12.4? Or on the next release? Or indepenently?
Status: NEW → ASSIGNED
Flags: needinfo?(alexis+bugs)
Whiteboard: [qa+]
Reporter | ||
Comment 2•10 years ago
|
||
this is independent of 0.12.4, the next big release (0.13) should have it, though, so we can start to do some QOS.
Flags: needinfo?(alexis+bugs)
Updated•10 years ago
|
Assignee: dwilson → bwong
Comment 3•10 years ago
|
||
Does loop-server currently have specs for videur ready? If so, where are they located? Is it an HTTP endpoint or a static file with the application?
Flags: needinfo?(alexis+bugs)
Reporter | ||
Comment 4•10 years ago
|
||
Thanks for taking this Benson, I don't actually find anything in the loop repo, but I know we have thought about what to do already and which API would need to have videur rules. Not sure if we have a videur specfile yet or not. Tarek,might now that :)
Flags: needinfo?(alexis+bugs) → needinfo?(tarek)
Reporter | ||
Comment 5•10 years ago
|
||
Benson, this is now in loop-server, we have the videur spec in there, see https://github.com/mozilla-services/loop-server/commit/029006c5ff4900766e3558b1ca9c22f8ebb65f8b
Flags: needinfo?(tarek)
Comment 6•10 years ago
|
||
Is this for Benson or Dean? What is the priority for this update? We still have not finished 0.12.5. Either way, would like to test this before pushing to Prod, so I will open a separate bug for that...
Updated•10 years ago
|
Summary: Deploy videur on loop-stage → Please deploy videur on Loop-Server Stage
Comment 7•10 years ago
|
||
The plan is to *not* disrupt the current loop dev/qa/ops cycle with another new thing. So this is what I'm going to do: - deploy a new separate staging stack with videur lua and loop-server videur code - this will use the same stage redis instance - it'll have its own loop-videur.stage.mozaws.net endpoint - QA will load test it w/ :tarek's new QA code (to be merged) - verifies that current functionality is unaffectred - verifies that videur is not introducing new and unwanted side effects - verifies that rate limiting is working, see: https://github.com/mozilla-services/loop-server/pull/231 - create an ops plan to merge the videur changes into main puppet-config code - deploy to real stage
Comment 8•10 years ago
|
||
Not sure what this is: QA will load test it w/ :tarek's new QA code (to be merged) So, I guess default to "old" Dev IAM (where I think loop-server Stage still lives) This will still need to be officially scheduled - my guess is early next week.
Comment 9•10 years ago
|
||
So, 0.12.5 + videur changes to Stage in old Dev IAM please Need the videur load test changes in a branch of that I guess I will test it as I have time
Comment 10•10 years ago
|
||
We can probably get some help on QA-ing this (security side of things) from Yvan's team... Added a getinfo
Flags: needinfo?(yboily)
Comment 11•10 years ago
|
||
What's the schedule for this testing? 0.12.5 is finally out the door so we can put some focus here before we move on to 0.12.6 or 0.13.0
Flags: needinfo?(yboily) → needinfo?(bwong)
Updated•10 years ago
|
Flags: needinfo?(yboily)
Comment 12•10 years ago
|
||
Going to need to see the rate limiting code (https://github.com/mozilla-services/loop-server/pull/231/files) back ported into the 12.x branch, hopefully a version 0.12.6 before we start implementing a demo deploy of this.
Flags: needinfo?(bwong)
Assignee | ||
Comment 13•9 years ago
|
||
Here are the next steps, as I understand them: 1.) Write an RPM build script for Videur. a.) Requires the lua-sec RPM available from our yum repository. b.) Requires the following luarocks not available from our yum repository: lua-resty, lua-cjson, lrexlib-posix, date. c.) Write build scripts, or otherwise include RPMs for items in step b. 2.) Write a Puppet module for Videur. 3.) Modify loop-server Puppet module, and loop-server Cloudformation document to: a.) Arbitrarily enable/disable Videur based on a Cloudformation parameter. b.) Override api-specs.json options included with loop-server application. 4.) Verify IP rate limiting works by running load test against Videur enabled loop-server stack. 5.) Verify throughput isn't unacceptably impacted by running a load test against a Videur enabled loop-server stack with upper limits set sufficiently high enough not to trigger rate limiting.
Comment 14•9 years ago
|
||
FYI: Current plan is: next release (by this Wed.) loop-server 0.15.5 - which will have one fix for Bug 1125788 (Move to a bigger redis instance for loop) afterwards: loop-server 0.15.6 - will go out with one fix for Videur only so we can properly test
Flags: needinfo?(yboily)
QA Contact: rpappalardo
Comment 15•9 years ago
|
||
"Override api-specs.json options included with loop-server application." Right now the loop-server app points to that file to serve it as a static document on the /api-specs URL This URL can be anything as long as you properly configure Nginx
Comment 16•9 years ago
|
||
(In reply to Bob Micheletto [:bobm] from comment #13) > Here are the next steps, as I understand them: > > 1.) Write an RPM build script for Videur. > a.) Requires the lua-sec RPM available from our yum repository. > b.) Requires the following luarocks not available from our yum > repository: lua-resty, lua-cjson, lrexlib-posix, date. > c.) Write build scripts, or otherwise include RPMs for items in step b. > 2.) Write a Puppet module for Videur. > 3.) Modify loop-server Puppet module, and loop-server Cloudformation > document to: > a.) Arbitrarily enable/disable Videur based on a Cloudformation parameter. > b.) Override api-specs.json options included with loop-server application. > 4.) Verify IP rate limiting works by running load test against Videur > enabled loop-server stack. > 5.) Verify throughput isn't unacceptably impacted by running a load test > against a Videur enabled loop-server stack with upper limits set > sufficiently high enough not to trigger rate limiting. Hi Tarek - since it only makes sense to test videur within the context of a specific product, do we need a separate staging env for this or could we simply add a loadtest to loop-server that would exercise videur? I'm not sure then how we might separate videur test from loop-server test (or even if that would be necessary).
Flags: needinfo?(tarek)
Comment 17•9 years ago
|
||
Hello Bob, We have a specific LoadTest that can verify videur configuration. https://github.com/mozilla-services/loop-server/pull/232
Reporter | ||
Comment 18•9 years ago
|
||
Bob, we can add this to the current staging configuration and see if the loadtest (actually, any loadtest should fail after a certain amount of requests, I believe, depending how videur is configured).
Updated•9 years ago
|
Flags: needinfo?(tarek)
Reporter | ||
Comment 19•9 years ago
|
||
Adding Bob as a NI on this, I think we can move forward.
Flags: needinfo?(bobm)
Comment 20•9 years ago
|
||
bobm noted some anomalous 307s, 404s, malformed reqs, etc. last Fri. Noting here for reference so we can test for similar after videur is in place: 5:04 PM <bobm> Weird anomalous spike in 307 requests for loop-server traffic. 5:05 PM <bobm> Starting at about 4:37 PST 5:05 PM <bobm> 00:37 UTC 5:05 PM <bobm> er.. 307 status 5:06 PM <benbangert> bobm: I will assume loop-push is fine.... :) 5:06 PM <bobm> Also 404s. 5:06 PM <bobm> I'll assume the same. 5:07 PM → msander joined (msander@moz-gjt6gc.jjrr.25u9.1010.2600.IP) 5:07 PM <bobm> Looks like some malformed requests. 5:08 PM <bobm> A lot of GETs on bad looking version strings. 5:08 PM <bobm> Getting some samples. 5:09 PM <bobm> GET /v0/v2.2/XXXXXXXXXXXXXXX HTTP/1.1
Comment 21•9 years ago
|
||
Reassigning this to :bobm. Not sure if it is still applicable.
Assignee: bwong → bobm
Comment 22•9 years ago
|
||
It is still something we want for loop. We can try to work on it after the 0.17.4 deploy in production.
Updated•9 years ago
|
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
Assignee | ||
Updated•7 years ago
|
Flags: needinfo?(bobm)
You need to log in
before you can comment on or make changes to this bug.
Description
•