Closed Bug 1055587 Opened 11 years ago Closed 11 years ago

Deploy Release 0.4.5 to msisdn-gateway Stage

Categories

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Product:
Cloud Services
Component:
Operations: Deployment Requests - DEPRECATED
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: rhubscher, Assigned: bobm)

References

Details

(Whiteboard: [qa+])

No description provided.
Blocks: 1055589
- uploaded the 0.4.4-1 RPM to our S3 yum repo - the CFN in us-east-1 dev account just needs to be updated w/ the new AppVersion as this is just a bug fix - assigning issue to :bobm
Assignee: bwong → bobm
Do we know if this is a regular/scheduled release or a break/fix release? Looks like a reg release to me, but we should try to get it out this week pending OPs and QA current schedules... REF: https://github.com/mozilla-services/msisdn-gateway/releases/tag/0.4.4 https://github.com/mozilla-services/msisdn-gateway/compare/0.4.4...master
Status: NEW → ASSIGNED
Whiteboard: [qa+]
The updated package has been deployed to Stage.
Verified code is deployed and the service is running on a single m3.medium instance: Instance: ec2-54-81-60-236 Code: msisdn-gateway-svcops 0.4.4-1 x86_64 54825600 puppet-config-msisdn 20140812210007-1 x86_64 13723 Verified that we will be talking to the mock server for load testing. So, the new CLI tool did not work for me at all (I bounced that bug). The webapp works but apparently not against Stage, so not helpful in this case. Moving on to a short load test for today, a longer load test tomorrow...
The 30min load test looks good as well: https://loads.services.mozilla.com/run/abc8f3d4-fb38-4083-b2fb-58e1376283b2 I will follow-up tomorrow with a bit more stress/load and some review of the logs.
jbonacci you link is wrong, it is a bugfix release: https://github.com/mozilla-services/msisdn-gateway/compare/0.4.3...0.4.4
James, as an additional note, I find it very useful to have the information you put in this bug, about the instance and the configuration that's being used. Just so you know :)
A new bug fix as been made today. Please update Stage with 0.4.5 release.
Summary: Deploy Release 0.4.4 to msisdn-gateway Stage → Deploy Release 0.4.5 to msisdn-gateway Stage
OK. My links are still usable, but yours is better! But now I guess I focus on https://github.com/mozilla-services/msisdn-gateway/compare/0.4.4...0.4.5 ;-) Ok, that fix looks important. We just started yesterday afternoon, so I am sure we can roll forward to 0.4.5 in Stage. I just caught :bobm in IRC
This has been deployed to stage.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Thanks. Will start on this in about 90min...
:alexis - https://bugzilla.mozilla.org/show_bug.cgi?id=1055587#c8 It's a "value added service" compliments of QA... :-)
OK, we have a single-install deploy of msisdn: ec2-54-226-36-100 Code: msisdn-gateway-svcops 0.4.5-1 x86_64 54832649 puppet-config-msisdn 20140812210007-1 x86_64 13723 Verified processes, logs, and files /data/msisdn-gateway/config/production.json has expected entries for omxen Looking at the yaml files... /etc/puppet/yaml/app/msisdn_gateway.yaml ...etc... supportedLanguages: - "en-US" - "fr" - "db-LB" ...etc... Note the influence of Persona in that third line/language... ;-) /etc/puppet/yaml/app/msisdn_gateway.stage.yaml Verified again the omxen server info https://msisdn.stage.mozaws.net returns {"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN Gateway","version":"0.4.5","homepage":"https://github.com/mozilla-services/msisdn-gateway/","endpoint":"https://msisdn.stage.mozaws.net"} https://msisdn.stage.mozaws.net/__heartbeat__ returns {"storage":true} Moving on to load testing...
so, a 2000 "hit" load test amounts to about 65 minutes. Results look good. See link above or this pastebin: https://jbonacci.pastebin.mozilla.org/6078198 nginx logs have the usual 200s, but I am also seeing 204s and 400s. Last time we saw this, there was a problem with the mock server. See https://bugzilla.mozilla.org/show_bug.cgi?id=1038772#c30 This should be investigated.
So :bobm kicked the mock server and I started a quick load test. I am still seeing 204s and 400s. With release 0.4.3, we originally saw the same issue, but it cleared up ... https://bugzilla.mozilla.org/show_bug.cgi?id=1038772 So, I just need the 204s and 400s reviewed to make sure those are acceptable for msisdn-gateway hitting the mock server.
Flags: needinfo?(rhubscher)
A little poking around by me in the updated loadtest.py file shows a "MT Flow failed" generates a 204 a collision or wrong code would give a 400 :rfkelly confirmed via https://github.com/mozilla-services/msisdn-gateway/commit/1372eb02acedaa0fd1a5ec7ac525bfbf3e2838d4 OK, so we are good to go to Prod... /me updates my docs
Status: RESOLVED → VERIFIED
Flags: needinfo?(rhubscher)
204 and 400 are fully normal. 204 means 200 without content which is the case for many endpoint. 400 is because the loadtest is trying wrong codes as I explained earlier.
Well you had it before :)
(In reply to Rémy Hubscher (:natim) from comment #19) > 204 and 400 are fully normal. > > 204 means 200 without content which is the case for many endpoint. > 400 is because the loadtest is trying wrong codes as I explained earlier. Maybe this can be documented briefly in the loadtest
I have added some *status-200*, *status-204* and *status-400* counters to the loadtest to let you know how many of these code where expected. Also I changed the *oxmen-message-collision* to as *oxmen-collision-PROBABLY-WRONG* so that if you see this counter know something went wrong. https://github.com/mozilla-services/msisdn-gateway/commit/e41251a0d794fcfd43603fb8544cac032a7669ed
Yep, it has been documented on my wiki, but these changes will be quite helpful. Will try it out on the next release...
You need to log in before you can comment on or make changes to this bug.