Deploy Release 0.4.5 to msisdn-gateway Stage

VERIFIED FIXED

Status

VERIFIED FIXED
4 years ago
4 years ago

People

(Reporter: rhubscher, Assigned: bobm)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [qa+])

Comment hidden (empty)
(Reporter)

Updated

4 years ago
Blocks: 1055589
- uploaded the 0.4.4-1 RPM to our S3 yum repo 
- the CFN in us-east-1 dev account just needs to be updated w/ the new AppVersion as this is just a bug fix
- assigning issue to :bobm
Assignee: bwong → bobm
Do we know if this is a regular/scheduled release or a break/fix release?
Looks like a reg release to me, but we should try to get it out this week pending OPs and QA current schedules...

REF:
https://github.com/mozilla-services/msisdn-gateway/releases/tag/0.4.4
https://github.com/mozilla-services/msisdn-gateway/compare/0.4.4...master
Status: NEW → ASSIGNED
Whiteboard: [qa+]
(Assignee)

Comment 3

4 years ago
The updated package has been deployed to Stage.
Verified code is deployed and the service is running on a single m3.medium instance:
Instance: ec2-54-81-60-236
Code:
msisdn-gateway-svcops 0.4.4-1 x86_64 54825600
puppet-config-msisdn 20140812210007-1 x86_64 13723

Verified that we will be talking to the mock server for load testing.

So, the new CLI tool did not work for me at all (I bounced that bug).
The webapp works but apparently not against Stage, so not helpful in this case.

Moving on to a short load test for today, a longer load test tomorrow...
The 30min load test looks good as well:
https://loads.services.mozilla.com/run/abc8f3d4-fb38-4083-b2fb-58e1376283b2

I will follow-up tomorrow with a bit more stress/load and some review of the logs.
(Reporter)

Comment 7

4 years ago
jbonacci you link is wrong, it is a bugfix release: https://github.com/mozilla-services/msisdn-gateway/compare/0.4.3...0.4.4
James, as an additional note, I find it very useful to have the information you put in this bug, about the instance and the configuration that's being used. Just so you know :)
(Reporter)

Comment 9

4 years ago
A new bug fix as been made today. Please update Stage with 0.4.5 release.
Summary: Deploy Release 0.4.4 to msisdn-gateway Stage → Deploy Release 0.4.5 to msisdn-gateway Stage
OK. My links are still usable, but yours is better!
But now I guess I focus on https://github.com/mozilla-services/msisdn-gateway/compare/0.4.4...0.4.5
;-)

Ok, that fix looks important.
We just started yesterday afternoon, so I am sure we can roll forward to 0.4.5 in Stage.

I just caught :bobm in IRC
(Assignee)

Comment 11

4 years ago
This has been deployed to stage.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Thanks. Will start on this in about 90min...
:alexis - https://bugzilla.mozilla.org/show_bug.cgi?id=1055587#c8
It's a "value added service" compliments of QA...
:-)
OK, we have a single-install deploy of msisdn: ec2-54-226-36-100

Code:
msisdn-gateway-svcops 0.4.5-1 x86_64 54832649
puppet-config-msisdn 20140812210007-1 x86_64 13723

Verified processes, logs, and files

/data/msisdn-gateway/config/production.json has expected entries for omxen

Looking at the yaml files...

/etc/puppet/yaml/app/msisdn_gateway.yaml
...etc...
        supportedLanguages: 
            - "en-US"
            - "fr"
            - "db-LB"
...etc...

Note the influence of Persona in that third line/language...
;-)



/etc/puppet/yaml/app/msisdn_gateway.stage.yaml
Verified again the omxen server info


https://msisdn.stage.mozaws.net returns
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN Gateway","version":"0.4.5","homepage":"https://github.com/mozilla-services/msisdn-gateway/","endpoint":"https://msisdn.stage.mozaws.net"}


https://msisdn.stage.mozaws.net/__heartbeat__ returns
{"storage":true}


Moving on to load testing...
so, a 2000 "hit" load test amounts to about 65 minutes.
Results look good.
See link above or this pastebin: https://jbonacci.pastebin.mozilla.org/6078198

nginx logs have the usual 200s, but I am also seeing 204s and 400s.
Last time we saw this, there was a problem with the mock server.
See https://bugzilla.mozilla.org/show_bug.cgi?id=1038772#c30

This should be investigated.
So :bobm kicked the mock server and I started a quick load test.
I am still seeing 204s and 400s.

With release 0.4.3, we originally saw the same issue, but it cleared up ...
https://bugzilla.mozilla.org/show_bug.cgi?id=1038772

So, I just need the 204s and 400s reviewed to make sure those are acceptable for msisdn-gateway hitting the mock server.
Flags: needinfo?(rhubscher)
A little poking around by me in the updated loadtest.py file shows
a "MT Flow failed" generates a 204
a collision or wrong code would give a 400

:rfkelly confirmed via
https://github.com/mozilla-services/msisdn-gateway/commit/1372eb02acedaa0fd1a5ec7ac525bfbf3e2838d4


OK, so we are good to go to Prod...
/me updates my docs
Status: RESOLVED → VERIFIED
Flags: needinfo?(rhubscher)
(Reporter)

Comment 19

4 years ago
204 and 400 are fully normal.

204 means 200 without content which is the case for many endpoint.
400 is because the loadtest is trying wrong codes as I explained earlier.
(Reporter)

Comment 20

4 years ago
Well you had it before :)
(In reply to Rémy Hubscher (:natim) from comment #19)
> 204 and 400 are fully normal.
> 
> 204 means 200 without content which is the case for many endpoint.
> 400 is because the loadtest is trying wrong codes as I explained earlier.

Maybe this can be documented briefly in the loadtest
(Reporter)

Comment 22

4 years ago
I have added some *status-200*, *status-204* and *status-400* counters to the loadtest to let you know how many of these code where expected.

Also I changed the *oxmen-message-collision* to as *oxmen-collision-PROBABLY-WRONG* so that if you see this counter know something went wrong.

https://github.com/mozilla-services/msisdn-gateway/commit/e41251a0d794fcfd43603fb8544cac032a7669ed
Yep, it has been documented on my wiki, but these changes will be quite helpful.
Will try it out on the next release...
You need to log in before you can comment on or make changes to this bug.