1102207 - Disable Mobile Originated endpoint in production

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task, P1)

Description

[Rémy Hubscher (:natim)]

We want to disable all mobileOriginated endpoints for MSISDN in production until we get specific agreement with operators that the MO message will be free.

Basically removing all of those:

https://github.com/mozilla-services/puppet-config/blob/master/msisdn/yaml/app/msisdn_gateway.prod.yaml#L23-L41

Comment 1

[Maria Angeles Oteo (:oteo)]

Rémy, when do you think this issue will be handled?

Comment 2

[Rémy Hubscher (:natim)]

Probably with next release with was due by QA for today.

Comment 3

[Maria Angeles Oteo (:oteo)]

Rémy, any news about this?

Comment 4

[Tarek Ziadé (:tarek)]

This is getting quite urgent, Dean can you give us a status today ?  thx

Comment 5

[Dean Wilson (:deanw)]

I can do a new staging deploy with these configs merged in if you can find someone to verify it. We'll also need to arrange a time where people are around to verify the prod push.

Comment 6

[Dean Wilson (:deanw)]

Should this change also have the moVerifierMapping: removed or does it need to be present but empty? (removing would be nice for cleanliness.)

I assume this has to happen in stage and dev. Is this correct?

Comment 7

[Tarek Ziadé (:tarek)]

> I can do a new staging deploy with these configs merged in if you can find someone to verify it

If 0.5.1 is the version deployed in stage it means we have *zero* change in the code. So we can probably bypass most of the QA work. 

What we can do is have it deployed on stage and Maria can probably try it and let us know if everything's ok. then push the config change in prod. Adding Richard to have his green light on this.

> Should this change also have the moVerifierMapping: removed or does it need to be present but empty?

I guess it should be removed.

Comment 8

[Dean Wilson (:deanw)]

:tarek :natim is this change correct? https://github.com/mozilla-services/puppet-config/compare/bz-1102207-msisdn?expand=1

Comment 9

[Tarek Ziadé (:tarek)]

LGTM

Comment 10

[Dean Wilson (:deanw)]

I'll make the change to staging now. I can also confirm we're running 0.5.1.

Comment 11

[Tarek Ziadé (:tarek)]

Thanks a lot Dean

Comment 12

[Dean Wilson (:deanw)]

Staging has been redeployed with the new config -

https://msisdn.stage.mozaws.net/

$ rpm -qa | grep msisd
msisdn-gateway-svcops 0.5.1-1 x86_64 67505404

and the value has gone from the config file.

Comment 13

[Richard Pappalardo [:rpapa][:rpappalardo]]

(In reply to Tarek Ziadé (:tarek) from comment #7)
> > I can do a new staging deploy with these configs merged in if you can find someone to verify it
> 
> If 0.5.1 is the version deployed in stage it means we have *zero* change in
> the code. So we can probably bypass most of the QA work. 
> 
> What we can do is have it deployed on stage and Maria can probably try it
> and let us know if everything's ok. then push the config change in prod.
> Adding Richard to have his green light on this.

OK by me.

Comment 14

[Dean Wilson (:deanw)]

OK, who needs to agree that this can be promoted to prod? And can it wait until everyone is back on Monday?

Comment 15

[Maria Angeles Oteo (:oteo)]

(In reply to Dean Wilson (:deanw) from comment #14)
> OK, who needs to agree that this can be promoted to prod? And can it wait
> until everyone is back on Monday?

I have tried to verify the flow pointing to Stage server but I am not receiving the verification code, I suppose this is due to SMSs are not sent for real in that server, they are collected (https://wiki.mozilla.org/Services/Mobile-ID)

Tarek, if you want me to verify it before pushing to Production, you could apply the change in development server so I can confirm if all is fine, wdyt?


Otherwise,

Comment 16

[Alexis Metaireau (:alexis)]

I just changed the configuration of the dev server. Let us know if that's better.

Thanks

Comment 17

[Maria Angeles Oteo (:oteo)]

Yes, it's working as expected, be free to push it to Production and thanks a lot to all of you!

Comment 18

[Alexis Metaireau (:alexis)]

Thanks for checking. Dean, can you push that to prod? Thanks!

Comment 19

[Dean Wilson (:deanw)]

This is scheduled to go to prod in the next hour or two. I'll update the ticket once it's done.

Comment 20

[Dean Wilson (:deanw)]

The new deploy is at https://msisdnprod1-ms-elb-iiky1vvo28ey-621229108.us-west-2.elb.amazonaws.com/

Waiting for alexis to have a look before i repoint dns from the old stack.

Comment 21

[Alexis Metaireau (:alexis)]

It seems to be working but I would like to have Maria's feedback as well. Can you actually test that for us? Thanks!

Comment 22

[Maria Angeles Oteo (:oteo)]

Sorry for answering so late, bank holiday today in Spain... what do you want me to test? pointing to Production sever?

Comment 23

[Alexis Metaireau (:alexis)]

Please test using this url: https://msisdnprod1-ms-elb-iiky1vvo28ey-621229108.us-west-2.elb.amazonaws.com/

Comment 24

[Maria Angeles Oteo (:oteo)]

I've configure the server you mentioned, in the beginning there was a problem as the certificate was invalid, but I solve it by going to that URL in the FxOS browser and accepting a permanent exception after doing that, the device does not seem to connect to the server see here the logcat: https://pastebin.mozilla.org/7875612

Comment 25

[Alexis Metaireau (:alexis)]

Well, from what I see it seems that it worked okay. Why do you say it's not working ?

Comment 26

[Maria Angeles Oteo (:oteo)]

based on the log "I/Gecko   (29695): 1418119116025        Hawk    DEBUG       Response text: {"code":401,"errno":110,"error":"Bad mac"}" 

and on the fact that the UI is stalled at the "connecting" phase... is that log expected? Do you think we are doing something wrong in the client?

Comment 27

[Alexis Metaireau (:alexis)]

The bad mac is probably due to the fact we're using a server with a different domain name that the client is expecting + the SSL problem, so it's a red herring (not a real problem that would happen in prod).

I believe we should be good to go for the prod deploy. A better way to test this would be to actually deploy staging in a way that's sending actual SMS and not using oxmen (so we can validate this configuration is working properly), or we could actually check oxmen to see if messages are there, waiting to be sent.

I'm handing this off to Tarek since he worked on oxmen.

Comment 28

[Tarek Ziadé (:tarek)]

> The bad mac is probably due to the fact we're using a server with a different domain name that the client is expecting + the SSL problem, so it's a red herring (not a real problem that would happen in prod).


Yeah well. we have a weird testing environment. and we need a real stage one.

see https://wiki.mozilla.org/Services/Mobile-ID#

In the meantime, if everything worked in Dev, it's good enough for me for a prod push.

Comment 29

[Alexis Metaireau (:alexis)]

Dean, can you switch the DNS over?

Comment 30

[Dean Wilson (:deanw)]

This has been done. the new version should be visible shortly.

I'll leave the old stack up for a few hours in case we need to roll back.

Comment 31

[Richard Pappalardo [:rpapa][:rpappalardo]]

(In reply to Tarek Ziadé (:tarek) from comment #28)
> > The bad mac is probably due to the fact we're using a server with a different domain name that the client is expecting + the SSL problem, so it's a red herring (not a real problem that would happen in prod).
> 
> Yeah well. we have a weird testing environment. and we need a real stage one.
> 
> see https://wiki.mozilla.org/Services/Mobile-ID#

On the wiki it outlines a 'Real' stage and a 'Load' stage.  Do we need both or should we just have one staging environment that looks like prod?

Comment 32

[Tarek Ziadé (:tarek)]

We need both because when we do load testing we want to fake the sms gateway so we don't send thousands of real SMSs

Comment 33

[Rémy Hubscher (:natim)]

Isn't the https://msisdn-dev.stage.mozaws.net/ server not enough for manual testing?