1181181 - Please re-deploy release 0.17.7 to loop-server STAGE - CONFIG CHANGE: autopush

Status: RESOLVED WONTFIX

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Richard Pappalardo [:rpapa][:rpappalardo]]

Re-deploying loop-server to STAGE with config change to begin testing using autopush in place of Loop-Push.  NOTE: This won't be promoted to PROD until we've sync-up again w/ loop team and have sign off.

per bbangert:
The new infrastructure is actually the scalable one, the current version has hit its limit and is falling over which is why we'd like to switch Hello over to using the new one.

We just want to be sure that the persistent message store is not going to cause an issue, so some extensive testing of cases due to that should be run.

Comment 1

[Dean Wilson (:deanw)]

This has been deployed to stage -
https://loopsvrstg1-lo-ELB-1C1M4QD84DRQ-546893858.eu-west-1.elb.amazonaws.com

DNS has not been pointed to this host

grep -A 2 pushServerURIs /data/loop-server/config/settings.json 
    "pushServerURIs": [
      "wss://autopush.stage.mozaws.net/"
    ],

Comment 2

[Richard Pappalardo [:rpapa][:rpappalardo]]

TESTING

- bobm and I also did a few e2e tests using autopush on STAGE today and it worked for us.
- I believe Standard8 has also been testing Loop on DEV with autopush and seems to be working fine in that 
  environment as well.  
- Concern is more around what will happen to users when we do the DNS switch from LoopPush to autopush.
  (aka: will there be caching and how would we handle this).  
- bobm and I tried to test migration scenarios (moving from LoopPush to autopush) on STAGE, but LoopPush
  was down.  bobm to follow-up w/ deanw on Mon. to troubleshoot Loop w/ LoopPush STAGE deployment errors.
  once that's fixed, we'll re-attempt.

MIGRATION NOTES

- deamw, bobm, travis, tarek and i met this morning to discuss migration from Loop-Push to autopush.
- Need to ensure we have a way for new client code to move to the new simple push URL even
  if old URL is still valid.
- We could enable loop-server to have a list of both push servers until TTL expires (30 days?) and meanwhile
  get messages back from both websockets.
- If user base is <100K users, we can probably just take the burst.  If we anticipate larger number of users, we 
  should probably have some kind of transition script (like we did for redis migration)
- Plan is to try and have a follow-on meeting with Mark, Ben to make sure we're not missing anything

Comment 3

[Ben Bangert [:benbangert]]

MIGRATION:

So, this is my theory. Push production (autopush) will hand out different endpoint URL's than loop-push does. So when ppl reconnect and end up on autopush, they'll be handed a new URL. Unfortunately there's no client code to detect that they are in a new cluster, so all existing push endpoint links won't work, and it won't resubscribe.

So the question is, under what cases will the client renew its subscription (and thus realize it has a new endpoint)?

Comment 4

[Rémy Hubscher (:natim)]

I my understanding of the client, each time a new websocket connection is done, the client will get a new URL and will register it on the loop-server.

Comment 5

[Rémy Hubscher (:natim)]

Answer received by email:

> I'm away for a week, but I definitely want the client side devs to do some testing wrt what happens when a client comes back online if a notification has been received whilst they are offline.
>
> I can lead that testing but not until I get back.
>
> Mark.

Comment 6

[Mark Banner (:standard8)]

(In reply to Dean Wilson (:deanw) from comment #1)
> This has been deployed to stage -
> https://loopsvrstg1-lo-ELB-1C1M4QD84DRQ-546893858.eu-west-1.elb.amazonaws.com

I've not done any testing yet, additionally the host above seems to no-longer exist.

Comment 7

[Dean Wilson (:deanw)]

This stack no longer exists. As it was not confirmed as working we didn't migrate it to the new IAM. We can discuss adding a new version in the loop weekly meeting.

Comment 8

[Richard Pappalardo [:rpapa][:rpappalardo]]

Hello (Loop) migration from Loop-push to autopush tentatively scheduled for Tues., 8/5 so we can have a few things in place (including having Mark back from PTO)

pre-requisites:
1. adding statsd metrics (+ moving statsd metrics to DD)
   - loop-server dev
2. gathering/reviewing data from statsd
   - Ops
3. test caching scenarios  on STAGE (DNS switch)
   - Ops,QA

Comment 9

[Ben Bangert [:benbangert]]

Does Push infra need any additional statsd metrics? We're already covered on most operations as is.

Comment 10

[Rémy Hubscher (:natim)]

> Does Push infra need any additional statsd metrics? We're already covered on most operations as is.

What we decided during yesterday meeting is to add Push StatsD counters on loop-server side.

Comment 11

[Richard Pappalardo [:rpapa][:rpappalardo]]

(In reply to Rémy Hubscher (:natim) from comment #10)
> > Does Push infra need any additional statsd metrics? We're already covered on most operations as is.
> 
> What we decided during yesterday meeting is to add Push StatsD counters on
> loop-server side.

These counters have just been released in a new tag so, halting 0.17.7 re-deploy in favor of:

Bug 1187293 - Please deploy release 0.17.8 to loop-server STAGE - CONFIG CHANGE: autopush