1141860 - Please deploy pushgo 1.5 to Loop-Push Production

Status: VERIFIED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Richard Pappalardo [:rpapa][:rpappalardo]]

Per irc w/ :oremj, we'll deploy to prod tomorrow (Wed.,3/11) @ 10:30am PST

Comment 1

[Richard Pappalardo [:rpapa][:rpappalardo]]

Hey Jeremy, pls ping me once you've deployed the ELB.  I'd like to verify loop-push before we switch over DNS.
Thanks!

Comment 2

[Karl Thiessen [:kthiessen, he/him]]

Verified by rpapa and kthiessen.

Comment 3

[Richard Pappalardo [:rpapa][:rpappalardo]]

Thanks, Karl!  There was an issue with deployment that could be prevented by adding an additional test in future.  
Posting notes below for reference.

----------------------
Deployment Summary
----------------------

Prod deployment was verified via normal testing by kthiessen, rpapa placing loop calls with FF restart and clean profiles.  Though DNS was switched to new Loop-push stack, old stack was not shut down right away, causing Loop to inadvertently remain in a 'degraded' state for many users who would have had old loop-push server cached (and thus unable to receive new notifications).  Issue was resolved once ops shutdown old loop-push server and connections began climbing again.

Normal testing wouldn't catch this, so going forward we should add an additional test with an existing profile and no FF restart (retaining cache) from pre- through post-deployment.  

Notes from IRC:

benbangert noted:
- I'm seeing split out goroutines per host, as expected, but not per-host split out connection counts
- also, reconnects are way down
- ppl are not reconnecting when they drop

benbangert verified with following test:
- already had a conversation link for an existing open Firefox, 
- I opened Chrome and used it, Firefox got no update
- restarted Firefox, did the link in Chrome again, Firefox got the update
- i.e., existing clients open before the cutover do not seem to have reconnected to the new cluster, so they are missing notifications

Summary of issue:
- if old stack isn't shut off, many users will have old loop-push cached and receive no new notifications
- people that haven't reconnected will get no notification of new calls
- reconnection can only happen if the router/firewall drops the old connection, and if our server drops it
- none of this affects webrtc at all, this is purely notifications notification