1194030 - Deploy EOL-inator place holder service for Sync 1.1 Termination

Status: RESOLVED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Bob Micheletto [:bobm]]

Stand up service to send EOL messaging to hold out Sync 1.1 clients after the service has been terminated.

Requirements:
* Best combination of migration and EOL messaging.
  - Users shouldn't be under the impression that they're still syncing.
  - Users with migration capable Fx agents should be presented with the migration option. 
* Measures active EOL user base.
* Avoids causing the behavior describe here: https://bugzilla.mozilla.org/show_bug.cgi?id=1014411#c16

Comment 1

[Ryan Kelly [:rfkelly]]

As described in Bug 1180575 Comment 11 I think a production-ready version of my "sync11eol-inator" service from Bug 1014411 will be the best way to go here:

  https://github.com/mozilla-services/sync11-eol-server

It sends the necessary headers to trigger the guided migration, but it also causes a "sync encountered an error" bar to appear which will signal to the user that things are no longer working as they expect.  It's even got an ancient r+ from :telliott that I'm going to roll over, so we just need to dust it off and figure out the details of deploying it in production.

Comment 2

[Ryan Kelly [:rfkelly]]

Attachment: sync11eol-fxa-credentials.diff

Bob, as we discussed briefly last week, I think we need to add "meta/fxa_credentials" handling to this server in order to support people migrating from an android device.  The attached patch does the following:

* allows reads and writes of "meta/fxa_credentials" without sending the hard EOL
* increases the default cache expiry from 60 seconds to 2 days

It's the second part I'm mostly concerned about.  For this to be useful, we have to keep the fxa_credentials blog around long enough for the Android client to discover and act on it.  But I don't want to put us in the position of storing lots of data forever.

Thoughts?

Comment 3

[Ryan Kelly [:rfkelly]]

Attachment: sync11eol-authz-handling.diff

Another thought about running this sync1.1 EOLinator in production: currently it has no authentication whatsoever.  If you show up and ask to read/write the data stored for a user, it'll happily oblige regardless of what credentials you provide.

Originally this was OK, because it was only storing /meta/global and /crypto/keys which are not sensitive data.  But if we start storing the fxa_credentials blob as well, perhaps we need to add authentication.

If we do want to add it, here's a sketch of something that might work, following a precedent set way back in Bug 877930 when we were looking at standing up some sync nodes in AWS with cached credentials.  The idea is to derive the memcached key for an item from a HMAC of the provided Authorization header:

   key = HMAC-SHA256(secret_key, authorization_info) + "/" + item_name

Anyone is allowed to write claiming to be any user. But to read the data back, you need to provide the same Authorization header again in order to look it up in memcache.

The downside of this approach is that we'd have cheaply-verified hashes of user passwords sitting around in memcache.  If the memcached keys leaked, they could open up the ability to brute-force guess passwords for a particular user.  I laid out some of the pros and cons of the approach back in Bug 877930 and they'd all apply here as well.

Mark, thoughts?

Comment 4

[Ryan Kelly [:rfkelly]]

> * Measures active EOL user base.

Bob, are you interested in any app-level metrics from this thing, or will nginx logs tell us pretty much all we need to know about the ongoing use of this tiny service?

Comment 5

[Ryan Kelly [:rfkelly]]

> we'd have cheaply-verified hashes of user passwords sitting around in memcache

We could of course PBKDF2 or even scrypt that thing, but at the cost of increasing the CPU load of running this service.  An earlier version of Bug 877930 used scrypt and it was a performance disaster, because it had to scrypt the password on every request.

Comment 6

[Mark Hammond [:markh] [:mhammond]]

Comment on attachment 8652723 [details] [diff] [review]
sync11eol-authz-handling.diff

Review of attachment 8652723 [details] [diff] [review]:
-----------------------------------------------------------------

I don't know :)

* The fxa_credentials written by desktop is misnamed - it has the user's email address, uid, verified flag and any relevant custom prefs they may have set (which will almost be none). So there isn't much that is truly sensitive (ie, there are no actual credentials)

* If we drop these on the floor, the user experience isn't *that* bad - they just need to know what username they used on each of their devices (ie, each device acts as though it is the first device being migrated). Sadly, mobile is a different story - IIUC, migration will not automatically start without the sentinel.

* Given the above and speaking only for desktop, it seems difficult to justify too much risk simply to keep a "nice" UX for users who probably don't really care about Sync and have managed to ignore the migration notifications for this long.

The mobile issue is a concern though, but even then it's not a deal breaker IIUC - the user can still manually reconfigure Sync.

Ryan, do we have any insights into the breakdown of these reluctant upgraders? eg, how active they are and how many are multi-device users?

Nick, please see comment 3 for the back-story.

Comment 7

[Ryan Kelly [:rfkelly]]

Actually, I can never remember which BSOs are encrypted and which are not, but the encryption might also provide protection here and mean we can get away without auth.

> Ryan, do we have any insights into the breakdown of these reluctant upgraders?

Bob, do we have any insights into the propertion of Android users in our reluctant-upgrader population?

Comment 8

[Mark Hammond [:markh] [:mhammond]]

(In reply to Ryan Kelly [:rfkelly] from comment #7)
> Actually, I can never remember which BSOs are encrypted and which are not,
> but the encryption might also provide protection here and mean we can get
> away without auth.

Ah, yeah, good point - the sentinel *is* encrypted, which seems to lower the risk significantly - the worst that I could see happening is someone maliciously writing a bad migration sentinel that the "real" client will discard.

Comment 9

[Bob Micheletto [:bobm]]

Comment on attachment 8652712 [details] [diff] [review]
sync11eol-fxa-credentials.diff

Sounds good to me.

Comment 10

[Bob Micheletto [:bobm]]

(In reply to Ryan Kelly [:rfkelly] from comment #4)
> > * Measures active EOL user base.
> 
> Bob, are you interested in any app-level metrics from this thing, or will
> nginx logs tell us pretty much all we need to know about the ongoing use of
> this tiny service?

Processing the Nginx logs as if they were normal server storage logs should be able to tell us everything we need to know.  Mostly we want to know how many users are hitting the service.

Comment 11

[Bob Micheletto [:bobm]]

(In reply to Ryan Kelly [:rfkelly] from comment #7)

> Bob, do we have any insights into the propertion of Android users in our
> reluctant-upgrader population?

Just under 15%.

Comment 12

[Ryan Kelly [:rfkelly]]

Comment on attachment 8652723 [details] [diff] [review]
sync11eol-authz-handling.diff

I'm going to drop the auth-handling patch, based on discussion here and IRL we're happy to depend on sync's strong encryption to protect the user here.

Comment 13

[Ryan Kelly [:rfkelly]]

fxa_credentials handling added in:
https://github.com/mozilla-services/sync11-eol-server/commit/e74c166dd1d35de5b50d0e5723df2c520abfd488

Bob, I've prepared tag v0.1.0 in the above repo, should be ready for deployment.

Comment 14

[Nick Alexander :nalexander [he/him]]

(In reply to Ryan Kelly [:rfkelly] from comment #12)
> Comment on attachment 8652723 [details] [diff] [review]
> sync11eol-authz-handling.diff
> 
> I'm going to drop the auth-handling patch, based on discussion here and IRL
> we're happy to depend on sync's strong encryption to protect the user here.

Sorry, late to this party -- but I agree with this choice.  If an attacker writes a bad meta/fxa_credentials, the failure mode for Android would be around [1].  We'd drop the record on the floor (silently) and not do anything new until info/collections revealed a new meta/fxa_credentials was available.  Certainly not ideal, but not catastrophic either.

[1] https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/sync/MigrationSentinelSyncStage.java?offset=2300#206

Comment 15

[Bob Micheletto [:bobm]]

Stage EOLinator deployed:

$ curl -v https://eolinator.stage.mozaws.net/1.1/FAKEUSER/meta/global
* About to connect() to eolinator.stage.mozaws.net port 443 (#0)
*   Trying 52.3.92.115...
* Connected to eolinator.stage.mozaws.net (52.3.92.115) port 443 (#0)
> GET /1.1/FAKEUSER/meta/global HTTP/1.1
> User-Agent: curl/7.29.0
> Host: eolinator.stage.mozaws.net
> Accept: */*
> 
< HTTP/1.1 513 SERVICE EOL
< Content-Type: application/json; charset=UTF-8
< Date: Thu, 10 Sep 2015 23:07:36 GMT
< Server: openresty/1.7.0.1
< X-Weave-Alert: {"url": "https://support.mozilla.org/kb/how-update-new-sync", "message": "The sync1.1 service has been shut down", "code": "hard-eol"}
< Content-Length: 1
< Connection: keep-alive
< 
* Connection #0 to host eolinator.stage.mozaws.net left intact

Comment 16

[Bob Micheletto [:bobm]]

Where should we redirect https://services.mozilla.com/privacy-policy to? https://accounts.firefox.com/legal/privacy seems like a reasonable choice, but perhaps there's a better one.

Comment 17

[Ryan Kelly [:rfkelly]]

https://accounts.firefox.com/legal/privacy seems like an appropriate choice, I don't think new-sync has a separate document.

Comment 18

[Bob Micheletto [:bobm]]

Production EOLinator service deployed using standard Cloud Operations deployment pipeline.

$ curl -v https://eolinator.services.mozilla.com/1.1/NOTREAL/meta/global
* About to connect() to eolinator.services.mozilla.com port 443 (#0)
*   Trying 54.187.119.69...
* Connected to eolinator.services.mozilla.com (54.187.119.69) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_DHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
* 	subject: CN=*.services.mozilla.com,O=Mozilla Foundation,L=Mountain View,ST=CA,C=US
* 	start date: Apr 08 00:00:00 2014 GMT
* 	expire date: Oct 26 12:00:00 2016 GMT
* 	common name: *.services.mozilla.com
* 	issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US
> GET /1.1/NOTREAL/meta/global HTTP/1.1
> User-Agent: curl/7.29.0
> Host: eolinator.services.mozilla.com
> Accept: */*
> 
< HTTP/1.1 513 SERVICE EOL
< Content-Type: application/json; charset=UTF-8
< Date: Thu, 24 Sep 2015 00:27:42 GMT
< X-Weave-Alert: {"url": "https://support.mozilla.org/kb/how-update-new-sync", "message": "The sync1.1 service has been shut down", "code": "hard-eol"}
< Content-Length: 1
< Connection: keep-alive
< 
* Connection #0 to host eolinator.services.mozilla.com left intact

Closing this bug.