980289 - Server need to override previous SimplePush URL when registering a new one

Status: VERIFIED FIXED

(Hello (Loop) :: Server, defect)

Description

[Alexis Metaireau (:alexis)]

While Bug 971993 provides storage for the simplepush urls, we should have a mechanism that make them expire after some amount of time.

We need input from the simple push project folks, so I'm putting a needinfo on JR and Dougt here: for how long a simple push url is valid for? Or asked differently, after how long should I consider it not valid anymore?

Comment 1

[JR Conlin [:jrconlin,:jconlin]]

In terms of the spec, a push URL is valid until it's not. There are several ways that this could happen, which are internal to the SimplePush mechanism. 

In terms of the production server, there's no such thing as an "invalid URL".

I believe that you may misunderstand the external server's role of the SimplePush URL. Put simply, there is only one SimplePush URL per recipient. It's provided by the client (and endpoint) which requests the SimplePush URL, or receives a SimplePush URL refresh event.

The server should immediately discard any previous SimplePush URL it may have had for the recipient and only use the most recent SimplePush URL.

Comment 2

[Rémy Hubscher (:natim)]

For now, we can identify a user but not a User Agent (client device).

We need to have an unique identifier per User Agent.

On the server it means that we have to store an association of (UserID, UserAgentID and SimplePush URL) with a unique key on the (UserID, UserAgentID).

I am not sure we can currently use the FxA assertion to get this information or if the client need to send additionnal information to us.

Comment 3

[Rémy Hubscher (:natim)]

For the MLP, we temporary don't use FxA but cookie based session.

In that case we will have a different session for each UserAgent.

I propose to store only one SimplePush URL per session for now and then when we will be able to use FxA we will add the UserAgent Identifier.

Chris: Is it possible from a FxA assertion to get a UserAgent Unique ID?

Comment 4

[JR Conlin [:jrconlin,:jconlin]]

tl;dr: You shouldn't be retaining old SimplePush endpoints for any reason. 

Simplepush works by providing 3rd party servers with an unique URL for an endpoint (yes, I know that's vague, because that's all that is required and gives us flexibility in how we provide that service.) The endpoint resolves to a User Agent ID + a Channel ID. User Agent points to a given device + account (if I remember correctly) and Channel ID points to a given app.

The ChannelID is actually determined by the SimplePush client running on the User Agent. It is a simple map of ID to the corresponding application. 

When a user "logs off" of a mobile device, it is my understanding that it is a complete system shift. That means that all apps associated with the prior account are shut down. This means that all SimplePush clients no longer get updates, since they no longer have any information about the other accounts. Granted, I'm not a huge fan of this approach, but this is not a decision I get to make. 

I'm happy to go into details about how the server is blissfully ignorant in handling endpoint URLs, but I think it's a bit out of scope for this bug.

As for your needs, I'd offer the following:

1. Simplepush handles generating the UserAgentID, ChannelID and endpoint for you. 
2. You get to see the endpoint, because that's all that you need to know for SimplePush.
3. You get an endpoint that is 100% valid and will get to your logged in device until you get a new endpoint, at that time, your server needs to start using that endpoint.

Chances are EXCEPTIONALLY good that IF you get a new endpoint, all of the endpoints for a given user will change. There's no need to send to multiple endpoints (for overlap or any other reason), unless you're specifically using multiple channels for a given user. Even in that case, you just need to use the latest endpoint, relayed by your application on the user's device.

Comment 5

[Rémy Hubscher (:natim)]

Attachment: Link to GitHub PR.

Comment 6

[Rémy Hubscher (:natim)]

This patch is a solution alexis, jrconlin and I chose for the MLP, but the question for ckarlof still stands for the MVP

Comment 7

[Chris Karlof [:ckarlof]]

(In reply to Rémy Hubscher (:natim) from comment #3)

> Chris: Is it possible from a FxA assertion to get a UserAgent Unique ID?

It doesn't exist now, but it's possible. Can you give me a list of requirements for this id? For example, how long does it need to persist? Does it need to persist if the user logs out of a device and logs back in? If a user gets forcibly logged out (e.g., because of password reset on another device), does it need to persist across that re-authentication? When do you need this by?

Comment 8

[Rémy Hubscher (:natim)]

Yes for our use case, we want a User/UserAgent unique identifier.

It would be nice if it was persistant for one User on one UserAgent even if the user logs out and logs back in.

We have a workaround for the MLP but we'd like to have it for the MVP (we should land the loop feature inside FF32 if I am not wrong)

Comment 9

[JR Conlin [:jrconlin,:jconlin]]

(In reply to Chris Karlof [:ckarlof] from comment #7)
> (In reply to Rémy Hubscher (:natim) from comment #3)
> 
> > Chris: Is it possible from a FxA assertion to get a UserAgent Unique ID?
> 
> It doesn't exist now, but it's possible. Can you give me a list of
> requirements for this id? For example, how long does it need to persist?
> Does it need to persist if the user logs out of a device and logs back in?
> If a user gets forcibly logged out (e.g., because of password reset on
> another device), does it need to persist across that re-authentication? When
> do you need this by?

Is https://wiki.mozilla.org/Identity/Firefox_Accounts#Verifier no longer correct? The Verifier will no longer return a FxA ID as part of the verification? 

I currently try to store the user based on this ID, if it is not present, I will generate a SHA256 from the email. It was my understanding that this ID was unique to a user independent of the device they were connecting to. Has this changed?

Comment 10

[Chris Karlof [:ckarlof]]

(In reply to JR Conlin [:jrconlin,:jconlin] from comment #9)
> Is https://wiki.mozilla.org/Identity/Firefox_Accounts#Verifier no longer
> correct? The Verifier will no longer return a FxA ID as part of the
> verification? 

Since the FxA Sync in Fx29 whip-cracking began in late Dec, no one has given that page much love. I'm sure there is some inaccurate information on it. I'll try to resolve that soon.

Our FxA assertion format is not clearly documented, but it's just the standard Persona BiD format, but the principle.email = <FxA uid>@accounts.firefox.com. The domain of that email could be something else if you're not using the production FxA server. We also supply some additional top claims, "fxa-generation" and "fxa-lastAuthAt", and we could provide more, if necessary. For example, we don't currently signal the user's actual email address here, but that would be an easy change, if required. 

Here are the docs that cover this (although they could be more clear): 
https://github.com/mozilla/fxa-auth-server/blob/master/docs/api.md#post-v1certificatesign

> I currently try to store the user based on this ID, if it is not present, I
> will generate a SHA256 from the email. It was my understanding that this ID
> was unique to a user independent of the device they were connecting to. Has
> this changed?

Your understanding is correct. There is no need to hash the email. As stated above, the principle.email provided in FxA assertions is a namespaced FxA identifier and can be used "as-is" as a stable FxA user identifier.

Comment 11

[JR Conlin [:jrconlin,:jconlin]]

Thanks for the update. I understand that the product is in flux and that I shouldn't make any hard requirements or designs based off of it, but I'm still a little bit sadder.

Sigh. The following is not related to this bug, but might be useful.

I hash the email mostly because what I'm storing could be derived as personal information. By hashing, I at least attempt to obscure what devices you have so that bad guys and three letter government agencies would have a harder time getting information about your location and activities. It was hoped that by having an externally generated identifier provided by the log in system, I'd have something that was a meaningless, Long String of Crap that I could store. 

Generating the hash does allow me to separate data from processing and still allows me to worry a tiny bit less about the data store being compromised, but it does mean that it is possible for a pen registry or other mechanism to be injected and that information disclosed. 

It's still the same sort of info being tracked and stored. I was just happy that there was the chance to offer a bit more privacy.

Comment 12

[Chris Karlof [:ckarlof]]

I don't understand your response. Currently, FxA assertions contain no personal information.

Comment 13

[JR Conlin [:jrconlin,:jconlin]]

Ah, perhaps this has changed since the last time I looked at the spec, which was a few months ago.

As I understood, the FxA validator returns an user email (or at the least, an identifier in the pattern of "local@domain"). In this case, the identifier might be [e-mail] or like, since the identifier would be chosen by a user.

If that's the case, then it would be reasonably simple to determine that ckarlof is you, and therefor the location information that we store would reflect to your movements. While the string of letters may not be a social security or government issued ID number, they're still fairly easy to associate to a user.

By hashing the email identifier, I attempt to make theft of the data less meaningful. 

If the validator is no longer returning an email like ID and is instead returning a meaningless ID, then you're right, and I'm worried about nothing. Is this the case?

Comment 14

[Rémy Hubscher (:natim)]

I created bug 984288 to discuss about the privacy concerns with FxA assertions. Let's continue the discussion there and keep this bug for expiration of SimplePush URLs.

Comment 15

[Alexis Metaireau (:alexis)]

(In reply to Chris Karlof [:ckarlof] from comment #7)
> Can you give me a list of requirements for this id?
Sure!

> For example, how long does it need to persist?
Ideally, this information would be attached to the user identity, and would be persisted in the profile of the user.

> Does it need to persist if the user logs out of a device and logs back in?
Yes, we want a way to identify a user device, and we don't really care about the fact he got disconnected.

> If a user gets forcibly logged out (e.g., because of password reset on
> another device), does it need to persist across that re-authentication?

I guess so; What we want to achieve here is to have a way to identify user's devices, and to get a different id for each of these devices.

We don't want/need to have this device ID shared accross users / profiles of the same browser.

> When do you need this by?
We would need this for loop MVP, apparently that would be Fx34, so 2014-07-21 in m-c.

Comment 16

[Alexis Metaireau (:alexis)]

For MLP we simply don't expire the simple push URLs, and will consider to have one SP URL per session cookie.

For MVP, we will use FxA assertions with the feature we're discussing here.

Comment 17

[Mark Banner (:standard8)]

Comment on attachment 8390652 [details] [review]
Link to GitHub PR.

Looks good. r=Standard8

Comment 18

[Rémy Hubscher (:natim)]

https://github.com/mozilla/loop-server/commit/fe657be829dea4507afd2cf68a681868511687f4

Comment 19

[James Bonacci [:jbonacci]]

Closing out the older bugs. The code is newer than this commit.