1109809 - Change FxA/Basket API integration to use queues and notifications

Status: RESOLVED INCOMPLETE

(Cloud Services :: Server: Firefox Accounts, defect)

Description

[Chris Karlof [:ckarlof]]

This will help address reliability issues on both ends by giving Basket more control over when and how they process the notifications.

Comment 1

[Ryan Kelly [:rfkelly]]

This bug is part of a 2015Q1 goal for the FxA team.  I've started fleshing out an "fxa-notification-server" API that could be the core of a solution here:

  https://github.com/mozilla/fxa-notification-server/blob/master/docs/api.md

The world I'd like to see goes something like this:

* The FxA auth server starts publishing account-create and account-delete events to the above-linked API, and forgets all about its special-case handling of Basket notifications.

* The Basket service gets OAuth relier credentials that allow it to query the above-linked API to find and process newly-created accounts.  It can apply whatever logic it needs to handle them without cooperation from the FxA auth server.

* We put a nice high-level API for the above into https://github.com/mozilla/PyFxA, so that Basket can focus on its own logic rather than on interfacing with FxA.

* As a bonus, Basket can use the same technique to find account-deletion events and process them per Bug 1066384.

The basket-side consumer of this API could run in a variety of different ways - as a persistent background process perhaps, or spawned in response to a push notification, or even via a simple cronjob.

The important thing is that we switch from "FxA pushes new accounts into Basket" to "Basket pulls new-account data out of FxA".  Logic would look something like this:

    while True:
        events = [grab a batch of events from fxa-notification-server]
        while events:
            for evt in events:
                if evt.typ == "create":
                    [pull email, locale, etc data from fxa-profile-server]
                    [add or update account in basket database]
                elif evt.type == "delete":
                    [unlink account in basket database]
            [acknowledge processing of this batch of events]
            events = [grab the next batch of events from fxa-notification-server]
        [go back to sleep until e.g. woken by push notification]


Paul, what do you think about this approach as a high-level concept?

Comment 2

[Paul [:pmac] McLanahan]

I love it. I don't think we need anything even as fancy as a push notification. We can simply run on a schedule and queue jobs on our end to process. All of our processing code is in a celery (async) task already, so it'd just be a matter of firing them off via a FxA-pull script instead of the view we're using now.

I say go for it, and as soon as you're ready I don't think the basket mods are going to be difficult at all.

Comment 3

[Ryan Kelly [:rfkelly]]

Great, thanks for the fast feedback, I will keep you updated when things are ready on our end.

This approach may also help us deal with accounts created before the basket integration was set up, per Bug 1083527.

Comment 4

[Paul [:pmac] McLanahan]

Good point. You should be able to load up the queue with whatever accounts you want :)

Comment 5

[Ryan Kelly [:rfkelly]]

We're doing this now, see e.g. 1066384.  But there's no useful context in this ancient bug so I'm closing it out.