1198633 - Contacts sync architecture discussion

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Sync, defect)

Description

[Michiel de Jong [:michielbdejong]]

In addition to storing/syncing the history and passwords of a device (in development), and making bookmarks and open tabs from other devices available (planned), we want to sync contacts. There are several ways to do this, and syncing contacts touches on importing contacts from other sources, as well as on exporting contacts to other sources, and on merging contacts from multiple sources (e.g. merging an email-only contact with a phonenumber-only contact of the same physical person). Apart from that, the desire to self-host may be bigger for contacts than for other types of data, for instance if a user already runs a self-hosted contacts server.

For history, passwords, bookmarks and open tabs we use the Syncto bridge, to integrate with the existing Firefox Sync service, where people already have data stored from other Firefox products.

Where should we store a user's contacts? Options:

1. Add a collection to Sync and access it over Syncto.
Pro's:
  * no extra servers to deploy
Con's: 
  * the Sync server can wipe out a user's data at will, and this is probably more of a problem for contacts data than for bookmarks data.
  * still need separate mechanism for import/export with other (silo) addressbooks

2. Deploy a Kinto server and access it with kinto.js
Pro's:
  * no extra code needed on the client (Synchronizer already uses kinto.js)
  * Unlike Sync servers, Kinto servers will not wipe a user's data
  * Removes dependency on Syncto, at least for contacts
Con's:
  * if a user already self-hosts a contacts server, it is likely to have a CardDav API, but obviously not a Kinto API. So self-hosters need to deploy their own Kinto server (but they need to deploy their own FxA and Syncto server anyway)
  * this architecture allows for importing contacts from Gmail/Facebook/Outlook, but periodically importing changes from there is more complex, as is exporting to there

3. Deploy a CardDav server and extend Synchronizer so that it talks CardDav
Pro's:
  * self-hosters with existing CardDav servers can import/export to that for free.
  * people with access to a third-party CardDav server can do the same.
  * if Synchronizer uses CardDav instead of Kinto for contacts sync, then we remove the need for separate import/export functionality from/to third-party CardDav server, as long as the user only wants import/export with one third-party source (see also next option)
Con's:
  * if people use a third-party CardDav server that's incompatible with our client code, we're in for a lot of debugging and error checking
  * Sync is hard, and it will be quite a lot of work (and take more time) to make Synchronizer handle all edge-cases of not only the Kinto protocol, but also the CardDav protocol
  * It is not clear if and how a user who syncs their phone contacts with for instance Outlook, wants all their Outlook contacts to appear on their phone, and also wants all their SIM-card contacts to appear in their Outlook addressbook
  * If a user syncs their phone with their Outlook, they might still want to import their Facebook/Twitter/Gmail contacts, in which case we still need separate import/export functionality for that, and it's also not clear whether a user wants to see all their Twitter contacts in their phone addressbook, also wants them in their Outlook addressbook.
  * Not all silos have a CardDav API, so once we go down this route, we might see ourselves implementing sync not only for Kinto and for CardDav, but also for the proprietary Contacts APIs of various silos.
  * It is unclear whether many people have one central place where all their contacts are before they FTU their phone. People may also have e.g. their work contacts on a CardDav-compatible server, but their private contacts on a different one, or on some silo, or not really collected anywhere at all.

4. Deploy both Kinto and CardDav servers, storing CardDav credentials on Kinto
Pro's:
  * We can link multiple CardDav servers to one account, resolving some of the issues of option 3.
Con's:
  * Syncing to multiple sources is more complex than syncing to just one source
  * It's not clear how to deal with contacts that exist on several sources

5. Store "master" addressbook on Kinto, and remember which data was imported (read-only) from which source
Pro's:
  * allows for importing new changes from external sources without doing a full new import
  * does not require two-way sync / conflict detection and resolution in CardDav and silo-specific clients
  * allows users to import email-only contacts from Outlook without having to push e.g. phonenumber-only SIM-card contacts to there

6. Same as 5, but with automatic read/write sync-back to sources
Pro's:
  * nice to have :)
Con's:
  * more work :)

Comment 1

[Michiel de Jong [:michielbdejong]]

By the way, end-to-end encryption (where the storage provider does not get to see the data, like we already do for bookmarks etc.) is possible with:

1: yes
2: yes
3: no
4: no
5: yes
6: yes

Comment 2

[Sean Lee [:seanlee][:weilonge]]

Thanks a lot for Michiel's research.

In Contacts app, the contacts list can be imported from Gmail and Outlook. (only import but export.)

If contacts synchronization includes Gmail and Outlook, we have to consider that the import/export behavior between Contacts app and FxSync are compatible.

Comment 3

[Dale Harvey (:daleharvey)]

CardDav is the industry standard for syncing address book details so I believe full support for syncing with CardDav is a fairly hard requirement. In terms of simplicity I would probably lean towards using carddav entirely and using firefox accounts for the auth since it just means less moving parts. If the Synchroniser approach is working well with syncto however then it seems reasonable to keep the client code the same and have synchroniser talk to carddav

Comment 4

[Francisco Jordano [:arcturus] [:francisco]]

+1 to carddav first, as a bare minimum for interoperability.

Another point is, what happen with users that don't understand what's a carddav server, should they also be able to store their data 'outside' of the phone, and by outside could mean using that data in a normal browser (bug 1198122). That's an interesting question that we should check with the product team.

Btw, contacts allow merging right now, but since we just have a data source (for saving), mozcontacts, we check against that.

Comment 5

[Michiel de Jong [:michielbdejong]]

Summary of the current status: there are a lot of options, and we don't know yet. :)

Maybe we can continue this discussion (both for approach and for road map / time planning) when we're all together in Mozlando in December, but for now we're concentrating our efforts only on syncing collections that already exist on FxSync, and contacts is not in that list.

Comment 6

[Paul Theriault [:pauljt] (no longer reading bugmail)]

(In reply to Michiel de Jong [:michielbdejong] from comment #1)
> By the way, end-to-end encryption (where the storage provider does not get
> to see the data, like we already do for bookmarks etc.) is possible with:
> 
> 1: yes
> 2: yes
> 3: no
> 4: no
> 5: yes
> 6: yes

Unless Mozilla as an organisation is considering a shift in its approach to user data, I would expect that end-to-end encryption would be a must have requirement. However that would seem to contradict the advice in comment 3 & 4 that Carddav is a must have.

Does Mozilla want to get into the business of storing private user data? That's a question above my pay-grade.

Is there a middle-ground approach here? IE Support carddav to sync with external contact repositories, and then use Sync to move contacts between Firefox instances? Actually I guess this is basically what Michiel is suggesting in option 1, with "separate mechanism for import/export with other (silo) addressbooks" being carddav.

Comment 7

[Michiel de Jong [:michielbdejong]]

Discussion continued(In reply to Paul Theriault [:pauljt] from comment #6)
> Does Mozilla want to get into the business of storing private user data?

We're now discussing a focus more on improving the existing one-shot contacts import to make it repeatable, add more choice of sources (especially CardDAV), and adding export to whatever sources the user imported from. So then we let the user choose their data storage provider, and Mozilla provides just the device, the cloud behind it is an open market which is independent from the device you buy.
 
> Is there a middle-ground approach here? IE Support carddav to sync with
> external contact repositories, and then use Sync to move contacts between
> Firefox instances?

If a user uses Gmail contacts or Outlook contacts or their own server with CardDAV, then they can already move contacts between Firefox instances via that. So it saves us a lot of work. :)

We're in the process of creating a mailing list (bug 1221920) and will continue this discussion there.

Comment 8

[Michiel de Jong [:michielbdejong]]

Recent discussions (which are not on there because it hadn't been created yet, but needinfo me if you want me to forward the thread to you) have focused more on syncing contacts with a non-Mozilla server (CardDAV or other), and less on Mozilla offering any hosting.

So that's quite an important change of perspective, for FxOS Data Sync in general.

Closing this discussion-bug as we move the discussion to https://lists.mozilla.org/listinfo/dev-fxos-sync, please subscribe and post there.