446444 - [meta] Sync for Thunderbird

Status: ASSIGNED

(Thunderbird :: General, enhancement)

Description

[Dan Mosedale (:dmosedale, :dmose)]

Comment 1

[Dan Mosedale (:dmosedale, :dmose)]

Attachment: minor refactoring, patch 1 (diff -w, for review)

This patch does some minor refactoring to make it Weave amenable to embedding in multiple applications.  It also adds a Thunderbird embedding.  A decent portion of the patch is code motion.  I've squashed all the regressions to the Firefox embedding that I've found, but that's not to say that there aren't more.

Note that to test the Thunderbird embedding, a build with a patch to Thunderbird to include loginmanager is required.  I'll attach that to this bug for the interested.

Comment 2

[Dan Mosedale (:dmosedale, :dmose)]

Attachment: minor refactoring, patch 1 (checked in)

Comment 3

[Dan Mosedale (:dmosedale, :dmose)]

Note that this patch merely hooks up the UI and gets signin working; it doesn't yet do anything.  That's up next...

Comment 4

[Dan Mosedale (:dmosedale, :dmose)]

Attachment: make Tb build with login manager

Thanks to Standard8 for putting this baby together.

Comment 5

[Dan Mills [:thunder]]

Comment on attachment 330604 [details] [diff] [review]
minor refactoring, patch 1 (diff -w, for review)

Looks good, though I haven't actually tested it.

Note this patch includes an unrelated change to set the username/password during verifyPassphrase (but it's ok to commit that too).

Comment 6

[Dan Mosedale (:dmosedale, :dmose)]

Pushed.

Comment 7

[Dan Mosedale (:dmosedale, :dmose)]

Using weave in a debug build asserts; see bug 447882.  The easy, but clearly incorrect, workaround is to comment out the assertion in secasn1d.c in NSS.

Comment 8

[Dan Mosedale (:dmosedale, :dmose)]

Attachment: make Tb build with login manager, comm-central portion

Comment 9

[Dan Mosedale (:dmosedale, :dmose)]

Attachment: make Tb build with login manager, mozilla-central portion

Comment 10

[Dan Mosedale (:dmosedale, :dmose)]

Attachment: tb sync engines WIP patch, v1

Here's a work-in-progress patch that does some basic syncing of identities and accounts.  There are still a bunch of issues, and identities don't seem to be working quite correctly at the moment.  A number of other engines are going to be necessary; nsIMsgIncomingServers and SMTP servers are at the top of the list.

Comment 11

[Mark Banner (:standard8)]

Comment on attachment 331220 [details] [diff] [review]
make Tb build with login manager, comm-central portion

TB now builds with login manager, so this patch is obsolete.

Comment 12

[Mark Banner (:standard8)]

Dan's not working on this at the moment, and quite a bit of weave has been rewritten, so the patches here may not be of much use.

However, we'd like to see this happen as it'd be a good tool for Thunderbird and I think would be suitable as a student project (or indeed for anyone else to pick up on).

Comment 13

[Peter Lairo]

Requesting blocking Weave 1.0. 

After browsing, e-mail is the most commonly performed activity on the web; and the hardest to set up if your settings (IMAP, SMTP, etc.) are away on your "other" PC.

Comment 14

[thanhbv]

I have written a Firefox extension (clicknlearn: http://code.google.com/p/clicknlearn/ ) that have a Weave engine to synchronize some data. So I think I have some experiences.
I'm quite busy now so can't take & work on this bug :D.
But if someone takes bug & need a co-worker then I am very please.

Comment 15

[Mike Connor [:mconnor]]

This will not block Weave 1.0.

Comment 16

[Shane Caraveo (:mixedpuppy)]

I've just implemented a patch for weave that allows me to create a weave extension implementing thunderbird specific support.  I currently (works on my box) have address books syncing between profiles.  

I choose the route of a second extension to avoid having tb specific code in weave.  This should also allow any other addons (at least demonstrate how to) or even other xul apps to add support for using weave.

You can see it at http://bitbucket.org/mixedpuppy/weaver/

I'll post the weave patch I'm using here.

Comment 17

[Shane Caraveo (:mixedpuppy)]

Attachment: weave patch

patch to allow me to add functionality for thunderbird via a second addon.

Comment 18

[Ed Lee :Mardak]

Comment on attachment 439105 [details] [diff] [review]
weave patch

>+++ b/source/modules/service.js
>+try {
>+  Cu.import("resource://weave/engines/bookmarks.js", Weave);
>+} catch(e) {
>+  // fails on thunderbird due to inclusion of utils.js for places
>+  // bug 548382, it doesn't seem like anything from utils.js is
>+  // directly used in bookmarks.js
>+}
PlacesUtils.archiveBookmarksFile is provided by places' utils.js file.

Comment 19

[Ed Lee :Mardak]

  createRecordFromCard: function ABStore_createRecordFromCard(card) {
    let record = new ABRec();
    record.id = getItemUUID(card, this._book, this._log);
    let data = {};
    let properties = card.properties;
    while (properties.hasMoreElements()) {
      var property = properties.getNext().QueryInterface(Ci.nsIProperty);
      this._log.debug("  prop: "+property.name+" value "+property.value);
      if (this.__ignoreProperties.indexOf(property.name) < 0)
        data[property.name] = property.value;
    }
    record.json = JSON.stringify(data);
    return record;
  },

Hrmm.. I wonder if there's a better way to store the data instead of collapsing it into a JSON string. I suppose the main issue is that nsIAbCard has a large number of default properties as well as can take any other properties, like the UUID one you added.

http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/public/nsIAbCard.idl

Comment 20

[Shane Caraveo (:mixedpuppy)]

(In reply to comment #18)
> (From update of attachment 439105 [details] [diff] [review])
> >+++ b/source/modules/service.js
> >+try {
> >+  Cu.import("resource://weave/engines/bookmarks.js", Weave);
> >+} catch(e) {
> >+  // fails on thunderbird due to inclusion of utils.js for places
> >+  // bug 548382, it doesn't seem like anything from utils.js is
> >+  // directly used in bookmarks.js
> >+}
> PlacesUtils.archiveBookmarksFile is provided by places' utils.js file.

ah, wonder how I missed that before...anyway, this was a quick hack to get up and running, I don't really like it.

Comment 21

[Shane Caraveo (:mixedpuppy)]

(In reply to comment #19)

> Hrmm.. I wonder if there's a better way to store the data instead of collapsing
> it into a JSON string. I suppose the main issue is that nsIAbCard has a large
> number of default properties as well as can take any other properties, like the
> UUID one you added.

That's exactly why I did it this way.  My first attempt was just to export a vcard from the nsIAbCard, but that didn't export all the data.

Comment 22

[Igor Velkov]

Doing the same I saw a properties witd default alues didn't enumerates.

Comment 23

[Dan Mosedale (:dmosedale, :dmose)]

I wonder how card it would be to whip up a JS component that exported addrbook cards as PoCo JSON...

Comment 24

[Shane Caraveo (:mixedpuppy)]

(In reply to comment #23)
> I wonder how card it would be to whip up a JS component that exported addrbook
> cards as PoCo JSON...

was just looking at poco now that I am aware of it...

Comment 25

[Dan Mills [:thunder]]

This is hot, guys.

That is all.

Comment 26

[Shane Caraveo (:mixedpuppy)]

Attachment: weave patch

diff from weave-ext repo on bitbucket

Comment 27

[Shane Caraveo (:mixedpuppy)]

I've put together an xpi that bundles weave for tb, and weaver, the extension providing tb specific code and abcard syncing.  

For weave, I forked into a weave-ext repo on bitbucket (http://bitbucket.org/mixedpuppy) and created a branch for the modifications, weaver also exists there in another repo.  The xpi is in the downloads section of the weaver repo.

When the weave team has bandwidth, I'd like to talk about how to start getting the tb support properly back into weave.  

There is one item that would be nice to have earlier feedback on, which is the change in prefs.js to use the application id as part of WEAVE_PREFS_GUID.  That change would not be compatible with the current weave addon, I need to handle that better.  I've attached the diff for feedback.

Comment 28

[Leho Kraav (:macmaN @lkraav)]

is there anyone actively hacking on this? last update seems to be from 7 months ago.

Comment 29

[Shane Caraveo (:mixedpuppy)]

(In reply to comment #28)
> is there anyone actively hacking on this? last update seems to be from 7 months
> ago.

The latest code is in my bitbucket account at http://bitbucket.org/mixedpuppy (weave-ext + weaver projects), last updated 3 months ago.  I'm going to be updating again soon to get on the latest sync code, and then I'll be pushing new bugs with patches to get them integrated into sync.

Comment 30

[Leho Kraav (:macmaN @lkraav)]

thanks for the update, already clicked follow on your repository a bit earlier, will chip in when i can.

Comment 31

[Shane Caraveo (:mixedpuppy)]

I'm moving patch reviews to bug 621198, if there are general comments they can continue here.

Comment 34

[Allison Naaktgeboren :ally]

sync triage: given the decision to stop development on thunderbird, is this bug still valid?

Comment 35

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Since this is still something the community could possibly step up and do, I don't think we should close the bug. Or is there any part of this bug that requires massive non-community work?

Comment 36

[Allison Naaktgeboren :ally]

:Fallen, 
It costs Mozilla a nontrivial amount of money(in staff & hardware) to support additional engines. The real expense of a sync feature is support, not generation.

If you produced the perfect engine magically tomorrow, we might not be able to take it because we might not be able to pay for it. 

As far as I know, there's no way for the community to help with that. (though if there is, I'd love to hear about it)

That must sound deeply unfair. I am sorry. I wish that was not the case.

Comment 37

[Jens Müller (:tessarakt)]

Well, nobody says that Mozilla has to provide the sync servers ... People can just use their private webhosting packages ...

Comment 38

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Sync is configured with the Mozilla Servers by default, so that might not work out because it will put up expectations. Isn't the server side laid out for any sync engine? I thought its just a matter of creating a client side sync engine, which could possibly be done in an extension? Of course it would mean more traffic, but as you know the user base of Thunderbird isn't as large as the Firefox userbase. Additionally there is an amount of enterprise users, which are probably not interested in synchronizing email account settings using weave.

I think I'd leave the final decision to someone from the Thunderbird team, but it would be sad to bury this.

Comment 39

[Philip Chee]

Everyone please ignore Ally Naaktgeboren. What he says isn't Mozilla policy.
The current Sync architecture is explicitly designed to allow extensions to plug in custom sync engines. There are multiple extensions on addons.mozilla.org that register custom engines to sync their data to the mozilla sync servers. Examples:

https://addons.mozilla.org/en-us/firefox/addon/reqpolsync/
http://adblockplus.org/development-builds/firefox-sync-support-added

There was even an official Mozilla GSoC mailnews sync project:
http://hariniachala.blogspot.com/2010/04/google-summer-of-code-2010.html
http://hariniachala.blogspot.com/2010/06/adding-thunderbird-support-to-new-mail.html

Comment 40

[Mike Connor [:mconnor]]

This is one of those impossibly broad bugs to judge impact for, which I think is contributing to the somewhat polarized discussion here.

"Sync account settings, prefs, etc" doesn't concern me at all.  If every single Tbird user signs up... I'll actually be pretty happen with that!

That said, "Sync read/unread state of newsgroups" is probably a massive chunk of write-happy data (I think Kairo ended up with 155k records when he tried), and would lead to a pretty savage chunk of load.  Doing this might result in blocking Thunderbird clients... so let's not do that.

Comment 41

[Mike Connor [:mconnor]]

(In reply to Philip Chee from comment #39)
> What he says isn't Mozilla policy.

Just to be clear here, there isn't an official Mozilla policy either way.  There's a tradeoff and discussion for everything we do.  If an add-on or other client causes problems for the service, we can and will take steps to prevent that.  If Thunderbird support was deemed an unsupportable expense, we would have to block those clients.  Having an architecture that's designed to be highly flexible and open does not mean we're giving the community a blank cheque to do whatever they want.  TANSTAAFL applies here, as in all things.

Comment 42

[Allison Naaktgeboren :ally]

Philip Chee, I am not expressing concerns about the client. I am expressing concerns about the mozilla owned & supported servers to which this may connect, which is the impression I got from the patch. 


What I don't want is for you guys to invest lots of time writing an engine only to have it blocked on the server because of a complication no one warned you about. I don't want that for anyone.

I/We'd be happy to provide design review to ensure that it's not going to generate excessive cost/load. 

Please don't call me 'he', as I am a 'she'.

Comment 43

[Mike Conley (:mconley) (:⚙️)]

I'd be happy starting simply with TB prefs, accounts, and add-ons.

I think that's an attainable goal, and highly unlikely to be a storage quota threat.

Comment 44

[Philipp von Weitershausen [:philikon]]

(In reply to Mike Conley (:mconley) from comment #43)
> I'd be happy starting simply with TB prefs, accounts, and add-ons.

You should be able to get prefs and add-ons for nearly free, meaning at very little or no review cost to the Sync team.

Comment 45

[Gregory Szorc [:gps]]

I think it makes little sense for Thunderbird to incur the effort to port Sync *right now*. I am currently making significant changes to the Firefox Sync JS code base and it will be much easier (and likely less buggy) if the Thunderbird port waited until my work is mostly done. But, if someone wants to go through at least double the effort to get things maybe 3 months earlier, I won't stop you.

Comment 46

[Mike Conley (:mconley) (:⚙️)]

(In reply to Gregory Szorc [:gps] from comment #45)
> I think it makes little sense for Thunderbird to incur the effort to port
> Sync *right now*. I am currently making significant changes to the Firefox
> Sync JS code base and it will be much easier (and likely less buggy) if the
> Thunderbird port waited until my work is mostly done. But, if someone wants
> to go through at least double the effort to get things maybe 3 months
> earlier, I won't stop you.

That's good to know. I've also heard rumblings of Persona being used as an authentication mechanism, though that might just be rumours.

Waiting for stability makes sense. Or, alternatively, working off of your target spec.

Comment 47

[Gregory Szorc [:gps]]

(In reply to Mike Conley (:mconley) from comment #46)
> That's good to know. I've also heard rumblings of Persona being used as an
> authentication mechanism, though that might just be rumours.
> 
> Waiting for stability makes sense. Or, alternatively, working off of your
> target spec.

Persona will be leveraged, yes. The master tracking bug is bug 744321.

Comment 48

[Philip Chee]

> Philip Chee, I am not expressing concerns about the client. I am expressing concerns
> about the mozilla owned & supported servers to which this may connect, which is the
> impression I got from the patch. 

Thank you for the clarification. That appears to be an old patch. The GSoC project demonstrated that syncing messages was impractical.

> Please don't call me 'he', as I am a 'she'.
Oh very sorry. I didn't know that Naaktgeboren is a female name.

Comment 49

[Igor Velkov]

> syncing messages was impractical.

... but syncing of tags was very, very useful for imap.

Comment 50

[Andrea Monni]

We should probably create new bugs with focus on a narrow and given set of items to synchronize, such as prefs, tags, etc.

Right now I don't think it makes sense to think about synchronizing messages (or contacts for that matter) as we have industry standards good and accepted enough not to have to reinvent the wheel.

Comment 51

[Robert Kaiser]

(In reply to Mike Connor [:mconnor] from comment #40)
> That said, "Sync read/unread state of newsgroups" is probably a massive
> chunk of write-happy data (I think Kairo ended up with 155k records when he
> tried), and would lead to a pretty savage chunk of load.  Doing this might
> result in blocking Thunderbird clients... so let's not do that.

Just FYI, that was doing state for every single message as a separate object to sync, which is probably the wrong way to go there. If doing one object per group/folder is feasible, it probably could be done with much less storage space. So, we learned a good deal from that GSoC experiment in case we ever want to go in that direction again. :)

Comment 52

[Richard Newman [:rnewman]]

Moving this to Thunderbird for feature tracking.

Most likely this bug as-is should be closed; if Thunderbird ends up integrating Sync (ideally a future rewrite), then we can file individual bugs for engines and any support that's necessary from the Services Integration team for the main Sync codebase.

Bug 621198 can be used to track the latter if you would like.

Comment 54

[Quick_Silver]

Does anybody know about the actual status of Thunderbird implementing the sync feature ?

Comment 55

[Richard Newman [:rnewman]]

I haven't heard any news. But broadly, now is not a great time to be taking dependencies on Sync; there's a lot of stuff up in the air, so there's a lot of risk of wasted effort.

Comment 56

[Quick_Silver]

I'm a little bit confused now ...
Reading the German wikipedia, I stumbled upon some information, that says "Thunderbird won't be actively developed any further, only security updates will be made".
=> http://de.wikipedia.org/wiki/Mozilla_Thunderbird

Sounds to me, as if there won't be an official sync functionality ...
So the efforts that one invests here won't be wasted !

Comment 57

[Mark Banner (:standard8)]

(In reply to Quick_Silver from comment #56)
> I'm a little bit confused now ...
> Reading the German wikipedia, I stumbled upon some information, that says
> "Thunderbird won't be actively developed any further, only security updates
> will be made".
> => http://de.wikipedia.org/wiki/Mozilla_Thunderbird

That is wrong, the statement only pertains to changes made by Mozilla employees. The community is welcome to provide (and has been) new features & other changes to Thunderbird.

Comment 59

[Craig Arno]

I hope that like Firefox Sync, what gets synced in Thunderbird can be enabled/disabled.

As a user I'd like to see the following items Sync:

- Adaptive Junk Mail Controls (learned data)
- Account Settings (by individual account as different "seats" have different needs)
- Add-ons
- Add-on Settings, (Lightning in particular)
- Filter config

  Things I don't really need Sync'ed and would like to disable if they exist:
- Addressbook (handled by CardDAV through SOGo Connector to OwnCloud Server with HTTPS)
- Calendar (currently handled using CalDAV to OwnCloud Server with HTTPS)

I use a self hosted Sync Server (OwnCloud).  The new Sync 1.5/2.0 efforts will temporarily disrupt Self hosted (OwnCloud) Sync; https://bugzilla.mozilla.org/show_bug.cgi?id=989756 until the new system can be implemented; https://github.com/owncloud/mozilla_sync/issues/33 It seems Firefox Accounts/FxA also needs to be implemented before Sync can be moved to the new 1.5/2.0 efforts.

I'm particularly keen on seeing Junk Mail Adaptive data sync'ed.  The reason is I had Junkmail all tuned up and working on my Linux workstation and decided to put Thunderbird on my Windows based Laptop.  One of the first things that happened when Junk mail filtering was enabled is most of my now dead parents emails were flagged as "junk" and removed.  Fortunately they weren't deleted, but I spent the remainder of the evening locating and retrieving as many of these emails as I could.  I believe a careful implementation of Sync in this area (i.e. disable junk mail controls first, then sync adaptive data, then re-enable junkmail controls with newly sync'ed adaptive data) would go a long way toward addressing the experience I had.  Both Thunderbird instances (I actually use about 10 of these) connect to the same IMAPS mail server(s).

For Account settings, I might be inside company A in the morning and company B in the afternoon.  I wouldn't want company A settings or email to show up at company B and the reverse as a result of using Sync.  I wouldn't want either one to show up at home, unless I specifically had authorization from that employer and wanted this feature.  I otherwise use IMAPS with local storage turned off to monitor the accounts I need.

Lastly, I'd like to thank everyone who's looking at this.  I'm impressed with all the effort I see and have read here.  So Thank you.

Comment 60

[Future]

I think this is a great idea. The service could be set up, as already mentioned here, accordingly the sync for Firefox : Thunderbird Sync.

Containing following components of a profile to be focused in sync:
- Filter
- Email Accounts (Folder settings, Signatures, Server settings...)
- Settings (General, View, Security)
- Addons + their configuration (addons should be synced before settings...otherwise newly added addons will overwrite synced settings with addon default settings)
- Passwords (with feature to sync master password)
- Contacts/Calendar not included, because they can be synced in many different ways (IMAP folder, dropbox, webdav, card- and caldav). In order to stay compatible with web standards, I would suggest Thunderbird to implement a standard CardDav functionality (as CalDav already works). But this is another feature request (address book to use CardDAV to enable sync (like apple)).

Looking forward for this feature. Would be nice, if the sync service is provided as update for the Mozilla weave server.

Comment 61

[Daniele "Mte90" Scasciafratte]

any news about this ticket?

Comment 64

[Alexandre]

I think it is not a bug. It is a question of new feature that could be implemented at Thunderbird to solve some problems when we lost the operation system or trye to use another computer.The firefox Sync feature is a great way to solve some problems of lost bookmarks. But at this moment if you lose your filter rules you will got a big problem.

Please , add the firefox sync feature on the Thunderbird.

Comment 65

[Antoine Turmel [:GeekShadow]]

BTW, there is a bounty for this issue:
https://www.bountysource.com/issues/3488825-sync-for-thunderbird

Comment 66

[Future]

I am using Owncloud as well and therefore I am highly interested, if this Thunderbird sync will end up at the latest weave implementation (1.5???) or the old one (1.1) currently working with Owncloud.

The latest (1.5) is not implemented yet with Owncloud (https://github.com/owncloud/mozilla_sync/issues/33). Would be nice, if Thunderbird sync is working together with current Owncloud Sync app (https://apps.owncloud.com/content/show.php/Mozilla+Sync?content=161793)

Comment 68

[Prashant Savaj]

Sorry to Trouble you Cosmin, But i had not found sync. of Thunderbird Question on Bugzilla, So I Posted 

Again Sorry fro that

& Thanks to inform me that Sync for Thunderbird is already in progress.

Best of Luck with Lot of Good Wishes to you & your Team.

Comment 69

[Thomas Sisson]

I vote for this and hope to see it in an Earlybird release. Thanks for all the work.

Tom Sisson

Comment 70

[Chris Andrichak]

+want
Would love to not to have to re-do my "allow remote links from xxxx email" each time I change computers or otherwise have to reset my TBird install. 

And the bountysource.com link above shows $145 fwiw.

Comment 71

[Antoine Turmel [:GeekShadow]]

I think Thunderbird could be done using WebExtensions API in the near future see https://bugzilla.mozilla.org/show_bug.cgi?id=1396172

From comment #60 every features need a WebExtension API

Comment 72

[u601362]

Very sad this has not been implemented already

Comment 75

[Pander]

Personally, I would prefer to use the infra and account I also use for Firefox. This would save on setting up and maintaining a seperate infra and reduce the number of accounts and serviced I use. Would that be possible?

Comment 76

[Pander]

Is there a way to crowdfund this?

Comment 77

[glennie]

Hi,
It will be nice to have it.
Is there a way prioritize this issue? May be with a crowdfunding?
Kind regards

Comment 78

[Geoff Lankow (:darktrojan) [away until April 11]]

I am working on this. You can stop asking now.

I'm turning this bug into a meta-bug. The work will happen in other bugs as there's enough comments here already.

Comment 79

[Geoff Lankow (:darktrojan) [away until April 11]]

If you just received an email about bug 1793378 being marked as fixed, and you're thinking now's the time to try Sync, please read this post I made to the Daily mailing list first.