Open Bug 446444 Opened 16 years ago Updated 2 days ago

[meta] Sync for Thunderbird

Categories

(Thunderbird :: General, enhancement)

enhancement

Tracking

(Not tracked)

People

(Reporter: dmosedale, Unassigned)

References

(Depends on 1 open bug, Blocks 1 open bug, )

Details

(Keywords: meta, Whiteboard: [GS])

Attachments

(3 files, 5 obsolete files)

This patch does some minor refactoring to make it Weave amenable to embedding in multiple applications.  It also adds a Thunderbird embedding.  A decent portion of the patch is code motion.  I've squashed all the regressions to the Firefox embedding that I've found, but that's not to say that there aren't more.

Note that to test the Thunderbird embedding, a build with a patch to Thunderbird to include loginmanager is required.  I'll attach that to this bug for the interested.
Attachment #330604 - Flags: review?(thunder)
Attachment #330604 - Attachment description: minor refactoring, patch 1 → minor refactoring, patch 1 (diff -w, for review)
OS: Mac OS X → All
Hardware: PC → All
Note that this patch merely hooks up the UI and gets signin working; it doesn't yet do anything.  That's up next...
Attached patch make Tb build with login manager (obsolete) — — Splinter Review
Thanks to Standard8 for putting this baby together.
Comment on attachment 330604 [details] [diff] [review]
minor refactoring, patch 1 (diff -w, for review)

Looks good, though I haven't actually tested it.

Note this patch includes an unrelated change to set the username/password during verifyPassphrase (but it's ok to commit that too).
Attachment #330604 - Flags: review?(thunder) → review+
Pushed.
Attachment #330610 - Attachment description: minor refactoring, patch 1 (for checkin) → minor refactoring, patch 1 (checked in)
Attachment #330610 - Attachment is patch: true
Attachment #330610 - Attachment mime type: application/octet-stream → text/plain
Using weave in a debug build asserts; see bug 447882.  The easy, but clearly incorrect, workaround is to comment out the assertion in secasn1d.c in NSS.
Attachment #330611 - Attachment is obsolete: true
Attachment #331220 - Attachment is patch: true
Attachment #331220 - Attachment mime type: application/octet-stream → text/plain
Depends on: 448199
Attachment #330604 - Attachment is obsolete: true
Attached patch tb sync engines WIP patch, v1 — — Splinter Review
Here's a work-in-progress patch that does some basic syncing of identities and accounts.  There are still a bunch of issues, and identities don't seem to be working quite correctly at the moment.  A number of other engines are going to be necessary; nsIMsgIncomingServers and SMTP servers are at the top of the list.
Depends on: 449480
Target Milestone: -- → Future
Comment on attachment 331220 [details] [diff] [review]
make Tb build with login manager, comm-central portion

TB now builds with login manager, so this patch is obsolete.
Attachment #331220 - Attachment is obsolete: true
Attachment #331221 - Attachment is obsolete: true
Blocks: 274840
Dan's not working on this at the moment, and quite a bit of weave has been rewritten, so the patches here may not be of much use.

However, we'd like to see this happen as it'd be a good tool for Thunderbird and I think would be suitable as a student project (or indeed for anyone else to pick up on).
Assignee: dmose → nobody
Requesting blocking Weave 1.0. 

After browsing, e-mail is the most commonly performed activity on the web; and the hardest to set up if your settings (IMAP, SMTP, etc.) are away on your "other" PC.
Flags: blocking-weave1.0?
Component: Weave → General
Product: Mozilla Labs → Weave
QA Contact: weave → general
I have written a Firefox extension (clicknlearn: http://code.google.com/p/clicknlearn/ ) that have a Weave engine to synchronize some data. So I think I have some experiences.
I'm quite busy now so can't take & work on this bug :D.
But if someone takes bug & need a co-worker then I am very please.
This will not block Weave 1.0.
Flags: blocking-weave1.0? → blocking-weave1.0-
Component: General → Experimental Clients
QA Contact: general → experimental.clients
I've just implemented a patch for weave that allows me to create a weave extension implementing thunderbird specific support.  I currently (works on my box) have address books syncing between profiles.  

I choose the route of a second extension to avoid having tb specific code in weave.  This should also allow any other addons (at least demonstrate how to) or even other xul apps to add support for using weave.

You can see it at http://bitbucket.org/mixedpuppy/weaver/

I'll post the weave patch I'm using here.
Attached patch weave patch (obsolete) — — Splinter Review
patch to allow me to add functionality for thunderbird via a second addon.
Comment on attachment 439105 [details] [diff] [review]
weave patch

>+++ b/source/modules/service.js
>+try {
>+  Cu.import("resource://weave/engines/bookmarks.js", Weave);
>+} catch(e) {
>+  // fails on thunderbird due to inclusion of utils.js for places
>+  // bug 548382, it doesn't seem like anything from utils.js is
>+  // directly used in bookmarks.js
>+}
PlacesUtils.archiveBookmarksFile is provided by places' utils.js file.
  createRecordFromCard: function ABStore_createRecordFromCard(card) {
    let record = new ABRec();
    record.id = getItemUUID(card, this._book, this._log);
    let data = {};
    let properties = card.properties;
    while (properties.hasMoreElements()) {
      var property = properties.getNext().QueryInterface(Ci.nsIProperty);
      this._log.debug("  prop: "+property.name+" value "+property.value);
      if (this.__ignoreProperties.indexOf(property.name) < 0)
        data[property.name] = property.value;
    }
    record.json = JSON.stringify(data);
    return record;
  },

Hrmm.. I wonder if there's a better way to store the data instead of collapsing it into a JSON string. I suppose the main issue is that nsIAbCard has a large number of default properties as well as can take any other properties, like the UUID one you added.

http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/public/nsIAbCard.idl
(In reply to comment #18)
> (From update of attachment 439105 [details] [diff] [review])
> >+++ b/source/modules/service.js
> >+try {
> >+  Cu.import("resource://weave/engines/bookmarks.js", Weave);
> >+} catch(e) {
> >+  // fails on thunderbird due to inclusion of utils.js for places
> >+  // bug 548382, it doesn't seem like anything from utils.js is
> >+  // directly used in bookmarks.js
> >+}
> PlacesUtils.archiveBookmarksFile is provided by places' utils.js file.

ah, wonder how I missed that before...anyway, this was a quick hack to get up and running, I don't really like it.
(In reply to comment #19)

> Hrmm.. I wonder if there's a better way to store the data instead of collapsing
> it into a JSON string. I suppose the main issue is that nsIAbCard has a large
> number of default properties as well as can take any other properties, like the
> UUID one you added.

That's exactly why I did it this way.  My first attempt was just to export a vcard from the nsIAbCard, but that didn't export all the data.
Doing the same I saw a properties witd default alues didn't enumerates.
I wonder how card it would be to whip up a JS component that exported addrbook cards as PoCo JSON...
(In reply to comment #23)
> I wonder how card it would be to whip up a JS component that exported addrbook
> cards as PoCo JSON...

was just looking at poco now that I am aware of it...
This is hot, guys.

That is all.
Attached patch weave patch — — Splinter Review
diff from weave-ext repo on bitbucket
Attachment #439105 - Attachment is obsolete: true
I've put together an xpi that bundles weave for tb, and weaver, the extension providing tb specific code and abcard syncing.  

For weave, I forked into a weave-ext repo on bitbucket (http://bitbucket.org/mixedpuppy) and created a branch for the modifications, weaver also exists there in another repo.  The xpi is in the downloads section of the weaver repo.

When the weave team has bandwidth, I'd like to talk about how to start getting the tb support properly back into weave.  

There is one item that would be nice to have earlier feedback on, which is the change in prefs.js to use the application id as part of WEAVE_PREFS_GUID.  That change would not be compatible with the current weave addon, I need to handle that better.  I've attached the diff for feedback.
is there anyone actively hacking on this? last update seems to be from 7 months ago.
(In reply to comment #28)
> is there anyone actively hacking on this? last update seems to be from 7 months
> ago.

The latest code is in my bitbucket account at http://bitbucket.org/mixedpuppy (weave-ext + weaver projects), last updated 3 months ago.  I'm going to be updating again soon to get on the latest sync code, and then I'll be pushing new bugs with patches to get them integrated into sync.
thanks for the update, already clicked follow on your repository a bit earlier, will chip in when i can.
I'm moving patch reviews to bug 621198, if there are general comments they can continue here.
Depends on: 621198
Summary: make weave work in Thunderbird → make sync (formely weave) work in Thunderbird
sync triage: given the decision to stop development on thunderbird, is this bug still valid?
Since this is still something the community could possibly step up and do, I don't think we should close the bug. Or is there any part of this bug that requires massive non-community work?
:Fallen, 
It costs Mozilla a nontrivial amount of money(in staff & hardware) to support additional engines. The real expense of a sync feature is support, not generation.

If you produced the perfect engine magically tomorrow, we might not be able to take it because we might not be able to pay for it. 

As far as I know, there's no way for the community to help with that. (though if there is, I'd love to hear about it)

That must sound deeply unfair. I am sorry. I wish that was not the case.
Well, nobody says that Mozilla has to provide the sync servers ... People can just use their private webhosting packages ...
Sync is configured with the Mozilla Servers by default, so that might not work out because it will put up expectations. Isn't the server side laid out for any sync engine? I thought its just a matter of creating a client side sync engine, which could possibly be done in an extension? Of course it would mean more traffic, but as you know the user base of Thunderbird isn't as large as the Firefox userbase. Additionally there is an amount of enterprise users, which are probably not interested in synchronizing email account settings using weave.

I think I'd leave the final decision to someone from the Thunderbird team, but it would be sad to bury this.
Everyone please ignore Ally Naaktgeboren. What he says isn't Mozilla policy.
The current Sync architecture is explicitly designed to allow extensions to plug in custom sync engines. There are multiple extensions on addons.mozilla.org that register custom engines to sync their data to the mozilla sync servers. Examples:

https://addons.mozilla.org/en-us/firefox/addon/reqpolsync/
http://adblockplus.org/development-builds/firefox-sync-support-added

There was even an official Mozilla GSoC mailnews sync project:
http://hariniachala.blogspot.com/2010/04/google-summer-of-code-2010.html
http://hariniachala.blogspot.com/2010/06/adding-thunderbird-support-to-new-mail.html
This is one of those impossibly broad bugs to judge impact for, which I think is contributing to the somewhat polarized discussion here.

"Sync account settings, prefs, etc" doesn't concern me at all.  If every single Tbird user signs up... I'll actually be pretty happen with that!

That said, "Sync read/unread state of newsgroups" is probably a massive chunk of write-happy data (I think Kairo ended up with 155k records when he tried), and would lead to a pretty savage chunk of load.  Doing this might result in blocking Thunderbird clients... so let's not do that.
(In reply to Philip Chee from comment #39)
> What he says isn't Mozilla policy.

Just to be clear here, there isn't an official Mozilla policy either way.  There's a tradeoff and discussion for everything we do.  If an add-on or other client causes problems for the service, we can and will take steps to prevent that.  If Thunderbird support was deemed an unsupportable expense, we would have to block those clients.  Having an architecture that's designed to be highly flexible and open does not mean we're giving the community a blank cheque to do whatever they want.  TANSTAAFL applies here, as in all things.
Philip Chee, I am not expressing concerns about the client. I am expressing concerns about the mozilla owned & supported servers to which this may connect, which is the impression I got from the patch. 


What I don't want is for you guys to invest lots of time writing an engine only to have it blocked on the server because of a complication no one warned you about. I don't want that for anyone.

I/We'd be happy to provide design review to ensure that it's not going to generate excessive cost/load. 

Please don't call me 'he', as I am a 'she'.
I'd be happy starting simply with TB prefs, accounts, and add-ons.

I think that's an attainable goal, and highly unlikely to be a storage quota threat.
(In reply to Mike Conley (:mconley) from comment #43)
> I'd be happy starting simply with TB prefs, accounts, and add-ons.

You should be able to get prefs and add-ons for nearly free, meaning at very little or no review cost to the Sync team.
I think it makes little sense for Thunderbird to incur the effort to port Sync *right now*. I am currently making significant changes to the Firefox Sync JS code base and it will be much easier (and likely less buggy) if the Thunderbird port waited until my work is mostly done. But, if someone wants to go through at least double the effort to get things maybe 3 months earlier, I won't stop you.
(In reply to Gregory Szorc [:gps] from comment #45)
> I think it makes little sense for Thunderbird to incur the effort to port
> Sync *right now*. I am currently making significant changes to the Firefox
> Sync JS code base and it will be much easier (and likely less buggy) if the
> Thunderbird port waited until my work is mostly done. But, if someone wants
> to go through at least double the effort to get things maybe 3 months
> earlier, I won't stop you.

That's good to know. I've also heard rumblings of Persona being used as an authentication mechanism, though that might just be rumours.

Waiting for stability makes sense. Or, alternatively, working off of your target spec.
(In reply to Mike Conley (:mconley) from comment #46)
> That's good to know. I've also heard rumblings of Persona being used as an
> authentication mechanism, though that might just be rumours.
> 
> Waiting for stability makes sense. Or, alternatively, working off of your
> target spec.

Persona will be leveraged, yes. The master tracking bug is bug 744321.
> Philip Chee, I am not expressing concerns about the client. I am expressing concerns
> about the mozilla owned & supported servers to which this may connect, which is the
> impression I got from the patch. 

Thank you for the clarification. That appears to be an old patch. The GSoC project demonstrated that syncing messages was impractical.

> Please don't call me 'he', as I am a 'she'.
Oh very sorry. I didn't know that Naaktgeboren is a female name.
> syncing messages was impractical.

... but syncing of tags was very, very useful for imap.
We should probably create new bugs with focus on a narrow and given set of items to synchronize, such as prefs, tags, etc.

Right now I don't think it makes sense to think about synchronizing messages (or contacts for that matter) as we have industry standards good and accepted enough not to have to reinvent the wheel.
(In reply to Mike Connor [:mconnor] from comment #40)
> That said, "Sync read/unread state of newsgroups" is probably a massive
> chunk of write-happy data (I think Kairo ended up with 155k records when he
> tried), and would lead to a pretty savage chunk of load.  Doing this might
> result in blocking Thunderbird clients... so let's not do that.

Just FYI, that was doing state for every single message as a separate object to sync, which is probably the wrong way to go there. If doing one object per group/folder is feasible, it probably could be done with much less storage space. So, we learned a good deal from that GSoC experiment in case we ever want to go in that direction again. :)
Moving this to Thunderbird for feature tracking.

Most likely this bug as-is should be closed; if Thunderbird ends up integrating Sync (ideally a future rewrite), then we can file individual bugs for engines and any support that's necessary from the Services Integration team for the main Sync codebase.

Bug 621198 can be used to track the latter if you would like.
Component: Firefox Sync: Other Clients → General
Flags: blocking-weave1.0-
Product: Mozilla Services → Thunderbird
Summary: make sync (formely weave) work in Thunderbird → Sync for Thunderbird
Target Milestone: Future → ---
Does anybody know about the actual status of Thunderbird implementing the sync feature ?
I haven't heard any news. But broadly, now is not a great time to be taking dependencies on Sync; there's a lot of stuff up in the air, so there's a lot of risk of wasted effort.
I'm a little bit confused now ...
Reading the German wikipedia, I stumbled upon some information, that says "Thunderbird won't be actively developed any further, only security updates will be made".
=> http://de.wikipedia.org/wiki/Mozilla_Thunderbird

Sounds to me, as if there won't be an official sync functionality ...
So the efforts that one invests here won't be wasted !
(In reply to Quick_Silver from comment #56)
> I'm a little bit confused now ...
> Reading the German wikipedia, I stumbled upon some information, that says
> "Thunderbird won't be actively developed any further, only security updates
> will be made".
> => http://de.wikipedia.org/wiki/Mozilla_Thunderbird

That is wrong, the statement only pertains to changes made by Mozilla employees. The community is welcome to provide (and has been) new features & other changes to Thunderbird.
I hope that like Firefox Sync, what gets synced in Thunderbird can be enabled/disabled.

As a user I'd like to see the following items Sync:

- Adaptive Junk Mail Controls (learned data)
- Account Settings (by individual account as different "seats" have different needs)
- Add-ons
- Add-on Settings, (Lightning in particular)
- Filter config

  Things I don't really need Sync'ed and would like to disable if they exist:
- Addressbook (handled by CardDAV through SOGo Connector to OwnCloud Server with HTTPS)
- Calendar (currently handled using CalDAV to OwnCloud Server with HTTPS)

I use a self hosted Sync Server (OwnCloud).  The new Sync 1.5/2.0 efforts will temporarily disrupt Self hosted (OwnCloud) Sync; https://bugzilla.mozilla.org/show_bug.cgi?id=989756 until the new system can be implemented; https://github.com/owncloud/mozilla_sync/issues/33 It seems Firefox Accounts/FxA also needs to be implemented before Sync can be moved to the new 1.5/2.0 efforts.

I'm particularly keen on seeing Junk Mail Adaptive data sync'ed.  The reason is I had Junkmail all tuned up and working on my Linux workstation and decided to put Thunderbird on my Windows based Laptop.  One of the first things that happened when Junk mail filtering was enabled is most of my now dead parents emails were flagged as "junk" and removed.  Fortunately they weren't deleted, but I spent the remainder of the evening locating and retrieving as many of these emails as I could.  I believe a careful implementation of Sync in this area (i.e. disable junk mail controls first, then sync adaptive data, then re-enable junkmail controls with newly sync'ed adaptive data) would go a long way toward addressing the experience I had.  Both Thunderbird instances (I actually use about 10 of these) connect to the same IMAPS mail server(s).

For Account settings, I might be inside company A in the morning and company B in the afternoon.  I wouldn't want company A settings or email to show up at company B and the reverse as a result of using Sync.  I wouldn't want either one to show up at home, unless I specifically had authorization from that employer and wanted this feature.  I otherwise use IMAPS with local storage turned off to monitor the accounts I need.

Lastly, I'd like to thank everyone who's looking at this.  I'm impressed with all the effort I see and have read here.  So Thank you.
I think this is a great idea. The service could be set up, as already mentioned here, accordingly the sync for Firefox : Thunderbird Sync.

Containing following components of a profile to be focused in sync:
- Filter
- Email Accounts (Folder settings, Signatures, Server settings...)
- Settings (General, View, Security)
- Addons + their configuration (addons should be synced before settings...otherwise newly added addons will overwrite synced settings with addon default settings)
- Passwords (with feature to sync master password)
- Contacts/Calendar not included, because they can be synced in many different ways (IMAP folder, dropbox, webdav, card- and caldav). In order to stay compatible with web standards, I would suggest Thunderbird to implement a standard CardDav functionality (as CalDav already works). But this is another feature request (address book to use CardDAV to enable sync (like apple)).

Looking forward for this feature. Would be nice, if the sync service is provided as update for the Mozilla weave server.
I think Thunderbird could be done using WebExtensions API in the near future see https://bugzilla.mozilla.org/show_bug.cgi?id=1396172

From comment #60 every features need a WebExtension API
Keywords: helpwanted

Very sad this has not been implemented already

Type: defect → enhancement

Personally, I would prefer to use the infra and account I also use for Firefox. This would save on setting up and maintaining a seperate infra and reduce the number of accounts and serviced I use. Would that be possible?

Is there a way to crowdfund this?

Hi,
It will be nice to have it.
Is there a way prioritize this issue? May be with a crowdfunding?
Kind regards

Depends on: 1782646

I am working on this. You can stop asking now.

I'm turning this bug into a meta-bug. The work will happen in other bugs as there's enough comments here already.

Assignee: nobody → geoff
Status: NEW → ASSIGNED
Keywords: student-projectmeta
Summary: Sync for Thunderbird → [meta] Sync for Thunderbird
Depends on: 1782647
Depends on: 1782648
Depends on: 1783308
Depends on: 1793378
Severity: normal → S3
Depends on: 1802283

If you just received an email about bug 1793378 being marked as fixed, and you're thinking now's the time to try Sync, please read this post I made to the Daily mailing list first.

Depends on: 1797125
Duplicate of this bug: 274840
Depends on: 1805109
No longer blocks: 274840
Depends on: 1805324
See Also: → 1833177
Assignee: geoff → nobody
Status: ASSIGNED → NEW
Duplicate of this bug: 1852564
Blocks: 1852564
No longer duplicate of this bug: 1852564
See Also: → 1859260
Duplicate of this bug: 1864699
Duplicate of this bug: 1719843
You need to log in before you can comment on or make changes to this bug.