79525 - (sieve) Support Sieve

Status: NEW

(MailNews Core :: Filters, enhancement)

Description

[Ben Bucksch (:BenB)]

RFC3028.

Comment 1

[Julian Missig]

I hope this is considered for post-1.0. It would be really great to export
filters to Sieve from Mozilla.

Comment 2

[Barry de la Rosa]

Nearing IETF standard stage. Plese support!

Comment 3

[Marc Bejarano]

we are AT ietf standard stage :)  well.. as standard as things get, meaning a true RFC (3028).

the question is, once we have sieve scripts being output by the filter UI, how do we transfer them to the server for server-side filtering?  via the managesieve protocol ala mulberry and cmu's sieveshell?  or via ACAP?

in any case, this is definitely the future.  IMAP sucks without server-side filtering.

http://www.nwfusion.com/news/2002/0715chaos.html

marc

Comment 4

[(not reading, please use [e-mail] instead)]

mass re-assign.

Comment 5

[[:nickname]]

Seems to be no developer interest :) Sadly. Even Squirrelmail provides a plugin
which supports sieve (though I never got it to work ..).

Comment 6

[Julian Missig]

Well, there doesn't even seem to be all that much user interest either. That's a shame.

Comment 7

[[:nickname]]

Well, I'm "interested" but lack a) knowledge to implement it and b) time for a)
;-) However I'ld definitely donate for anyone working on this (that's true for
most of the bugs I'm monitoring anyway).

Comment 8

[kael]

Mozilla/Thunderbird must support the SIEVE mail filtering language.

RFC 3028: "It is designed to be implementable on either a mail client or mail
server. It is meant to be extensible, simple, and independent of access
protocol, mail architecture, and operating system" -
http://www.ietf.org/rfc/rfc3028.txt?number=3028.

Also, the *SIEVE flag extension* Internet-Draft:
draft-melnikov-sieve-imapflags-06.txt
http://www.melnikov.ca/mel/Drafts/draft-melnikov-sieve-imapflags-06.txt must be
supported by Thunderbird.

"The extension allows to set both [IMAP] system flags and [IMAP] keywords".

May that bug be condsidered as important.

Comment 9

[Michael Liebl]

That would be great!

Comment 10

[Bill]

Support for Sieve filters makes Thunderbird that much more viable in a corporate
environment. I know I support this idea and hopefully someone can make this happen. 

Comment 11

[Garth Wallace]

As nice as this would be, a SIEVE extension for Mozilla seems more likely than
built-in support.

Comment 12

[Mike Brodbelt]

I would also love to see this implemented. It would make Thunderbird into a more viable corporate mail client, and provide another area where mozilla software can displace Exchange.

Comment 13

[kael]

Thunderbird now supports an extension for MANAGESIEVE http://sieve.mozdev.org . Hopefully, it'll be implemented in Thunderbird natively.

Comment 14

[Marc Bejarano]

fyi: the thunderbird version of this bug is bug 264705.  it was resolved WONTFIX.

Comment 15

[(not reading, please use [e-mail] instead)]

sorry for the spam.  making bugzilla reflect reality as I'm not working on these bugs.  filter on FOOBARCHEESE to remove these in bulk.

Comment 16

[Florian Fritz]

ALL mail filtering should be excluded from thunderbird, and be an extension. than evry user could use the extension that does what he needs.

Comment 17

[David Gravereaux]

RFC updated, now at 5228: https://tools.ietf.org/html/rfc5228

Comment 18

[Michiel van Leeuwen (email: mvl+moz@)]

Attachment: totally WIP patch

This patch is totally work in progress. I didn't even look at how to download script etc. Just wanted to have it up somewhere. comments are welcome, as long as it's not about the parts I just didn't write yet :)

Comment 20

[Joshua Cranmer [:jcranmer]]

A tiny nit I have: I would rather the directory go in mailnews/base/search/sieve/ or perhaps mailnews/base/sieve/, to be more consistent with the existing implementation.

Comment 21

[Prognathous]

The Mulberry mail client includes a very nice Sieve GUI, and is now open source under the Apache 2 License. More here:

http://trac.mulberrymail.com/mulberry/wiki/WikiStart

Prog.

Comment 22

[Michiel van Leeuwen (email: mvl+moz@)]

Attachment: more WIP

More work going on. Didn't move the files yet.
Now does some serializing.
It's not a 1:1 mapping from sieve to mailnews internal filters (yet), so there will be dataloss when parsing and then serializing.
If you want to play with it, some test xpcshell files are included. (play on a code-level basis that is, it's not ready for real use. Not ready at all)

Comment 23

[Michiel van Leeuwen (email: mvl+moz@)]

Attachment: Another revision

Another revision of the patch. Main change is that managesieve work has now started.

Comment 24

[Michiel van Leeuwen (email: mvl+moz@)]

Attachment: Basic functionality

New patch. This patch has the first steps on integrating all modules into thunderbird. If you set the right preferences, it should be possible to read sieve rules from a server and display them. Storing doesn't work yet. (serializing to text should work, but you can't call that).

prefs (change "server3" to whatever account you want this to apply to):
user_pref("mail.server.server3.sieve.enabled", true);
user_pref("mail.server.server3.sieve.host", "localhost");
user_pref("mail.server.server3.sieve.server", "localhost");

There is a very limited test server included, plus a script to start it: runServer.sh. But that contains hardcoded paths to my objdir, so you need to change it.

Comment 25

[Thomas Schmid]

Hi,

while looking at your patch I've seen that your are implementing the ManageSieve protocol. 

If you want to save time, you can use the the code base from my project at sieve.mozdev.org. It is an complete implementation of ManageSieve, with support for Referrals, Proxy Authentication, and TLS. Further more it is compatible to all major Sieve server implementations. Which took some years to get that far... ;-)

Regards

Thomas

Comment 26

[Michiel van Leeuwen (email: mvl+moz@)]

Attachment: rough wip patch

Patch is still very rough, especially the part that makes nsIMsgFilterList async. Serves as (rather large) example on how to use the pluggable lists from bug 464770.

Comment 28

[Michiel van Leeuwen (email: mvl+moz@)]

Attachment: patch v5

More work done. Things are coming together now. Patch from bug 464770 is needed.
The list of working sieve commands is still quite low, but that's the easiest part to expand.

One important thing I added is support for async. The sieve script is downloaded on the background using normal necko async calls. In order to make the filters show in the UI, I added a notification. So I didn't add any methods to nsIMsgFilterList, but the interface did change a bit.

Comment 29

[Michiel van Leeuwen (email: mvl+moz@)]

To use the patch, you need to apply the patch in bug 464770. Then, some prefs need to be set:

user_pref("mail.server.server3.filter.type", "local");
user_pref("mail.server.server3.filter.editable.separate", true);
user_pref("mail.server.server3.filter.editable.type", "sieve");
user_pref("mail.server.server3.sieve.server", "localhost");

(change "server3" to whatever account you want this to apply to)

Comment 30

[Kent James (:rkent)]

I'm not sure of the intention of the SIEVE implementation here. It seems to me that there are two possible approaches to SIEVE:

1) use SIEVE as a method of implementing the existing filter searches and actions. Users would not see the scripts, only the existing filter dialog.

2) implement a basic managesieve implementation into the core code, so that SIEVE scripts that were written by users were uploaded, selected, editable, etc. by the standard mailnews code.

What is the intention here? I believe it is 1), is that correct?

Comment 31

[Joshua Cranmer [:jcranmer]]

(In reply to comment #30)
> I'm not sure of the intention of the SIEVE implementation here. It seems to me
> that there are two possible approaches to SIEVE:
> 
> 1) use SIEVE as a method of implementing the existing filter searches and
> actions. Users would not see the scripts, only the existing filter dialog.
> 
> 2) implement a basic managesieve implementation into the core code, so that
> SIEVE scripts that were written by users were uploaded, selected, editable,
> etc. by the standard mailnews code.
> 
> What is the intention here? I believe it is 1), is that correct?

Actually, the intent of the bug is to implement a client-side editor for server-side filters, accessible via managesieve; at least, that is how mvl has treated it.

Comment 32

[Ben Bucksch (:BenB)]

The intend of this bug is 1) (or something close to that). Sorry for not having made that explicit in my initial comment when I filed the bug.

Users should be able to specify filters in our existing filters dialog (or whatever is better and replaces it), and be able to specify (via a checkbox) whether the filter(s) should apply on the server side already or on the client side.

If Sieve allows more than our current filters do, that could be exposed in a user-friendly manner, too.

Being able to edit a Sieve language text file and upload it is nor something average users would do, not the goal of this bug and I don't think it belongs in Thunderbird core. Less than 0.1% of users would know the Sieve language or want to learn it, essentially only very technical types, so that belongs in an extension. The backend code which uploads/downloads the filter file may be useful for the filter UI as well, though, so that's useful code that could be in core.

Comment 33

[Ben Bucksch (:BenB)]

I don't want to discourage work on this, though. As said, the backend code will be very useful and is one of the hard parts. Once we have that, the Sieve language <-> UI code can be built in JS.

Comment 34

[Dan Mosedale (:dmosedale, :dmose)]

1 feels like the right high-level goal to me as well.  2, however, could be useful to a lot of people, and probably useful before all the work in 1 is complete, so it would probably make a fine extension.

Comment 35

[Michiel van Leeuwen (email: mvl+moz@)]

(In reply to comment #30)
> 1) use SIEVE as a method of implementing the existing filter searches and
> actions. Users would not see the scripts, only the existing filter dialog.

I am not planning on replacing the current filters and searches. I'm only planning on showing filters that are stored on a server. Those filters will not be 'run' by thunderbird.

> 2) implement a basic managesieve implementation into the core code, so that
> SIEVE scripts that were written by users were uploaded, selected, editable,
> etc. by the standard mailnews code.

I am planning on having managesieve, but currently not a full clients. Creating new scripts, multiple scripts etc might be added later, but not now. Also, raw script will not be seen by the user. The managesieve backend would be pretty much invisible.

So i'm planning a mix of 1 and 2. It's just like jcranmer said: an editor for filters stored on a server.

Comment 36

[Jelle de Jong]

Hi all, I have been following this bug report for a while and I am wondering if we are on the same side of the idea or not.

This is what the bug 481190 report is about:
 
1) Get a native feature to import and export mail filters in the open sieve standard.

2) Use the current mail filter dialog only with a open sieve standard import and export. I think it is a good idea to store native filters as sieve but this is not an requirement.

3) An additional GPL compatible add-on so we can add the ManageSieve functionality and make it sync/edit with ManageSieve server mail filtering systems. (Add-on would be fine, but native in the core code would be great to)

What are your goals?

Comment 37

[Ben Bucksch (:BenB)]

http://tools.ietf.org/html/rfc5703

Comment 38

[Michiel van Leeuwen (email: mvl+moz@)]

As requested, a status update:
The attached patch is still the latest version. I did experiments on preferences UI to enable sieve, but that didn't get me anywhere yet.

What needs to be done (not necessarily in this order):
* clean up code (remove commented-out code, remove test code, add comments, add license blocks, etc)
* add (preferences) UI to enable sieve, set host name, etc.
* test, test and more test (especially the conversion of sieve rules to mozilla-internal filter rules)
* figure out a way to make the filter UI deal with the async nature of downloading filter rules. The UI shouldn't block during this operation.
* Add a way to name sieve rules, as the current filter UI is based on named rules, but sieve doesn't have that concept. Maybe use comments?
* check which filter actions/rules/etc need to be disabled in the UI, and see if any need to be added
* release an extension or drive sieve into the core product

Comment 39

[Ludovic Hirlimann [:Usul]]

May I add unit test to the list above ? Plus having unit test will help the testing part of it.

Comment 41

[Nikolay Shopik]

There is adddon adding support managesieve http://sieve.mozdev.org/

Comment 42

[Jelle de Jong]

(In reply to comment #41)
> There is adddon adding support managesieve http://sieve.mozdev.org/

The plugin is not free software. Mozilla Thunderbird should use the sieve format internally by default, with the GUI for the filters and the auto changes when moving directories. There should be an option to use the internal sieve location or an other managesieve server. The http://sieve.mozdev.org/ is no where near a plugin that can be used in a production environment.

With kind regards,

Jelle de Jong

Comment 43

[Charles]

(In reply to comment #42)
> (In reply to comment #41)
>> There is adddon adding support managesieve http://sieve.mozdev.org/

> The plugin is not free software.

Hmmm... not sure why you say that:

http://sieve.mozdev.org/index.html

"The extension is free and open source software, it is made available to you under the terms of the GNU Affero General Public License (AGPLv3). This means you may use, copy and distribute it to others. You are also welcome to modify the source code as you want to meet your needs. "

> Mozilla Thunderbird should use the sieve format internally by default,
> with the GUI for the filters and the auto changes when moving directories.
> There should be an option to use the internal sieve location or an other
> managesieve server. The http://sieve.mozdev.org/ is no where near a plugin
> that can be used in a production environment.

I don't use it myself, but have heard from others that it works fine...

Comment 44

[Nikolay Shopik]

Michiel van Leeuwen, are you still working on this?

Comment 45

[Michiel van Leeuwen (email: mvl+moz@)]

No, I'm not currently working on this. If someone wants to take over, I'd be happy to help and explain whatever code I already wrote.

Comment 46

[Selek Respa]

https://github.com/thsmi/sieve implements this.
Would be a good foundation for implementing it upstream in TB directly.

Comment 47

[Ulf Zibis]

Please add bug 297852 to the "see also" list (unfortunately I'm not grated to do that).

Comment 48

[Mokile]

Please support ManageSieve protocol (RFC 5804) in Thunderbird. This would be really helpful.