Closed Bug 11033 Opened 25 years ago Closed 22 years ago

Filter after the fact (run filters on mail you've already received - apply filters on the fly)

Categories

(MailNews Core :: Filters, enhancement, P2)

enhancement

Tracking

(Not tracked)

VERIFIED FIXED
mozilla1.2alpha

People

(Reporter: sspitzer, Assigned: Bienvenu)

References

()

Details

(Keywords: helpwanted, Whiteboard: [Hixie-P0][Hixie-1.0])

Attachments

(2 files, 2 obsolete files)

Perhaps Based on selection?
Summary: [HELP WANTED] Filter after the fact (run filters on mail you've already received) → Filter after the fact (run filters on mail you've already received)
Whiteboard: HELP WANTED
Summary: Filter after the fact (run filters on mail you've already received) → [HELP WANTED] Filter after the fact (run filters on mail you've already received)
Target Milestone: M15
QA Contact: lchiang → laurel
Blocks: 9417
A superset of this is bug 9417.  Marking dependency, mark as dupe if you want.
*** Bug 10885 has been marked as a duplicate of this bug. ***
Would anyone mind if I put HELP WANTED on 9417?
Bulk-resolving requests for enhancement as "later" to get them off the Seamonkey
bug tracking radar. Even though these bugs are not "open" in bugzilla, we
welcome fixes and improvements in these areas at any time. Mail/news RFEs
continue to be tracked on http://www.mozilla.org/mailnews/jobs.html
Reopen mail/news HELP WANTED bugs and reassign to nobody@mozilla.org
Keywords: helpwanted
Summary: [HELP WANTED] Filter after the fact (run filters on mail you've already received) → Filter after the fact (run filters on mail you've already received)
Whiteboard: HELP WANTED
Target Milestone: M15
*** Bug 61917 has been marked as a duplicate of this bug. ***
*** Bug 82237 has been marked as a duplicate of this bug. ***
*** Bug 83986 has been marked as a duplicate of this bug. ***
*** Bug 84366 has been marked as a duplicate of this bug. ***
I believe this should be naving's. (We talked about it the other day.) Reassign.
Assignee: nobody → naving
Component: Mail Back End → Filters
Keywords: mozilla1.0
*** Bug 65136 has been marked as a duplicate of this bug. ***
*** Bug 87855 has been marked as a duplicate of this bug. ***
fixing summary to catch more cases in queries.
Summary: Filter after the fact (run filters on mail you've already received) → Filter after the fact (run filters on mail you've already received - apply filters on the fly)
*** Bug 88210 has been marked as a duplicate of this bug. ***
*** Bug 45609 has been marked as a duplicate of this bug. ***
If anyone is not working on this then I will tackle it. I want to be able to 
set a filter up that will automatically move my mail to subfolders of sent-mail 
that are dated each month so I will have:

+ sent-mail
   sent-mail-jan-2001
   sent-mail-feb-2001
   sent-mail-mar-2001

and so on and so forth. I figure I will have to implement filter after the fact 
for this as well as extend the current filter creation GUI to allow for 
creating new folders. Also it would be nice if the folders were in date order 
rather than alpha order in the UI as they are now. Not sure how we could do 
that in a general way though.
Simple to order folders by date: Name them "Sent-Mail-2001-01, ..-2001-02, etc.
A more powerful filtering mechanism would allow a script to run on each message
so you could grab the date and synthesize the folder name!
*** Bug 90979 has been marked as a duplicate of this bug. ***
Keywords: mostfreq
True but that is rather inflexible... A user might not like to have them in 
this order. They may like September-2001 or Sept-2001 and it would be nice if 
it handled it. Your suggestion is probably what I will use though...
*** Bug 91690 has been marked as a duplicate of this bug. ***
*** Bug 94097 has been marked as a duplicate of this bug. ***
*** Bug 98986 has been marked as a duplicate of this bug. ***
Targetting this for 0.9.6, may get to it earlier. There are some other 
things that need to be done before we can do this one. 
Status: NEW → ASSIGNED
Target Milestone: --- → mozilla0.9.6
*** Bug 100711 has been marked as a duplicate of this bug. ***
I really need this -- I just switched to Mozilla, so I have 7000 e-mails in my
INBOX that I would like to filter! :-)
Whiteboard: [Hixie-P0][Hixie-1.0]
ian - there's a hacky workaround to achieve the affect - I've used it once or twice:
- go to your INBOX, select all, and mark them all unread.
- quit mozilla
- go into the IMAP mail folder for that server, and delete INBOX.msf and (if
it's there) INBOX.
- start up mozilla and log back in.

it will think your inbox was previously empty, and that you just recieved 7000
email...
Re: Hackey work-around from alecf@netscape.com -
That isn't much help if one isn't using IMAP, or if the messages were already
deleted from the server.  
oops, I should have mentioned that the hack was IMAP only.. it wasn't intended
as a replacement, just as a hack in case someone wanted to do this TODAY instead
of waiting for this bug to be fixed :)
Blocks: 102231
*** Bug 104246 has been marked as a duplicate of this bug. ***
nominating
Keywords: nsbeta1
Are we going to allow filter after the fact to work on inbox or on all 
folders under that account. It makes sense to do only for inbox. jglick, 
do we have a spec for this feature. 
I think it makes a lot of sense for it to run on folders other than the inbox..
I would use that feature all the time if I could. For instance, I have a
subfolder "Outside" which contains all non-netscape.com mail. I'd sure love to
re-filter that mail into subfolders...
Unless there's a major technical hurdle (I can't think of one), it may as well
be generic enough to run on any folder ... but yes, if it's just done for Inbox
that will be sufficient for 90% of people I think.
I definitely think, if as jkeiser says, it's not too much of a problem, that
this should be doable on any/all folders of an account.
It logical to allow filtering of any *selected* folder. why shouldn't you do 
that for other if you can for inbox? To avoid painful accidents I'd maybe vote 
for having a warning or confirmation box before processing folders other than 
inbox.
If it could filter also newsgroup folders that would solve another ancient bug
17483.
Filter spec is here:
http://www.mozilla.org/mailnews/specs/filters/#Msg Filters D

It was never discussed whether filter after the fact should include subfolders 
or not (cause we never got close enough to having it working?). Maybe the ui 
needs to be changed to allow the user to select what folder(s) they want the 
filter run on?
Attached file design doc
Attachment #54316 - Attachment is obsolete: true
Attachment #54316 - Attachment is obsolete: false
Attachment #54316 - Attachment mime type: text/plain → text/html
david/seth, what do you think about design doc ?
Personally, I'd like to see this feature as a menu item in the "Message" menu.
It should filter all the currently selected messages.
Responding to the Design Doc:
* "enabled filters" --> "selected filter(s)"  I don't know about anyone else but
I have a pretty big list to run down twiddling the enabled flag.

* Modal progress indicator could be per-folder similar to the way some install
programs show a "per-disk" progress.

* I'm not sure why you need do anything special to accomodate arbitrary headers
-- the list of headers /in the query/ are not at all unknown.  There's so much
variation in headers already I don't imagine there's any difficulty if messages
don't have an instance of the header in question.

I don't know IMAP but how is that done now?  This looks to me identical to
running a search query - maybe allowing a more complex search or more complex
actions.
I think we need to rethink the UI.  I don't think that going to the filters
dialog and doing this action should run this filter on every single folder in an
account.

Here are some suggestions (and they might have already been suggested, I've just
glanced through this bug).

1.  If you do this through the filter dialog it just runs it on the Inbox in
that account.
2.  We add a menu item that lets you do this on the currently selected folder. 
The menu item would either bring up some new UI that would let you choose the
filter/folder (with current folder pre-selected), or the menu item would be a
menu with a list of filters for that account.

Some other possibilities that I'd like to throw out:

1.  Take it out of the filter dialog completely and just let it be executed from
a menu.
2.  If we leave it in the filter dialog, either let it bring up some UI that
lets you select the folder(s) the user wants to do this on or have some UI in
the filter dialogs that lets you choose the folder(s) the user wants to do this too.
Keywords: nsbeta1nsbeta1+
Priority: P3 → P2
Why not just do it similar to how Outlook Express does it?  Have a button in the 
filter area that says Apply Now...  You click on Apply Now, another UI comes up, 
you select the rule you want to apply, click the browse button to select the 
folder to apply it to (defaulting to Inbox), and click Apply Now.  We could also 
have the Select All and Select None buttons so that you can apply all your rules 
in one go if you so wished.  It even shows a description of what the rule does 
in a small window (basically just shows the rule itself).  There is also an 
include subfolders checkbox in case the chosen folder has subfolders.

In any case, let's not make it like GroupWise 5.5.  I don't know if GroupWise 6 
is any different, but in 5.5 you have to select all the messages you want the 
rule to apply to before you run the rule against existing messages.
I like the idea of a menu which displays the filter list for a folder:
Run filter on this folder > All filters
                            Filter 1
                            Filter 2

etc

of course the wording would need to be a bit more terse :)
I'd expect this feature to be that all filters be run on a selected folder, like
with new mail just that you can choose which folder(s).  Do we really need UI to
choose which filter to run too?
I always assumed I'd have the ability to pick which filter I wanted to run.  I
think we should have an option for all, but I think we should be able to choose
an individual filter as wel..
I don't think a menu item for each filter is a good UI, a dialog with a list of
filters could be better. The dialog could pre-select the filter you used last,
so that it would be easy to re-run a filter, e.g. after jumping into another
folder. (The Message|Move Message and Message|Copy Message menus have the same
problem and are really annoying.)

One thing I'd like to be considered is the relation between filters and message
searching. Manually run filters are essentially stored searches with attached
actions. Maybe the UI could be unified, e.g. have "Store as filter" in the
search dialog etc.
Target Milestone: mozilla0.9.6 → mozilla0.9.7
Target Milestone: mozilla0.9.7 → mozilla1.0
*** Bug 114608 has been marked as a duplicate of this bug. ***
Keywords: nsbeta1+nsbeta1-
Target Milestone: mozilla1.0 → mozilla1.2
No longer blocks: 102231
I'm sure you hear this all the time, but I really don't understand why it takes 
so long to implement this feature. IMO this is really a basic function which is 
essential for a 1.0 release and although I'm no programmer I can't see why this 
should be so difficult to implement. Please reconsider the target milestone.
Sorry for the rant.

simi
The milestone isn't going to be reconsidered as long as Navin owns the bug.  

You are right, it wouldn't take very long to implement this feature.  It's the
fact that there are about 3600 other mail bugs/features that prevent this from
getting implemented.
why not just open the mailfolder, and send each mail out via a loopback
interface back to myself to fire the mail filter rules? That should be easiest
to get the effects early. Talk about efficiency later...
Severity: normal → enhancement
Whiteboard: [Hixie-P0][Hixie-1.0] → [Hixie-P0]
*** Bug 122587 has been marked as a duplicate of this bug. ***
*** Bug 123502 has been marked as a duplicate of this bug. ***
Whiteboard: [Hixie-P0] → [Hixie-P0][Hixie-1.0]
Keywords: mail3
I feel that the default behavior should be apply all filters to the selected
folder(s) and any subfolders.  This would suffice for all of my needs.  I have
been extremely impressed by the way that KMail handles this.  This would allow
everything to be re-processed if one selected the mail server, but usually only
the inbox would be selected.  KMail only processes selected messages, and while
almost always sufficient it fails if one wants to process a hierarchy of
folders.  Perhaps there should be a switch to turn off recursive descent, though
I can only think of twice when I would have used such an option.

If one were to get fancier than that, then I would be more in favor of allowing
regular expressions into the recognitions semantics than most of the other
obvious options.  And some facility for filtering by character set family.  Or
language, but that seems more difficult, and it would generally suffice to
reject messages with several non-western european letters in a row.  I don't
read any language written in Cyrillic or Hiragana or ....  (I recognize that
this isn't actually a part of the same project, but it appears closely related,
and seems [to me] more worthy of the effort than complex dialogs, or rebuilding
the whole UI.  Also a lot simpler.)
*** Bug 136565 has been marked as a duplicate of this bug. ***
*** Bug 137908 has been marked as a duplicate of this bug. ***
*** Bug 139794 has been marked as a duplicate of this bug. ***
This bug seems to be about both the front end and the backend for refiltering
messages. IMO the backend should be a function that will apply a list of filters
to a mail folder. The frontend should be responsible to interact with the
user to find out which filters to apply, and to which folders and invoke the
backend as often as necessary to process all folders.
A first start for the frontend that would be trivial to implement is to
add a button "Run now" to the "tools"->"message filters" dialog that 
would simply run all enabled filters on the *currently viewed list of
messages*. I think the expected run now behaviour would be to only run the
filters on those messages that are displayed in the same window. 
Additional front ends for filtering folders recursively 
by right clicking on them etc. could be added later. 
To sum up, the current concensus seems to be:

* Have a button in the filter list dialog, clicking that will bring up another
dialog.
* The new dialog will have options like which folders to process, and which
filters (one specific or all) to use.
* Applying the filter(s) on the specified folder(s) will pop up a progress
dialog, similar to the one in compose, where you will be able to track the
progress of the filtering.

There are lots of cool additions we could make, but let's file new bugs for
those features, so we don't block the progress of the basic implementation.

BTW, when implementing this, we need to make sure we don't filter the same
messages multiple times; e.g., moving a message from folder A to folder B, and
then filtering the messsages in folder B would re-filter the moved message.
Actually, you could just put a progressbar on the options dialog, instead of
creating a new one, and use that when the filtering starts.
*** Bug 145630 has been marked as a duplicate of this bug. ***
*** Bug 146031 has been marked as a duplicate of this bug. ***
*** Bug 147162 has been marked as a duplicate of this bug. ***
*** Bug 147675 has been marked as a duplicate of this bug. ***
What about "remove messages put to this box more that X month ago" to trash?
Approximately like in KMail (there count received date - not put-into-theis-box
date).
One reason I want to 'filter after the fact' is that sometimes I read mail in
some other IMAP client.  Then I go back to Netscape, and my mail is all
un-categorized.
*** Bug 152200 has been marked as a duplicate of this bug. ***
I'm really confused/frustrated here.  I use IMAP and mail doesn't magically
"already exist/after the fact" in my folders.  folderXYZ is functionally exactly
the same as INBOX.  Why should they be processed any differently?

In other words, I use procmail to sort and deliver my mail to ~30 mailboxes.  I
don't use Mozilla to sort anything.  I have one filter rule:

If X-Bmilter contains "=True", then label as SPAM.

Most people I talk to appear to only be aware of how POP3 works and are really
unfamiliar with IMAP and the ability to have more than one mailbox on the
server.  IMAP lets you have your inbox plus any number of mailboxes...and even
lets you use public shared mailboxes.

All of these mailboxes are "new mail" that has to be downloaded.  Why aren't
filters applied to these mailboxes when mail is fetched?

Put at another angle, I don't want to have to a) fetch mail, b) open a menu for
each mailbox folder, c) run selected filters.  That's very tedious and time
consuming.

I want to be able to a) define a list of filters b) on each mailbox folder, be
able to pick which filters I want applied to that mailbox, normally [x] All

Discussion?
don't be frustrated. what you're talking about is a feature above and beyond the
scope of THIS bug. nobody could even begin to implement the feature you describe
without first implementing this one. file a new bug, and make it dependent on
this one.
I had already made 152200 and it was duped against this bug.  I'm reopening it.
Blocks: 152200
Re: comment 70 : what he is suggesting is not a  duplicate of this bug. He wants 
1) To be able to filter other mailboxes, not just Inbox. 
2) To be able to specify which filters apply to which mailboxes

He didn't ask about:
3) Apply such filters manually or automatically 

Clearly only 3) depends on this bug 
*** Bug 154002 has been marked as a duplicate of this bug. ***
What I feel would be best for filters:
- filters should be applicable on the two "flows" of email, which are "into
account" (therefore when mail comes into the main inbox) and "out of account"
(ie when being sent).
- in addition to that, the user should be able to manually apply (or reapply)
filters on any folder (including inbox, sent,...: this is useful, amongst
others,  if you read occasionally your mail with another mail client, which does
not know about your filters and therefore moves all your incoming mail into the
inbox without filtering)
Please, this bug is not a general bugs where you can ask for new features. File
new bugs for new features. This bug *only* tracks the ground work that needs to
be done to allow to run filters on an arbitrary folder per comment 60.

Do not comment here - file new bugs.
Just an FYI for you Vincent, (comment 74) there are only two "flows" of email
for pop3 accounts.  imap can have numerous mailboxes.  Please read over bug
115200 that was referenced in comment 68 through comment 72.
*** Bug 159093 has been marked as a duplicate of this bug. ***
*** Bug 163446 has been marked as a duplicate of this bug. ***
taking; I've got this mostly coded up.
Assignee: naving → bienvenu
Status: ASSIGNED → NEW
Cool :) Go Bienvenu!
Blocks: majorbugs
Attached patch work in progress on this (obsolete) — Splinter Review
this is what I have so far - it's not ready to checkin, but it shows the basic
approach.
I chose the Search approach (as opposed to iterating over the msg hdrs applying
the filters in turn) because, now that, thx to Navin, we have custom imap
headers,  we can't just use the hdrs stored in the db, and if we had to
download the extra headers from the server anyway, we might as well just do the
search on the server. This approach allows the code to be completely generic -
we don't care if the filters are on imap, local or news folders, because search
handles all of those, and the filter action application is handled by
nsIMsgFolder interfaces. By getting all the search results back before applying
the filter , we can apply the action all at once. And Search has the nice
attribute of already being asynchronous so it doesn't lock up the UI.

The backend code supports applying an arbitrary list of filters to an arbitrary
list of folders. I have a lot of testing to do, and I need to improve the error
handling code - we might want to prompt the user if filter execution fails to
see if they want to continue or not, for example.

I also need to incorporate the filter logging stuff that Seth has been working
on.
Call me crazy, but I sometimes use search from an offline state - does that mean
that I can apply filters while offline as well? That would be very cool..
yes, forgot to mention that, it should just work while offline because Search
detects when you're offline (except that custom headers won't work because we
won't have them).

Another detail about the interface - I only apply the enabled filters, so if the
UI  allows the user to select disabled filters to run, when the UI code copies
the filters to a new list, it needs to enable them.
*** Bug 167922 has been marked as a duplicate of this bug. ***
will this make it into 1.2?
it should make the 1.2 beta and then 1.2 - it won't make the 1.2 alpha, obviously.
Attached patch proposed fix (obsolete) — Splinter Review
This patch is ready for review, I believe.

I tested the following things:

IMAP filters, local filters, filter logging, single filter execution, multiple
filter execution (can't test execution on multiple folders yet since there's no
UI for it). I've verified that if the local .msf file is missing or out of
date, we'll reparse the mailbox and then apply the filters. I've also verified
that the nsMsgFilterAfterTheFact object does get released when done, and that
in the cae of the server being down, an error message is displayed; then ask
the user if they want to continue filter execution or not. 

There are a couple issues outstanding - the PAB presence filter stuff Scott is
landing, and handling of the stop button, but I need Seth to do the UI for that
first and it's pretty useful as is.


Navin, can you review? Thx!
Comment on attachment 98938 [details] [diff] [review]
proposed fix

why are we using temp filter list ? why not just iterate over enabled filters
in the original filterList. 

Also before doing filter move you might want to check for bogus folders
missing new line at end of filter.properties
Navin, I initially implemented it the way you suggest, but ran into the problem
that if the user selects disabled filters in the filter dialog to apply, we'd
need to clone the filters in order to enable them and run them. Cloning filters
would be painful if you did a deep clone because of all the member variables in
the filter and rule, so it was easier to do it the way I ended up rewriting it.

It would be hard to get an invalid folder from the filter picker or the folder
pane but I can try to verify it - check for null parent, you think?
oh, sorry, the move target - yeah, I can try to verify that.
Attached patch proposed fixSplinter Review
patch to address Navin's comments (and fix menu item text for apply filter
after the fact).
Attachment #98468 - Attachment is obsolete: true
Attachment #98938 - Attachment is obsolete: true
Comment on attachment 98977 [details] [diff] [review]
proposed fix

r=naving looks good. I saw UI is already in but looks like it is not hooked
upto select which filters I want to run.
Attachment #98977 - Flags: review+
right, Seth checked in the UI w/o it being hooked up, so that clicking the run
button did nothing.
No, I meant if I want to apply 1st, 3rd and 5th filter out of 6 filters I have.
Your code does not handle that. It adds all enabled filters to the temp filter
list. 
Comment on attachment 98977 [details] [diff] [review]
proposed fix

sr=sspitzer

laurel is going to be so happy!  how many years have we wanted this?
Attachment #98977 - Flags: superreview+
it shouldn't - I thought it was just iterating over the filters in the
selection. It looks like to me it's just going over the selection.
in FilterListDialog.js, this code iterates over the selected filters in the
filter list (Seth actually wrote this, so I can't take any credit for it)

  var sel = gFilterTree.view.selection;
  for (var i = 0; i < sel.getRangeCount(); i++) {
    var start = {}, end = {};
    sel.getRangeAt(i, start, end);
    for (var j = start.value; j <= end.value; j++) {
      var curFilter = getFilter(j);
      if (curFilter)
      {
        filterList.insertFilterAt(start.value - j, curFilter);
      }
    }
  }
It is ok I saw the part of the patch for mailWindowOverlay.js. It is right you 
have no way of selecting filters here. 

Index: base/resources/content/mailWindowOverlay.js
===================================================================
RCS file: /cvsroot/mozilla/mailnews/base/resources/content/mailWindowOverlay.js,v
retrieving revision 1.131
diff -u -w -r1.131 mailWindowOverlay.js
--- base/resources/content/mailWindowOverlay.js	14 Aug 2002 22:12:13 -0000	1.131
+++ base/resources/content/mailWindowOverlay.js	12 Sep 2002 23:44:01 -0000
@@ -1216,6 +1216,39 @@
     }
 }
 
+function MsgApplyFilters()
+{
+  var filterService =
Components.classes["@mozilla.org/messenger/services/filters;1"].getService(Components.interfaces.nsIMsgFilterService);
+
+  var preselectedFolder = GetFirstSelectedMsgFolder();
+  var selectedFolders =
Components.classes["@mozilla.org/supports-array;1"].createInstance(Components.interfaces.nsISupportsArray);
+  selectedFolders.AppendElement(preselectedFolder);
+         
+  var curFilterList = preselectedFolder.getFilterList(msgWindow);
+  // create a new filter list and copy over the enabled filters to it.
+  // We do this instead of having the filter after the fact code ignore
+  // disabled filters because the Filter Dialog filter after the fact
+  // code would have to clone filters to allow disabled filters to run,
+  // and we don't support cloning filters currently.
+  var tempFilterList = filterService.getTempFilterList(preselectedFolder);
+  var numFilters = curFilterList.filterCount;
+  // make sure the temp filter list uses the same log stream
+  tempFilterList.logStream = curFilterList.logStream;
+  tempFilterList.loggingEnabled = curFilterList.loggingEnabled;
+  var i = 0; 
+  var newFilterIndex = 0;
+  for (; i < numFilters; i++)
+  {
+    var curFilter = curFilterList.getFilterAt(i);
+    if (curFilter.enabled)
+    {
+      tempFilterList.insertFilterAt(newFilterIndex, curFilter);
+      newFilterIndex++;
+    }
+  }
+  filterService.applyFiltersToFolders(tempFilterList, selectedFolders, msgWindow);
+}
fix checked in. Follow up issues will be handled by new bugs.
Status: NEW → RESOLVED
Closed: 22 years ago
Resolution: --- → FIXED
bienvenu wrote:
> fix checked in. Follow up issues will be handled by new bugs.

What's the bugid for the "nebiros"-bustage on tinderbox ? 
I believe it's been fixed already by mkapply.
*** Bug 171457 has been marked as a duplicate of this bug. ***
marking verified using current trunk builds... implementation in, bugs found
will be logged separately.
Status: RESOLVED → VERIFIED
Blocks: 66425
Product: MailNews → Core
No longer blocks: majorbugs
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: