886893 - one filter error is interpreted as a error for the whole batch of filtered message

Status: RESOLVED INCOMPLETE

(Thunderbird :: Filters, defect)

Description

[fabrice.neyret]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36

Steps to reproduce:

I have filters that move emails to some place if fitting some criterion.
One day, an incoming message made the filter action failing.
I don't know if the reason of the failing is important for the nasty thing that occured then (see below), but action was "move to a folder in another IMAP account", and this day it had to move an email which size was more than the maximum allowed by the provider of the target folder. (Anyway you cannot request max email size of an IMAP account, and you cannot filter using "subjet contains X or Y or Z, and size < max").


Actual results:

From then, the filter tries to act on inbox, succeed to act on some new mails, but fails on the same mail than the 1st time. The problem is that it considers wrongly that *all* mails action has failed despite the copy part of the move did worked. So mails are not deleted from inbox.
Day after day, things get worst: new + old mails obeying the criterion are copied instead of moved. After a few days, hundreds of copies of each mail accumulates in the target folder. Up to reaching provider quota (if some emails have attachements... just counts the Gb). After this, you just loose all next emails sent to this account as long as you don't notice the nasty thing. :-(


Expected results:

1: in the batch of mails to move, only one mail should be marked as failed, not all.
    Or if the copy is considered as fail, the possibly (partly)copied emails should be (temptatively)deleted for coherency.
2: if a filter action failed on an email, it should be flaged as erroneous and not tried again and again.
3: it should be great that at least thunderbird can detect that the fail is due to exceeding a limit on the target IMAP account provider (I guess the action got an error message with some returned information). It would be even greater if it could know it before and not try doing the forbiden action. And the very best would be to access this info in the filters ("size < accounts.maxsize").

Comment 1

[WADA:World Anti-bad-Duping Agency]

If multiple mails hits with same filter rule or different rules, and if the multiple mails are moved to same target folder, "Move by filter" is done by single "bulk move" operation, and if within same IMAP sever, it's done by single or two IMAP command.
(A) no "Move" extension support, move within same IMAP account
  At INBOX : (A-1) uid copy  smallest_uid:largest_uid Move_Target_Folder
             (A-2) uid store smallest_uid:largest_uid +Flags(\Deleted)
(B) "Move" extension support, move within same IMAP account
  At INBOX : (b-1) uid move smallest_uid:largest_uid Move_Target_Folder
(See IMAP log of bug 892424 comment #5)

If connection error between (A-1) and (A-2), timeout while (A-1), etc. occurs, phenomenon of "*all* mails action has failed despite the copy part of the move did worked" is pretty normal.
Is your server connection pretty stable?
When multiple mails are requested by "uid copy a:b,c:d,...x:y Target", does your server return OK respnse within reasonable period?

If "move from IMAP folder to IMAP folder of different account", "Move mesages by filter" may be "Bulk copy" + "Bulk delete".
(C) "move from IMAP folder to IMAP folder of different account"   
  IMAP-1: (C-1) uid fetch smallest_uid:largest_uid body.peek[]
  IMAP-2: (C-2-a) no MULTIAPPEND support   :"append" for each mail
          (C-2-b) with MULTIAPPEND support : sinle "append" command
  IMAP-1: (C-3) uid store smallest_uid:largest_uid +Flags(\Deleted)
If this kind of "Bulk move" or "Batch type move" is executed, and if connection loss or timeout occurs, phenomenon of "*all* mails action has failed despite the copy part of the move did worked" is pretty normal too.

Read bug 892424 comment #4, and get minimum IMAP log, and check IMAP level flow by yourself.
Is something wrong seen in IMAP log like bug 892424?

For request of improvements in "filter copy/move error notification".
It's different and independent issue from "Tb's bad behaviou in filter copy/move", and big changes like bug 695671 comment #37 is needed for it. Open separate/independent bug for it, or watch other bugs, please.

Comment 2

[fabrice.neyret]

Note that this often occurs in a very common situation: the target provider has a quota of size max for a mail. This makes the move fail systematically, and thunderbird will consider the whole batch of move to have failed. Worse: any following filter run will fail since the large mail remains on place.

Thunderbird offers no way to test this value, and do not analyse the error code to figure this cause.

This common case is excessively nasty, because each filter run will try to move an increasing number of emails to the target since each batch is assumed failed. This results in HUNDREDS of copys of (often big) emails to the target folder. Up to reaching the account global quota.

So, this bug has probably caused mail account failure of many users.
This should really be fixed URGENTLY.
Moreover, I don't know any turnaround since filter rules do not know "maxsize" and cannot mix and/or (needed for the rule "my criterions AND size<maxsize")

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to fabrice.neyret from comment #2)

Before adding comments of merely complaint on "Tb's behaviour in filter move is far diferent from your expectation" repeatedly, please attach IMAP log data to this bug which will be an evidence of "Tb's behaviour in filter move is not correect, or inappropriate, or inadequate" with "specific or special conditions, such as maxsize, which occurs in your environment".

Comment 4

[fabrice.neyret]

"please attach IMAP log data": please tell me how to proceed.

"specific or special conditions, such as maxsize, which occurs in your environment": as far as I know, all providers on earth impose a max size for incoming emails.
So it is by no way specific to me.
Just check the max value for your provider, try, and see.

Moreover, I would say that having thunderbird filling an account up to blocking it in such an easily reproductable way IS an especially DAMAGING PURE BUG, not a "complaint" about "expectations".  

So if you are too ashamed to face that and be polite, just hide in a hole and let serious people fix serious things. Your behavior is just damaging confidence in free software development.

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to fabrice.neyret from comment #4)
> as far as I know, all providers on earth impose a max size for incoming emails.
> So it is by no way specific to me.
> Just check the max value for your provider, try, and see.

How can Tb know such size limit?
As fa as I know, there is no way to know such size limit in IMAP. Only avialable "size limit" like one is "Mbox size" relevant one in QUOTA.
And, such size limitation is usually:
  Depends on server,
  Different beween "receiving from SMTP" and "append"(upload). 
How can "mail size limit in my server" be relevant to your problem?
How can "mail size limit in serverA" be relevant to "mail size limit in serverB"? (you are referring "move from serverA to serverB", aren't you?)

> Your behavior is just damaging confidence in free software development.

Here is bugzilla.mozilla.org for "developers"(not for helping users) where to report Tb's actual bug to Tb developers and to ask them to fix Tb's bug. Not "Customer Center".
All Tb developers are volunteers in development of software called Thunderbird, which is also a "free software". Even if any people can open "BUG at B.M.O" at any time, and even if any people can request anything to developers of Tb, if requesting to fix actual bug to developers, sufficient data for problem analysis should be provided to developers by reporter.

I believe that simply saying "Tb has bug" and ordering developers "just fix bug　quickly" won't help developers to fix bug of a "free software". I believe saying "Tb has bug" at B.M.O without required data for problem analysis is merely a "complaint to developers".

> "please attach IMAP log data": please tell me how to proceed.

IIRC, I already pointed bug for it. You haven't read pointed bug's comment?

Because your case is "filter move from serverA to serverB", all of followings are relevant to your problem.

(A) Tb's bug around "append".
Tb had problem of Bug 693353. This kind of problem may still exist in recent Tb.
If NO is returned while "append" due to "mail size limit" or "Quota limit", similar problem may occur.
If such problem is involved, "talking about 'one filter error is interpreted as a error' without resolving such problem" is nonsence.
See also bug 542723.

(B) Tb's problem around "filter move of multiple mails".
As I already wrote by example of "MULTIAPPEND", Tb's behavior depends on IMAP extensions which is supported/not supported by IMAP server.
"When Tb can detect error" depends on Tb's behavior, and "How and when Tb actually detects error" depends on current program code.
And, as I already wrote by pointing a bug, Tb has problem in error detection/error handling during filter execution.
To resolve such issues, problem analysis by specific case is needed. Without finding solution of specific case, general solution can not be found usually.
This is reason why I requested you to provide IMAP log to developers.

(C) Problem in notification/reporting of error while filterring.
I already pointed a bug.

Because issue around "append" is relevant, UIDPLUS, MULTIAPPEND, is important extension.
Get minimum IMAP log with minimum configuration, with normal case, with several pretty small test mails first, and check CAPABILITY response, flow around "fetch from move source server" and "append to move target server" etc..
If flow of normal case is provided, develers may be possible to guess Tb's behavior when "NO to append due to mail size limit".
See IMAP log I attached to bug 892424. That log was different(Yahoo! IMAP) from bug opener's case(Gmail IMAP), but it was suficient for problem analysis, and it's readable because of "minimum".
Note: "Minimum configuration" I called:
- Stop any automatic IMAP server access.
  new mail check, mail purge, filter for other servers, ...
- Or, new profile, with two relevant IMAP accounts only

Comment 6

[WADA:World Anti-bad-Duping Agency]

Attachment: IMAP log : Filter move of 3 mails from Yahoo! IMAP to Gmail IMAP

FYI.
Quick check result, with "filter move from Yahoo! IMAP to Gmail IMAP".
"Filter move of multiple mails from ServerA to ServerB" was:
(1) Fetch multiple mails from INBOX of serverA,
(2) Copy(append) multiple mails to MboxB of serverB
    Because Gmail doesn't support MULTIAPPEND, one append command was
    used for upload of each mail data.
    I don't know Tb's behavior when MULTIAPPEND is supportd.
(3) Delete(store Flags \Deleted) multiple mails from INBOX of serverA.
Each step is a "Batch type job" and "Error during append command for a mail" in step (2) == "failure of Batch type job of step (2)".
So, step (3) is not executed if error occurs during step (2).
(i.e. data loss in Bug 693353 is perhaps already resolved, as far as error detection is correctly done in any case. If "NO to append" is still not correctly processed, same problem as Bug 693353 occurs before problem of "one filter error is interpreted as a error" you call.)
Note: Checked with Offline-Use=Off to make it simple. I don't know difference when mail data is downloaded to offline-store file by auto-sync during filter execution.

This is current implementation of message filter.
1. Check filter rules for all new mails.
2. For all hit mails, do non Copy/Move/Delete actions.
3. For each "Copy Target Folder", do "Copy to Folder"
4. For each "Move/Delete Target Folder", do "Move to Folder, or Delete".
(bug 892424 is for problem of "4. is executed before end of 3.")

Are you requesting "complete filter execution per a new mail" in any case? (from Local/IMAP to Local/same-IMAP/Diferent-IMAP, if IMAP, from/to Offline-use=Off/Offline-Use=Off, from/to FileStore==BerkleyStore/MaildirStore, ...)
Or simply saying "Tb has bug" at B.M.O for developers, without checking current implementation?

If latter, please note that "different behavior of a software from your expectation" is not always bug(flaw in program code) of a software.
If latter, your behavior precisely, is an example of following, isn't it?
> Your behavior is just damaging confidence in free software development

If former, please note that a reason of "batch type job" is performance reason in IMAP.
If Batch Type, (1) uid fetch x:y body.peek[], (2) single append for multiple mails(with MULTAPPEND), (3) uid store x:y Flags \Deleted(as done currently), is possible. This is better than "uid fetch N" for each mail, "append" for each mail, "uid store N Flags \Deleted" for each mail, from perspective of IMAP server access efficiency.
I think big implementation change will be needed for your request, even though I also think "complete filter execution per a new mail" is better from perspective of "error detection/recovery", and "too many UIDs in an IMAP command" should be avoided at as many places as possible(see bug 846123 for very many mails(UIDs) in single "uid copy" command when mail copy within an IMAP server).

Comment 7

[WADA:World Anti-bad-Duping Agency]

To [e-mail](bug opener):

Is your report of comment #0 based on your detailed observation of phenomenon which actually occured due "mail size limit" when "append"(upload of mail)?
If so, and if you server returns "NO" to append, instead of killing connection immediately when mail which exceeds "mail size limit" is uploaded, it means "Bug 693353 is already fixed".

Comment 8

[fabrice.neyret]

(In reply to WADA from comment #5)
> Fabrice wrote:
> > Just check the max value for your provider, try, and see.

> How can Tb know such size limit?
> As fa as I know, there is no way to know such size limit in IMAP. Only
> avialable "size limit" like one is "Mbox size" relevant one in QUOTA.

I don't know the IMAP API, but I would be surprised that there are no command to get server's limits and policy as well as no usable error return code + msg. May be it is the case (which would be really bad).
If TB has really no way to predict predictable fails (which are partially not failed), I guess a safer solution would be to not coalescent emails for IMAP to IMAP transfers. (It might be the reason why TB used to be non reliable to move dirs with subdirs. Don't know if still the case nowadays).

> And, such size limitation is usually:
>   Depends on server,

off course (destination server).

>   Different beween "receiving from SMTP" and "append"(upload). 

possibly, but for the 3 servers I tested the IMAP and SMTP per message limit was consistent. Oppositely, many providers allow sending emails larger than they accept.


> How can "mail size limit in my server" be relevant to your problem?
> How can "mail size limit in serverA" be relevant to "mail size limit in
> serverB"? (you are referring "move from serverA to serverB", aren't you?)

the relevant mail size limit is obviously the one of the targer server. 
If I can move to it, it's because TB knows my accounts and serveurs A and B. So if a getlimits command or an error status is available, TB can make use of it.


> Here is bugzilla.mozilla.org for "developers"(not for helping users) where
> to report Tb's actual bug to Tb developers and to ask them to fix Tb's bug.
> Not "Customer Center".

I have been a developper for decades. In R&D of several companies, and teaching, and now as a CS researcher. I'm on several GPL products.
Still, I'm horrified the way ubuntu/linuxLibs/thunderbird/firefox treat serious users and bug reports. Really, after 30 years of activism I now sometime hesitate to recommand serious use of free tools because of that, and maybe soon, even for myself. E.g. pages and pages of discussions - often silly - for years about a bug that is very probably a "" protection missing in a path (bug for svn+ssh: url in clicable-link extensions but due to base libs). TB regressing on several ways, and visibly no longer developped by people using it intensively (with hundreds of subdirs, hundreds of email/day, several accounts). Same for Ubuntu (concerning NFS home dirs, multiuser per machine and multimachine per users, NFS4, not speaking of "not a warning but no worry error messages" in logfiles).
Ok I stop here with my "old green ages complaint" and get back to problem solving mode.

> I believe that simply saying "Tb has bug" and ordering developers "just fix
> bug　quickly" won't help developers to fix bug of a "free software". I
> believe saying "Tb has bug" at B.M.O without required data for problem
> analysis is merely a "complaint to developers".

The interface requires fields "step to reproduces", "bad result obtained", "expected result", and I fullfilled these. This set often contains all the pertinent information.
BTW I just don't know how to get the trace you ask for (if available for non-crashes), and you didn't replied on this point so I still don't know.


> > "please attach IMAP log data": please tell me how to proceed.
> IIRC, I already pointed bug for it. You haven't read pointed bug's comment?

The point is that I'm a developper (maybe old school) but I just don't understand your technical posts, even after reading several time. And I don't know the meaning of IIRC (I just googled it). So I just could do nothing with them.


> Because your case is "filter move from serverA to serverB", all of
> followings are relevant to your problem.
> (A) Tb's bug around "append".
> Tb had problem of Bug 693353. This kind of problem may still exist in recent
> Tb.
> If NO is returned while "append" due to "mail size limit" or "Quota limit",
> similar problem may occur.
> If such problem is involved, "talking about 'one filter error is interpreted
> as a error' without resolving such problem" is nonsence.
> See also bug 542723.

ok, these might share common causes even if concerning other user-level behavior.
Note that user-level behavior is also important, since connected to the level of consequences and ability to turn-around or not.

> (B) Tb's problem around "filter move of multiple mails".
> As I already wrote by example of "MULTIAPPEND", Tb's behavior depends on
> IMAP extensions which is supported/not supported by IMAP server.

ok.

> "When Tb can detect error" depends on Tb's behavior, and "How and when Tb
> actually detects error" depends on current program code.

I'm not sure about the distinction between "TB" and "current program code" (maj/min version ?)

> And, as I already wrote by pointing a bug, Tb has problem in error
> detection/error handling during filter execution.

ok. anyway, note that it behaves exactly the same when I manually select a bunch of emails and drop them to the folder on the other IMAP server, so it does not seem directly related to filtering per se.


> To resolve such issues, problem analysis by specific case is needed. Without
> finding solution of specific case, general solution can not be found usually.
> This is reason why I requested you to provide IMAP log to developers.

I would be really happy to provide it if you explain me how to get it (as asked on my previous post).
...
gotcha: https://wiki.mozilla.org/MailNews:Logging
cool ! 
-> which modules options + other Protocol Logging options do you want me to set ? all level 4 ? (is it resource intensive or can I let them set permanently ?)
Thanks.


> See IMAP log I attached to bug 892424. That log was different(Yahoo! IMAP)
> from bug opener's case(Gmail IMAP), but it was suficient for problem
> analysis, and it's readable because of "minimum".
> Note: "Minimum configuration" I called:
> - Stop any automatic IMAP server access.
>   new mail check, mail purge, filter for other servers, ...
> - Or, new profile, with two relevant IMAP accounts only

ok, it is a full experimental set-up + resulting logs, not a simple on-the-fly trace.

Comment 9

[fabrice.neyret]

(In reply to WADA from comment #6)
> Are you requesting "complete filter execution per a new mail" in any case?
> (from Local/IMAP to Local/same-IMAP/Diferent-IMAP, if IMAP, from/to
> Offline-use=Off/Offline-Use=Off, from/to
> FileStore==BerkleyStore/MaildirStore, ...)

no comprendo (like 80% of your post). Please remind that some developpers are not TB developper ;-).

> Or simply saying "Tb has bug" at B.M.O for developers, without checking
> current implementation?

my TB version is up to date. Anyway I just upgraded my ubuntu version, I don't know if this impacts the level of TB update I can get.


> If former, please note that a reason of "batch type job" is performance
> reason in IMAP.

of course. As long as this does not decay reliability.

Comment 10

[fabrice.neyret]

(In reply to WADA from comment #7)
> Is your report of comment #0 based on your detailed observation of
> phenomenon which actually occured due "mail size limit" when "append"(upload
> of mail)?

due to mail size limit: 99.9% sure (I simplified the experiments down to manually moving a set of 4, 2 then 1 email). assuming "append" = "move mail" at user-level.

> If so, and if you server returns "NO" to append, instead of killing
> connection immediately when mail which exceeds "mail size limit" is
> uploaded, 

don't know (and not sure to understand).

> it means "Bug 693353 is already fixed".

ok. since with TB version ? I'll compare to my TB versions in my lab.

Comment 11

[WADA:World Anti-bad-Duping Agency]

"append" is IMAP command which is used for upload of mail.
See IMAP log I attached(a minimum log).
(1) At INBOX of Yahoo!  : "uid fetch NN body.peek[]" x 3 times
(2) At Gmail-A of Gmail : "append" x 3 times for upload mail
(3) At INBOX of Yahoo!  : "uid store x:y +Flafgs(\Deleted)"

Get IMAP log, and check server response when mail size exceeds limit, and check Tb's behavior in log after the error, please.
Following parameter, which is placed at bottom of bug 892424 comment #4, is useful in many IMAP related testing.
> timestamp,imap:5,msgbiff:5,ImapAutoSync:5,MsgCopyService:5,IMAPOFFLINE:5,MsgPurge:5,MSGDB:5
However, "timestamp,imap:5" only is mandatory to check "response to append". And, "reducing parameter" can perhaps do nothing on "huge log file size", because 99.9999...% of log data is for huge mail data which exceeds server side size limit...
For ease of "removing log for data line", test with crafted mail may be better.
1. Edit .eml file by Text Editor.
  An attachment part of test mail
  Content-Type: application/pdf
  Content-Transfer-Encoding: base64
  all data lines in this part is same data like next.
  AAAAAAAA .............  AAAAAAAAAAA[CRLF]
  => Log lines for data of "AAA ... AAA" can be easily removed
     by pretty simple/small script.
2. Drag&Drop the .eml file to hread pane of an Mbox of Local Folders.
   (Import of .eml file via Drag&Drop)
3. You can use this mail for test any time.
4. Because test to see response to "append", following can be used;
   Drag&Drop the copied mail in the Mbox or .eml file itself
   to Thread Pane of an IMAP Mbox.

Comment 12

[WADA:World Anti-bad-Duping Agency]

> > "When Tb can detect error" depends on Tb's behavior, and "How and when Tb
> > actually detects error" depends on current program code.
> I'm not sure about the distinction between "TB" and "current program code" (maj/min version ?)

"Tb" in the context is a "running instance of software called Thunderbird" from perspective of IMAP sever, OS, network components, etc.
(A) If socket close occurs at server or connection loss happens, it's notified to Tb by TCP/IP sooner or later. So,　"Tb" *can* detect/process it.
(B) However, "Tb's current Message Filter code or IMAP code properly detects it, or properly processes it" is absolutely independent from incident of (A). It depends on current code. Not on design or concept of Tb.
Coding style used at many places of Mozilla's modules is:
  Call someone. If RC!=0, show assertion only when Debug build
  then set RC to ZERO in order to continue processing without error,
  or silently stops processing.
It's far different from OS of host system, big host applications I know.
  Usually, write as "If RC!=Expected one or correct one, Abend",
  in order to surely report error always.
  If unexpected Abend is reported by test or by customer,
  add additional/required RC handling logic if the RC is correct or
  possible one, or fix bug in module who returned wrong return code.
  Repeat it forever...
  "Don't propagate error" is philosophy in host system, to avoid system
  down, to avoid wrong balance of bank account in Data Base.
This is mainly because Mozilla is merely a client application, instead of a component in server system or host system. And, this is a reason of weakness of Tb in error handling.
Read bug 695671 comment #37, please.

Comment 13

[Wayne Mery (:wsmwk)]

fabrice, what are your thoughts on comment 11 and comment 12?

Comment 14

[Wayne Mery (:wsmwk)]

I still can't tell what comment 0 is about.
Is this only about filters failing because account quota was hit?
If yes, this issue is covered in another bug report.
If not, please clarify.