384735 - Copy to folder by message filter locks dest folder - copy of second or later mail by filter won't complete might see JS Error of "folderSummaryInfoEl.parseFolder is not a function" in newmailalert.js

Status: VERIFIED FIXED

(MailNews Core :: Backend, defect)

Description

[Ian Ribas]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
Build Identifier: versão 2.0.0.4 (20070604)

I create a filter with the following conditions:

- From is in address book Personal
- Subject doesn't contain ...

And the following actions:

Mark as read
Copy to ....

The filter is to be applied to the Inbox folder of the "Local Folders" account. It is a kind of "archive" filter.

I hav two email accounts, both saving new messages on the Inbox folder of the "Local Folders" account.

The problem is that, when receiving messages, only the first message received that matches the filter is copied to the chosen folder, the others are marked as read but not copied. Stranger still, I cannot copy or move messages to the folder even by hand, the folder seems somehow "blocked". If I restart thunderbird I can then copy the messages to the folder.

In the filter log, all actions are logged ok, even those that have not really happened! I see the all messages have been trated by the filter because they are marked as read, but the copy action does not work, even if it is logged as if it did.

I'm using thunderbird 2.0.0.4, brazillian portuguese version, on windows XP english.

Below a copy of the msgFilterRules.dat file:

version="8"
logging="yes"
name="Arquivamento"
enabled="yes"
type="1"
action="Mark read"
action="Copy to folder"
actionValue="mailbox://nobody@Local%20Folders/Antigos-2007"
condition="AND (from,is in ab,moz-abmdbdirectory://abook.mab) AND (subject,doesn't contain,[Echecs])"


Reproducible: Always

Steps to Reproduce:
1. Create filter with a "From is in address book" condition and a "copy to folder" action on local folders/Inbox
2. Make sure you are in your own address book
3. Send two or more test messages to yourself
Actual Results:  
Only first message is copied to folder and folder is "blocked".

Expected Results:  
All messages should have been copied.

Comment 1

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> Actual Results:  
> Only first message is copied to folder and folder is "blocked".

It sounds that filter is waiting for lock or copy failed due to dead lock or something wrong, although culprit of dead lock or something wrong can be "copy by filter" himself.
Processes I can imagine which possibly access to folder of Antigos-2007 are;
 - Message filter for other accounts or "Local Folders"
 - Junk filter of other account (if mail is moved to Antigos-2007 by filter)
 - Internal re-build index if something wrong occurred during filtering
 - Automatic compact folder
 - Check for Retention Policy
 
Questions to clarify.
(Q1) POP3? IMAP? If POP3, Global Inbox? (sounds POP3 and Global Inbox)
(Q2) When POP3 and Global Inbox, both message filter for account and message
     filter of account for Global Inbox(usually Local Folders) can be defined.
     Do you have filter for both? (2 accounts, then 3 filters can exist)
(Q3) Do you have other filter rule which moves(not copy) mail to Antigos-2007?
     Please check all three message filters, if multiple message filters.
(Q4) Do you enable "allow antivirus clients to quarantine individual messages"?
(Q5) Do you enable "Check for new messages at startup"?
     If yes, will problem occur when options are changed?
        - Disable "Check for new messages at startup"
        - Disable "Check new message every NN minutes"
      (Case-1) Get Msgs
      (Case-2) Get All New Message  
(Q6) Do you install extension(add-on)?
     If yes, will problem occur even when safe-mode of Thunderbird?

Comment 2

[Ian Ribas]

Answers:

Q1: It is effectively POP3 and Global Inbox, sorry for not mentioning it earlier.

Q2: I tried the filter for the Global Inbox only, so only one filter exists for the Global Inbox folder.

Q3: No other filters exist.

Q4: "allow antivirus clients to quarantine individual messages" is not enabled.

Q5: The two "Check ..." options were set. Tried taking them off and nothing changed. Tried case-1 and case-2 also to no avail.

Q6: I have QuickFile installled. Tried safe mode and the provblem continues.

Other than that:
- "automatic compact folder" seems to be off.
- There was a "retention policy" for the Inbox folder as "Use server defaults" (or something that would translate to that), changed to "Never delete" and problem persists. The terget folder has the same "Never delete" policy.
- Is there a way to check if index rebuild is going on?

Anyway, thanks for the prompt answer! Even if mine was not as swift.

Comment 3

[WADA:World Anti-bad-Duping Agency]

(Q7) Do you use anti-virus software? Are file of Antigos-2007 and Antigos-2007
     excluded from scan target of anti-virus software?
(Q8) If not excluded, what will happen when "Local Folders" directory is excluded
     from scan target of anti-virus software.

Comment 4

[WADA:World Anti-bad-Duping Agency]

Problem is re-created with Thunderbird version 3.0a1pre (20070612) (Win-XP SP2).
 Two mails on POP3 server to be filtered. Get Msgs => only first one is copied.
 (Tb's mail directory is not excluded from scan target of avast! 4)
Next error in Error Console.
> Error: folderSummaryInfoEl.parseFolder is not a function
> Source File: chrome://messenger/content/newmailalert.js
> Line: 88
After above problem,
 1. Move of a mail(drag&drop) to the copy target folder of filter does nothing.
 2. Compact folder, Re-build index says Done Compaction or Done.
 3. Move mail(D&D) from the folder to other healthy folder worked fine.
 4. Same action as step 1 still does nothing.
Above JS error seems to have locked the folder when copy of first mail by filter. Note: I didn't do test case of no scan by avast! yet.

Confirming.

Comment 5

[WADA:World Anti-bad-Duping Agency]

Ian Ribas(bug opener), is there above error message in Error Console when the problem occur on Tb 2.0? 

Comment 6

[WADA:World Anti-bad-Duping Agency]

Changing to Core because error in messenger/content/newmailalert.js.

Comment 7

[WADA:World Anti-bad-Duping Agency]

CC-ing to David Bienvenu.
Bug 307540, Bug 331391, Bug 368383, Bug 376235 look to be reports of some phenomena in this bug and/or reports of problem after this bug.
Bug 289683 is possibly one of oldest report of problem, I'm not sure though.

Comment 8

[WADA:World Anti-bad-Duping Agency]

Above JS Error occurr just after copy of first filtered mail, i.e. problem is recreated with only one mail and your message filter.

Comment 9

[WADA:World Anti-bad-Duping Agency]

I tested after stop standard shield(on-access scan)) of avast. Above JS error was issued, and problem occurred even when no scan by avast.
Ian Ribas, please ignore (Q7)/(Q8) in Comment #3.

Comment 10

[WADA:World Anti-bad-Duping Agency]

Changing summary, because cause is JS error just after first copy by filter, and because subsequent problem is not only "copy of second or later mail by filter won't complete".
Ian Ribas(bug opener):
Please modify/correct bug summary if you feel it's inappropriate.

Comment 11

[David :Bienvenu]

I believe that js error in newmailalert.js (which is TB only) is just a symptom of the underlying problem, and not the cause. The underlying problem seems to be that the db is marked invalid - I haven't reproduced this bug to verify it, however. I will try that today.

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #11)
Oh, David has come!
Possibly problem when "copy to a folder under Local Folders" in POP3 account's message filter. I tested with filter of "If (condition), Add a Tag, Copy to a folder under Local Folders", according to bug opener's comment #0.

Comment 13

[WADA:World Anti-bad-Duping Agency]

Filter rule I used in problem re-creation test.
> action="AddTag"
> actionValue="tag-01"
> action="Copy to folder"
> actionValue="mailbox://nobody@Local%20Folders/filter-copy-test"
> condition="OR (from,contains,[e-mail])"
(my real mail address is replaced by [e-mail] before paste)

Comment 14

[David :Bienvenu]

OK, I've reproduced this problem locally in 2.0.0.4 and the trunk - the issue is that for some reason the first copy request with that folder as a target never gets cleared in the copy service, and all the other copy requests get in line behind it. I'm not sure why it doesn't get cleared, since the copy does seem to be getting executed.

Comment 15

[David :Bienvenu]

Attachment: proposed fix

When using the global inbox, you want to copy from m_downloadFolder, which is the global inbox, not m_rootFolder, which is the incoming server's inbox. I'm not sure why the copy worked for the first message at all with the old code.

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #15)
> Created an attachment (id=268866) [details]
> proposed fix

> I'm not sure why the copy worked for the first message at all with the old code.

Oh, sorry. I forgot to mention very important thing.
Reporter's case is Global Inbox case. But when I tested, I tested with account of no Global Inbox which I usually use. So probably m_downloadFolder==m_rootFolder in my case, and mail is copied to "Local Folders" by filter.
Can this explain why?

Comment 17

[Ian Ribas]

Just to say that I don't get anything in the Error Console ... And to thank
again for the prompt answers. Waiting eagerly for the fix to be released ;-)

Comment 18

[David :Bienvenu]

I'm not sure what's going on here - m_downloadFolder and m_rootFolder are the same folder, it turns out, if I'm reading the code correctly. That explains why the first copy works at all. Where we're failing is in the copy service code that compares pointer values for the srcFolder nsISupports pointer for the copy. Somehow the QI's are coming out different, which doesn't make any sense. I guess I'll have to debug it more to figure out why my change seemed to work, and if there's some other way to fix it.

Comment 19

[David :Bienvenu]

adjusting summary since it's the copy by filter that's the interesting part. I take back what I said earlier - m_downloadFolder and m_rootFolder aren't the same. The reason the copy works at all is because we use the msgHdr->GetFolder() result as the source folder for the copy url, but we check for the src folder passed into the copy operation when trying to clean up the request. This causes confusion. So the fix is good, but I might want to try to cleanup the msgcopyservice code so that we can't get into this situation again...

Comment 20

[David :Bienvenu]

fixed on trunk - only fixes global inbox case.

Comment 21

[David :Bienvenu]

Comment on attachment 268866 [details] [diff] [review]
proposed fix

request 2.0.0.x approval - simple fix that only affects pop3 copy filter actions.

Comment 22

[David :Bienvenu]

What's still broken is when you have a pop3 filter that does a copy to a local folder followed by a move to a local folder. The copy happens asynchronously, but the move is synchronous, and clears the bits out from under the copy operation. That's Bug 376235, which is really unrelated to this issue.

Comment 23

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #20)
> fixed on trunk - only fixes global inbox case.
Is my non Global Inbox case covered by other bug?

Comment 24

[David :Bienvenu]

WADA, I can't really say. Can you try a build from tomorrow or later and see if it works? Do you have any filter that does a move of the message that your test filter is doing a copy of? It doesn't have to be the same filter...

Comment 25

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #24)
> Do you have any filter that does a move of the message (snip)
No. I tested with "Add Tag" and "Copy to folder" only.
I'll test build your patch is landed on, and feed back to you.

Comment 26

[WADA:World Anti-bad-Duping Agency]

To David Bienvenu:
Question before test - can "quarantine" option have relation to problem?
Should I test both quarantine=Off case and quarantine=On case?

Comment 27

[David :Bienvenu]

quarantining is definitely one variable in the equation, so testing both cases would be good.

Comment 28

[Scott MacGregor]

Comment on attachment 268866 [details] [diff] [review]
proposed fix

a=mscott for mailnews only change to 1.8.1.5.

Comment 29

[David :Bienvenu]

fixed for 1.8.1.5

Comment 30

[WADA:World Anti-bad-Duping Agency]

Tested with Tb version 3.0a1pre (2007070804) (Win-XP SP2).
(Case-1 : quarantine option =OFF)
   Two mails were copied by filter successfully. No lock of folders was observed.
   => VERIFIED
(Case-2 : quarantine option =ON)
   Two mails were NOT copied by filter. No message filter log.
   => Re-open

Comment 31

[WADA:World Anti-bad-Duping Agency]

Additional result.
"Run Filter on Folder" works well even after problem of Case-2.

Comment 33

[David :Bienvenu]

I think the quarantine case is a different issue/bug. Did it work before this change?

Comment 34

[David :Bienvenu]

I'm going to re-resolve this as fixed, and open a different bug for the quarantine case.

Comment 35

[David :Bienvenu]

see Bug 387361  for the quarantine case.

Comment 36

[WADA:World Anti-bad-Duping Agency]

Changing to VERIFIED, according to result of quarantine=OFF case in comment #30.

Comment 38

[Carsten Book [:Tomcat]]

verified fixex 1.8.1.5 using Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.5) Gecko/20070716 Thunderbird/2.0.0.5 ID:2007071611 (Thunderbird 2.0.0.5 RC1) and the steps to reproduce from this bug. 

Comment 39

[glenn n. davis]

I seem to have this problem but i will state it in my own words:
i cannot delete this folder i cannot copy to it i cannot move it
 
the mail sort filter gives an error:
ALERT!  The messages could not be moved or copied to the folder(foldername)because writing to folder failed.To gain diskspace....
I tried the remedy but disk space is not the issue.

I now have this issue with thunderbird 2 as well
I can rename the folder but the renamed folder cannot be moved or deleted or repositioned either!
It is listed as fixed but it still cannot be gotten rid of.
the file is effectively locked.