816304 - Filters that copy messages crash with maildir [@ mozalloc_abort | mozalloc_handle_oom | moz_xrealloc | nsTArray_base<nsTArrayDefaultAllocator>::EnsureCapacity | nsTArray<void*, nsTArrayDefaultAllocator>::AppendElements<void*> | nsPresArena::State::Free ]

Status: VERIFIED FIXED

(MailNews Core :: Filters, defect)

Description

[stef]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:17.0) Gecko/17.0 Firefox/17.0
Build ID: 20121120042814

Steps to reproduce:

Created a filter to copy messages into another folder. 
Manually run the filter to test it.
Was happening with TB 16, moved to TB 17 and did not improve.
With Maildir on KDE.
Same happens also with TB 16 and 17 with Maildir on Windows.


Actual results:

Thunderbird crashes. Every time.
When I relaunch it, the message has actually been copied as expected.


Expected results:

No crash, just copy.

Comment 1

[Wayne Mery (:wsmwk)]

stefmorph, please see https://developer.mozilla.org/En/How_to_get_a_stacktrace_for_a_bug_report#Linux

Comment 2

[:aceman]

Or the crash ID from about:crashes.

Comment 3

[stef]

(In reply to :aceman from comment #2)
> Or the crash ID from about:crashes.

Here is the crash id:
http://crash-stats.mozilla.com/report/index/bp-0e0dbf53-484c-4e14-a66e-0db592121201

And the filter:

version="9"
logging="no"
name="Copy Tagged Mail"
enabled="yes"
type="17"
action="Copy to folder"
actionValue="mailbox://nobody@Sent%20Archive/Tagged%20Mail"
condition="AND (tag,isn't empty,remindit.expired)"

Same behavior also in safemode.

Comment 4

[:aceman]

It is interesting that the filter definition also saves the third argument even though there is none in the filter editor UI. I'd expect it to be "(tag,isn't empty,)", but hopefully it is ignored at execution time. But it can store privacy sensitive data, that the user does not expect to be there. E.g. that you have a "remindit.expired" tag.

I created a filter you have pasted (even with the bogus "remindit.expired" value) but I couldn't get the crash on TB20.
What do you mean with the Maildir reference in the report? Have you manually toggled TB from using mbox to the new maildir-lite format in that account?

Comment 5

[stef]

Yes, I was surprised too to find the third argument. I manually removed it to see if it mattered but TB crashed also with the third argument removed.

I am using maildir for mail storage. I am coming from Evolution, which uses maildir, and I like this idea that every mail is a separate message. Makes backups a lot easier. So I have googled around to see if TB can support maildir, and have found that it does, even if it is still experimental. So I have manually replaced

mail.serverDefaultStoreContractID = “@mozilla.org/msgstore/berkeleystore;1″
with
mail.serverDefaultStoreContractID = “@mozilla.org/msgstore/maildirstore;1″

and then recreated all of my stores. Works pretty well, except for this crash with the filters. What do you suggest? Shall I move to TB 20? Go back to mbox?

Comment 6

[:aceman]

OK, so there may be a problem in the experimental maildir implementation.
Does it crash also with any other filter rules?

It is just strange that I do not see anything filters related in the crash log you posted.

Comment 7

[stef]

Here are two more crashes. This time I am on TB 18.0 (I upgraded today), safe mode, and run it from Vista, so a totally different environment, with maildir of course. The first crash is with the same filter we discussed in the past days:

http://crash-stats.mozilla.com/report/index/bp-951fbcf9-5037-40c9-a462-a335f2121204

The second is with a different filter:

http://crash-stats.mozilla.com/report/index/bp-862c51d6-4cfe-4aea-8b36-b8a472121204

Here is the new filter:
version="9"
logging="yes"
name="Subject contains: SAS"
enabled="yes"
type="17"
action="Move to folder"
actionValue="mailbox://nobody@Local%20Folders/Posta%20inviata"
condition="AND (subject,contains,SAS)"

Hope this helps. If you need more info or more experiments, I am available to try.

Comment 8

[:aceman]

Now, these logs are much better, there are filter functions in the stack. I wonder what nsMsgFilterAfterTheFact::`vector deleting destructor' operation is. Maybe some new/malloc/free/free[] mismatch ? Rkent, are you able to track this kind of stuff?

Comment 9

[Tony Mechelynck [:tonymec]]

Related with bug 793524?

Comment 10

[Kent James (:rkent)]

I strongly suspect the addref/release mismatch, which occurs if there is more than once call to onEndExecution. The crash itself should be fixed in bug 555539, but there is probably an extra call to one of the parents of onEndExecution that should be fixed as well, hence bugs like this are not a dup of bug 555539

At some point I am hoping that one of us (possibly me) gets motivated to attack all of the maildir bugs. As for tracking, maildir should be in either the summary or whiteboard so that we can find them.

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to Kent James (:rkent) from comment #10)
> I strongly suspect the addref/release mismatch, which occurs if there is
> more than once call to onEndExecution. The crash itself should be fixed in
> bug 555539, but there is probably an extra call to one of the parents of
> onEndExecution that should be fixed as well, hence bugs like this are not a
> dup of bug 555539
> 
> At some point I am hoping that one of us (possibly me) gets motivated to
> attack all of the maildir bugs. As for tracking, maildir should be in either
> the summary or whiteboard so that we can find them.

irving, does this put the kaibosh to your thought that this might be related to Bug 815012?

regardless, could you take on this bug and the very lonely bug 555539?

Comment 12

[Wayne Mery (:wsmwk)]

note also, bug 791966 has a patch for maildir issue

Comment 13

[:Irving Reid (No longer working on Firefox)]

I agree that this bug no longer seems related to bug 815012. Probably best to mark these bugs helpwanted and put them on The List, rather than specifically assigning them to me - I haven't managed to scrape together much bug fixing time lately.

Comment 14

[Wayne Mery (:wsmwk)]

there is a crapload of signatures with "EnsureCapacity"

https://crash-stats.mozilla.com/query/query?product=Thunderbird&version=Thunderbird%3A17.0&version=Thunderbird%3A16.0.2&version=Thunderbird%3A15.0.1&version=Thunderbird%3A20.0a1&version=Thunderbird%3A19.0a2&version=Thunderbird%3A18.0b1&version=Thunderbird%3A17.0esr&version=Thunderbird%3A17.0b3&range_value=2&range_unit=weeks&date=01%2F01%2F2013+02%3A32%3A02&query_search=signature&query_type=contains&query=EnsureCapacity&reason=&build_id=&process_type=any&hang_type=any&do_query=1

Comment 15

[Wayne Mery (:wsmwk)]

dupe of bug 797710?

Comment 16

[Wayne Mery (:wsmwk)]

Do you still crash when using a current version?

Comment 17

[stef]

Yes, it still does. I am using now version 24.5 under windows 8.1. I tested again today and nothing seems to have changed.

Comment 18

[:aceman]

Do you have a fresh crash ID?

Comment 19

[stef]

Yes I have - here it is:

UUID 	f15596ac-de8a-4fd1-a63f-fe7d92140529

Comment 20

[:aceman]

The stack of this crash is different now:
0 	xul.dll 	nsMsgFilterAfterTheFact::ApplyFilter(bool *) 	mailnews/base/search/src/nsMsgFilterService.cpp (http://hg.mozilla.org/releases/comm-esr24/annotate/c74538a5f310/mailnews/base/search/src/nsMsgFilterService.cpp#l543)
1 	xul.dll 	nsMsgFilterAfterTheFact::OnSearchDone(tag_nsresult) 	mailnews/base/search/src/nsMsgFilterService.cpp
2 	xul.dll 	nsMsgSearchSession::NotifyListenersDone(tag_nsresult) 	mailnews/base/search/src/nsMsgSearchSession.cpp
3 	xul.dll 	nsMsgSearchSession::TimerCallback(nsITimer *,void *) 	mailnews/base/search/src/nsMsgSearchSession.cpp

The maildir implementation is not yet fully polished so things like this are expected to happen. Fortunately we have a GSoC project to fix some of the maildir problems. Maybe this will be fixed somewhere in that work.

Comment 21

[Wayne Mery (:wsmwk)]

bp-f15596ac-de8a-4fd1-a63f-fe7d92140529 is nsMsgFilterAfterTheFact::ApplyFilter(bool *)

2	xul.dll	nsMsgSearchSession::NotifyListenersDone(tag_nsresult)	mailnews/base/search/src/nsMsgSearchSession.cpp 
hg@0 561 nsCOMPtr<nsIMsgSearchNotify> listener;
asutherland@5504 562 m_iListener = 0;
irving@11027     563 while (m_iListener != -1 && m_iListener < (signed)m_listenerList.Length())
hg@0             564 {
asutherland@5504 565 listener = m_listenerList[m_iListener];
ehsan@13324      566 int32_t listenerFlags = m_listenerFlagList[m_iListener++];
asutherland@5504 567 if (!listenerFlags || (listenerFlags & nsIMsgSearchSession::onSearchDone))
asutherland@5504 568 listener->OnSearchDone(aStatus); 

1 xul.dll nsMsgFilterAfterTheFact::OnSearchDone(tag_nsresult) mailnews/base/search/src/nsMsgFilterService.cpp
hg@0 402 if (continueExecution)
hg@0 403 return m_searchHits.IsEmpty() ? RunNextFilter() : ApplyFilter(); 

0 	xul.dll 	nsMsgFilterAfterTheFact::ApplyFilter(bool *)
hg@0 529 nsCOMPtr<nsIMsgCopyService> copyService = do_GetService(NS_MSGCOPYSERVICE_CONTRACTID, &rv);
hg@0 530 if (copyService)
kent@3238 531 {
kent@3238 532 rv = copyService->CopyMessages(m_curFolder, m_searchHitHdrs,
kent@3238 533 destIFolder, actionType == nsMsgFilterAction::MoveToFolder,
bugzilla@10318
534 this, m_msgWindow, false);
kent@3238 535 // We'll continue after a copy, but not after a move
kent@3238 536 if (NS_SUCCEEDED(rv) && actionType == nsMsgFilterAction::CopyToFolder
kent@3238 537 && actionIndex < numActions - 1)
kent@3238 538 m_nextAction = actionIndex + 1;
kent@3238 539 else
kent@3238 540 m_nextAction = 0; // OnStopCopy tests this to move to next filter
kent@3330 541 // Tell postplugin filters if we are moving the message.
kent@3330 542 if (actionType == nsMsgFilterAction::MoveToFolder)
ehsan@13324 543 for (uint32_t i = 0; i < m_searchHits.Length(); i++)
...then...
kent@3330 544 m_curFolder->OrProcessingFlags(m_searchHits[i],
kent@3330 545 nsMsgProcessingFlags::FilterToMove);

Comment 22

[Wayne Mery (:wsmwk)]

bug 797710 related?

bug 537017 related?

Comment 23

[WADA:World Anti-bad-Duping Agency]

Checked with trunk nightly. (2015/3/15 build)
> Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101 Thunderbird/39.0a1

Filter for copy/move from Mbox of Gmail IMAP(MaildirStore, Offline-Use=On) to FolderX of Local Folders(MaildirStore).
   At Mbox of Gmail IMAP, if(...), add Tag Important, Copy to FolderX of Local Folders, Move to FolderX of Local Folders
3 mails are normally copied 2 times to FolderX\cur directory of Local Folders, and deleted from Mbox of Gmail IMAP.


Messaage filter works, and crash does not occur.

Comment 24

[mcmurchy1917-bugzilla]

I'm getting this crash with Filters on a MailDir implementation of Thunderbird Version 31.5.0. Can I get a copy of the patched version to see whether it fixes my system?

Comment 25

[Kent James (:rkent)]

Please use the latest beta of Thunderbird 38 which has much improved maildir support.

Comment 26

[mcmurchy1917-bugzilla]

Thanks Kent James.

Beta of Thunderbird 38 solved the crash problem whilst filtering messages.

As far as my testing reveals the only bug I've still got is moving/deleting multiple messages. What bug report is tracking this - had a quick look but couldn't spot it.

Comment 27

[Kent James (:rkent)]

I am not aware of an issue with moving/deleting multiple messages in TB 38 maildir. Can you please file a bug with a description of your issue?

Comment 28

[WADA:World Anti-bad-Duping Agency]

(In reply to mcmurchy1917-bugzilla from comment #26)
> As far as my testing reveals the only bug I've still got is moving/deleting multiple messages.

What do you call by "moving" and "deleting"? Next?
  (a) "moving" = Filter Action of "Move to folder including Trash"
  (b) "deleting = Filter Action of "Delete", regardless of "imap delete del" if imap
As Filter Action of "Delete" had independent problem of bug 695671, please separate problem in (a) and (b).

Comment 29

[mcmurchy1917-bugzilla]

I've raised Bug 1152651 for Maildir moving multiple messages.

Comment 30

[Thierry Blanc]

38.6.0
Trying to migrate to maildir. When copying single message from one account to a SUBfolder of another account, TB crashes. If I copy to the directory above, no crash.