1158578 - Crash in nsImapCacheStreamListener::OnStopRequest when compacting IMAP account

Status: VERIFIED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Mason]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0 SeaMonkey/2.33.1
Build ID: 20150321194901

Steps to reproduce:

When SeaMonkey decides to compact folders (to save storage space), the process runs for 10-20 seconds, works for a few folders, then crashes, leaving the current account in a corrupt state (clicking a message shows the body of a different message).

This might be a dupe of bug 1066998, but I have a different crash signature.

My mailnews setup is moderately complex:

8 IMAP accounts (TLS IMAP port 993)
2 POP3 accounts (TLS POP3 port 995)
2 NNTP accounts (clear text port 119)
1 NNTP account (TLS NNTP port 563)

Comment 1

[Mason]

Crash Report
[@ nsCOMPtr_base::assign_with_AddRef(nsISupports*) | nsImapMockChannel::Close() ]

bp-b4894bd5-cdf0-4ba2-8cc8-cf9f62150426
bp-51cb6fae-063f-42c9-a385-ba8092150214
bp-10b03a26-38e9-4288-aa13-0795b2141025
bp-eeec13b8-7f96-458e-a2f7-ba9da2140919

nsCOMPtr_base::assign_with_AddRef(nsISupports*)
nsImapMockChannel::Close()
nsImapCacheStreamListener::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsInputStreamPump::OnStateStop()
nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*)
nsInputStreamReadyEvent::Run()
nsThread::ProcessNextEvent(bool, bool*)
NS_ProcessNextEvent(nsIThread*, bool)
nsThread::Shutdown()
nsRunnableMethodImpl<void ( mozilla::dom::DOMStorageCacheBridge::*)(void), void, 0>::Run()
nsThread::ProcessNextEvent(bool, bool*)
NS_ProcessNextEvent(nsIThread*, bool)
nsThread::Shutdown()
nsRunnableMethodImpl<void ( mozilla::dom::DOMStorageCacheBridge::*)(void), void, 0>::Run()
nsThread::ProcessNextEvent(bool, bool*)
NS_ProcessNextEvent(nsIThread*, bool)
mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*)
MessageLoop::RunHandler()
MessageLoop::Run()
nsBaseAppShell::Run()
nsAppShell::Run()
nsAppStartup::Run()
XREMain::XRE_mainRun()
XREMain::XRE_main(int, char** const, nsXREAppData const*)
XRE_main
do_main
NS_internal_main(int, char**)
wmain
__tmainCRTStartup
BaseThreadInitThunk
__RtlUserThreadStart
_RtlUserThreadStart

Comment 2

[Mason]

I have another crash signature, might be related.

Crash Report
[@ nsCOMPtr_base::assign_with_AddRef(nsISupports*) | nsImapCacheStreamListener::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult) ]

bp-277399c2-b5c2-4fb7-8d24-3851d2150426

This seems to be the same signature from bug 1066998

nsCOMPtr_base::assign_with_AddRef(nsISupports*)
nsImapCacheStreamListener::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsInputStreamPump::OnStateStop()
nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*)
nsOutputStreamReadyEvent::Run()
nsThread::ProcessNextEvent(bool, bool*)
NS_ProcessNextEvent(nsIThread*, bool)
nsThread::Shutdown()
nsRunnableMethodImpl<tag_nsresult ( nsIThread::*)(void), void, 1>::Run()
nsThread::ProcessNextEvent(bool, bool*)
NS_ProcessNextEvent(nsIThread*, bool)
nsThread::Shutdown()
nsRunnableMethodImpl<tag_nsresult ( nsIThread::*)(void), void, 1>::Run()
nsThread::ProcessNextEvent(bool, bool*)
NS_ProcessNextEvent(nsIThread*, bool)
nsThread::Shutdown()
nsRunnableMethodImpl<tag_nsresult ( nsIThread::*)(void), void, 1>::Run()
nsThread::ProcessNextEvent(bool, bool*)
NS_ProcessNextEvent(nsIThread*, bool)
nsThread::Shutdown()
nsRunnableMethodImpl<tag_nsresult ( nsIThread::*)(void), void, 1>::Run()
nsThread::ProcessNextEvent(bool, bool*)
NS_ProcessNextEvent(nsIThread*, bool)
mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*)
MessageLoop::RunHandler()
MessageLoop::Run()
nsBaseAppShell::Run()
nsAppShell::Run()
nsAppStartup::Run()
XREMain::XRE_mainRun()
XREMain::XRE_main(int, char** const, nsXREAppData const*)
XRE_main
do_main
NS_internal_main(int, char**)
wmain
__tmainCRTStartup
BaseThreadInitThunk
__RtlUserThreadStart
_RtlUserThreadStart

Comment 3

[Mason]

In both stacks, it seems we crash in
nsCOMPtr_base::assign_with_AddRef(nsISupports*)

Comment 4

[Mason]

Manually requesting "Compact folders" on one of my IMAP account.

Crash Report
[@ nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult) ]

bp-b55add11-aac4-400e-bb85-ae47a2150426

Back trace is 118 functions deep!

nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)
nsImapMockChannel::ReadFromImapConnection()
nsImapMockChannel::AsyncOpen(nsIStreamListener*, nsISupports*)
nsImapService::GetMessageFromUrl(nsIImapUrl*, int, nsIMsgFolder*, nsIImapMessageSink*, nsIMsgWindow*, nsISupports*, bool, nsIURI**)
nsImapService::StreamMessage(char const*, nsISupports*, nsIMsgWindow*, nsIUrlListener*, bool, nsACString_internal const&, bool, nsIURI**)
nsOfflineStoreCompactState::CopyNextMessage(bool&)
XREMain::XRE_mainRun()
XREMain::XRE_main(int, char** const, nsXREAppData const*)
XRE_main
do_main
NS_internal_main(int, char**)
wmain
__tmainCRTStartup
BaseThreadInitThunk
__RtlUserThreadStart
_RtlUserThreadStart

Comment 5

[Mason]

One more for the road, with an even deeper stack (128 deep!)

[@ nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult) ]

bp-286da79d-e41a-4743-8ad4-bfe842150426

Crash Address: 0x5a5a5a5a

Trying to use a function pointer in a freed struct?

Comment 6

[Mason]

browser.cache.use_new_backend comes with the following comment:

Preference for switching the cache backend, can be changed freely at runtime
0 - use the old (Darin's) cache
1 - use the new cache back-end (cache v2)

In SM 2.33, default is 0; I changed it to 1.

I was then able to compact my 8 IMAP accounts without crashing.

I hear that Thunderbird 38 going to fix the problem. How?

Comment 7

[Philip Chee]

(In reply to Mason from comment #6)
> browser.cache.use_new_backend comes with the following comment:
> 
> Preference for switching the cache backend, can be changed freely at runtime
> 0 - use the old (Darin's) cache
> 1 - use the new cache back-end (cache v2)
> 
> In SM 2.33, default is 0; I changed it to 1.
> 
> I was then able to compact my 8 IMAP accounts without crashing.
> 
> I hear that Thunderbird 38 going to fix the problem. How?
Probably by switching to the new cache backend.

Comment 8

[Philip Chee]

Mason, thank you for the detailed analysis.
CC rkent and jcranmer
Hope the mailnews backend experts can identify the cause.

Comment 9

[Mason]

You're welcome.

As I stated in comment #1, this bug might be a dupe of bug 1066998

And setting browser.cache.use_new_backend to 1 seems to work around the problem.

Comment 10

[Mason]

Uh-oh, I got a crash even with browser.cache.use_new_backend set to 1.
(Manually compacting my main IMAP account)
bp-0c5493cc-eab4-4914-926c-2c1d32150506

Comment 11

[Mason]

Two more crashes, one with use_new_backend=1
bp-6e0912d8-d61e-4633-9234-c64732150508

the second with use_new_backend=0
bp-4ea81517-53ce-4a38-8d3d-641ce2150508

Please note that the crash signatures are different!

Does the full state of about:config vars appear in crash reports?

Comment 12

[Wayne Mery (:wsmwk)]

bp-6e0912d8-d61e-4633-9234-c64732150508
 nsCOMPtr_base::assign_with_AddRef(nsISupports*) | nsImapCacheStreamListener::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult) More Reports Search
 0 	xul.dll	nsCOMPtr_base::assign_with_AddRef(nsISupports*)	xpcom/glue/nsCOMPtr.cpp
1 	xul.dll	nsImapCacheStreamListener::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)	e:/builds/slave/rel-c-rel-w32-bld/build/mailnews/imap/src/nsImapProtocol.cpp:8678
2 	xul.dll	nsInputStreamPump::OnStateStop()	netwerk/base/src/nsInputStreamPump.cpp 

bp-4ea81517-53ce-4a38-8d3d-641ce2150508
[@ nsCOMPtr_base::assign_with_AddRef(nsISupports*) | nsImapMockChannel::Close() ]
0 	xul.dll	nsCOMPtr_base::assign_with_AddRef(nsISupports*)	xpcom/glue/nsCOMPtr.cpp
1 	xul.dll	nsImapMockChannel::Close()	e:/builds/slave/rel-c-rel-w32-bld/build/mailnews/imap/src/nsImapProtocol.cpp:8763
2 	xul.dll	nsImapCacheStreamListener::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult)	e:/builds/slave/rel-c-rel-w32-bld/build/mailnews/imap/src/nsImapProtocol.cpp:8679
3 	xul.dll	nsInputStreamPump::OnStateStop()	netwerk/base/src/nsInputStreamPump.cpp 

let's focus in bug 1066998

Comment 13

[Wayne Mery (:wsmwk)]

Mason, are you still crashing when using a newer version?

Comment 14

[Mason]

I have a few crashes, but they weren't submitted to Socorro.

0c557e2e-7778-4953-b3c6-0ab5a567fed3	04/02/2016	12:19
1065ab20-a2cc-4fca-8ac3-b29f133258d0	10/01/2016	09:20
a1e67aad-c63d-46f1-af1b-bfa226466530	31/12/2015	11:00
607bdd74-b40a-4bc8-8c32-5afe5b40c91b	20/11/2015	00:25
53a393d1-26cc-430a-83d1-1675a96cba06	20/11/2015	00:17
3e0ca5cd-ef7b-487c-ace7-6b0c374756f9	20/11/2015	00:10
f10f133e-c402-43d9-9247-8d916972a79c	20/10/2015	23:16
91052d26-07ea-45ec-9c19-87a12541581b	20/10/2015	21:32
a73006f9-2b82-4a18-8e0f-8b60d4644472	11/10/2015	17:11

Is there a way to force-push them to the crash server?

Comment 15

[Mason]

(In reply to Mason from comment #14)
> I have a few crashes, but they weren't submitted to Socorro.
> Is there a way to force-push them to the crash server?

I was middle-clicking the crash reports, but only left-clicking works.
They are all Flash-related.

So the last time I had a crash compacting IMAP folders was August 2015 in SM 2.33.1
bp-16d42ef2-2dd8-4f53-b2f9-9776a2150808

browser.cache.use_new_backend is set to 0

Comment 16

[Magnus Melin [:mkmelin]]

Wonder if this got worse now that we we switched away from the old backend.

I've had at least two crashes recently 
bp-666b58a3-339a-454a-9e40-c12b52161020: @ nsImapCacheStreamListener::OnStopRequest
https://dxr.mozilla.org/comm-central/rev/cf775e0371ea81bbd017c36c54d95c611ec44e86/mailnews/imap/src/nsImapProtocol.cpp#8726

bp-a3ec1a6f-09f4-4339-8894-b01762161014: @ nsImapCacheStreamListener::OnStopRequest (but lines off)

+ related to that as it trails down to cache entries (OnCacheEntryAvailable)
bp-c709ae41-0545-4282-bb03-c9fa02161020:  @ nsOfflineStoreCompactState::OnStopRequest
https://dxr.mozilla.org/comm-central/rev/cf775e0371ea81bbd017c36c54d95c611ec44e86/mailnews/base/src/nsMsgFolderCompactor.cpp#1106

Comment 17

[Wayne Mery (:wsmwk)]

magnus nightly build bp-02192508-a95e-41af-bdaa-c38f22170205
lussnig bp-e8587a1b-3eed-4f23-81ab-d4b5f2170213

Comment 18

[Wayne Mery (:wsmwk)]

(In reply to Magnus Melin from comment #16)
> Wonder if this got worse now that we we switched away from the old backend.

Magnus, what "old backend" are you referring to?

Comment 19

[Mason]

I assume he's referring to browser.cache.use_new_backend = 0

Comment 20

[Magnus Melin [:mkmelin]]

Yes, I was referring to using the new cache backend. 
The crash report is a bit hard to interpret, but apparently something in nsImapMockChannel::Close https://dxr.mozilla.org/comm-central/rev/424cdd3c5239dd57a3110b6468bb79ca9d005eaa/mailnews/imap/src/nsImapProtocol.cpp#8807

Comment 21

[Wayne Mery (:wsmwk)]

Mason, 
I've looked up your current crashes and they are completely different, and infrequent. (last one was 3 months ago bp- 2016-12-28 20:27:28 )   Have you changed your setup?

Comment 22

[Mason]

Hello Wayne,

My mail/news setup has not changed ;-)

I think the latest crash signatures you see are for bug 1323688 (which seems to have been fixed in SM 2.49)

I've just compacted my main IMAP account, and the operation completed smoothly.

Regards.

Comment 23

[Wayne Mery (:wsmwk)]

(In reply to Magnus Melin from bug 1316416 comment #51)
> The compact crash should be bug 1158578. It's very common for me, happens
> mostly on my mozilla folder which is fairly big. I suspect size/timing makes
> it more common, and also I suspect it's related to the new cache backend we
> started using.

Yes, this is what concerns me about going live with 52.0. I've had it on the back burner, because I don't know how bad this will get 


(In reply to Magnus Melin from comment #16)
> Wonder if this got worse now that we we switched away from the old backend.

Seems likely. But I can't say for sure.  cache2 was enabled 2016-08-30 via bug 1021843 in TB51.0a1.

Crash rate is higher in recent months for reporters with compact in their comments.  The following shows nightly crash count 55.0a1 > 54.0a1 > 53.0a1. But we have no september data, so for what follows I cannot provide 51.0a1 vs 50.0a1 comparison.  For non-release versions in the last 6 months visit the following link, then click version tab, then sort on versions column https://crash-stats.mozilla.com/search/?user_comments=~ompact&release_channel=%21release&product=Thunderbird&date=%3E%3D2016-09-30T19%3A03%3A50.000Z&date=%3C2017-03-30T19%3A03%3A50.000Z&_sort=-date&_facets=signature&_facets=version&_columns=date&_columns=signature&_columns=version&_columns=build_id&_columns=platform&_columns=user_comments#facet-version

rank count signature
1     14   js::SavedStacks::insertFrames 
2      8   js::jit::InlineFrameIterator::resetOn   
3      7   nsCOMPtr_base::assign_assuming_AddRef | nsImapMockChannel::Close   
4      5   nsOfflineStoreCompactState::OnStopRequest   
5      4   nsScriptSecurityManager::CanCreateWrapper   

collectively for version 52.0b4 these signatures total 90+ crashes which, if all the crash reports are compact related, rank coampact as a #2 crash - https://crash-stats.mozilla.com/topcrashers/?product=Thunderbird&version=52.0b4&_facets_size=200

(To put in perspective using statistics from release versions, 45.6.0 and 45.7.1 each had 10 only crash comments per month.  45.8.0 has been out 3 weeks and not a single compact comment yet - not sure what to make of that)

Comment 24

[Jorg K (CEST = GMT+2)]

OK, we meet again here ;-)

Looking at two reports at random
https://crash-stats.mozilla.com/report/index/bc525551-ed76-41a3-a8bb-5ffb52170330
https://crash-stats.mozilla.com/report/index/e40728e3-1e9d-49f4-bb8b-69a672170329
I see the same thing:

Crashed at nsMsgMailSession.cpp:147

NS_IMETHODIMP
nsMsgMailSession::OnItemPropertyFlagChanged(nsIMsgDBHdr *aItem,
                                            nsIAtom *aProperty,
                                            uint32_t aOldValue,
                                            uint32_t aNewValue)
{
  NOTIFY_FOLDER_LISTENERS(propertyFlagChanged, OnItemPropertyFlagChanged, <=== 147
                          (aItem, aProperty, aOldValue, aNewValue));
  return NS_OK;
}

The macro is this:

#define NOTIFY_FOLDER_LISTENERS(propertyflag_, propertyfunc_, params_) \
  PR_BEGIN_MACRO                                                       \
  nsTObserverArray<folderListener>::ForwardIterator iter(mListeners);  \
  while (iter.HasMore()) {                                             \
    const folderListener &fL = iter.GetNext();                         \
    if (fL.mNotifyFlags & nsIFolderListener::propertyflag_)            \
      fL.mListener->propertyfunc_ params_;                             \
  }                                                                    \
  PR_END_MACRO

The crashes happen at the XPCOM boundary, since we seem to he calling a listener implemented in JS.

I think "crash master" Kent needs to help us here ;-)

I should mention for Kent that the crash signature has some JS details and is tracked in bug 1316416 which sadly is a mixed bag of all things that end up in JS processing due to various causes. The challenge is to eliminate these causes one by one, starting here ;-)

Comment 25

[Kent James (:rkent)]

At least one error I looked at https://crash-stats.mozilla.com/report/index/e40728e3-1e9d-49f4-bb8b-69a672170329 is a stack overflow, which could make sense with the deep stack. There's a repetitive cycle of nsImapMockChannel::OpenCacheEntry() (called 5 times) that is suspicious.

So I see two possibilities. One, you are in some sort of loop, and need to figure out why. Two, all of those calls are legitimate, but combined they overflow the stack. Then you need to exit at some point to stop the stack growth and add a call to post the request to the main window event loop rather than doing things like "return cacheStorage->AsyncOpenURI(newUri, extension, cacheAccess, this);"

Comment 26

[Jorg K (CEST = GMT+2)]

(In reply to Kent James (:rkent) from comment #25)
> There's a repetitive cycle of nsImapMockChannel::OpenCacheEntry()
> (called 5 times) that is suspicious.
Yes, I see that now in both of these:
https://crash-stats.mozilla.com/report/index/bc525551-ed76-41a3-a8bb-5ffb52170330
https://crash-stats.mozilla.com/report/index/e40728e3-1e9d-49f4-bb8b-69a672170329

They all come from nsImapProtocol.cpp:9346:

  // This is where is gets complicated. The entire message is in the cache,
  // but we don't know whether it's suitable for use. Its meta data
  // might indicate that the message is incomplete.
  mTryingToReadPart = true;
  return cacheStorage->AsyncOpenURI(newUri, extension, cacheAccess, this);

Surely that code has changed 500% in bug 1021843 and for the parts in bug 629738.

What I don't understand is that three calls from there, we're at the same spot without ever going through OnCacheEntryAvailable(). That does appear later in the stack and it always calls ReadFromImapConnection() which means that the cache entry was indeed not in the cache or in the cache and unusable.

I think what's going on here well be hard to understand without a reproducible case. So basically there seems to be some logic error in the interaction of cache2 with compaction. User lussnig talked about a "race condition" in bug 1316416 comment #47 and he may not be far off.

I might play around with compacting IMAP folders, trying both with and without offline use. Maybe there is some message with an unexpected parts arrangement. But then Magnus crashes on his bugmail folder and these messages don't even have parts, or do you get your bugmail as HTML, Magnus?

Comment 27

[Wayne Mery (:wsmwk)]

Perhaps bienvenu's bug 619390 comment 3 is useful

My guess is that this happens when we're compacting an offline store, and a message in the folder is not in the offline store, but is in the disk/memory cache, so we try to stream it to write to the new, compacted offline store. I've tried to reproduce that scenario but it hasn't led to any crashes. But that scenario is a bit tricky to reproduce, so I'll have to do a bit of debugging to see if I've actually reproduced it. Normally, we don't hit the ::Close() call in the destructor of the mock channel, and I haven't reproduced that.

Comment 29

[Magnus Melin [:mkmelin]]

Caught it in debugger. I'll have to look at the trace sometime later (the build is not trunk, a bit old, mostly still it should be the same).

Thread 1 "thunderbird" received signal SIGSEGV, Segmentation fault.
nsCOMPtr_base::~nsCOMPtr_base (this=this@entry=0x7fffffff8d90, 
    __in_chrg=<optimized out>)
    at /opt/comm-central/src/obj-x86_64-pc-linux-gnu/dist/include/nsCOMPtr.h:294
294	      NSCAP_RELEASE(this, mRawPtr);
(gdb) bt
#0  nsCOMPtr_base::~nsCOMPtr_base (this=this@entry=0x7fffffff8d90, 
    __in_chrg=<optimized out>)
    at /opt/comm-central/src/obj-x86_64-pc-linux-gnu/dist/include/nsCOMPtr.h:294
#1  0x00007fffe941fb4a in nsOfflineStoreCompactState::CopyNextMessage (
    this=this@entry=0x7fffbeaa3000, done=@0x7fffffff8e13: false)
    at /opt/comm-central/src/obj-x86_64-pc-linux-gnu/dist/include/nsCOMPtr.h:801
#2  0x00007fffe941fd60 in nsOfflineStoreCompactState::OnStopRequest (
    this=0x7fffbeaa3000, request=<optimized out>, ctxt=<optimized out>, 
    status=<optimized out>)
    at /opt/comm-central/src/mailnews/base/src/nsMsgFolderCompactor.cpp:1104
#3  0x00007fffe9530a1b in nsImapMockChannel::ReadFromImapConnection (
    this=this@entry=0x7fff990e0b70)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9472
#4  0x00007fffe9532b47 in nsImapMockChannel::OnCacheEntryAvailable (
    this=0x7fff990e0b70, entry=0x0, aNew=<optimized out>, 
    aAppCache=<optimized out>, status=<optimized out>)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9170
#5  0x00007fffe98c4d0a in mozilla::net::CacheEntry::InvokeAvailableCallback (
    this=this@entry=0x7fffa68a1ec0, aCallback=...)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:875
#6  0x00007fffe98c51bb in mozilla::net::CacheEntry::InvokeCallback (
    this=this@entry=0x7fffa68a1ec0, aCallback=...)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:812
#7  0x00007fffe98c3be7 in mozilla::net::CacheEntry::InvokeCallbacks (
    this=this@entry=0x7fffa68a1ec0, aReadOnly=aReadOnly@entry=true)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:678
#8  0x00007fffe98c3d01 in mozilla::net::CacheEntry::InvokeCallbacks (
    this=0x7fffa68a1ec0)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:620
#9  0x00007fffe98c59b7 in mozilla::net::CacheEntry::Open (
    this=this@entry=0x7fffa68a1ec0, aCallback=..., 
    aTruncate=aTruncate@entry=false, aPriority=aPriority@entry=false, 
    aBypassIfBusy=aBypassIfBusy@entry=false)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:338
#10 0x00007fffe98c5abf in mozilla::net::CacheEntry::AsyncOpen (
    this=0x7fffa68a1ec0, aCallback=aCallback@entry=0x7fff990e0b78, 
    aFlags=aFlags@entry=2)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:312
#11 0x00007fffe98d411c in mozilla::net::CacheStorage::AsyncOpenURI (
    this=0x7fffc0fd15e0, aURI=<optimized out>, aIdExtension=..., aFlags=2, 
    aCallback=0x7fff990e0b78)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheStorage.cpp:112
---Type <return> to continue, or q <return> to quit---
#12 0x00007fffe9531316 in nsImapMockChannel::OpenCacheEntry (
    this=this@entry=0x7fff990e0b70)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9287
#13 0x00007fffe953177a in nsImapMockChannel::AsyncOpen (this=0x7fff990e0b70, 
    listener=<optimized out>, ctxt=0x7fffb49a88c8)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9610
#14 0x00007fffe954ad46 in nsImapService::GetMessageFromUrl (
    this=this@entry=0x7fffd3d61510, aImapUrl=<optimized out>, 
    aImapAction=aImapAction@entry=268435510, aImapMailFolder=<optimized out>, 
    aImapMessage=<optimized out>, aMsgWindow=aMsgWindow@entry=0x7fffc7f78ba0, 
    aDisplayConsumer=0x7fffbeaa3000, aConvertDataToText=false, 
    aURL=0x7fffffff9b38)
    at /opt/comm-central/src/mailnews/imap/src/nsImapService.cpp:1109
#15 0x00007fffe954b4d9 in nsImapService::StreamMessage (this=0x7fffd3d61510, 
    aMessageURI=<optimized out>, aConsumer=0x7fffbeaa3000, 
    aMsgWindow=0x7fffc7f78ba0, aUrlListener=<optimized out>, 
    aConvertData=<optimized out>, aAdditionalHeader=..., aLocalOnly=true, 
    aURL=0x7fffffff9b38)
    at /opt/comm-central/src/mailnews/imap/src/nsImapService.cpp:1210
#16 0x00007fffe941fad3 in nsOfflineStoreCompactState::CopyNextMessage (
    this=this@entry=0x7fffbeaa3000, done=@0x7fffffff9bb3: false)
    at /opt/comm-central/src/mailnews/base/src/nsMsgFolderCompactor.cpp:1037
#17 0x00007fffe941fd60 in nsOfflineStoreCompactState::OnStopRequest (
    this=0x7fffbeaa3000, request=<optimized out>, ctxt=<optimized out>, 
    status=<optimized out>)
    at /opt/comm-central/src/mailnews/base/src/nsMsgFolderCompactor.cpp:1104
#18 0x00007fffe9530a1b in nsImapMockChannel::ReadFromImapConnection (
    this=this@entry=0x7fff990e0aa0)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9472
#19 0x00007fffe9532b47 in nsImapMockChannel::OnCacheEntryAvailable (
    this=0x7fff990e0aa0, entry=0x0, aNew=<optimized out>, 
    aAppCache=<optimized out>, status=<optimized out>)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9170
#20 0x00007fffe98c4d0a in mozilla::net::CacheEntry::InvokeAvailableCallback (
    this=this@entry=0x7fffa1df76e0, aCallback=...)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:875
#21 0x00007fffe98c51bb in mozilla::net::CacheEntry::InvokeCallback (
    this=this@entry=0x7fffa1df76e0, aCallback=...)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:812
#22 0x00007fffe98c3be7 in mozilla::net::CacheEntry::InvokeCallbacks (
    this=this@entry=0x7fffa1df76e0, aReadOnly=aReadOnly@entry=true)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:678
#23 0x00007fffe98c3d01 in mozilla::net::CacheEntry::InvokeCallbacks (
    this=0x7fffa1df76e0)
---Type <return> to continue, or q <return> to quit--- 
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:620
#24 0x00007fffe98c59b7 in mozilla::net::CacheEntry::Open (
    this=this@entry=0x7fffa1df76e0, aCallback=..., 
    aTruncate=aTruncate@entry=false, aPriority=aPriority@entry=false, 
    aBypassIfBusy=aBypassIfBusy@entry=false)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:338
#25 0x00007fffe98c5abf in mozilla::net::CacheEntry::AsyncOpen (
    this=0x7fffa1df76e0, aCallback=aCallback@entry=0x7fff990e0aa8, 
    aFlags=aFlags@entry=2)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:312
#26 0x00007fffe98d411c in mozilla::net::CacheStorage::AsyncOpenURI (
    this=0x7fffc0fd15e0, aURI=<optimized out>, aIdExtension=..., aFlags=2, 
    aCallback=0x7fff990e0aa8)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheStorage.cpp:112
#27 0x00007fffe9531316 in nsImapMockChannel::OpenCacheEntry (
    this=this@entry=0x7fff990e0aa0)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9287
#28 0x00007fffe953177a in nsImapMockChannel::AsyncOpen (this=0x7fff990e0aa0, 
    listener=<optimized out>, ctxt=0x7fffb49a82f8)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9610
#29 0x00007fffe954ad46 in nsImapService::GetMessageFromUrl (
    this=this@entry=0x7fffd3d61510, aImapUrl=<optimized out>, 
    aImapAction=aImapAction@entry=268435510, aImapMailFolder=<optimized out>, 
    aImapMessage=<optimized out>, aMsgWindow=aMsgWindow@entry=0x7fffc7f78ba0, 
    aDisplayConsumer=0x7fffbeaa3000, aConvertDataToText=false, 
    aURL=0x7fffffffa8d8)
    at /opt/comm-central/src/mailnews/imap/src/nsImapService.cpp:1109
#30 0x00007fffe954b4d9 in nsImapService::StreamMessage (this=0x7fffd3d61510, 
    aMessageURI=<optimized out>, aConsumer=0x7fffbeaa3000, 
    aMsgWindow=0x7fffc7f78ba0, aUrlListener=<optimized out>, 
    aConvertData=<optimized out>, aAdditionalHeader=..., aLocalOnly=true, 
    aURL=0x7fffffffa8d8)
    at /opt/comm-central/src/mailnews/imap/src/nsImapService.cpp:1210
#31 0x00007fffe941fad3 in nsOfflineStoreCompactState::CopyNextMessage (
    this=this@entry=0x7fffbeaa3000, done=@0x7fffffffa953: false)
    at /opt/comm-central/src/mailnews/base/src/nsMsgFolderCompactor.cpp:1037
#32 0x00007fffe941fd60 in nsOfflineStoreCompactState::OnStopRequest (
    this=0x7fffbeaa3000, request=<optimized out>, ctxt=<optimized out>, 
    status=<optimized out>)
    at /opt/comm-central/src/mailnews/base/src/nsMsgFolderCompactor.cpp:1104
#33 0x00007fffe9530a1b in nsImapMockChannel::ReadFromImapConnection (
    this=this@entry=0x7fff990e09d0)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9472
#34 0x00007fffe9532b47 in nsImapMockChannel::OnCacheEntryAvailable (
---Type <return> to continue, or q <return> to quit--- 
    this=0x7fff990e09d0, entry=0x0, aNew=<optimized out>, 
    aAppCache=<optimized out>, status=<optimized out>)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9170
#35 0x00007fffe98c4d0a in mozilla::net::CacheEntry::InvokeAvailableCallback (
    this=this@entry=0x7fffa3874b40, aCallback=...)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:875
#36 0x00007fffe98c51bb in mozilla::net::CacheEntry::InvokeCallback (
    this=this@entry=0x7fffa3874b40, aCallback=...)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:812
#37 0x00007fffe98c3be7 in mozilla::net::CacheEntry::InvokeCallbacks (
    this=this@entry=0x7fffa3874b40, aReadOnly=aReadOnly@entry=true)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:678
#38 0x00007fffe98c3d01 in mozilla::net::CacheEntry::InvokeCallbacks (
    this=0x7fffa3874b40)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:620
#39 0x00007fffe98c59b7 in mozilla::net::CacheEntry::Open (
    this=this@entry=0x7fffa3874b40, aCallback=..., 
    aTruncate=aTruncate@entry=false, aPriority=aPriority@entry=false, 
    aBypassIfBusy=aBypassIfBusy@entry=false)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:338
#40 0x00007fffe98c5abf in mozilla::net::CacheEntry::AsyncOpen (
    this=0x7fffa3874b40, aCallback=aCallback@entry=0x7fff990e09d8, 
    aFlags=aFlags@entry=2)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:312
#41 0x00007fffe98d411c in mozilla::net::CacheStorage::AsyncOpenURI (
    this=0x7fffc0fd15e0, aURI=<optimized out>, aIdExtension=..., aFlags=2, 
    aCallback=0x7fff990e09d8)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheStorage.cpp:112
#42 0x00007fffe9531316 in nsImapMockChannel::OpenCacheEntry (
    this=this@entry=0x7fff990e09d0)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9287
#43 0x00007fffe953177a in nsImapMockChannel::AsyncOpen (this=0x7fff990e09d0, 
    listener=<optimized out>, ctxt=0x7fffb49a5478)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9610
#44 0x00007fffe954ad46 in nsImapService::GetMessageFromUrl (
    this=this@entry=0x7fffd3d61510, aImapUrl=<optimized out>, 
    aImapAction=aImapAction@entry=268435510, aImapMailFolder=<optimized out>, 
    aImapMessage=<optimized out>, aMsgWindow=aMsgWindow@entry=0x7fffc7f78ba0, 
    aDisplayConsumer=0x7fffbeaa3000, aConvertDataToText=false, 
    aURL=0x7fffffffb678)
    at /opt/comm-central/src/mailnews/imap/src/nsImapService.cpp:1109
#45 0x00007fffe954b4d9 in nsImapService::StreamMessage (this=0x7fffd3d61510, 
    aMessageURI=<optimized out>, aConsumer=0x7fffbeaa3000, 
    aMsgWindow=0x7fffc7f78ba0, aUrlListener=<optimized out>, 
---Type <return> to continue, or q <return> to quit---    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:620
    aConvertData=<optimized out>, aAdditionalHeader=..., aLocalOnly=true, 
    aURL=0x7fffffffb678)
    at /opt/comm-central/src/mailnews/imap/src/nsImapService.cpp:1210
#46 0x00007fffe941fad3 in nsOfflineStoreCompactState::CopyNextMessage (
    this=this@entry=0x7fffbeaa3000, done=@0x7fffffffb6f3: false)
    at /opt/comm-central/src/mailnews/base/src/nsMsgFolderCompactor.cpp:1037
#47 0x00007fffe941fd60 in nsOfflineStoreCompactState::OnStopRequest (
    this=0x7fffbeaa3000, request=<optimized out>, ctxt=<optimized out>, 
    status=<optimized out>)
    at /opt/comm-central/src/mailnews/base/src/nsMsgFolderCompactor.cpp:1104
#48 0x00007fffe9530a1b in nsImapMockChannel::ReadFromImapConnection (
    this=this@entry=0x7fff990e0900)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9472
#49 0x00007fffe9532b47 in nsImapMockChannel::OnCacheEntryAvailable (
    this=0x7fff990e0900, entry=0x0, aNew=<optimized out>, 
    aAppCache=<optimized out>, status=<optimized out>)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9170
#50 0x00007fffe98c4d0a in mozilla::net::CacheEntry::InvokeAvailableCallback (
    this=this@entry=0x7fffa1df9f20, aCallback=...)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:875
#51 0x00007fffe98c51bb in mozilla::net::CacheEntry::InvokeCallback (
    this=this@entry=0x7fffa1df9f20, aCallback=...)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:812
#52 0x00007fffe98c3be7 in mozilla::net::CacheEntry::InvokeCallbacks (
    this=this@entry=0x7fffa1df9f20, aReadOnly=aReadOnly@entry=true)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:678
#53 0x00007fffe98c3d01 in mozilla::net::CacheEntry::InvokeCallbacks (
    this=0x7fffa1df9f20)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:620
#54 0x00007fffe98c59b7 in mozilla::net::CacheEntry::Open (
    this=this@entry=0x7fffa1df9f20, aCallback=..., 
    aTruncate=aTruncate@entry=false, aPriority=aPriority@entry=false, 
    aBypassIfBusy=aBypassIfBusy@entry=false)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:338
#55 0x00007fffe98c5abf in mozilla::net::CacheEntry::AsyncOpen (
    this=0x7fffa1df9f20, aCallback=aCallback@entry=0x7fff990e0908, 
    aFlags=aFlags@entry=2)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheEntry.cpp:312
#56 0x00007fffe98d411c in mozilla::net::CacheStorage::AsyncOpenURI (
    this=0x7fffc0fd15e0, aURI=<optimized out>, aIdExtension=..., aFlags=2, 
    aCallback=0x7fff990e0908)
    at /opt/comm-central/src/mozilla/netwerk/cache2/CacheStorage.cpp:112
#57 0x00007fffe9531316 in nsImapMockChannel::OpenCacheEntry (
    this=this@entry=0x7fff990e0900)
---Type <return> to continue, or q <return> to quit---#24 0x00007fffe98c59b7 in mozilla::net::CacheEntry::Open (
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9287
#58 0x00007fffe953177a in nsImapMockChannel::AsyncOpen (this=0x7fff990e0900, 
    listener=<optimized out>, ctxt=0x7fffa68f9ea8)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:9610
#59 0x00007fffe954ad46 in nsImapService::GetMessageFromUrl (
    this=this@entry=0x7fffd3d61510, aImapUrl=<optimized out>, 
    aImapAction=aImapAction@entry=268435510, aImapMailFolder=<optimized out>, 
    aImapMessage=<optimized out>, aMsgWindow=aMsgWindow@entry=0x7fffc7f78ba0, 
    aDisplayConsumer=0x7fffbeaa3000, aConvertDataToText=false, 
    aURL=0x7fffffffc418)
    at /opt/comm-central/src/mailnews/imap/src/nsImapService.cpp:1109
#60 0x00007fffe954b4d9 in nsImapService::StreamMessage (this=0x7fffd3d61510, 
    aMessageURI=<optimized out>, aConsumer=0x7fffbeaa3000, 
    aMsgWindow=0x7fffc7f78ba0, aUrlListener=<optimized out>, 
    aConvertData=<optimized out>, aAdditionalHeader=..., aLocalOnly=true, 
    aURL=0x7fffffffc418)
    at /opt/comm-central/src/mailnews/imap/src/nsImapService.cpp:1210
#61 0x00007fffe941fad3 in nsOfflineStoreCompactState::CopyNextMessage (
    this=this@entry=0x7fffbeaa3000, done=@0x7fffffffc493: false)
    at /opt/comm-central/src/mailnews/base/src/nsMsgFolderCompactor.cpp:1037
#62 0x00007fffe941fd60 in nsOfflineStoreCompactState::OnStopRequest (
    this=0x7fffbeaa3000, request=<optimized out>, ctxt=<optimized out>, 
    status=<optimized out>)
    at /opt/comm-central/src/mailnews/base/src/nsMsgFolderCompactor.cpp:1104
#63 0x00007fffe952c9a2 in nsImapCacheStreamListener::OnStopRequest (
    this=0x7fffa77c6a00, request=<optimized out>, aCtxt=<optimized out>, 
    aStatus=nsresult::NS_OK)
    at /opt/comm-central/src/mailnews/imap/src/nsImapProtocol.cpp:8810
#64 0x00007fffe96f769a in nsInputStreamPump::OnStateStop (this=0x7fffa8e140d0)
    at /opt/comm-central/src/mozilla/netwerk/base/nsInputStreamPump.cpp:715
#65 0x00007fffe96fca96 in nsInputStreamPump::OnInputStreamReady (
    this=0x7fffa8e140d0, stream=<optimized out>)
    at /opt/comm-central/src/mozilla/netwerk/base/nsInputStreamPump.cpp:433
#66 0x00007fffe967bd2b in nsInputStreamReadyEvent::Run (this=0x7fffa3cdcdd0)
    at /opt/comm-central/src/mozilla/xpcom/io/nsStreamUtils.cpp:96
#67 0x00007fffe96936f3 in nsThread::ProcessNextEvent (this=0x7ffff6b51250, 
    aMayWait=<optimized out>, aResult=0x7fffffffc797)
    at /opt/comm-central/src/mozilla/xpcom/threads/nsThread.cpp:1269
#68 0x00007fffe9693f7a in NS_ProcessNextEvent (aThread=<optimized out>, 
    aThread@entry=0x7ffff6b51250, aMayWait=aMayWait@entry=false)
    at /opt/comm-central/src/mozilla/xpcom/threads/nsThreadUtils.cpp:389
#69 0x00007fffe9a266a1 in mozilla::ipc::MessagePump::Run (this=0x7fffe64f3040, 
    aDelegate=0x7ffff6b6c260)
    at /opt/comm-central/src/mozilla/ipc/glue/MessagePump.cpp:96
---Type <return> to continue, or q <return> to quit---    this=this@entry=0x7fffa1df76e0, aCallback=..., 
#70 0x00007fffe99e986c in MessageLoop::RunHandler (this=<optimized out>)
    at /opt/comm-central/src/mozilla/ipc/chromium/src/base/message_loop.cc:231
#71 MessageLoop::Run (this=<optimized out>)
    at /opt/comm-central/src/mozilla/ipc/chromium/src/base/message_loop.cc:211
#72 0x00007fffeacd2e11 in nsBaseAppShell::Run (this=0x7fffdde54560)
    at /opt/comm-central/src/mozilla/widget/nsBaseAppShell.cpp:156
#73 0x00007fffeb94f0ea in nsAppStartup::Run (this=0x7fffdde2e3d0)
    at /opt/comm-central/src/mozilla/toolkit/components/startup/nsAppStartup.cpp:283
#74 0x00007fffeb9b5906 in XREMain::XRE_mainRun (this=this@entry=0x7fffffffcac0)
    at /opt/comm-central/src/mozilla/toolkit/xre/nsAppRunner.cpp:4492
#75 0x00007fffeb9b5f10 in XREMain::XRE_main (this=this@entry=0x7fffffffcac0, 
    argc=argc@entry=1, argv=argv@entry=0x7fffffffddf8, aConfig=...)
    at /opt/comm-central/src/mozilla/toolkit/xre/nsAppRunner.cpp:4670
#76 0x00007fffeb9b61a3 in XRE_main (argc=1, argv=0x7fffffffddf8, aConfig=...)
    at /opt/comm-central/src/mozilla/toolkit/xre/nsAppRunner.cpp:4761
#77 0x0000000000405d8d in do_main (argc=argc@entry=1, 
    argv=argv@entry=0x7fffffffddf8, envp=envp@entry=0x7fffffffde08)
    at /opt/comm-central/src/mail/app/nsMailApp.cpp:237
#78 0x0000000000405717 in main (argc=1, argv=0x7fffffffddf8, 
    envp=0x7fffffffde08) at /opt/comm-central/src/mail/app/nsMailApp.cpp:308

Comment 30

[Wayne Mery (:wsmwk)]

magnus, 
did you have good results?

Comment 31

[Magnus Melin [:mkmelin]]

Very annoying bug, but unfortunately no :(
 
My hunch is it's not the all regular "stuck in a loop" crash. It's doing a lot of stuff for many messages in a row. It's possible there is some missing "end" event that isn't being called so it consumes resources until it hits the roof. Maybe.

Comment 32

[Wayne Mery (:wsmwk)]

IT is good that you are crashing :)
Some of your other crashes

 nsOfflineStoreCompactState::CopyNextMessage  bp-3b89c399-08c2-44ff-aff3-a641d2170412
0 	xul.dll	nsOfflineStoreCompactState::CopyNextMessage(bool&)	C:/builds/moz2_slave/tb-rel-c-esr52-w32_bld-0000000/build/mailnews/base/src/nsMsgFolderCompactor.cpp:1052
1 	xul.dll	nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, nsresult)	C:/builds/moz2_slave/tb-rel-c-esr52-w32_bld-0000000/build/mailnews/base/src/nsMsgFolderCompactor.cpp:1104
2 	xul.dll	nsImapCacheStreamListener::OnStopRequest(nsIRequest*, nsISupports*, nsresult)	C:/builds/moz2_slave/tb-rel-c-esr52-w32_bld-0000000/build/mailnews/imap/src/nsImapProtocol.cpp:8741
3 	xul.dll	nsInputStreamPump::OnStateStop()	netwerk/base/nsInputStreamPump.cpp:714
4 	xul.dll	nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*)	netwerk/base/nsInputStreamPump.cpp:433 

 nsOfflineStoreCompactState::OnStopRequest bp-9da6e670-2966-45fb-90d5-abfdc2170327
 0 	libxul.so	nsOfflineStoreCompactState::OnStopRequest	/builds/slave/tb-c-cen-l64-ntly-000000000000/build/mailnews/base/src/nsMsgFolderCompactor.cpp:1107
1 	libxul.so	nsImapCacheStreamListener::OnStopRequest	/builds/slave/tb-c-cen-l64-ntly-000000000000/build/mailnews/imap/src/nsImapProtocol.cpp:8810
2 	libxul.so	nsInputStreamPump::OnStateStop	netwerk/base/nsInputStreamPump.cpp:715 


Whatever the cause, this is much worse in version 52 and severe enough to block unthrottling updates to all user. Please set the tracking flag

Comment 33

[Jorg K (CEST = GMT+2)]

Let's track this bug here instead of bug 1316416.

Repeating bug 1316416 comment #0:
This is #1 crash for 51.0a2, perhaps beginning an upswing in early September. Fairly rare prior to that. Also rare on c-c, with no crashes since 2016-10-27.  After the merge next week, need to evaluate a) whether the crashing on aurora stops, b) whether the crashing appears in beta.

If it received an upswing in September 2016 that's because cache2 (bug 1021843) landed 2016-08-30.

But the problem was certainly pre-existing as for example comment #4 or comment #12 from 2015 show.

(In reply to Magnus Melin from comment #16)
> Wonder if this got worse now that we we switched away from the old backend.

Looks like it. Comment #25 to comment #27 are a good read.

Comment 34

[Jorg K (CEST = GMT+2)]

I fixed bug 1355350 (pending review), so I have some cycles for this one.

What are the STR?
- IMAP folder (sync for offline or not? Apparently not, since only non-sync folders use cache).
- Delete a whole lot of messages.
- Compact, crash, right?

Tried that, didn't crash. So let me know.

Comment 35

[Magnus Melin [:mkmelin]]

I don't know the exact cause. For me it's my mozilla folder, which is IMAP and synced for offline. I think the amount of messages plays in, mine is around 6000 messages. (Using mark-as-deleted if that possibly could make any difference.)

Then delete some messages, compact the folder, wait for a few seconds. The actual expunge may or may not have the time to succeed on the server. After a few seconds it will just freeze, and then there's a crash in maybe 10-15 seconds.

Comment 36

[Andriy Gapon]

(In reply to Jorg K (GMT+2) from comment #34)
I have approximately the same configuration as in comment #35:
- IMAP folder
- selected for offline use
- deleted messages are marked as deleted
- about 1000 messages in the folder

For me, the crash does not happen every time.
It happens in about 10% of cases.

Comment 37

[Jan Horak [:jhorak]]

I guess the bug 1358140 is dupe of this one. Please have a look in comment 1 what I've been trying to find out, if that's any help.

Comment 38

[Jorg K (CEST = GMT+2)]

Thanks for the hint. The thing I don't understand is this: All the reporters say their folder is "selected for offline use" or "synced for offline". How on earth is it possible that the code goes through caching, see nsImapMockChannel::OnCacheEntryAvailable() followed by the involvement of the nsStreamListenerTee, which is supposed to write to the channel and the cache at the same time:
https://dxr.mozilla.org/comm-central/rev/5e4e889f13eb1fd5091f60921a1566c661f2c630/mailnews/imap/src/nsImapProtocol.cpp#9165

Further up in the stack: nsImapMockChannel::AsyncOpen() and nsImapMockChannel::OpenCacheEntry().

As far as I know, when the message is stored in a local folder for offline use, there is no caching, well, the local folder is the cache in this case. What am I missing?

Comment 40

[Matti Aarnio]

This is repeating reliably for me.
I have 2 separate Fedora 26 Linux machines (home, office), and both behave the same way.

This appeared when Fedora 25 upgraded from Thunderbird 45.x to Thunderbird-52.0.
Then I upgraded to Fedora 26, and the issue did not go away.
Lastly I tried currently available Thunderbird binary from mozilla.org at one of these machines, and the same thing happened.

About this IMAP INBOX:
 - Highest message id 85933
 - about 20 000 messages in the box (says mutt)


I had folder setting enabling offline use.
When I disabled that, no crashing, INBOX compacting is quick.

Comment 41

[Matti Aarnio]

Additional note: It was infinite recursion that killed my Thunderbird.
....
#43322 0x00007fffe76e34fd in  () at ./thunderbird/libxul.so
#43323 0x00007fffe786c64b in  () at ./thunderbird/libxul.so
#43324 0x00007fffe786f90a in  () at ./thunderbird/libxul.so
#43325 0x00007fffe77fdb60 in  () at ./thunderbird/libxul.so
#43326 0x00007fffe7810b42 in  () at ./thunderbird/libxul.so
#43327 0x00007fffe782839a in  () at ./thunderbird/libxul.so
#43328 0x00007fffe7ac44ab in  () at ./thunderbird/libxul.so
#43329 0x00007fffe7ab0175 in  () at ./thunderbird/libxul.so
#43330 0x00007fffe8a4703b in  () at ./thunderbird/libxul.so
#43331 0x00007fffe9025e36 in  () at ./thunderbird/libxul.so
#43332 0x00007fffe9070ba8 in  () at ./thunderbird/libxul.so
#43333 0x00007fffe9070e56 in  () at ./thunderbird/libxul.so
#43334 0x00007fffe90710d7 in XRE_main () at ./thunderbird/libxul.so
#43335 0x0000000000404d3c in _start ()
(gdb)

Comment 42

[Mason]

Bug reporter speaking. I am currently using
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49
and I have not had any mail-related crashes recently, other than bug 1323688.

Did TB and SM code diverge?

Comment 43

[Marcel Kneuer]

I am getting Crashes while compacting mails very often (since months). Now I can't compact anymore, because every time TB crashes without (before) compacting. I have Mozilla/5.0 (Windows NT 10.0; WOW64; rv:54.0) Gecko/20100101 Thunderbird/54.0a2

Here are my (compacting) crash reports of today:
https://crash-stats.mozilla.com/report/index/7171a24b-0e11-4221-915a-21fbc0170421
https://crash-stats.mozilla.com/report/index/7171a24b-0e11-4221-915a-21fbc0170421

Now I have repaired the folder, because I know that this helps for some days, until the crashes will start again.

Comment 44

[Marcel Kneuer]

And here is a crash report from my mac (same inbox as in Comment 43), related to this bug: https://crash-stats.mozilla.com/report/index/6500ed3f-86fb-4736-bf78-9412a0170414 . Mostly (as today) I get no crash information, but it is the same problem: https://crash-stats.mozilla.com/report/index/fec71d89-049c-47ed-8d60-7ff2e0170421

Comment 46

[Wayne Mery (:wsmwk)]

js::jit::InlineFrameIterator::::resetOn is #1 crash for nightly

Comment 47

[Jorg K (CEST = GMT+2)]

Looking at one of the crashes, here is what happens:

nsOfflineStoreCompactState::OnStopRequest() is done and decides to move onto the next message:
https://dxr.mozilla.org/comm-central/rev/5e4e889f13eb1fd5091f60921a1566c661f2c630/mailnews/base/src/nsMsgFolderCompactor.cpp#1129

CopyNextMessage()
https://dxr.mozilla.org/comm-central/rev/5e4e889f13eb1fd5091f60921a1566c661f2c630/mailnews/base/src/nsMsgFolderCompactor.cpp#1061

calls SteramMessage()
https://dxr.mozilla.org/comm-central/rev/5e4e889f13eb1fd5091f60921a1566c661f2c630/mailnews/imap/src/nsImapService.cpp#1210

And if the message is not stored locally, it will be retrieved from the server. And then it's downhill from there.

Needs some more digging, but something seems fishy. Why fetch a message from the server when the task is to compact (not download!) the folder?

Could the people experiencing the crash try to switch to off-line before compacting. That should/may stop those downloads which are failing.

It also looks like something is wrong with the listeners. I don't see why nsStreamListenerTee::OnStopRequest(), involved in filling the cache, should call nsOfflineStoreCompactState::OnStopRequest(), which causes the endless recursion.

Comment 48

[Jacques Amar]

> 
> Could the people experiencing the crash try to switch to off-line before
> compacting. That should/may stop those downloads which are failing.
> 

This is a crash on IMAP - inherently online - and technically server based. Going offline defeats the purpose of compacting.

Comment 49

[Peter Peltonen]

(In reply to Jacques Amar from comment #48)
> > 
> > Could the people experiencing the crash try to switch to off-line before
> > compacting. That should/may stop those downloads which are failing.
> > 
> 
> This is a crash on IMAP - inherently online - and technically server based.
> Going offline defeats the purpose of compacting.

I assume what was meant with "offline" was offline syncing. For me this solved the issue:

Right click inbox, select "synchronization" tab and deselect "Select his folder for offline use"

=> after this compacting worked

If I reeneable offline use and try compacting again, the problem comes back again.

Comment 50

[Jacques Amar]

(In reply to Peter Peltonen from comment #49)
> (In reply to Jacques Amar from comment #48)
> > > 
> > > Could the people experiencing the crash try to switch to off-line before
> > > compacting. That should/may stop those downloads which are failing.
> > > 
> > 
> > This is a crash on IMAP - inherently online - and technically server based.
> > Going offline defeats the purpose of compacting.
> 
> I assume what was meant with "offline" was offline syncing. For me this
> solved the issue:
> 
> Right click inbox, select "synchronization" tab and deselect "Select his
> folder for offline use"
> 
> => after this compacting worked
> 
> If I reeneable offline use and try compacting again, the problem comes back
> again.

Yep (and duh!). Sync off works as advertised here too.

Comment 51

[Jorg K (CEST = GMT+2)]

Attachment: 1158578-crash-100-percent.patch

Following my analysis in comment #47, this patch makes the compact crash 100% in the endless recursion that we are seeing.

The problem is, as described, that the message that is requested by the compaction is not in the local synced folder, a strange condition, that seems to occur at times, I don't know how.

In that case, IMAP tries to fetch the message from the server and that goes into endless recursion.

So now that we have a 100% reliable test case without having to shuffle data around, the fix shouldn't be far off.

To answer the previous questions: I meant: Go offline, pull out the LAN cable, disconnect the WiFi. This way the compaction cannot get the message from the server and there won't be a crash. But I noticed that compaction is not available in offline mode.

Comment 52

[Marcel Kneuer]

I think the real problem that produces this bug is that messages are not downloaded correctly. I have selected that messages always should be downloaded for offline use, so there is no reason why some messages shouldn't be there for offline use. It also makes clear why it works after rebuilding the index, because all messages are downloaded again at once and this works.

So maybe to fix this bug it is better to fix the download problem/errors.

Comment 53

[Jorg K (CEST = GMT+2)]

Summarising my findings for today:
The compactor calls nsImapService::StreamMessage() asking for a *local* message, either from the local sync'ed folder or the memory cache. The local folder says that it has the message, but when we come to retrieve it, it can't be retrieved and we go onto fetching it from the server and caching it locally in memory.

Simulating retrieval failure from the local sync'ed folder gives me a plethora of crashes.

Sadly the initial function StreamMessage() doesn't pass on the information that only a *local* message is required. By the time the local retrieval fails, this information isn't available any more and an attempt is made to fetch from the server.

If I read comment #52 correctly, "rebuilding the index" (how? - Repair folder?) and downloading again will avoid the problem since all local messages are really retrievable locally.

To be continued. I think, erroneously fetching from the server is not so bad as long as it doesn't crash ;-)

Comment 54

[Jan Horak [:jhorak]]

(In reply to Jorg K (GMT+2) from comment #53)
> Summarising my findings for today:
> The compactor calls nsImapService::StreamMessage() asking for a *local*
> message, either from the local sync'ed folder or the memory cache. The local
> folder says that it has the message, but when we come to retrieve it, it
> can't be retrieved and we go onto fetching it from the server and caching it
> locally in memory.
> 
> Simulating retrieval failure from the local sync'ed folder gives me a
> plethora of crashes.
> 
> Sadly the initial function StreamMessage() doesn't pass on the information
> that only a *local* message is required. By the time the local retrieval
> fails, this information isn't available any more and an attempt is made to
> fetch from the server.
Well, not completely, from backtrace reported by users I see the 
nsImapMockChannel::ReadFromImapConnection does not try to download message, because 
imapUrl->GetLocalFetchOnly(&localOnly) return localOnly=true, therefore it returns 
NS_MSG_ERROR_MSG_NOT_OFFLINE [1].

The NS_MSG_ERROR_MSG_NOT_OFFLINE is propagated to the nsOfflineStoreCompactState::OnStopRequest [2] (see the interesting comment [3] and the OnStopRequest later calls 'rv = CopyNextMessage(done);' [4].
Why is CopyNextMessage called from there? This cause the recursion in case the message failed to download because the previous call of CopyNextMessage is still on the stack.
The CopyNextMessage [5] should already process all messages in processed folder by using while loop, shouldn't it?

[1] https://dxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapProtocol.cpp#9492
[2] https://dxr.mozilla.org/comm-central/source/mailnews/base/src/nsMsgFolderCompactor.cpp#1086
[3] https://dxr.mozilla.org/comm-central/source/mailnews/base/src/nsMsgFolderCompactor.cpp#1095
[4] https://dxr.mozilla.org/comm-central/source/mailnews/base/src/nsMsgFolderCompactor.cpp#1129
[5] https://dxr.mozilla.org/comm-central/source/mailnews/base/src/nsMsgFolderCompactor.cpp?q=CopyNextMessage&redirect_type=direct#1031

Comment 55

[Jorg K (CEST = GMT+2)]

Jan, thanks for the comment. At least my poking around and semi-correct findings are causing a discussion now ;-) Glad I gave up at midnight to make space for someone with an analytical eye.

Would you like to propose a patch? Looks like we're getting closer.

Comment 56

[Jan Horak [:jhorak]]

Attachment: reproduction-patch-read-from-local-cache-fails.patch

This is the patch which will produce endless recursion simulating the case when the ReadFromLocalCache() fails.

Comment 57

[Jorg K (CEST = GMT+2)]

Umm, I attached pretty much the same earlier. Now, how do we fix it?

Comment 58

[Jorg K (CEST = GMT+2)]

Looking into the code a bit more:
CopyNextMessage() is called twice, once in nsOfflineStoreCompactState::StartCompacting() and the second time in nsOfflineStoreCompactState::OnStopRequest() to kick off the next message.

The while loop inside CopyNextMessage() breaks after successfully kicking off the streaming of a message,
https://dxr.mozilla.org/comm-central/rev/5e4e889f13eb1fd5091f60921a1566c661f2c630/mailnews/base/src/nsMsgFolderCompactor.cpp#1077
so moving on to the next message is done in the OnStopRequest().

The while loop *cannot* process more than one message, since the streaming is asynchronous. So it kicks off one message and exists the function, to be called again when that message is done. There is not even any recursion here, since OnStopRequest() only kicks off the next message before returning and completing the async thread.

I didn't write that code, but I guess this answers your question.

That said, I still don't see how we get into the recursive situation. When if NS_MSG_ERROR_MSG_NOT_OFFLINE is passed to OnStopRequest(), 'm_curIndex' is increased, so we move to the next message.

Comment 59

[Jan Horak [:jhorak]]

(In reply to Jorg K (GMT+2) from comment #58)
> Looking into the code a bit more:
> CopyNextMessage() is called twice, once in
> nsOfflineStoreCompactState::StartCompacting() and the second time in
> nsOfflineStoreCompactState::OnStopRequest() to kick off the next message.
> 
> The while loop inside CopyNextMessage() breaks after successfully kicking
> off the streaming of a message,
> https://dxr.mozilla.org/comm-central/rev/
> 5e4e889f13eb1fd5091f60921a1566c661f2c630/mailnews/base/src/
> nsMsgFolderCompactor.cpp#1077
> so moving on to the next message is done in the OnStopRequest().
> 
> The while loop *cannot* process more than one message, since the streaming
> is asynchronous. So it kicks off one message and exists the function, to be
> called again when that message is done. There is not even any recursion
> here, since OnStopRequest() only kicks off the next message before returning
> and completing the async thread.
> 
> I didn't write that code, but I guess this answers your question.
> 
> That said, I still don't see how we get into the recursive situation. When
> if NS_MSG_ERROR_MSG_NOT_OFFLINE is passed to OnStopRequest(), 'm_curIndex'
> is increased, so we move to the next message.
Yes we move to next message, but we still have previous call of CopyNextMessage on stack while calling another CopyNextMessage().

I think the problem lies there: https://dxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapProtocol.cpp?q=%2Bfunction%3AnsImapMockChannel%3A%3AReadFromImapConnection%28%29&redirect_type=single#9499
The OnStopRequest is called immediately there while the CopyNextMessage is still on stack. This normally wouldn't happened because if the cache hits, the result arrives asynchronously and processed by OnStopRequest while we're already out of CopyNextMessage.

Comment 60

[Jorg K (CEST = GMT+2)]

I don't think I agree. nsImapMockChannel::OpenCacheEntry() calls AsyncOpenURI() and that asynchronously transfers control to OnCacheEntryAvailable() which calls ReadFromImapConnection() which then calls OnStopRequest(). At the very latest CopyNextMessage() should have returned after OpenCacheEntry() returned. What am I missing?

I must be missing something since the crash dumps published here all seem to show endless recursions. I noticed that a few times the recursion actually stops in nsMsgMailSession::OnItemPropertyFlagChanged() just before the crash:
https://crash-stats.mozilla.com/report/index/bc525551-ed76-41a3-a8bb-5ffb52170330 from comment #26.

Comment 61

[Andriy Gapon]

(In reply to Jorg K (GMT+2) from comment #60)
> I must be missing something since the crash dumps published here all seem to
> show endless recursions.

In my case, originally reported in bug #1352334, there was no recursion.
Also, I would like to confirm that doing "Repair Folder" stops the crashes for me, at least for a while.

Comment 62

[Jan Horak [:jhorak]]

(In reply to Jorg K (GMT+2) from comment #60)
> I don't think I agree. nsImapMockChannel::OpenCacheEntry() calls
> AsyncOpenURI() and that asynchronously transfers control to
> OnCacheEntryAvailable() which calls ReadFromImapConnection() which then
> calls OnStopRequest(). At the very latest CopyNextMessage() should have
> returned after OpenCacheEntry() returned. What am I missing?
Um...you're right, it's more complicated :). Looks like CacheEntry::Open (called by CacheEntry::AsyncOpen) "fails" because Load has a cache miss - Load returns true only when in LOADING state [1] and instead of returning the Open calls InvokeCallbacks. And this later leads to another CopyNextMessage by this backtrace:
#0  nsOfflineStoreCompactState::CopyNextMessage(bool&) (this=, done=false)
#1  nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, nsresult)
#2  mozilla::net::nsStreamListenerTee::OnStopRequest(nsIRequest*, nsISupports*, nsresult)
#3  nsImapMockChannel::ReadFromImapConnection() (this=) 
#4  nsImapMockChannel::OnCacheEntryAvailable(nsICacheEntry*, bool, nsIApplicationCache*, nsresult) 
#5  mozilla::net::CacheEntry::InvokeAvailableCallback(mozilla::net::CacheEntry::Callback const&) 
#6  mozilla::net::CacheEntry::InvokeCallback(mozilla::net::CacheEntry::Callback&) (this=, aCallback=...) 
#7  mozilla::net::CacheEntry::InvokeCallbacks(bool) (this=, aReadOnly=false) 
#8  mozilla::net::CacheEntry::InvokeCallbacks() (this=) 
#9  mozilla::net::CacheEntry::Open(mozilla::net::CacheEntry::Callback&, bool, bool, bool) 
#10 mozilla::net::CacheEntry::AsyncOpen(nsICacheEntryOpenCallback*, unsigned int)

So callbacks are no longer called async. Suprisingly it's probably bad idea to expect that async calls always ends as a new event.
[1] http://searchfox.org/mozilla-central/source/netwerk/cache2/CacheEntry.cpp#448


I've been able to fix/workaround this by:
1. remove direct call of m_channelListener->OnStopRequestcreating from nsImapMockChannel::ReadFromImapConnection
2. add a runnable class which is dispatched to the current thread which do the m_channelListener->OnStopRequestcreating call.
See

Btw this is how the backtrace should look like for subsequent calls of CopyNextMessage when the message is available offline:
#0  nsOfflineStoreCompactState::CopyNextMessage(bool&) (this=0x7fff96d34000, done=@0x7fffffffbe3f: false)
#1  nsOfflineStoreCompactState::OnStopRequest(nsIRequest*, nsISupports*, nsresult) 
#2  nsImapCacheStreamListener::OnStopRequest(nsIRequest*, nsISupports*, nsresult)
#3  nsInputStreamPump::OnStateStop() (this=0x7fff7ece0ec0)
#4  nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*)
#5  nsInputStreamReadyEvent::Run() (this=0x7fff805fa6f0)
#6  nsThread::ProcessNextEvent(bool, bool*) (this=0x7ffff6bbd420, aMayWait=false, aResult=0x7fffffffc1df)
#7  NS_ProcessNextEvent(nsIThread*, bool) (aThread=0x7ffff6bbd420, aMayWait=false)

(backtrace are shortened for less verbose/fitting the message, I can attach full bt if you want).

> I must be missing something since the crash dumps published here all seem to
> show endless recursions. I noticed that a few times the recursion actually
> stops in nsMsgMailSession::OnItemPropertyFlagChanged() just before the crash:
> https://crash-stats.mozilla.com/report/index/bc525551-ed76-41a3-a8bb-
> 5ffb52170330 from comment #26.
Hm, I don't think it actually stopped, only the last processed message was available as offline and during that execution the stack overflow happened (see the frame number at the end of report).

Comment 63

[Jan Horak [:jhorak]]

Attachment: rather workaroung patch to call nsOfflineStoreCompactState::OnStopRequest from runnable

That's the patch relevant to previous comment.

Comment 64

[Jorg K (CEST = GMT+2)]

Comment on attachment 8861834 [details] [diff] [review]
rather workaroung patch to call nsOfflineStoreCompactState::OnStopRequest from runnable

Review of attachment 8861834 [details] [diff] [review]:
-----------------------------------------------------------------

Nice job! Let me do a try build on Windows for people to test.

::: mailnews/imap/src/nsImapProtocol.cpp
@@ +9505,4 @@
>    return rv;
>  }
>  
> +class nsReadFromImapConnectionFailure : public mozilla::Runnable

How about some more comments here.

@@ +9522,5 @@
> +  RefPtr<nsImapMockChannel> mImapMockChannel;
> +};
> +
> +
> +nsresult nsImapMockChannel::RunListeners()

I suggest to change the name here, maybe:
RunOnStopRequestFailure() ?

@@ +9679,4 @@
>        || imapAction == nsIImapUrl::nsImapMsgFetchPeek))
>        return NS_ERROR_FAILURE; // abort the running of this url....it failed a security check
>    }
> +  /*TEMPORARILY DONT DO THAT TO REPRODUCE PROBLEM

I'll compile a try build with this removed.

Comment 66

[Jorg K (CEST = GMT+2)]

Attachment: recursive-fix.patch - Jan's patch with a few tweaks

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=c6a23ce8cbe1d22633af16b31d045fa7957e448c

I'll publish the download link later.

Comment 67

[Jorg K (CEST = GMT+2)]

Anyone keen to try can download a Windows build here:
https://archive.mozilla.org/pub/thunderbird/try-builds/[e-mail]-c6a23ce8cbe1d22633af16b31d045fa7957e448c/try-comm-central-win32/

Other platforms upon request.

Comment 68

[lussnig]

Hi, i can check it in the evening :-)

Comment 69

[lussnig]

Got with the new Build an crash and submitted it. Maybe this will help.

Comment 70

[Jorg K (CEST = GMT+2)]

Kindly let us know the crash ID.

Comment 71

[Wayne Mery (:wsmwk)]

(In reply to lussnig from comment #69)
> Got with the new Build an crash and submitted it. Maybe this will help.

xul.dll@0x14b2b9 | xul.dll@0x14b364 | xul.dll@0x14c133 | xul.dll@0x23b3c8 | xul.dll@0x23cfaa | xul.dll@0x26ced1 | xul.dll@0x2a1bb4 | xul.dll@0x3aecb0 | xul.dll@0x3ae544 | xul.dll@0x1afec24 | xul.dll@0x1afeba8 | xul.dll@0x1c87baa | xul.dll@0x1c8658a | x... 
bp-958380fd-5851-4e9d-b2fa-8fd210170426

Comment 72

[lussnig]

bp-958380fd-5851-4e9d-b2fa-8fd210170426.txt was mine

Comment 73

[Jorg K (CEST = GMT+2)]

This crash report is sadly useless.

Personally, I'd take this patch since it doesn't make matters worse. I haven't tested it, but I trust Jan has run it with the hunk that stops the retrieval from the local folder and would force the endless recursion unless mitigated by the patch.

Jan, are you OK with my changes?

This bug has now become a collection pool of all problems related to IMAP compaction, but we started off fixing the recursion problem.

Or perhaps Magnus wants to do some testing here?

Comment 74

[Jorg K (CEST = GMT+2)]

Attachment: recursive-fix.patch - Jan's patch with a few tweaks (v2).

Fixed a comment.

Comment 75

[Jan Horak [:jhorak]]

(In reply to Jorg K (GMT+2) from comment #73)
> This crash report is sadly useless.
> 
> Personally, I'd take this patch since it doesn't make matters worse. I
> haven't tested it, but I trust Jan has run it with the hunk that stops the
> retrieval from the local folder and would force the endless recursion unless
> mitigated by the patch.
> 
> Jan, are you OK with my changes?
> 
> This bug has now become a collection pool of all problems related to IMAP
> compaction, but we started off fixing the recursion problem.
> 
> Or perhaps Magnus wants to do some testing here?

Looks good to me. Thanks for looking into it.

Comment 76

[Magnus Melin [:mkmelin]]

Comment on attachment 8862064 [details] [diff] [review]
recursive-fix.patch - Jan's patch with a few tweaks (v2).

Review of attachment 8862064 [details] [diff] [review]:
-----------------------------------------------------------------

Compacted and it didn't crash, which it often but not always did before

::: mailnews/base/src/nsMsgFolderCompactor.cpp
@@ +1094,5 @@
>  
>    // The NS_MSG_ERROR_MSG_NOT_OFFLINE error should allow us to continue, so we
>    // check for it specifically and don't terminate the compaction.
> +  if (rv == NS_MSG_ERROR_MSG_NOT_OFFLINE) {
> +    NS_WARNING("Message should be available as offline but it is not");

can we add something to identify which message?

Comment 77

[Jorg K (CEST = GMT+2)]

Attachment: recursive-fix.patch - Jan's patch with a few tweaks (v3).

OK, as Magnus requested, now printing out the message that's not available.

Actually, Magnus, you said it didn't crash, but did you see the warning?

Comment 78

[Jorg K (CEST = GMT+2)]

https://hg.mozilla.org/comm-central/rev/2f6c5a34d8f821c85ddd8f4ccdec89004fd540b8

So let's see where the crash statistics go with this landed. I don't expect a 100% cure, but it won't make things worse either.

Comment 79

[Jorg K (CEST = GMT+2)]

Comment on attachment 8862136 [details] [diff] [review]
recursive-fix.patch - Jan's patch with a few tweaks (v3).

Let's take this to TB 54 beta to see what it does ;-)

Comment 80

[Jorg K (CEST = GMT+2)]

TB 54 (beta):
https://hg.mozilla.org/releases/comm-beta/rev/e046c1af98b5cf74df0eb9d3a86c16cf3d26fdab

Comment 81

[Magnus Melin [:mkmelin]]

(In reply to Jorg K (GMT+2) from comment #77)
> Actually, Magnus, you said it didn't crash, but did you see the warning?

It wasn't a debug build so no warning would have been seen.

Comment 82

[Jacques Amar]

Sorry to interject ...
Since the issue seems to be related to a corrupt local folder (fixing that folder fixed my crashes), would ther be a way to actually detect a corrupt folder and give the end user an indication that they should fix?

Comment 83

[Wayne Mery (:wsmwk)]

Jorg asked about the impact of the patch landed 2017-04-26.  

I can say with certainty 
* js::SavedStacks::insertFrames on nightly builds is gone starting with 04-27 build. The average had been 15 crashes per day. So this patch appears to kill Bug 1316416
* js::jit::InlineFrameIterator::resetOn on nightly builds is gone starting with 04-27 build. So this kills #1 nightly-only crash signature

For other signatures we won't know whether there is impact until the patch ships in other versions, 54.0b1 and hopefully 52.1.1

Comment 84

[Jorg K (CEST = GMT+2)]

Well, I asked whether the IMAP compact crash is gone now, which has been a long-standing issue, but aggravated by the switch to cache2 in bug 1021843. I think the answer is yes, so let's declare this bug here fixed.

A very big thank you to Jan to analyse the problem and come up with a solution!!

That said: Bug 1316416 is a collection of stack overruns caused by various issues, most prominently this issue here bug also readily reproducible by bug 1343536 (obscure scenario). So I leave it to Wayne to close bug 1343536.

Comment 85

[Magnus Melin [:mkmelin]]

Yes, many thanks to Jan for figuring this out!

Comment 86

[Marcel Kneuer]

Did this patch also land on osx? My windows thunderbird is no fine, but osx ist still crashing starting compacting: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:54.0) Gecko/20100101 Thunderbird/54.0a2

Crash reports are always empty, so I can't help with this (https://crash-stats.mozilla.com/report/index/2e07c1de-a90d-4706-aab1-150200170430).

Comment 87

[Wayne Mery (:wsmwk)]

(In reply to Marcel Kneuer from comment #86)
> Did this patch also land on osx? My windows thunderbird is no fine, but osx
> ist still crashing starting compacting: Mozilla/5.0 (Macintosh; Intel Mac OS
> X 10.12; rv:54.0) Gecko/20100101 Thunderbird/54.0a2
> 
> Crash reports are always empty, so I can't help with this
> (https://crash-stats.mozilla.com/report/index/2e07c1de-a90d-4706-aab1-
> 150200170430).

You are running aurora channel, but that channel is obsolete and didn't get the patch. 
Nightly channel has the patch https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/ 
Or change to beta which will get it soon.

Comment 89

[Jorg K (CEST = GMT+2)]

TB 52 ESR:
https://hg.mozilla.org/releases/comm-esr52/rev/a003e013e18bc2f4ba96717387ea86d23045ce2e

Comment 90

[Wayne Mery (:wsmwk)]

For those affected, 52.1.1 *candidate for release* is available for testing at
Windows: https://archive.mozilla.org/pub/thunderbird/candidates/52.1.1-candidates/build1/win32/en-US/Thunderbird%20Setup%2052.1.1.exe
Mac: https://archive.mozilla.org/pub/thunderbird/candidates/52.1.1-candidates/build1/mac/en-US/Thunderbird%2052.1.1.dmg
Linux: https://archive.mozilla.org/pub/thunderbird/candidates/52.1.1-candidates/build1/linux-x86_64/en-US/thunderbird-52.1.1.tar.bz2

Comment 91

[Peter Peltonen]

I tried 52.1.1 on my Mac, here my experiences:

Good news:
- I switched my inboxes to sync for offline use again
- I deleted bunch of messages
- I compacted the inboxes and now my TB didn't crash immediately like before, yay

Bad news:
- After a few hours of running I noticed Mozilla crash reporter and my TB gone, so it had crashed after some time
- So I do not know if this is related to compacting or something else
- Details of the submitted crash report:

AdapterDeviceID: 0x08a0
AdapterVendorID: 0x10de
Add-ons: sogo-connector%40inverse.ca:31.0.1,filtaquilla%40mesquilla.com:1.3.2,%7Be2fda1a4-762b-4020-b5ad-a41df1933103%7D:5.4.1.1,%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D:52.1.1
BuildID: 20170509142926
ContentSandboxCapable: 1
ContentSandboxLevel: 0
CrashTime: 1494854528
EMCheckCompatibility: true
FramePoisonBase: 7ffffffff0dea000
FramePoisonSize: 4096
InstallTime: 1494851460
Notes: AdapterVendorID: 0x10de, AdapterDeviceID: 0x08a0FP(D00-L1100-W00000000-T0100) GL Context? GL Context+ 
ProductID: {3550f703-e582-4d05-9a08-453d09bdfdc6}
ProductName: Thunderbird
ReleaseChannel: release
SafeMode: 0
SecondsSinceLastCrash: 16846
StartupCrash: 0
StartupTime: 1494851460
TelemetryEnvironment: {"build":{"applicationId":"{3550f703-e582-4d05-9a08-453d09bdfdc6}","applicationName":"Thunderbird","architecture":"x86-64","buildId":"20170509142926","version":"52.1.1","vendor":null,"platformVersion":"52.1.1","xpcomAbi":"x86_64-gcc3","hotfixVersion":null,"architecturesInBinary":"i386-x86_64"},"partner":{"distributionId":null,"distributionVersion":null,"partnerId":null,"distributor":null,"distributorChannel":null,"partnerNames":[]},"system":{"memoryMB":8192,"virtualMaxMB":null,"cpu":{"count":2,"cores":2,"vendor":"GenuineIntel","family":6,"model":23,"stepping":10,"l2cacheKB":3072,"l3cacheKB":null,"speedMHz":2400,"extensions":["hasMMX","hasSSE","hasSSE2","hasSSE3","hasSSSE3","hasSSE4_1"]},"os":{"name":"Darwin","version":"16.5.0","locale":"en-FI"},"hdd":{"profile":{"model":null,"revision":null},"binary":{"model":null,"revision":null},"system":{"model":null,"revision":null}},"gfx":{"D2DEnabled":null,"DWriteEnabled":null,"ContentBackend":"Skia","adapters":[{"description":null,"vendorID":"0x10de","deviceID":"0x08a0","subsysID":null,"RAM":null,"driver":null,"driverVersion":null,"driverDate":null,"GPUActive":true}],"monitors":[{"screenWidth":1280,"screenHeight":800,"scale":1}],"features":{"compositor":"none"}}},"settings":{"blocklistEnabled":true,"e10sEnabled":false,"e10sCohort":"unknown","telemetryEnabled":false,"locale":"en-US","update":{"channel":"release","enabled":true,"autoDownload":false},"userPrefs":{"app.update.auto":false,"browser.cache.disk.capacity":358400},"addonCompatibilityCheckEnabled":true,"isDefaultBrowser":null},"profile":{"creationDate":13725},"addons":{"activeAddons":{"[e-mail]":{"blocklisted":false,"description":"A DAV plugin for keeping addressbooks and events in sync","name":"Inverse SOGo Connector","userDisabled":false,"appDisabled":false,"version":"31.0.1","scope":1,"type":"extension","foreignInstall":false,"hasBinaryComponents":false,"installDay":16482,"updateDay":16482,"isSystem":false},"[e-mail]":{"blocklisted":false,"description":"Mail filter custom actions and searches","name":"FiltaQuilla","userDisabled":false,"appDisabled":false,"version":"1.3.2","scope":1,"type":"extension","foreignInstall":false,"hasBinaryComponents":false,"installDay":17105,"updateDay":17105,"isSystem":false},"{e2fda1a4-762b-4020-b5ad-a41df1933103}":{"blocklisted":false,"description":"Integrated Calendaring & Scheduling for your Email client","name":"Lightning","userDisabled":false,"appDisabled":false,"version":"5.4.1.1","scope":1,"type":"extension","foreignInstall":false,"hasBinaryComponents":false,"installDay":14813,"updateDay":17301,"isSystem":false}},"theme":{"id":"{972ce4c6-7e08-4474-a285-3208198ce6fd}","blocklisted":false,"description":"The default theme.","name":"Default","userDisabled":false,"appDisabled":false,"version":"52.1.1","scope":4,"foreignInstall":false,"hasBinaryComponents":false,"installDay":15162,"updateDay":17296},"activePlugins":[],"activeGMPlugins":{},"activeExperiment":{},"persona":null}}
Theme: classic/1.0
Throttleable: 1
UptimeTS: 3083.34533823
Vendor: 
Version: 52.1.1
useragent_locale: en-US
This report also contains technical information about the state of the application when it crashed.

Comment 92

[Wayne Mery (:wsmwk)]

(In reply to Peter Peltonen from comment #91)
> I tried 52.1.1 on my Mac, here my experiences:
> 
> Good news:
> - I switched my inboxes to sync for offline use again
> - I deleted bunch of messages
> - I compacted the inboxes and now my TB didn't crash immediately like
> before, yay

Thanks for that info.

> Bad news:
> - After a few hours of running I noticed Mozilla crash reporter and my TB
> gone, so it had crashed after some time
> - So I do not know if this is related to compacting or something else
> - Details of the submitted crash report:

Unfortunately, the "details" data is never of any use. We would need your crash ID https://support.mozilla.org/en-US/kb/mozilla-crash-reporter#w_viewing-crash-reports

Comment 93

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #92)
> 
> Unfortunately, the "details" data is never of any use. We would need your
> crash ID
> https://support.mozilla.org/en-US/kb/mozilla-crash-reporter#w_viewing-crash-
> reports

I do not see any Mac crash reports for 52.1.1, so you may need to submit it at Help > Troubleshooting.  Another possibility is  the crash reporter failed.

Comment 94

[Peter Peltonen]

Ok, did not know about the crash report id, thanks for clarification. Here is the crash report I think (did not resubmit anything, so strange you did not see it before):

https://crash-stats.mozilla.com/report/index/b1652bd6-6720-4479-a6b5-2a9a30170515

I do have experienced similar crashes not related to compacting with 52.1.0 as well, I think this earlier from the same day was one:

https://crash-stats.mozilla.com/report/index/605b97dc-0079-44b4-ba0a-b0cc80170515

So perhaps this is a separate bug? I don't understand those reports, so hard for me to say...

Haven't happened after that first crash with 52.1.0. I will let you know if it does.

Comment 95

[Jorg K (CEST = GMT+2)]

Different issue, Wayne, can you please file a bug for those crashes.

Comment 96

[Wayne Mery (:wsmwk)]

(In reply to Peter Peltonen from comment #94)
> Ok, did not know about the crash report id, thanks for clarification. Here
> is the crash report I think (did not resubmit anything, so strange you did
> not see it before):
> 
> https://crash-stats.mozilla.com/report/index/b1652bd6-6720-4479-a6b5-
> 2a9a30170515

bug 1365136. And afaict unrelated to this bug

Comment 97

[Wayne Mery (:wsmwk)]

(I find no record of there ever being a js::jit::InlineFrameIterator::::resetOn, so removing)

All of these signatures are gone in 52.1.1
[@ nsImapMockChannel::Close ]
[@ nsOfflineStoreCompactState::CopyNextMessage ]
[@ nsOfflineStoreCompactState::OnStopRequest ]
[@ nsScriptSecurityManager::CanCreateWrapper ]
[@ nsImapCacheStreamListener::OnStopRequest ]

plus
mozilla::mailnews::MsgDBReporter::GetPath  bug 1353704
js::SavedStacks::insertFrames   Bug 1316416 

and some users had reported Empty signature.

All totalled, I expect 11-15% of crashes wiped out by this patch.
v.fixed

Thanks Jan and Jorg and everyone involved!