480878 - Generic crash [@ memcpy | nsSSLThread::requestRecvMsgPeek(nsNSSSocketInfo*, void*, int, int, unsigned int)], [@ nsSSLThread::requestRecvMsgPeek(nsNSSSocketInfo*, void*, int, int, unsigned int)], [@ nsSSLThread::requestRecvMsgPeek]

Status: RESOLVED WORKSFORME

(Core Graveyard :: Security: UI, defect)

Description

[Chris Pepper]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1
Build Identifier: 2.0.0.19 (2.0.0.19)

No idea what caused this.

Reproducible: Didn't try

Steps to Reproduce:
No recipe, sorry.
Actual Results:  
Crash

Expected Results:  
No crash.

Log attached.

Comment 1

[Chris Pepper]

Attachment: MOSX crashlog

Comment 2

[Wayne Mery (:wsmwk)]

Also seen with thunderbird trunk 3.0b2 20090223175111 and 3.0b3pre.  Haven't found any prior to 3.0b2.  Almost all crashes are after long periods, 2-10 hours

[@ memcpy | nsSSLThread::requestRecvMsgPeek(nsNSSSocketInfo*, void*, int, int, unsigned int) ]
bp-f18e5ba9-19f1-45d7-9c65-2a7652090226

bp-a2845726-3398-4b88-9a16-747d62090321
0  	mozcrt19.dll  	memcpy  	memcpy.asm:348
1 	thunderbird.exe 	nsSSLThread::requestRecvMsgPeek 	security/manager/ssl/src/nsSSLThread.cpp:171
2 	thunderbird.exe 	PSMRecv 	security/manager/ssl/src/nsNSSIOLayer.cpp:1869
3 	thunderbird.exe 	nsSocketTransport::IsAlive 	netwerk/base/src/nsSocketTransport2.cpp:1808
4 	thunderbird.exe 	nsImapProtocol::CanHandleUrl 	nsImapProtocol.cpp:2008
5 	thunderbird.exe 	nsImapIncomingServer::GetImapConnection 	nsImapIncomingServer.cpp:705

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

This is a crash in PSM code.  Adding PSM folks to CC list.

Comment 4

[timeless]

Most likely the optimizer is optimizing 169/171 and 581/583
169           PRInt32 return_amount = NS_MIN(amount, si->mThreadData->mSSLResultRemainingBytes);
171           memcpy(buf, si->mThreadData->mSSLRemainingReadResultData, return_amount);

581         PRInt32 return_amount = NS_MIN(amount, si->mThreadData->mSSLResultRemainingBytes);
583         memcpy(buf, si->mThreadData->mSSLRemainingReadResultData, return_amount);

588           si->mThreadData->mSSLRemainingReadResultData = nsnull;

If I'm right, then 588 probably happened, and we're crashing reading a null pointer. Figuring out the legality of such a call sequence is up to someone else.

Comment 5

[Nelson Bolyard (seldom reads bugmail)]

Timeless, 
Are you saying that you think this is a compiler optimization fault?

Is there anything about the lines at 
http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/security/manager/ssl/src/nsSSLThread.cpp&rev=1.16&mark=169-173#169
that appears to be unsafe/improper code?

Likewise, I would ask the same questions for 
http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/security/manager/ssl/src/nsSSLThread.cpp&rev=1.16&mark=581-588#580

Comment 6

[timeless]

no, i'm just explaining why it crashed at 171 instead of 169.

what's unsafe is that there isn't a null check. however i don't know the application logic (and don't want to figure it out), so i'm not certain if that null is supposed to be possible.

Comment 7

[Honza Bambas (:mayhemer)]

First, access to this property is not thread safe while it is accessed on multiple threads, check http://mxr.mozilla.org/mozilla1.8/ident?i=mSSLRemainingReadResultData for file nsSSLThread.cpp.

Second, we may probably experience call to memcpy before we assign  bstd.mSSLRemainingReadResultData = bstd.mSSLDataBuffer or before we call mThreadData->ensure_buffer_size. I have to walk all the code paths (lot of work) to try to figure out.

I never experienced this crash my self.

Comment 8

[Jesse Ruderman]

Honza, will you be able to fix those things?  If not, can you at least split them into separate bugs?

Comment 9

[Honza Bambas (:mayhemer)]

(In reply to comment #8)
> Honza, will you be able to fix those things?  If not, can you at least split
> them into separate bugs?

I have to analyze the code. I have some more important stuff at the moment, so I can get to this within few weeks. If this is more priority let me know please.

Comment 10

[Honza Bambas (:mayhemer)]

-> me

I'll take a look at this one.

Comment 11

[Wayne Mery (:wsmwk)]

is #32 and climbing, so approaching topcrash status for thunderbird 3.0. Thunderbird signature is now memcpy | nsSSLThread::requestRecvMsgPeek(nsNSSSocketInfo*, void*, int, int, unsigned int)

in firefox it's quite rare, only 5 crashes in 2 weeks for all FF releases. eg bp-bc3514a3-0d55-428a-baf9-54c2c2091206 ... FF signature is still [@ nsSSLThread::requestRecvMsgPeek(nsNSSSocketInfo*, void*, int, int, unsigned int) ]

Comment 12

[Peter Harvey]

I had a core dump today in Thunderbid 2.0.0.23 with a very similar stack trace:

libc_hwcap1.so.1`_lwp_kill+0x15(1, b, 80461f8, fecaaffe)
libc_hwcap1.so.1`raise+0x22(b, 8046210, 0, 807117a)
void nsProfileLock::FatalSignalHandler(int)+0xf1(b, 0, 804630c, ffff, fdfb2a00, fed91000)
libc_hwcap1.so.1`__sighndlr+0x15(b, 0, 804630c, 807116c)
libc_hwcap1.so.1`call_user_handler+0x2af(b)
libc_hwcap1.so.1`sigacthandler+0xdf(b, 0, 804630c)
libc_hwcap1.so.1`memcpy+0x22(bce4400, 8046607, 1, 2, 0, 1)
libpipnss.so`int PSMRecv(PRFileDesc*,void*,int,int,unsigned)+0x8a(9e60508, 8046607, 1, 2, 0, feab6fdc)
libnspr4.so`PR_Recv+0x1d(9e60508, 8046607, 1, 2, 0, 804660c)
libnecko.so`unsigned nsSocketTransport::IsAlive(int*)+0xa1(b889a88, 8046664, 804673c, fc1717bd)
libmail.so`unsigned nsImapProtocol::CanHandleUrl(nsIImapUrl*,int*,int*)+0xc7(dfe75f0, bb87928, 8046794, 8046798)
libmail.so`unsigned nsImapIncomingServer::GetImapConnection(nsIEventQueue*,nsIImapUrl*,nsIImapProtocol**)+0x1fc(9554e70, 81109d0, bb87928, 804684c)
libmail.so`unsigned nsImapIncomingServer::GetImapConnectionAndLoadUrl(nsIEventQueue*,nsIImapUrl*,nsISupports*)+0x41(9554e70, 81109d0, bb87928, 0)
libmail.so`unsigned nsImapService::GetImapConnectionAndLoadUrl(nsIEventQueue*,nsIImapUrl*,nsISupports*,nsIURI**)+0x1b6(8b7d0a0, 81109d0, bb87928, 0, 8046a04, 8046938)
libmail.so`unsigned nsImapService::UpdateFolderStatus(nsIEventQueue*,nsIMsgFolder*,nsIUrlListener*,nsIURI**)+0x263(8b7d0a0, 81109d0, 8742d5c, 9554f0c, 8046a04, 8046a10)
libmail.so`unsigned nsImapMailFolder::UpdateStatus(nsIUrlListener*,nsIMsgWindow*)+0x90(8742d38, 9554f0c, 0, fc12b16a)
libmail.so`unsigned nsImapIncomingServer::OnStopRunningUrl(nsIURI*,unsigned)+0x170(9554e70, d405694, 0, fbf70e44)
libmail.so`unsigned nsUrlListenerManager::BroadcastChange(nsIURI*,nsUrlNotifyType,unsigned)+0x12b(ab1ac98, d405694, 1, 0)
libmail.so`unsigned nsUrlListenerManager::OnStopRunningUrl(nsIMsgMailNewsUrl*,unsigned)+0x25(ab1ac98, d405694, 0, fbf55ea6)
libmail.so`unsigned nsMsgMailNewsUrl::SetUrlState(int,unsigned)+0xc2(d405694, 0, 0, 0)
libmail.so`unsigned nsImapMailFolder::SetUrlState(nsIImapProtocol*,nsIMsgMailNewsUrl*,int,unsigned)+0xf1(8742d38, e00f558, d405694, 0, 0, b1f3178)
libxpcom_core.so`XPTC_InvokeByIndex+0x51(8742e3c, 1f, 4, bb8e200)
libxpcom_core.so`void*EventHandler(PLEvent*)+0x3c(d0511d0, 0, 8046c08, feba21a6)
libxpcom_core.so`PL_ProcessPendingEvents+0x17d(8255fc0, 81109d0, fec1a1a0, feba4be5)
libxpcom_core.so`unsigned nsEventQueueImpl::ProcessPendingEvents()+0xdd(81109d0, 0, 8046cb8, fe288310)
libwidget_gtk2.so`int event_processor_callback(_GIOChannel*,GIOCondition,void*)+0x18(8620158, 1, 81109d0, fe2caa82)
libglib-2.0.so.0.2200.0`g_io_unix_dispatch+0x3e(86201a0, fc836524, 81109d0, 8046d20)
libglib-2.0.so.0.2200.0`g_main_context_dispatch+0x262(80d0220, c8, a5a94e0, a)
libglib-2.0.so.0.2200.0`g_main_context_iterate+0x483(80d0220, 1, 1, 80ae678)
libglib-2.0.so.0.2200.0`g_main_loop_run+0x1dd(8620210, 8620210, 8046de0, fe57e726)
libgtk-x11-2.0.so.0.1800.0`gtk_main+0xb7(808c898, 84d3980, 8046e58, fc600c56, 84d5d48, fc7a5210)
libwidget_gtk2.so`unsigned nsAppShell::Run()+0x36(84d5d48, fc7a5210, 0, fc600c34)
libtoolkitcomps.so`unsigned nsAppStartup::Run()+0x2e(84d3980, 80746a4, 0, 8064c65)
XRE_main+0x381f(1, 804713c, 808c870, 805b028, 80470f8)
main+0x28(1, 804713c, 8047144, 8047130)
_start+0x7d(1, 8047314, 0, 807272c, 8072744, 80473b5)

This was on OpenSolaris (snv_126, x64). No obvious trigger but I think I've seen this before recently and it was associated with opening a new IMAP connection.

Comment 13

[Wayne Mery (:wsmwk)]

top 25 crash for TB3.0.0 combining windows and Mac crashes, so fixing this is very much wanted

bp-e328dce5-e002-4c98-8359-0a5d52091216
"Using a mailbox with an untrusted certificate. The according untrusted-certificate-message pops up everytime, TB tries to download E-Mails from this mailbox."

bp-6d1ef113-fc50-4adb-93f1-079312091213 (no extensions)

bp-4bbe1737-4b45-42a1-a0bc-daf7b2091222 every time sending a message

Comment 14

[Liam Merwick]

Version:         2.0.0.23 (2.0.0.23)
Code Type:       X86 (Native)
OS Version:      Mac OS X 10.6.2 (10C540)

I've hit this twice in the last couple of days. There was no obvious trigger (TB was in the background when it happened). I had tried TB3.0 recently but reverted back to TB2.0 due to stability issues which I couldn't track down (waiting for OS X 10.6.3 to try again with TB3.0.1). Previously I had been running TB2.0 for a long time without ever hitting it. The only extension I have is Nostalgy.

Comment 15

[Kai Engert (:KaiE:)]

It would be interesting to know the parameter values of the function calls in the stack, it seems we don't have that?

Comment 16

[Kai Engert (:KaiE:)]

I see "crash address" is being reported as 0x0.

Does this mean memcpy attempts to write to a destination buffer at address 0x0 ?

PSMRecv and nsSSLThread::requestRecvMsgPeek get "bug" as input, they don't change it.

Caller nsSocketTransport::IsAlive uses

   char c;
   PRInt32 rval = PR_Recv(fd, &c, 1, PR_MSG_PEEK, 0); 

How could &c ever become 0x0 ??

Comment 17

[Kai Engert (:KaiE:)]

(In reply to comment #16)
> 
> PSMRecv and nsSSLThread::requestRecvMsgPeek get "bug" as input, they don't
> change it.

not "bug", I mean "buf"

Comment 18

[timeless]

si->mThreadData->mSSLRemainingReadResultData could be null, memcpy would crash.

Comment 19

[Wayne Mery (:wsmwk)]

#30 crash for v3.0.3.  (counterintuitive but) it's bit unusual for us to see our top crashes on other branches, so citing...
bp-eace7a4e-a962-40e0-b06e-6d15a2100322 3.2a1pre
bp-64a3791a-aa3c-44a8-b6a3-5243d2100316 3.1b1

pmed 3.0.3 reporter bp-79baa7db-a87c-4413-91ef-951f82100317

Comment 20

[Wayne Mery (:wsmwk)]

a couple crashes mention thunderbird was idle (from their perspective)

bp-1abade11-b93a-4179-a28c-d3f522100207 (david)
bp-4b368ce3-3d7f-4f26-a2b1-3b4cd2100214

pmed also 
bp-28531f87-3806-4278-8b24-de9a72100226 (scott)
bp-0ee8e7e2-2ee3-48ac-8509-875bc2100405 (bjorn)
bp-46783f61-5dcc-4391-9086-8e2a12100406 (infos)

Comment 21

[Michael]

Got an email asking for details about when the crash occurs.

The crash always occurs when I leave my computer locked over night. Thunderbird is running in the background along with a few other apps. I'm a software developer - for the first 10 hours after I leave each night the memory and CPU are maxed out building all my projects for the following days work. So there is likely lots of memory swapping if Thunderbird is trying to do anything.

Comment 22

[Orion Poplawski]

Same here.  See https://bugzilla.redhat.com/show_bug.cgi?id=641422.  Seems simple - don't call memcpy if si->mThreadData->mSSLRemainingReadResultData is null (return_amount will be zero too).

Comment 23

[timeless]

from brc 641422 comment 2:
#3  nsSSLThread::requestRecvMsgPeek (si=0x7ff51c7ea700, buf=0x7fffd591c07f,
amount=1, flags=2, timeout=<value optimized out>) at
/usr/include/bits/string3.h:52
        threadLock = {<nsAutoLockBase> = {<No data fields>}, mLock =
0x7ff51c4a71c0, mLocked = 1}
        realSSLFD = <value optimized out>

Code:

          PRInt32 return_amount = NS_MIN(amount,
si->mThreadData->mSSLResultRemainingBytes);

          memcpy(buf, si->mThreadData->mSSLRemainingReadResultData,
return_amount);

(gdb) print amount
$2 = 1
(gdb) print si->mThreadData->mSSLResultRemainingBytes
$3 = 0
(gdb) print si->mThreadData->mSSLRemainingReadResultData
$1 = 0x0

Comment 24

[Wayne Mery (:wsmwk)]

topcrash for v3.1.7 so would be great to get this on branch.
semi frequent on trunk, eg bp-abe86663-f3aa-47a9-aafa-e38202110208 linux
linux signature nsSSLThread::requestRecvMsgPeek

Comment 25

[Ludovic Hirlimann [:Usul]]

Honza , would you have more time to look into this now ?

Comment 26

[Honza Bambas (:mayhemer)]

Hopefully, if anyone wants to work on this immediately, feel free to take it and drop a note.

Comment 27

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Crashes involving nsSSLThread in it should be fixed by bug 674147, which removes that class (and thread) entirely.

Comment 28

[Sheila Mooney]

On the desktop we have about 15 crashes with the signature = nsSSLThread::requestRecvMsgPeek(nsNSSSocketInfo*, void*, int, int, unsigned int) on all versions over the past 4 weeks. For Thunderbird, the volume is much higher. Leaving as a top crash for now.

Comment 29

[Wayne Mery (:wsmwk)]

indeed, for thunderbird this is ~top 15

Comment 30

[Wayne Mery (:wsmwk)]

recheck when version 11 hits the streets

Comment 31

[Wayne Mery (:wsmwk)]

[@ libsystem_c.dylib@0x27cf1 ] looks like matching signature for v8 and 9.0.1

note, according to bug 674147 this is still at risk for staying in version 11 because Bug 542832 is not yet fixed, although it is making progress. perhaps it will land within a couple weeks?

Comment 32

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

This is impossible since nsSSLThread is gone with the patch for bug 674147. If we have to back out bug 674147 for any reason, we will have to reopen a bunch of bugs, including this one. We will know to do this because I am marking bug 674147 as blocking all of these bugs.