1017721 - Random Assertion failure: rc != 0 (destroyed timer off its target thread!) on shutdown EventTokenBucket

Status: RESOLVED DUPLICATE of bug 1024765

(Core :: Networking, defect)

Description

[Bob Clary [:bc] (inactive)]

Attachment: windbg stack

I have seen 

Assertion failure: rc != 0 (destroyed timer off its target thread!), at c:\work\
mozilla\builds\nightly\mozilla\xpcom\threads\TimerThread.cpp:279

on shutdown on Windows very often in bughunter's crash automation testing but have not been able to find reproducible steps. In just the last few weeks I have seen this assertion on over 400 urls though it has been occurring for much longer. This is not restricted to the Nightly branch. It appears to be random. I can only reproduce the assertion about 10% of the time when testing the urls where the assertion has previously been seen.

I have this captured in windbg for https://www.theclymb.com/ on the build machine for bughunter: win61i32-bld.bughunter.ateam.phx1.mozilla.com. I will leave it available for the until tomorrow when I need to create new builds.

I would appreciate some help with diagnosing this assertion. It is causing problems with regular crash testing due to its frequency. I've filed this under Firefox:General since it may be specific to the offending timer, but I don't know how to determine it.

Comment 1

[Benjamin Smedberg]

We need to know the timer client. I'm not sure whether you can get this out of the assertion or not: at http://hg.mozilla.org/mozilla-central/annotate/5add3261493b/xpcom/threads/TimerThread.cpp#l279 the timer object has already been destroyed, so probably timer->mCallbackType and timer->mCallback are already dead.

Attachment 8415866 [details] [diff] has some code which will help make this more reproducible; we should probably make that NS_WARNING a MOZ_ASSERT in general. And when you hit that, the debugger will have the real timer data.

See bug 881413 for a version of this that we fixed.

Comment 2

[Bob Clary [:bc] (inactive)]

Attachment: bug881413-updated.patch

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #1)
> We need to know the timer client. I'm not sure whether you can get this out
> of the assertion or not: at
> http://hg.mozilla.org/mozilla-central/annotate/5add3261493b/xpcom/threads/
> TimerThread.cpp#l279 the timer object has already been destroyed, so
> probably timer->mCallbackType and timer->mCallback are already dead.
> 

Yes, they are.

> Attachment 8415866 [details] [diff] has some code which will help make this
> more reproducible; we should probably make that NS_WARNING a MOZ_ASSERT in
> general. And when you hit that, the debugger will have the real timer data.

Great. I've attached the refreshed patch. Fwiw, prior to this patch when trying to reproduce this assertion the "WARNING: An event was posted to a thread that will never run it (rejected)" appears just before the assertion but can occur without the assertion being fired.

I've deployed this patch to win32 workers and see it firing the false assertion.

I've caught a different url <https://www.consignacaoalagoas.com.br/ES1/index.html> in the assertion

[2416] WARNING: An event was posted to a thread that will never run it (rejected
): file c:\work\mozilla\builds\nightly\mozilla\xpcom\threads\nsThread.cpp, line
462
Assertion failure: false, at c:\work\mozilla\builds\nightly\mozilla\xpcom\thread
s\nsThread.cpp:463

I'll attach the stack and locals next.

Comment 3

[Bob Clary [:bc] (inactive)]

Attachment: consignacaoalagoas stack + locals

Comment 4

[Benjamin Smedberg]

What's interesting here is aEvent->mTimer.mRawPtr->mCallback.i

which is a nsITimerCallback* but we really need to know the underlying type.

Comment 5

[Bob Clary [:bc] (inactive)]

Attachment: another url locals

Are these the right ones?

          mCallbackType	0x00 ''
          mType	0x00 ''

Comment 6

[Bob Clary [:bc] (inactive)]

fyi, I still have this assertion in windbg and can answer any other questions about the data you may have. Just ping me on irc.

Comment 7

[Benjamin Smedberg]

      __vtcast_mozilla::net::EventTokenBucket	class mozilla::net::EventTokenBucket

Looks like networking needs to cancel a timer before shutdown, probably the one here: http://hg.mozilla.org/mozilla-central/annotate/6a984e21c2ca/netwerk/base/src/EventTokenBucket.cpp#l306

Comment 9

[Bob Clary [:bc] (inactive)]

Resetting the os and platform since this assertion appears on Windows XP and Windows 7 for both 32 and 64bit.

mcmanus, this bug is a major source of pain for me in crash testing. Is there anything I can do to help get it resolved?

Comment 10

[Patrick McManus [:mcmanus]]

good first bug - I think jason should get one of the newer necko team members to work on it. I'll be happy to mentor.

Comment 11

[Jason Duell]

Sounds good--it's on my list.

Comment 12

[Bob Clary [:bc] (inactive)]

This and all other Assertion failure: rc != 0 failures were fixed by bug 1024765. This was the top crash in bughunter but I was unable to reproduce it after bug 1024765 was fixed and testing over 400 urls on all three branches and all three platforms.