Closed Bug 529544 Opened 10 years ago Closed 10 years ago

random crashes during test_browserGlue_corrupt.js and test_browserGlue_restore.js involving [@ mozilla::storage::Connection::Close()]

Categories

(Toolkit :: Places, defect, critical)

defect
Not set
critical

Tracking

()

RESOLVED FIXED
mozilla1.9.3a1

People

(Reporter: mak, Assigned: hsivonen)

References

Details

(Keywords: crash, intermittent-failure)

Crash Data

Attachments

(3 files)

These tests are simulating shutdown, with db close and the stack looks related.

 0  ld-2.5.so + 0x7f2
    eip = 0x00a6f7f2   esp = 0xbfc36be4   ebp = 0xbfc36bf0   ebx = 0x00002c23
    esi = 0xbfc36c90   edi = 0x00bc9ff4   eax = 0x00000000   ecx = 0x00002c23
    edx = 0x00000006   efl = 0x00000202
    Found by: given as instruction pointer in context
 1  libc-2.5.so + 0x2a450
    eip = 0x00abb451   esp = 0xbfc36bf8   ebp = 0xbfc36d1c
    Found by: previous frame's frame pointer
 2  libnspr4.so!PR_Abort [prlog.c:c3f5eaa30cdf : 548 + 0x4]
    eip = 0x00120602   esp = 0xbfc36d24   ebp = 0xbfc36d2c
    Found by: previous frame's frame pointer
 3  libxul.so!_ZL5AbortPKc [nsDebugImpl.cpp:c3f5eaa30cdf : 370 + 0x4]
    eip = 0x020b6944   esp = 0xbfc36d34   ebp = 0xbfc36d3c
    Found by: previous frame's frame pointer
 4  libxul.so!NS_DebugBreak_P [nsDebugImpl.cpp:c3f5eaa30cdf : 350 + 0xd]
    eip = 0x020b6f54   esp = 0xbfc36d44   ebp = 0xbfc3716c
    Found by: previous frame's frame pointer
 5  libxul.so!mozilla::storage::Connection::Close() [mozStorageConnection.cpp:c3f5eaa30cdf : 522 + 0x37]
    eip = 0x01e13288   esp = 0xbfc37174   ebp = 0xbfc3722c
    Found by: previous frame's frame pointer
 6  libxul.so!xptiInterfaceEntry::EnsureResolved(xptiWorkingSet*) + 0x2f
    eip = 0x020c25ff   esp = 0xbfc37234   ebp = 0xbfc37248
    Found by: previous frame's frame pointer
 7  libxul.so!XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) [xpcwrappednative.cpp:c3f5eaa30cdf : 2727 + 0x39]
    eip = 0x00e9eb80   esp = 0xbfc37250   ebp = 0xbfc37658
    Found by: previous frame's frame pointer
 8  libxul.so!XPC_WN_CallMethod(JSContext*, JSObject*, unsigned int, int*, int*) [xpcwrappednativejsops.cpp:c3f5eaa30cdf : 1756 + 0x15]
    eip = 0x00eabb8e   esp = 0xbfc37660   ebp = 0xbfc37738
    Found by: previous frame's frame pointer
 9  libmozjs.so!js_Invoke [jsinvoke.cpp:c3f5eaa30cdf : 1375 + 0x2f]
    eip = 0x006c03aa   esp = 0xbfc37740   ebp = 0xbfc37828
    Found by: previous frame's frame pointer
10  libmozjs.so!js_Interpret [jsops.cpp:c3f5eaa30cdf : 2309 + 0x26]
    eip = 0x006ac94e   esp = 0xbfc37830   ebp = 0xbfc37e28
    Found by: previous frame's frame pointer
11  libmozjs.so!js_Invoke [jsinvoke.cpp:c3f5eaa30cdf : 1383 + 0xa]
    eip = 0x006c03f3   esp = 0xbfc37e30   ebp = 0xbfc37f18
    Found by: previous frame's frame pointer
12  libxul.so!nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS*, unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) [xpcwrappedjsclass.cpp:c3f5eaa30cdf : 1696 + 0x2a]
    eip = 0x00e96cd4   esp = 0xbfc37f20   ebp = 0xbfc382c8
    Found by: previous frame's frame pointer
13  libxul.so!nsXPCWrappedJS::CallMethod(unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) [xpcwrappedjs.cpp:c3f5eaa30cdf : 570 + 0x35]
    eip = 0x00e8d8c1   esp = 0xbfc382d0   ebp = 0xbfc382f8
    Found by: previous frame's frame pointer
14  libxul.so!PrepareAndDispatch [xptcstubs_gcc_x86_unix.cpp:c3f5eaa30cdf : 95 + 0x3a]
    eip = 0x020c347a   esp = 0xbfc38300   ebp = 0xbfc383e8
    Found by: previous frame's frame pointer
15  libxul.so!nsThread::ProcessNextEvent(int, int*) [nsThread.cpp:c3f5eaa30cdf : 527 + 0x18]
    eip = 0x020a7139   esp = 0xbfc383f0   ebp = 0xbfc38458
    Found by: previous frame's frame pointer
16  libxul.so!NS_ProcessNextEvent_P(nsIThread*, int) [nsThreadUtils.cpp : 250 + 0x1f]
    eip = 0x0203cfac   esp = 0xbfc38460   ebp = 0xbfc38498
    Found by: previous frame's frame pointer
17  libxul.so!nsThread::Shutdown() [nsThread.cpp:c3f5eaa30cdf : 468 + 0x12]
    eip = 0x020a75da   esp = 0xbfc384a0   ebp = 0xbfc38508
    Found by: previous frame's frame pointer
18  libxul.so!nsHtml5Module::ReleaseStatics() [nsHtml5Module.cpp:c3f5eaa30cdf : 95 + 0x17]
    eip = 0x018e9b99   esp = 0xbfc38510   ebp = 0xbfc38518
    Found by: previous frame's frame pointer
19  libxul.so!nsLayoutStatics::Shutdown() [nsLayoutStatics.cpp:c3f5eaa30cdf : 380 + 0x4]
    eip = 0x0114c524   esp = 0xbfc38520   ebp = 0xbfc38528
    Found by: previous frame's frame pointer
20  libxul.so!nsLayoutStatics::Release() [nsLayoutStatics.cpp:c3f5eaa30cdf : 408 + 0x4]
    eip = 0x0114c58c   esp = 0xbfc38530   ebp = 0xbfc38548
    Found by: previous frame's frame pointer
21  libxul.so!_ZL8Shutdownv [nsLayoutModule.cpp:c3f5eaa30cdf : 386 + 0x4]
    eip = 0x01147c5d   esp = 0xbfc38550   ebp = 0xbfc38568
    Found by: previous frame's frame pointer
22  libxul.so!LayoutShutdownObserver::Observe(nsISupports*, char const*, unsigned short const*) [nsLayoutModule.cpp:c3f5eaa30cdf : 322 + 0x4]
    eip = 0x01147e10   esp = 0xbfc38570   ebp = 0xbfc38588
    Found by: previous frame's frame pointer
23  libxul.so!nsObserverList::NotifyObservers(nsISupports*, char const*, unsigned short const*) [nsObserverList.cpp:c3f5eaa30cdf : 130 + 0x34]
    eip = 0x02056351   esp = 0xbfc38590   ebp = 0xbfc385b8
    Found by: previous frame's frame pointer
24  libxul.so!nsObserverService::NotifyObservers(nsISupports*, char const*, unsigned short const*) [nsObserverService.cpp:c3f5eaa30cdf : 182 + 0x1f]
    eip = 0x0205746b   esp = 0xbfc385c0   ebp = 0xbfc385f8
    Found by: previous frame's frame pointer
25  libxul.so!mozilla::ShutdownXPCOM(nsIServiceManager*) [nsXPComInit.cpp:c3f5eaa30cdf : 688 + 0x3b]
    eip = 0x02046052   esp = 0xbfc38600   ebp = 0xbfc38678
    Found by: previous frame's frame pointer
26  libxul.so!NS_ShutdownXPCOM_P [nsXPComInit.cpp:c3f5eaa30cdf : 655 + 0xa]
    eip = 0x02046517   esp = 0xbfc38680   ebp = 0xbfc38688
    Found by: previous frame's frame pointer
27  libxpcom.so!NS_ShutdownXPCOM [nsXPComStub.cpp:c3f5eaa30cdf : 179 + 0xa]
    eip = 0x00c4c777   esp = 0xbfc38690   ebp = 0xbfc38698
    Found by: previous frame's frame pointer
28  xpcshell!main [xpcshell.cpp:c3f5eaa30cdf : 1910 + 0xb]
    eip = 0x0804e52a   esp = 0xbfc386a0   ebp = 0xbfc387a8
    Found by: previous frame's frame pointer
Severity: normal → critical
Keywords: crash
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1258584124.1258594890.29284.gz
Linux mozilla-central debug test everythingelse on 2009/11/18 14:42:04
s: moz2-linux-slave05
something must have regressed this
looking at tindeboxpushlog Phil is correct, this started after html5 push
Let's see if this patch fixes it.
Assignee: nobody → hsivonen
Status: NEW → ASSIGNED
http://hg.mozilla.org/mozilla-central/rev/2a0941f8bef9

Let's see if that's actually a fix for this...
Hmm. I guess the patch doesn't help if the orange unit test runs in the same process as one of the test cases that toggle the html5.enable pref. Does it?

Alternatively, the problem is the remaining strong ref to the main thread.
The tests that touch the pref are
parser/htmlparser/tests/mochitest/test_bug502091.html
parser/htmlparser/tests/mochitest/test_compatmode.html
content/base/test/test_bug503473.html
The third test was already disabled for other reasons.
http://hg.mozilla.org/mozilla-central/rev/f737a47efbce

Now the HTML5 parser should be out of the way for real.
Blocks: 527896
So test_browserGlue_corrupt.js and test_browserGlue_restore.js run from xpcshell, right? And xpcshell tests run during the lifetime of the same browser process as Mochitests?

Are the assumptions of the simulated shutdown and how the browser is expected to come back after the simulation documented somewhere?
so, i was bad recalling (sorry), those tests are not shutting down anything, they are creating some file in the profile folder, initializing browserGlue Service, initializing Places history service, waiting 1s (well i can probably change this since we should have a notification now), douing a couple checks...
So, this is even more strange, i thought there was more interaction with the env, but did confusion with other tests we have.

And i doubt xpcshell and mochitests could share the same process, in any way
what change did end up being effective to solve this (since looks like the crashes are gone)?
(In reply to comment #16)
> what change did end up being effective to solve this (since looks like the
> crashes are gone)?

Making the HTML5 parser not create the parser thread or hold a strong ref to the main thread until the HTML5 parser is used, and disabling mochitests that used the HTML5 parser.
(In reply to comment #17)
> Making the HTML5 parser not create the parser thread or hold a strong ref to
> the main thread until the HTML5 parser is used, and disabling mochitests that
> used the HTML5 parser.

Were all of those needed? so looks like creating the HTML 5 parser enables these crashes, anything that creates the parser.
sounds scary, and i can't understand why just these tests and not others.
is HTML5 shutdown running the event loop for any reason?
(In reply to comment #18)
> (In reply to comment #17)
> > Making the HTML5 parser not create the parser thread or hold a strong ref to
> > the main thread until the HTML5 parser is used, and disabling mochitests that
> > used the HTML5 parser.
> 
> Were all of those needed?

I don't know. It's possible that only avoiding the parser thread creation or only avoiding the strong ref to the main thread would have been sufficient.

> so looks like creating the HTML 5 parser enables
> these crashes, anything that creates the parser.
> sounds scary, and i can't understand why just these tests and not others.

Me, neither. :-(

(In reply to comment #19)
> is HTML5 shutdown running the event loop for any reason?

Not on the main thread. I don't know what nsIThread::Shutdown() does precisely for the parser thread.
the only thing i could argue here is that there is some interaction between HTML5 threads shutdown and Storage threads shutdown, i can't see anything in Places that could jump in the middle. Could be that when storage will stop running the events loop at shutdown this will be fine. That is bug 496019.
Shawn could know more about these interactions.
(In reply to comment #20)
> Not on the main thread. I don't know what nsIThread::Shutdown() does precisely
> for the parser thread.
I guess you call it from the main thread?  Calling shutdown on a thread will spin the calling thread's event loop.
(In reply to comment #22)
> (In reply to comment #20)
> > Not on the main thread. I don't know what nsIThread::Shutdown() does precisely
> > for the parser thread.
> I guess you call it from the main thread?

Yes. (On XPCOM shutdown.)

> Calling shutdown on a thread will spin the calling thread's event loop.

What would be the proper way to shut down the parser thread at app shutdown so that it doesn't disrupt these test cases?
(In reply to comment #23)
> What would be the proper way to shut down the parser thread at app shutdown so
> that it doesn't disrupt these test cases?
Per https://wiki.mozilla.org/XPCOM_Shutdown, I think you want to shutdown the thread at xpcom-shutdown-threads.
Summary: random crashes during test_browserGlue_corrupt.js and test_browserGlue_restore.js involving [@mozilla::storage::Connection::Close()] → random crashes during test_browserGlue_corrupt.js and test_browserGlue_restore.js involving [@ mozilla::storage::Connection::Close()]
Attachment #414270 - Flags: review?(sdwilsh)
Comment on attachment 414270 [details] [diff] [review]
Shut down the thread from an observer, restore previously disabled tests

r=sdwilsh
Attachment #414270 - Flags: review?(sdwilsh) → review+
Blocks: 483015
Thanks for the r. Pushed:
http://hg.mozilla.org/mozilla-central/rev/653323b08913
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla1.9.3a1
Crash Signature: [@ mozilla::storage::Connection::Close()]
Whiteboard: [orange]
You need to log in before you can comment on or make changes to this bug.