Closed Bug 855185 Opened 7 years ago Closed 6 years ago

identity/tests/chrome/test_sandbox.xul, browser_identity_UI.js | application crashed [@ PL_DHashTableEnumerate] after Assertion failure: (*(uint32_t*)(table->entryStore + ((uint32_t)1 << (32 - (table)->hashShift)) * table->entrySize)) > 0

Categories

(Core :: Networking, defect, critical)

Product:
Core
Component:
Networking
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla23

People

(Reporter: philor, Assigned: mcmanus)

References

Details

(Keywords: assertion, crash, intermittent-failure)

Crash Data

Attachments

(2 files)

crash test v0
5.71 KB, patch
mayhemer
: review+
Details | Diff | Splinter Review
patch 0
6.80 KB, patch
mayhemer
: review+
Details | Diff | Splinter Review 
https://tbpl.mozilla.org/php/getParsedLog.php?id=21136511&tree=Mozilla-Inbound
Rev3 WINNT 5.1 mozilla-inbound debug test mochitest-other on 2013-03-26 18:40:15 PDT for push cee46001a526
slave: talos-r3-xp-036

19:04:04     INFO -  Identity sandbox: Creating sandbox for: http://mochi.test:8888/chrome/toolkit/identity/tests/chrome/sandbox_content_alert.html
19:04:04     INFO -  ++DOCSHELL 12295028 == 13 [id = 670]
19:04:04     INFO -  ++DOMWINDOW == 94 (22167D28) [serial = 3395] [outer = 00000000]
19:04:04     INFO -  ++DOMWINDOW == 95 (11780970) [serial = 3396] [outer = 22167D28]
19:04:04     INFO -  Assertion failure: (*(uint32_t*)(table->entryStore + ((uint32_t)1 << (32 - (table)->hashShift)) * table->entrySize)) > 0, at e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/obj-firefox/xpcom/build/pldhash.cpp:751
19:04:04     INFO -  nsStringStats
19:04:04     INFO -   => mAllocCount:        2047304
19:04:04     INFO -   => mReallocCount:       530215
19:04:04     INFO -   => mFreeCount:         2011633  --  LEAKED 35671 !!!
19:04:04     INFO -   => mShareCount:        5881973
19:04:04     INFO -   => mAdoptCount:         115587
19:04:04     INFO -   => mAdoptFreeCount:     115580  --  LEAKED 7 !!!
19:04:04  WARNING -  TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/toolkit/identity/tests/chrome/test_sandbox.xul | Exited with code -2147483645 during test run
19:04:04     INFO -  INFO | automation.py | Application ran for: 0:20:30.797000
19:04:04     INFO -  INFO | zombiecheck | Reading PID log: c:\docume~1\cltbld\locals~1\temp\tmpq6dhgppidlog
19:04:04     INFO -  ==> process 3404 launched child process 312
19:04:04     INFO -  ==> process 3404 launched child process 972
19:04:04     INFO -  ==> process 3404 launched child process 3584
19:04:04     INFO -  ==> process 3404 launched child process 3760
19:04:04     INFO -  ==> process 3404 launched child process 3828
19:04:04     INFO -  ==> process 3404 launched child process 3880
19:04:04     INFO -  ==> process 3404 launched child process 3940
19:04:04     INFO -  ==> process 3404 launched child process 1236
19:04:04     INFO -  ==> process 3404 launched child process 284
19:04:04     INFO -  ==> process 3404 launched child process 3096
19:04:04     INFO -  ==> process 3404 launched child process 3304
19:04:04     INFO -  INFO | zombiecheck | Checking for orphan process with PID: 312
19:04:04     INFO -  INFO | zombiecheck | Checking for orphan process with PID: 972
19:04:04     INFO -  INFO | zombiecheck | Checking for orphan process with PID: 3584
19:04:04     INFO -  INFO | zombiecheck | Checking for orphan process with PID: 3760
19:04:04     INFO -  INFO | zombiecheck | Checking for orphan process with PID: 3828
19:04:04     INFO -  INFO | zombiecheck | Checking for orphan process with PID: 3880
19:04:04     INFO -  INFO | zombiecheck | Checking for orphan process with PID: 3940
19:04:04     INFO -  INFO | zombiecheck | Checking for orphan process with PID: 1236
19:04:04     INFO -  INFO | zombiecheck | Checking for orphan process with PID: 284
19:04:04     INFO -  INFO | zombiecheck | Checking for orphan process with PID: 3096
19:04:04     INFO -  INFO | zombiecheck | Checking for orphan process with PID: 3304
19:04:19     INFO -  PROCESS-CRASH | chrome://mochitests/content/chrome/toolkit/identity/tests/chrome/test_sandbox.xul | application crashed [@ PL_DHashTableEnumerate]
19:04:19     INFO -  Crash dump filename: c:\docume~1\cltbld\locals~1\temp\tmp0oiaem\minidumps\b10fa1a8-205f-49e3-927d-92d3b2bb4c3e.dmp
19:04:19     INFO -  Operating system: Windows NT
19:04:19     INFO -                    5.1.2600 Service Pack 2
19:04:19     INFO -  CPU: x86
19:04:19     INFO -       GenuineIntel family 6 model 23 stepping 10
19:04:19     INFO -       2 CPUs
19:04:19     INFO -  Crash reason:  EXCEPTION_BREAKPOINT
19:04:19     INFO -  Crash address: 0x378dece
19:04:19     INFO -  Thread 5 (crashed)
19:04:19     INFO -   0  xul.dll!PL_DHashTableEnumerate [pldhash.cpp:cee46001a526 : 751 + 0x56]
19:04:19     INFO -      eip = 0x0378dece   esp = 0x05eefdb8   ebp = 0x05eefdd4   ebx = 0x00000020
19:04:19     INFO -      esi = 0x074f068c   edi = 0x10261440   eax = 0x00000000   ecx = 0x32c2a9df
19:04:19     INFO -      edx = 0x10361f48   efl = 0x00000206
19:04:19     INFO -      Found by: given as instruction pointer in context
19:04:19     INFO -   1  xul.dll!nsBaseHashtable<nsCStringHashKey,nsAutoPtr<nsHttpConnectionMgr::nsConnectionEntry>,nsHttpConnectionMgr::nsConnectionEntry *>::Enumerate(PLDHashOperator (*)(nsACString_internal const &,nsAutoPtr<nsHttpConnectionMgr::nsConnectionEntry> &,void *),void *) [nsBaseHashtable.h:cee46001a526 : 223 + 0xe]
19:04:19     INFO -      eip = 0x020fdbb3   esp = 0x05eefddc   ebp = 0x05eefdf4
19:04:19     INFO -      Found by: call frame info
19:04:19     INFO -   2  xul.dll!nsHttpConnectionMgr::OnMsgClosePersistentConnections(int,void *) [nsHttpConnectionMgr.cpp:cee46001a526 : 2171 + 0xd]
19:04:19     INFO -      eip = 0x02102820   esp = 0x05eefdfc   ebp = 0x05eefe14
19:04:19     INFO -      Found by: call frame info
19:04:19     INFO -   3  xul.dll!nsHttpConnectionMgr::nsConnEvent::Run() [nsHttpConnectionMgr.h:cee46001a526 : 574 + 0xe]
19:04:19     INFO -      eip = 0x020fe3c8   esp = 0x05eefe0c   ebp = 0x05eefe14
19:04:19     INFO -      Found by: call frame info
19:04:19     INFO -   4  xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:cee46001a526 : 627 + 0xd]
19:04:19     INFO -      eip = 0x037e7564   esp = 0x05eefe1c   ebp = 0x05eefe6c
19:04:19     INFO -      Found by: call frame info
19:04:19     INFO -   5  xul.dll!NS_ProcessNextEvent_P(nsIThread *,bool) [nsThreadUtils.cpp:cee46001a526 : 238 + 0xc]
19:04:19     INFO -      eip = 0x0378c684   esp = 0x05eefe74   ebp = 0x05eefe80
19:04:19     INFO -      Found by: call frame info
19:04:19     INFO -   6  xul.dll!nsSocketTransportService::Run() [nsSocketTransportService2.cpp:cee46001a526 : 649 + 0x7]
19:04:19     INFO -      eip = 0x02073494   esp = 0x05eefe88   ebp = 0x05eefeb4
19:04:19     INFO -      Found by: call frame info
19:04:19     INFO -   7  xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:cee46001a526 : 627 + 0xd]
19:04:19     INFO -      eip = 0x037e7564   esp = 0x05eefebc   ebp = 0x05eeff0c
19:04:19     INFO -      Found by: call frame info
19:04:19     INFO -   8  xul.dll!NS_ProcessNextEvent_P(nsIThread *,bool) [nsThreadUtils.cpp:cee46001a526 : 238 + 0xc]
19:04:19     INFO -      eip = 0x0378c684   esp = 0x05eeff14   ebp = 0x05eeff20
19:04:19     INFO -      Found by: call frame info
19:04:19     INFO -   9  xul.dll!nsThread::ThreadFunc(void *) [nsThread.cpp:cee46001a526 : 265 + 0x7]
Crash Signature: [@ PL_DHashTableEnumerate | nsBaseHashtable<nsCStringHashKey,nsAutoPtr<nsHttpConnectionMgr::nsConnectionEntry>,nsHttpConnectionMgr::nsConnectionEntry *>::Enumerate(PLDHashOperator (*)(nsACString_internal const &,nsAutoPtr<nsHttpConnectionMgr::nsConnection…
Blocks: 855793
https://tbpl.mozilla.org/php/getParsedLog.php?id=21231530&tree=Firefox
Summary: identity/tests/chrome/test_sandbox.xul | application crashed [@ PL_DHashTableEnumerate] after Assertion failure: (*(uint32_t*)(table->entryStore + ((uint32_t)1 << (32 - (table)->hashShift)) * table->entrySize)) > 0 → identity/tests/chrome/test_sandbox.xul, browser_identity_UI.js | application crashed [@ PL_DHashTableEnumerate] after Assertion failure: (*(uint32_t*)(table->entryStore + ((uint32_t)1 << (32 - (table)->hashShift)) * table->entrySize)) > 0
Component: Identity → Networking
Blocks: 864705
Patrick, do you have the cycles to look into this? This is probably the most frequent version of this assert that we hit and it looks networking-related.
Flags: needinfo?(mcmanus)
(In reply to Ryan VanderMeulen [:RyanVM] from comment #169)
> Patrick, do you have the cycles to look into this? This is probably the most
> frequent version of this assert that we hit and it looks networking-related.

I'm going to take a crack at it tomorrow and see how hard it is - its not obvious to me what's going on from the stack.
Flags: needinfo?(mcmanus)
a try build with some more assertions shows this test accessing nsHttpConnectionMgr::mCT from the main thread (it is supposed to be the socket thread only).

https://tbpl.mozilla.org/php/getParsedLog.php?id=22386302&tree=Try&full=1
will all be marked dups of each other: 855793 859744 855185 857303

(I won't do that until there is a resolution to keep tbpl orange easier to do)
good news - https://tbpl.mozilla.org/?tree=Try&rev=b03517eec5ce looks promising.

the resetIPFamily() code that is triggered on a force reload runs on the wrong thread and creates parallel access to a hash table, which is wrong and consistent with the assertion failure.

I'll cobble together a more targeted patch and we'll see if that survives the volume of inbound,
if resetIPFamily() is the source of the bug it is also consistent that OnMsgClosePersistentConnections() is always on the stack as they are basically triggered by the same event.
Attached patch crash test v0 — — Splinter Review
Attachment #743600 - Flags: review?(honzab.moz)
Attached patch patch 0 — — Splinter Review
Attachment #743602 - Flags: review?(honzab.moz)
Attachment #743600 - Flags: review?(honzab.moz) → review+
Comment on attachment 743602 [details] [diff] [review]
patch 0

Review of attachment 743602 [details] [diff] [review]:
-----------------------------------------------------------------

r=honzab

Good catch.

::: netwerk/protocol/http/nsHttpConnectionMgr.cpp
@@ +2235,5 @@
>  {
>      LOG(("nsHttpConnectionMgr::OnMsgClosePersistentConnections\n"));
>      MOZ_ASSERT(PR_GetCurrentThread() == gSocketThread);
>  
> +    nsRefPtr<nsHttpConnectionInfo> ci = 

white space

::: netwerk/protocol/http/nsHttpConnectionMgr.h
@@ +99,5 @@
>  
>      // Close all idle persistent connections and prevent any active connections
> +    // from being reused. Optional connection info resets CI specific
> +    // information such as Happy Eyeballs history.
> +    nsresult ClosePersistentConnections(nsHttpConnectionInfo *);

When this method does more now, the name should change, like PerformShiftReloadOperations or so (not the best name, I know).
Attachment #743602 - Flags: review?(honzab.moz) → review+
(In reply to Honza Bambas (:mayhemer) from comment #192)
> @@ +99,5 @@
> >  
> >      // Close all idle persistent connections and prevent any active connections
> > +    // from being reused. Optional connection info resets CI specific
> > +    // information such as Happy Eyeballs history.
> > +    nsresult ClosePersistentConnections(nsHttpConnectionInfo *);
> 
> When this method does more now, the name should change, like

I agree. Thanks!

remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/907ba87c01bc
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/48af5d308778
the oranges in comments 198, 199, 200 are from trees that do not include the patches in comment 197
https://hg.mozilla.org/mozilla-central/rev/907ba87c01bc
https://hg.mozilla.org/mozilla-central/rev/48af5d308778

Thanks a million, Patrick!
Assignee: nobody → mcmanus
Status: NEW → RESOLVED
Closed: 6 years ago
Flags: in-testsuite+
Resolution: --- → FIXED
Target Milestone: --- → mozilla23
No longer blocks: 857540
Regressions: 1548822
You need to log in before you can comment on or make changes to this bug.