Closed Bug 855185 Opened 12 years ago Closed 11 years ago

identity/tests/chrome/test_sandbox.xul, browser_identity_UI.js | application crashed [@ PL_DHashTableEnumerate] after Assertion failure: (*(uint32_t*)(table->entryStore + ((uint32_t)1 << (32 - (table)->hashShift)) * table->entrySize)) > 0

Categories

(Core :: Networking, defect)

Product:
Core
Component:
Networking
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla23

People

(Reporter: philor, Assigned: mcmanus)

References

Details

(Keywords: assertion, crash, intermittent-failure)

Crash Data

Attachments

(2 files)

crash test v0
5.71 KB, patch
mayhemer
: review+
Details | Diff | Splinter Review
patch 0
6.80 KB, patch
mayhemer
: review+
Details | Diff | Splinter Review
https://tbpl.mozilla.org/php/getParsedLog.php?id=21136511&tree=Mozilla-Inbound Rev3 WINNT 5.1 mozilla-inbound debug test mochitest-other on 2013-03-26 18:40:15 PDT for push cee46001a526 slave: talos-r3-xp-036 19:04:04 INFO - Identity sandbox: Creating sandbox for: http://mochi.test:8888/chrome/toolkit/identity/tests/chrome/sandbox_content_alert.html 19:04:04 INFO - ++DOCSHELL 12295028 == 13 [id = 670] 19:04:04 INFO - ++DOMWINDOW == 94 (22167D28) [serial = 3395] [outer = 00000000] 19:04:04 INFO - ++DOMWINDOW == 95 (11780970) [serial = 3396] [outer = 22167D28] 19:04:04 INFO - Assertion failure: (*(uint32_t*)(table->entryStore + ((uint32_t)1 << (32 - (table)->hashShift)) * table->entrySize)) > 0, at e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/obj-firefox/xpcom/build/pldhash.cpp:751 19:04:04 INFO - nsStringStats 19:04:04 INFO - => mAllocCount: 2047304 19:04:04 INFO - => mReallocCount: 530215 19:04:04 INFO - => mFreeCount: 2011633 -- LEAKED 35671 !!! 19:04:04 INFO - => mShareCount: 5881973 19:04:04 INFO - => mAdoptCount: 115587 19:04:04 INFO - => mAdoptFreeCount: 115580 -- LEAKED 7 !!! 19:04:04 WARNING - TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/toolkit/identity/tests/chrome/test_sandbox.xul | Exited with code -2147483645 during test run 19:04:04 INFO - INFO | automation.py | Application ran for: 0:20:30.797000 19:04:04 INFO - INFO | zombiecheck | Reading PID log: c:\docume~1\cltbld\locals~1\temp\tmpq6dhgppidlog 19:04:04 INFO - ==> process 3404 launched child process 312 19:04:04 INFO - ==> process 3404 launched child process 972 19:04:04 INFO - ==> process 3404 launched child process 3584 19:04:04 INFO - ==> process 3404 launched child process 3760 19:04:04 INFO - ==> process 3404 launched child process 3828 19:04:04 INFO - ==> process 3404 launched child process 3880 19:04:04 INFO - ==> process 3404 launched child process 3940 19:04:04 INFO - ==> process 3404 launched child process 1236 19:04:04 INFO - ==> process 3404 launched child process 284 19:04:04 INFO - ==> process 3404 launched child process 3096 19:04:04 INFO - ==> process 3404 launched child process 3304 19:04:04 INFO - INFO | zombiecheck | Checking for orphan process with PID: 312 19:04:04 INFO - INFO | zombiecheck | Checking for orphan process with PID: 972 19:04:04 INFO - INFO | zombiecheck | Checking for orphan process with PID: 3584 19:04:04 INFO - INFO | zombiecheck | Checking for orphan process with PID: 3760 19:04:04 INFO - INFO | zombiecheck | Checking for orphan process with PID: 3828 19:04:04 INFO - INFO | zombiecheck | Checking for orphan process with PID: 3880 19:04:04 INFO - INFO | zombiecheck | Checking for orphan process with PID: 3940 19:04:04 INFO - INFO | zombiecheck | Checking for orphan process with PID: 1236 19:04:04 INFO - INFO | zombiecheck | Checking for orphan process with PID: 284 19:04:04 INFO - INFO | zombiecheck | Checking for orphan process with PID: 3096 19:04:04 INFO - INFO | zombiecheck | Checking for orphan process with PID: 3304 19:04:19 INFO - PROCESS-CRASH | chrome://mochitests/content/chrome/toolkit/identity/tests/chrome/test_sandbox.xul | application crashed [@ PL_DHashTableEnumerate] 19:04:19 INFO - Crash dump filename: c:\docume~1\cltbld\locals~1\temp\tmp0oiaem\minidumps\b10fa1a8-205f-49e3-927d-92d3b2bb4c3e.dmp 19:04:19 INFO - Operating system: Windows NT 19:04:19 INFO - 5.1.2600 Service Pack 2 19:04:19 INFO - CPU: x86 19:04:19 INFO - GenuineIntel family 6 model 23 stepping 10 19:04:19 INFO - 2 CPUs 19:04:19 INFO - Crash reason: EXCEPTION_BREAKPOINT 19:04:19 INFO - Crash address: 0x378dece 19:04:19 INFO - Thread 5 (crashed) 19:04:19 INFO - 0 xul.dll!PL_DHashTableEnumerate [pldhash.cpp:cee46001a526 : 751 + 0x56] 19:04:19 INFO - eip = 0x0378dece esp = 0x05eefdb8 ebp = 0x05eefdd4 ebx = 0x00000020 19:04:19 INFO - esi = 0x074f068c edi = 0x10261440 eax = 0x00000000 ecx = 0x32c2a9df 19:04:19 INFO - edx = 0x10361f48 efl = 0x00000206 19:04:19 INFO - Found by: given as instruction pointer in context 19:04:19 INFO - 1 xul.dll!nsBaseHashtable<nsCStringHashKey,nsAutoPtr<nsHttpConnectionMgr::nsConnectionEntry>,nsHttpConnectionMgr::nsConnectionEntry *>::Enumerate(PLDHashOperator (*)(nsACString_internal const &,nsAutoPtr<nsHttpConnectionMgr::nsConnectionEntry> &,void *),void *) [nsBaseHashtable.h:cee46001a526 : 223 + 0xe] 19:04:19 INFO - eip = 0x020fdbb3 esp = 0x05eefddc ebp = 0x05eefdf4 19:04:19 INFO - Found by: call frame info 19:04:19 INFO - 2 xul.dll!nsHttpConnectionMgr::OnMsgClosePersistentConnections(int,void *) [nsHttpConnectionMgr.cpp:cee46001a526 : 2171 + 0xd] 19:04:19 INFO - eip = 0x02102820 esp = 0x05eefdfc ebp = 0x05eefe14 19:04:19 INFO - Found by: call frame info 19:04:19 INFO - 3 xul.dll!nsHttpConnectionMgr::nsConnEvent::Run() [nsHttpConnectionMgr.h:cee46001a526 : 574 + 0xe] 19:04:19 INFO - eip = 0x020fe3c8 esp = 0x05eefe0c ebp = 0x05eefe14 19:04:19 INFO - Found by: call frame info 19:04:19 INFO - 4 xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:cee46001a526 : 627 + 0xd] 19:04:19 INFO - eip = 0x037e7564 esp = 0x05eefe1c ebp = 0x05eefe6c 19:04:19 INFO - Found by: call frame info 19:04:19 INFO - 5 xul.dll!NS_ProcessNextEvent_P(nsIThread *,bool) [nsThreadUtils.cpp:cee46001a526 : 238 + 0xc] 19:04:19 INFO - eip = 0x0378c684 esp = 0x05eefe74 ebp = 0x05eefe80 19:04:19 INFO - Found by: call frame info 19:04:19 INFO - 6 xul.dll!nsSocketTransportService::Run() [nsSocketTransportService2.cpp:cee46001a526 : 649 + 0x7] 19:04:19 INFO - eip = 0x02073494 esp = 0x05eefe88 ebp = 0x05eefeb4 19:04:19 INFO - Found by: call frame info 19:04:19 INFO - 7 xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:cee46001a526 : 627 + 0xd] 19:04:19 INFO - eip = 0x037e7564 esp = 0x05eefebc ebp = 0x05eeff0c 19:04:19 INFO - Found by: call frame info 19:04:19 INFO - 8 xul.dll!NS_ProcessNextEvent_P(nsIThread *,bool) [nsThreadUtils.cpp:cee46001a526 : 238 + 0xc] 19:04:19 INFO - eip = 0x0378c684 esp = 0x05eeff14 ebp = 0x05eeff20 19:04:19 INFO - Found by: call frame info 19:04:19 INFO - 9 xul.dll!nsThread::ThreadFunc(void *) [nsThread.cpp:cee46001a526 : 265 + 0x7]
Crash Signature: [@ PL_DHashTableEnumerate | nsBaseHashtable<nsCStringHashKey,nsAutoPtr<nsHttpConnectionMgr::nsConnectionEntry>,nsHttpConnectionMgr::nsConnectionEntry *>::Enumerate(PLDHashOperator (*)(nsACString_internal const &,nsAutoPtr<nsHttpConnectionMgr::nsConnection…
Blocks: 855793
https://tbpl.mozilla.org/php/getParsedLog.php?id=21231530&tree=Firefox
Summary: identity/tests/chrome/test_sandbox.xul | application crashed [@ PL_DHashTableEnumerate] after Assertion failure: (*(uint32_t*)(table->entryStore + ((uint32_t)1 << (32 - (table)->hashShift)) * table->entrySize)) > 0 → identity/tests/chrome/test_sandbox.xul, browser_identity_UI.js | application crashed [@ PL_DHashTableEnumerate] after Assertion failure: (*(uint32_t*)(table->entryStore + ((uint32_t)1 << (32 - (table)->hashShift)) * table->entrySize)) > 0
Component: Identity → Networking
Blocks: 864705
Patrick, do you have the cycles to look into this? This is probably the most frequent version of this assert that we hit and it looks networking-related.
Flags: needinfo?(mcmanus)
(In reply to Ryan VanderMeulen [:RyanVM] from comment #169) > Patrick, do you have the cycles to look into this? This is probably the most > frequent version of this assert that we hit and it looks networking-related. I'm going to take a crack at it tomorrow and see how hard it is - its not obvious to me what's going on from the stack.
Flags: needinfo?(mcmanus)
a try build with some more assertions shows this test accessing nsHttpConnectionMgr::mCT from the main thread (it is supposed to be the socket thread only). https://tbpl.mozilla.org/php/getParsedLog.php?id=22386302&tree=Try&full=1
will all be marked dups of each other: 855793 859744 855185 857303 (I won't do that until there is a resolution to keep tbpl orange easier to do)
good news - https://tbpl.mozilla.org/?tree=Try&rev=b03517eec5ce looks promising. the resetIPFamily() code that is triggered on a force reload runs on the wrong thread and creates parallel access to a hash table, which is wrong and consistent with the assertion failure. I'll cobble together a more targeted patch and we'll see if that survives the volume of inbound,
if resetIPFamily() is the source of the bug it is also consistent that OnMsgClosePersistentConnections() is always on the stack as they are basically triggered by the same event.
Attached patch crash test v0 — — Splinter Review
Attachment #743600 - Flags: review?(honzab.moz)
Attached patch patch 0 — — Splinter Review
Attachment #743602 - Flags: review?(honzab.moz)
Attachment #743600 - Flags: review?(honzab.moz) → review+
Comment on attachment 743602 [details] [diff] [review] patch 0 Review of attachment 743602 [details] [diff] [review]: ----------------------------------------------------------------- r=honzab Good catch. ::: netwerk/protocol/http/nsHttpConnectionMgr.cpp @@ +2235,5 @@ > { > LOG(("nsHttpConnectionMgr::OnMsgClosePersistentConnections\n")); > MOZ_ASSERT(PR_GetCurrentThread() == gSocketThread); > > + nsRefPtr<nsHttpConnectionInfo> ci = white space ::: netwerk/protocol/http/nsHttpConnectionMgr.h @@ +99,5 @@ > > // Close all idle persistent connections and prevent any active connections > + // from being reused. Optional connection info resets CI specific > + // information such as Happy Eyeballs history. > + nsresult ClosePersistentConnections(nsHttpConnectionInfo *); When this method does more now, the name should change, like PerformShiftReloadOperations or so (not the best name, I know).
Attachment #743602 - Flags: review?(honzab.moz) → review+
(In reply to Honza Bambas (:mayhemer) from comment #192) > @@ +99,5 @@ > > > > // Close all idle persistent connections and prevent any active connections > > + // from being reused. Optional connection info resets CI specific > > + // information such as Happy Eyeballs history. > > + nsresult ClosePersistentConnections(nsHttpConnectionInfo *); > > When this method does more now, the name should change, like I agree. Thanks! remote: https://hg.mozilla.org/integration/mozilla-inbound/rev/907ba87c01bc remote: https://hg.mozilla.org/integration/mozilla-inbound/rev/48af5d308778
the oranges in comments 198, 199, 200 are from trees that do not include the patches in comment 197
https://hg.mozilla.org/mozilla-central/rev/907ba87c01bc https://hg.mozilla.org/mozilla-central/rev/48af5d308778 Thanks a million, Patrick!
Assignee: nobody → mcmanus
Status: NEW → RESOLVED
Closed: 11 years ago
Flags: in-testsuite+
Resolution: --- → FIXED
Target Milestone: --- → mozilla23
No longer blocks: 857540
Regressions: 1548822
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: