Closed Bug 822036 Opened 13 years ago Closed 12 years ago

Intermittent test_theme.js, test_bug394300.js, test_redirect-caching_passing_wrap.js | test failed (with xpcshell return code: -1073741819) | application crashed [@ nsCacheEntry::RemoveDescriptor(nsCacheEntryDescriptor *,bool *)] [@ PR_Unlock]

Categories

(Core :: Networking: Cache, defect)

Product:
Core
Component:
Networking: Cache
Version:
20 Branch
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla20

People

(Reporter: RyanVM, Assigned: michal)

References

Details

(Keywords: crash, intermittent-failure)

Crash Data

Attachments

(1 file)

fix
945 bytes, patch
u408661
: review+
Details | Diff | Splinter Review
https://tbpl.mozilla.org/php/getParsedLog.php?id=17968072&tree=Mozilla-Inbound Rev3 WINNT 5.1 mozilla-inbound pgo test xpcshell on 2012-12-15 00:50:11 PST for push 6b9550f406de slave: talos-r3-xp-077 PROCESS-CRASH | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\extensions\test\xpcshell-unpack\test_theme.js | application crashed [@ nsCacheEntry::RemoveDescriptor(nsCacheEntryDescriptor *,bool *)] Crash dump filename: C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\extensions\test\xpcshell-unpack\fafeb6d1-ef56-4ce9-8d55-d9a1cf25a2e4.dmp Operating system: Windows NT 5.1.2600 Service Pack 2 CPU: x86 GenuineIntel family 6 model 23 stepping 10 2 CPUs Crash reason: EXCEPTION_ACCESS_VIOLATION_READ Crash address: 0x64 Thread 13 (crashed) 0 xul.dll!nsCacheEntry::RemoveDescriptor(nsCacheEntryDescriptor *,bool *) [nsCacheEntry.cpp:6b9550f406de : 223 + 0x3] eip = 0x0063df15 esp = 0x058ffe2c ebp = 0x058ffe2c ebx = 0x0150e924 esi = 0x054f9400 edi = 0x00000000 eax = 0x00000064 ecx = 0x054f9404 edx = 0x054f9404 efl = 0x00010246 Found by: given as instruction pointer in context 1 xul.dll!nsCacheService::CloseDescriptor(nsCacheEntryDescriptor *) [nsCacheService.cpp:6b9550f406de : 2553 + 0x8] eip = 0x00637baf esp = 0x058ffe34 ebp = 0x058ffe48 Found by: call frame info 2 xul.dll!nsCacheEntryDescriptor::Close() [nsCacheEntryDescriptor.cpp:6b9550f406de : 593 + 0x6] eip = 0x005275d4 esp = 0x058ffe50 ebp = 0x058ffe68 Found by: call frame info 3 xul.dll!nsCacheEntryDescriptor::`vector deleting destructor'(unsigned int) + 0x18 eip = 0x0063c8b8 esp = 0x058ffe70 ebp = 0x058ffe7c Found by: call frame info 4 xul.dll!nsCacheEntryDescriptor::Release() [nsCacheEntryDescriptor.cpp:6b9550f406de : 74 + 0x27] eip = 0x006d9a76 esp = 0x058ffe84 ebp = 0x058ffe88 Found by: call frame info 5 xul.dll!nsCOMPtr<nsIContent>::~nsCOMPtr<nsIContent>() + 0xd eip = 0x0055dade esp = 0x058ffe90 ebp = 0x058ffe9c Found by: call frame info 6 xul.dll!nsAsyncDoomEvent::~nsAsyncDoomEvent() + 0x16 eip = 0x009edd3c esp = 0x058ffe98 ebp = 0x058ffe9c Found by: call frame info 7 xul.dll!nsAsyncDoomEvent::`vector deleting destructor'(unsigned int) + 0xb eip = 0x009edd53 esp = 0x058ffea4 ebp = 0x058ffeac Found by: call frame info 8 xul.dll!nsRunnable::Release() [nsThreadUtils.cpp:6b9550f406de : 31 + 0x28] eip = 0x006beef9 esp = 0x058ffeb4 ebp = 0x058ffeb8 Found by: call frame info 9 xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:6b9550f406de : 633 + 0x7] eip = 0x005edf55 esp = 0x058ffec0 ebp = 0x058fff24 Found by: call frame info 10 xul.dll!nsThread::ThreadFunc(void *) [nsThread.cpp:6b9550f406de : 265 + 0xd] eip = 0x00646ba9 esp = 0x058fff2c ebp = 0x058fff44 Found by: call frame info 11 nspr4.dll!_PR_NativeRunThread [pruthr.c:6b9550f406de : 395 + 0x8] eip = 0x0032272f esp = 0x058fff4c ebp = 0x058fff68 Found by: call frame info
https://tbpl.mozilla.org/php/getParsedLog.php?id=17966026&tree=Mozilla-Inbound
Crash Signature: [@ nsCacheEntry::RemoveDescriptor(nsCacheEntryDescriptor *,bool *)] → [@ nsCacheEntry::RemoveDescriptor(nsCacheEntryDescriptor *,bool *)] [@ PR_Unlock]
Summary: Intermittent test_theme.js | test failed (with xpcshell return code: -1073741819) | application crashed [@ nsCacheEntry::RemoveDescriptor(nsCacheEntryDescriptor *,bool *)] → Intermittent test_theme.js, test_bug394300.js | test failed (with xpcshell return code: -1073741819) | application crashed [@ nsCacheEntry::RemoveDescriptor(nsCacheEntryDescriptor *,bool *)] [@ PR_Unlock]
https://tbpl.mozilla.org/php/getParsedLog.php?id=17965944&tree=Mozilla-Inbound
Summary: Intermittent test_theme.js, test_bug394300.js | test failed (with xpcshell return code: -1073741819) | application crashed [@ nsCacheEntry::RemoveDescriptor(nsCacheEntryDescriptor *,bool *)] [@ PR_Unlock] → Intermittent test_theme.js, test_bug394300.js, test_redirect-caching_passing_wrap.js | test failed (with xpcshell return code: -1073741819) | application crashed [@ nsCacheEntry::RemoveDescriptor(nsCacheEntryDescriptor *,bool *)] [@ PR_Unlock]
Blocks: 822074
Component: General → Networking: Cache
Product: Toolkit → Core
Keywords: crash
Assignee: nobody → michal.novotny
Attached patch fix — — Splinter Review
Crash reason: SIGSEGV Crash address: 0x60 Thread 9 (crashed) 0 libxul.so!nsCacheEntry::RemoveDescriptor(nsCacheEntryDescriptor*, bool*) [nsCacheEntry.cpp:c6023bc4dd3b : 223 + 0x0] eip = 0x01f333ad esp = 0xb10fe070 ebp = 0x00000000 ebx = 0x02e8ba34 esi = 0xb5528130 edi = 0x00000000 eax = 0x00000001 ecx = 0xb5528134 edx = 0x00000060 efl = 0x00210286 Found by: given as instruction pointer in context 1 libxul.so!nsCacheService::CloseDescriptor(nsCacheEntryDescriptor*) [nsCacheService.cpp:c6023bc4dd3b : 2554 + 0xf] eip = 0x01f391cc esp = 0xb10fe090 ebp = 0x00000000 ebx = 0x02e8ba34 esi = 0x00000000 edi = 0xb10fe0e8 Found by: call frame info 2 libxul.so!nsCacheEntryDescriptor::Close() [nsCacheEntryDescriptor.cpp:c6023bc4dd3b : 593 + 0x4] eip = 0x01f34e03 esp = 0xb10fe0c0 ebp = 0x00000000 ebx = 0x02e8ba34 esi = 0x00000000 edi = 0xb10fe0e8 Found by: call frame info 3 libxul.so!nsCacheEntryDescriptor::~nsCacheEntryDescriptor [nsCacheEntryDescriptor.cpp:c6023bc4dd3b : 99 + 0x7] eip = 0x01f34eea esp = 0xb10fe110 ebp = 0x00000000 ebx = 0x02e8ba34 esi = 0xb5528130 edi = 0x00000000 Found by: call frame info 4 libxul.so!nsCacheEntryDescriptor::~nsCacheEntryDescriptor [nsCacheEntryDescriptor.cpp:c6023bc4dd3b : 107 + 0x7] eip = 0x01f34f3c esp = 0xb10fe130 ebp = 0x00000000 ebx = 0x02e8ba34 esi = 0xb5528130 edi = 0x00000000 Found by: call frame info 5 libxul.so!nsCacheEntryDescriptor::Release() [nsCacheEntryDescriptor.cpp:c6023bc4dd3b : 72 + 0xb] eip = 0x01f34f9e esp = 0xb10fe150 ebp = 0x00000000 ebx = 0x02e8ba34 esi = 0xb5528130 edi = 0x00000000 Found by: call frame info 6 libxul.so!nsCacheListenerEvent::Run() [nsCacheService.cpp:c6023bc4dd3b : 1846 + 0x7] eip = 0x01f35379 esp = 0xb10fe180 ebp = 0x00000000 ebx = 0x02e8ba34 esi = 0xb551bba0 edi = 0x00000000 Found by: call frame info The program crashes at offset 0xfd33ad in libxul.so where is cmp %edx,0x60(%edi) Register edi is set at the beginning of nsCacheEntry::RemoveDescriptor() and AFAICS it should be "this" pointer. So at frame 1 the method RemoveDescriptor() was called on null pointer. We check whether the cache entry is non-null in nsCacheEntryDescriptor::~nsCacheEntryDescriptor() outside the cache lock and then again in nsCacheEntryDescriptor::Close() under the cache lock. But we release the lock while we close the streams and then we grab the lock again. The only possible race condition I see is during shutdown when the background thread releases the lock in nsCacheEntryDescriptor::Close() and on the main thread we call nsCacheService::ClearDoomList() which calls nsCacheEntry::DetachDescriptors() which clears the descriptor. The fix passes tryserver: https://tbpl.mozilla.org/?tree=Try&rev=a227d7abdeff I just don't understand how the race condition can happen in case of the test test_invalidport.js. I tried to reproduce it with debugger by manual thread synchronization, but I succeeded only when I commented out dooming the entry in nsHttpChannel::CloseCacheEntry(). The problem is that nsCacheListenerEvent has to hold the last reference to the descriptor at the time it releases it in Run() method. But the xpcshell test won't finish before nsHttpChannel::OnStopRequest() is called for the last channel and the channel holds the reference until OnStopRequest() is called. nsCacheListenerEvent::Run() is called before nsHttpChannel::OnStopRequest(), although on different threads. But since the channel fails to load, the entry is asynchronously doomed in nsHttpChannel::CloseCacheEntry() and this creates nsAsyncDoomEvent which holds the reference to the descriptor. I.e. when nsCacheListenerEvent::Run() is executed on the background thread either nsHttpChannel or nsAsyncDoomEvent references the descriptor.
Attachment #695498 - Flags: review?(bsmith)
Blocks: 825525
bsmith, ping for review on this top-orange. Cheers :-)
Blocks: 808997
Attachment #695498 - Flags: review?(bsmith) → review?(hurley)
Comment on attachment 695498 [details] [diff] [review] fix Review of attachment 695498 [details] [diff] [review]: ----------------------------------------------------------------- Looks good to me
Attachment #695498 - Flags: review?(hurley) → review+
https://hg.mozilla.org/mozilla-central/rev/3aa9fdcea821
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla20
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: