Closed Bug 1019934 Opened 11 years ago Closed 10 years ago

Intermittent 408431-1.html | Exited with code 1 during test run (after "AddressSanitizer: heap-use-after-free PeerConnectionImpl.cpp:1832 IsClosed")

Categories

(Core :: WebRTC, defect)

Product:
Core
Component:
WebRTC
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
firefox30 --- unaffected
firefox31 --- unaffected
firefox32 --- fixed
firefox33 --- affected
firefox-esr24 --- unaffected

People

(Reporter: RyanVM, Unassigned)

References

Details

(Keywords: crash, intermittent-failure)

Attachments

(2 files)

Manually symbolized trace
9.63 KB, text/plain
Details
v1
801 bytes, patch
michal
: review+
Details | Diff | Splinter Review
This started on May 30, but was getting mis-starred under bug 1017068. https://tbpl.mozilla.org/php/getParsedLog.php?id=40980021&tree=Mozilla-Central Ubuntu ASAN VM 12.04 x64 mozilla-central opt test crashtest on 2014-06-03 13:23:28 PDT for push 298b39b50ff7 slave: tst-linux64-spot-1033 13:28:31 INFO - ==1725==ERROR: AddressSanitizer: heap-use-after-free on address 0x6130002dc890 at pc 0x7fafa762851f bp 0x7fff41bf9670 sp 0x7fff41bf9668 13:28:31 INFO - READ of size 4 at 0x6130002dc890 thread T0 13:28:31 INFO - #0 0x7fafa762851e (/builds/slave/test/build/application/firefox/libxul.so+0x24d551e) 13:28:31 INFO - #1 0x7fafa764e430 (/builds/slave/test/build/application/firefox/libxul.so+0x24fb430) 13:28:31 INFO - #2 0x7fafa6474c95 (/builds/slave/test/build/application/firefox/libxul.so+0x1321c95) 13:28:31 INFO - #3 0x7fafa6334bfa (/builds/slave/test/build/application/firefox/libxul.so+0x11e1bfa) 13:28:31 INFO - #4 0x7fafa6c81989 (/builds/slave/test/build/application/firefox/libxul.so+0x1b2e989) 13:28:31 INFO - #5 0x7fafa6c2b810 (/builds/slave/test/build/application/firefox/libxul.so+0x1ad8810) 13:28:31 INFO - #6 0x7fafa8fdd537 (/builds/slave/test/build/application/firefox/libxul.so+0x3e8a537) 13:28:31 INFO - #7 0x7fafabf03508 (/builds/slave/test/build/application/firefox/libxul.so+0x6db0508) 13:28:31 INFO - #8 0x7fafabd72403 (/builds/slave/test/build/application/firefox/libxul.so+0x6c1f403) 13:28:31 INFO - #9 0x7fafabd732e3 (/builds/slave/test/build/application/firefox/libxul.so+0x6c202e3) 13:28:31 INFO - #10 0x7fafabd7412d (/builds/slave/test/build/application/firefox/libxul.so+0x6c2112d) 13:28:31 INFO - #11 0x48a2c7 (/builds/slave/test/build/application/firefox/firefox+0x48a2c7) 13:28:31 INFO - #12 0x7fafb4f2276c (/lib/x86_64-linux-gnu/libc.so.6+0x2176c) 13:28:31 INFO - #13 0x48972c (/builds/slave/test/build/application/firefox/firefox+0x48972c) 13:28:31 INFO - 0x6130002dc890 is located 144 bytes inside of 384-byte region [0x6130002dc800,0x6130002dc980) 13:28:31 INFO - freed by thread T0 here: 13:28:31 INFO - #0 0x471b41 (/builds/slave/test/build/application/firefox/firefox+0x471b41) 13:28:31 INFO - #1 0x7fafa761890c (/builds/slave/test/build/application/firefox/libxul.so+0x24c590c) 13:28:31 INFO - #2 0x7fafa637d509 (/builds/slave/test/build/application/firefox/libxul.so+0x122a509) 13:28:31 INFO - #3 0x7fafa637e0d9 (/builds/slave/test/build/application/firefox/libxul.so+0x122b0d9) 13:28:31 INFO - #4 0x7fafa637c1c9 (/builds/slave/test/build/application/firefox/libxul.so+0x12291c9) 13:28:31 INFO - previously allocated by thread T0 here: 13:28:31 INFO - #0 0x471d41 (/builds/slave/test/build/application/firefox/firefox+0x471d41) 13:28:31 INFO - #1 0x7fafb109cbed (/builds/slave/test/build/application/firefox/libmozalloc.so+0x1bed) 13:28:31 INFO - #2 0x7fafa76403e9 (/builds/slave/test/build/application/firefox/libxul.so+0x24ed3e9) 13:28:31 INFO - #3 0x7fafa871a5a4 (/builds/slave/test/build/application/firefox/libxul.so+0x35c75a4) 13:28:31 INFO - #4 0x7fafa921c538 (/builds/slave/test/build/application/firefox/libxul.so+0x40c9538) 13:28:31 INFO - SUMMARY: AddressSanitizer: heap-use-after-free ??:0 ?? 13:28:31 INFO - Shadow bytes around the buggy address: 13:28:31 INFO - 0x0c26800538c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c26800538d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c26800538e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c26800538f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c2680053900: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 13:28:31 INFO - =>0x0c2680053910: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd 13:28:31 INFO - 0x0c2680053920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 13:28:31 INFO - 0x0c2680053930: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c2680053940: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c2680053950: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c2680053960: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - Shadow byte legend (one shadow byte represents 8 application bytes): 13:28:31 INFO - Addressable: 00 13:28:31 INFO - Partially addressable: 01 02 03 04 05 06 07 13:28:31 INFO - Heap left redzone: fa 13:28:31 INFO - Heap right redzone: fb 13:28:31 INFO - Freed heap region: fd 13:28:31 INFO - Stack left redzone: f1 13:28:31 INFO - Stack mid redzone: f2 13:28:31 INFO - Stack right redzone: f3 13:28:31 INFO - Stack partial redzone: f4 13:28:31 INFO - Stack after return: f5 13:28:31 INFO - Stack use after scope: f8 13:28:31 INFO - Global redzone: f9 13:28:31 INFO - Global init order: f6 13:28:31 INFO - Poisoned by user: f7 13:28:31 INFO - Contiguous container OOB:fc 13:28:31 INFO - ASan internal: fe 13:28:31 INFO - ==1725==ABORTING 13:28:32 INFO - TEST-INFO | Main app process: killed by SIGHUP 13:28:32 WARNING - TEST-UNEXPECTED-FAIL | file:///builds/slave/test/build/tests/reftest/tests/dom/src/offline/crashtests/408431-1.html | Exited with code 1 during test run 13:28:32 INFO - INFO | automation.py | Application ran for: 0:02:12.725001 13:28:32 INFO - INFO | zombiecheck | Reading PID log: /tmp/tmpEkaGO9pidlog 13:28:32 INFO - WARNING | leakcheck | refcount logging is off, so leaks can't be detected! 13:28:32 INFO - REFTEST INFO | runreftest.py | Running tests: end. 13:28:32 ERROR - Return code: 1
Stacks are bogus, which is bad.
Blocks: 1020584
Depends on: 1020590
Attached file Manually symbolized trace —
Here's the manually symbolized trace, hope that helps until we fix the symbolizer bug.
Looks cache-related.
Component: DOM → Networking: Cache
This seems like duplicate of bug 971980. But here we have more info. I will duplicate after checking on this bug. Thanks!
(In reply to Honza Bambas (:mayhemer) from comment #10) > This seems like duplicate of bug 971980. But here we have more info. I > will duplicate after checking on this bug. Thanks! Err... overlook. Not related... This is new.
OK, more related to bug 1011771 landed on 2014-05-28. However, the stack trace looks broken, doesn't make much sense, so hard to say for sure...
Blocks: 1011771
When looking at the stack trace at bug 1020584 I am more sure this is related to bug 1011771. Seems like there still is a problem. Michal, would you agree?
Flags: needinfo?(michal.novotny)
I don't see any similarity with bug 1011771 here. There is not much information in the stack, but it seems to me that it is more related to bug 971980.
Flags: needinfo?(michal.novotny)
(In reply to Michal Novotny (:michal) from comment #14) > I don't see any similarity with bug 1011771 here. There is not much > information in the stack, but it seems to me that it is more related to bug > 971980. That was my first thought too (overlook), but look at the stack at [1] mainly at: freed by thread T0 here: #0 0x471b41 in __interceptor_free _asan_rtl_ #1 0x7fe135896c40 in Release /builds/slave/m-in-l64-asan-0000000000000000/build/netwerk/cache2/CacheFileChunk.cpp:77 Clearly double-delete of a CacheFileChunk object, the code we have touched in bug 1011771. [1] https://bug1020584.bugzilla.mozilla.org/attachment.cgi?id=8434480
Ah!!! I had once the same mistake... I realize now. We must not access mRefCnt in Release() after DispatchRelease() call since sometimes the object can already be freed... Sorry Michal, your patch v2 was correct with return a local copy of the ref counter in Release(). Will provide a patch - one line.
Assignee: nobody → honzab.moz
Status: NEW → ASSIGNED
Attached patch v1 — — Splinter Review
Attachment #8434838 - Flags: review?(michal.novotny)
Attachment #8434838 - Flags: review?(michal.novotny) → review+
(In reply to TBPL Robot from comment #25) This is on a run that definitely postdates comment 23 :(. Decoder, can we symbolize this one to see if anything looks different?
Flags: needinfo?(choller)
Keywords: leave-open
Here you go: ==1744==ERROR: AddressSanitizer: heap-use-after-free on address 0x613000284990 at pc 0x7fd6718c963f bp 0x7fff2d6e9950 sp 0x7fff2d6e9948 READ of size 4 at 0x613000284990 thread T0 #0 0x7fd6718c963e in CheckApiState build/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp:1832 #1 0x7fd6718ef550 in Run build/media/webrtc/signaling/../../../media/mtransport/runnable_utils_generated.h:122 #2 0x7fd670712765 in ProcessNextEvent build/xpcom/threads/nsThread.cpp:766 #3 0x7fd6705d225a in NS_ProcessNextEvent build/xpcom/glue/nsThreadUtils.cpp:263 #4 0x7fd670f206d9 in Run build/ipc/glue/MessagePump.cpp:95 #5 0x7fd670eca550 in RunInternal build/ipc/chromium/src/base/message_loop.cc:229 #6 0x7fd673267df7 in Run build/widget/xpwidgets/nsBaseAppShell.cpp:164 #7 0x7fd67619f618 in Run build/toolkit/components/startup/nsAppStartup.cpp:278 #8 0x7fd67600e503 in XRE_mainRun build/toolkit/xre/nsAppRunner.cpp:4012 #9 0x7fd67600f3e6 in XRE_main build/toolkit/xre/nsAppRunner.cpp:4083 #10 0x7fd67601023d in XRE_main build/toolkit/xre/nsAppRunner.cpp:4297 #11 0x48a2c7 in do_main build/browser/app/nsBrowserApp.cpp:282 #12 0x7fd67f1be76c in __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 #13 0x48972c in _start ??:0 0x613000284990 is located 144 bytes inside of 384-byte region [0x613000284900,0x613000284a80) freed by thread T0 here: #0 0x471b41 in __interceptor_free _asan_rtl_ #1 0x7fd6718b9a2c in Release build/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp:467 #2 0x7fd67061ad49 in ReleaseSliceNow build/xpcom/base/CycleCollectedJSRuntime.cpp:1066 #3 0x7fd67061b919 in ReleaseNow build/xpcom/base/CycleCollectedJSRuntime.cpp:1138 #4 0x7fd670619829 in OnGC build/xpcom/base/CycleCollectedJSRuntime.cpp:1238 previously allocated by thread T0 here: #0 0x471d41 in __interceptor_malloc _asan_rtl_ #1 0x7fd67b338bed in moz_xmalloc build/memory/mozalloc/mozalloc.cpp:52 #2 0x7fd6718e1509 in operator new build/obj-firefox/media/webrtc/signaling/signaling_ecc/../../../../dist/include/mozilla/mozalloc.h:201 #3 0x7fd6729af474 in _constructor build/obj-firefox/dom/bindings/./PeerConnectionImplBinding.cpp:1179 #4 0x7fd6734a5b78 in construct build/js/xpconnect/wrappers/XrayWrapper.cpp:1629
Flags: needinfo?(choller)
(In reply to Christian Holler (:decoder) from comment #27) > Here you go: Interesting, that looks more like bug 1018372. Glandium landed the fix for the ASAN issues (just waiting on merge to m-c), so we can wait at this point to see how things change once we start getting useful stacks and adjust this bug accordingly.
Summary: Intermittent 408431-1.html | Exited with code 1 during test run after "AddressSanitizer: heap-use-after-free ??:0 ??" error → Intermittent 408431-1.html | Exited with code 1 during test run (after "AddressSanitizer: heap-use-after-free PeerConnectionImpl.cpp:1832 IsClosed")
Component: Networking: Cache → WebRTC
CC me again if needed.
Assignee: honzab.moz → nobody
Status: ASSIGNED → NEW
Depends on: 1021928
Assignee: nobody → honzab.moz
Status: NEW → ASSIGNED
See Also: → 1018372
Assignee: honzab.moz → nobody
No longer blocks: 1011771
Status: ASSIGNED → NEW
Fixed by backout.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WORKSFORME
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Bill, it looks like maybe those WebRTC UAFs that started happening after your patch are back...
Flags: needinfo?(wmccloskey)
Status: REOPENED → RESOLVED
Closed: 11 years ago → 10 years ago
Flags: needinfo?(wmccloskey)
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: