Closed Bug 1019934 Opened 6 years ago Closed 6 years ago
.html | Exited with code 1 during test run (after "Address Sanitizer: heap-use-after-free Peer Connection Impl .cpp:1832 Is Closed")
This started on May 30, but was getting mis-starred under bug 1017068. https://tbpl.mozilla.org/php/getParsedLog.php?id=40980021&tree=Mozilla-Central Ubuntu ASAN VM 12.04 x64 mozilla-central opt test crashtest on 2014-06-03 13:23:28 PDT for push 298b39b50ff7 slave: tst-linux64-spot-1033 13:28:31 INFO - ==1725==ERROR: AddressSanitizer: heap-use-after-free on address 0x6130002dc890 at pc 0x7fafa762851f bp 0x7fff41bf9670 sp 0x7fff41bf9668 13:28:31 INFO - READ of size 4 at 0x6130002dc890 thread T0 13:28:31 INFO - #0 0x7fafa762851e (/builds/slave/test/build/application/firefox/libxul.so+0x24d551e) 13:28:31 INFO - #1 0x7fafa764e430 (/builds/slave/test/build/application/firefox/libxul.so+0x24fb430) 13:28:31 INFO - #2 0x7fafa6474c95 (/builds/slave/test/build/application/firefox/libxul.so+0x1321c95) 13:28:31 INFO - #3 0x7fafa6334bfa (/builds/slave/test/build/application/firefox/libxul.so+0x11e1bfa) 13:28:31 INFO - #4 0x7fafa6c81989 (/builds/slave/test/build/application/firefox/libxul.so+0x1b2e989) 13:28:31 INFO - #5 0x7fafa6c2b810 (/builds/slave/test/build/application/firefox/libxul.so+0x1ad8810) 13:28:31 INFO - #6 0x7fafa8fdd537 (/builds/slave/test/build/application/firefox/libxul.so+0x3e8a537) 13:28:31 INFO - #7 0x7fafabf03508 (/builds/slave/test/build/application/firefox/libxul.so+0x6db0508) 13:28:31 INFO - #8 0x7fafabd72403 (/builds/slave/test/build/application/firefox/libxul.so+0x6c1f403) 13:28:31 INFO - #9 0x7fafabd732e3 (/builds/slave/test/build/application/firefox/libxul.so+0x6c202e3) 13:28:31 INFO - #10 0x7fafabd7412d (/builds/slave/test/build/application/firefox/libxul.so+0x6c2112d) 13:28:31 INFO - #11 0x48a2c7 (/builds/slave/test/build/application/firefox/firefox+0x48a2c7) 13:28:31 INFO - #12 0x7fafb4f2276c (/lib/x86_64-linux-gnu/libc.so.6+0x2176c) 13:28:31 INFO - #13 0x48972c (/builds/slave/test/build/application/firefox/firefox+0x48972c) 13:28:31 INFO - 0x6130002dc890 is located 144 bytes inside of 384-byte region [0x6130002dc800,0x6130002dc980) 13:28:31 INFO - freed by thread T0 here: 13:28:31 INFO - #0 0x471b41 (/builds/slave/test/build/application/firefox/firefox+0x471b41) 13:28:31 INFO - #1 0x7fafa761890c (/builds/slave/test/build/application/firefox/libxul.so+0x24c590c) 13:28:31 INFO - #2 0x7fafa637d509 (/builds/slave/test/build/application/firefox/libxul.so+0x122a509) 13:28:31 INFO - #3 0x7fafa637e0d9 (/builds/slave/test/build/application/firefox/libxul.so+0x122b0d9) 13:28:31 INFO - #4 0x7fafa637c1c9 (/builds/slave/test/build/application/firefox/libxul.so+0x12291c9) 13:28:31 INFO - previously allocated by thread T0 here: 13:28:31 INFO - #0 0x471d41 (/builds/slave/test/build/application/firefox/firefox+0x471d41) 13:28:31 INFO - #1 0x7fafb109cbed (/builds/slave/test/build/application/firefox/libmozalloc.so+0x1bed) 13:28:31 INFO - #2 0x7fafa76403e9 (/builds/slave/test/build/application/firefox/libxul.so+0x24ed3e9) 13:28:31 INFO - #3 0x7fafa871a5a4 (/builds/slave/test/build/application/firefox/libxul.so+0x35c75a4) 13:28:31 INFO - #4 0x7fafa921c538 (/builds/slave/test/build/application/firefox/libxul.so+0x40c9538) 13:28:31 INFO - SUMMARY: AddressSanitizer: heap-use-after-free ??:0 ?? 13:28:31 INFO - Shadow bytes around the buggy address: 13:28:31 INFO - 0x0c26800538c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c26800538d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c26800538e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c26800538f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c2680053900: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 13:28:31 INFO - =>0x0c2680053910: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd 13:28:31 INFO - 0x0c2680053920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 13:28:31 INFO - 0x0c2680053930: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c2680053940: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c2680053950: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - 0x0c2680053960: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 13:28:31 INFO - Shadow byte legend (one shadow byte represents 8 application bytes): 13:28:31 INFO - Addressable: 00 13:28:31 INFO - Partially addressable: 01 02 03 04 05 06 07 13:28:31 INFO - Heap left redzone: fa 13:28:31 INFO - Heap right redzone: fb 13:28:31 INFO - Freed heap region: fd 13:28:31 INFO - Stack left redzone: f1 13:28:31 INFO - Stack mid redzone: f2 13:28:31 INFO - Stack right redzone: f3 13:28:31 INFO - Stack partial redzone: f4 13:28:31 INFO - Stack after return: f5 13:28:31 INFO - Stack use after scope: f8 13:28:31 INFO - Global redzone: f9 13:28:31 INFO - Global init order: f6 13:28:31 INFO - Poisoned by user: f7 13:28:31 INFO - Contiguous container OOB:fc 13:28:31 INFO - ASan internal: fe 13:28:31 INFO - ==1725==ABORTING 13:28:32 INFO - TEST-INFO | Main app process: killed by SIGHUP 13:28:32 WARNING - TEST-UNEXPECTED-FAIL | file:///builds/slave/test/build/tests/reftest/tests/dom/src/offline/crashtests/408431-1.html | Exited with code 1 during test run 13:28:32 INFO - INFO | automation.py | Application ran for: 0:02:12.725001 13:28:32 INFO - INFO | zombiecheck | Reading PID log: /tmp/tmpEkaGO9pidlog 13:28:32 INFO - WARNING | leakcheck | refcount logging is off, so leaks can't be detected! 13:28:32 INFO - REFTEST INFO | runreftest.py | Running tests: end. 13:28:32 ERROR - Return code: 1
https://tbpl.mozilla.org/php/getParsedLog.php?id=40980557&tree=Mozilla-Inbound We also have bug 1019533 for a recent timeout in this test.
Stacks are bogus, which is bad.
Here's the manually symbolized trace, hope that helps until we fix the symbolizer bug.
Component: DOM → Networking: Cache
This seems like duplicate of bug 971980. But here we have more info. I will duplicate after checking on this bug. Thanks!
(In reply to Honza Bambas (:mayhemer) from comment #10) > This seems like duplicate of bug 971980. But here we have more info. I > will duplicate after checking on this bug. Thanks! Err... overlook. Not related... This is new.
OK, more related to bug 1011771 landed on 2014-05-28. However, the stack trace looks broken, doesn't make much sense, so hard to say for sure...
When looking at the stack trace at bug 1020584 I am more sure this is related to bug 1011771. Seems like there still is a problem. Michal, would you agree?
I don't see any similarity with bug 1011771 here. There is not much information in the stack, but it seems to me that it is more related to bug 971980.
(In reply to Michal Novotny (:michal) from comment #14) > I don't see any similarity with bug 1011771 here. There is not much > information in the stack, but it seems to me that it is more related to bug > 971980. That was my first thought too (overlook), but look at the stack at  mainly at: freed by thread T0 here: #0 0x471b41 in __interceptor_free _asan_rtl_ #1 0x7fe135896c40 in Release /builds/slave/m-in-l64-asan-0000000000000000/build/netwerk/cache2/CacheFileChunk.cpp:77 Clearly double-delete of a CacheFileChunk object, the code we have touched in bug 1011771.  https://bug1020584.bugzilla.mozilla.org/attachment.cgi?id=8434480
Ah!!! I had once the same mistake... I realize now. We must not access mRefCnt in Release() after DispatchRelease() call since sometimes the object can already be freed... Sorry Michal, your patch v2 was correct with return a local copy of the ref counter in Release(). Will provide a patch - one line.
Assignee: nobody → honzab.moz
Status: NEW → ASSIGNED
Attachment #8434838 - Flags: review?(michal.novotny) → review+
(In reply to TBPL Robot from comment #25) This is on a run that definitely postdates comment 23 :(. Decoder, can we symbolize this one to see if anything looks different?
Here you go: ==1744==ERROR: AddressSanitizer: heap-use-after-free on address 0x613000284990 at pc 0x7fd6718c963f bp 0x7fff2d6e9950 sp 0x7fff2d6e9948 READ of size 4 at 0x613000284990 thread T0 #0 0x7fd6718c963e in CheckApiState build/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp:1832 #1 0x7fd6718ef550 in Run build/media/webrtc/signaling/../../../media/mtransport/runnable_utils_generated.h:122 #2 0x7fd670712765 in ProcessNextEvent build/xpcom/threads/nsThread.cpp:766 #3 0x7fd6705d225a in NS_ProcessNextEvent build/xpcom/glue/nsThreadUtils.cpp:263 #4 0x7fd670f206d9 in Run build/ipc/glue/MessagePump.cpp:95 #5 0x7fd670eca550 in RunInternal build/ipc/chromium/src/base/message_loop.cc:229 #6 0x7fd673267df7 in Run build/widget/xpwidgets/nsBaseAppShell.cpp:164 #7 0x7fd67619f618 in Run build/toolkit/components/startup/nsAppStartup.cpp:278 #8 0x7fd67600e503 in XRE_mainRun build/toolkit/xre/nsAppRunner.cpp:4012 #9 0x7fd67600f3e6 in XRE_main build/toolkit/xre/nsAppRunner.cpp:4083 #10 0x7fd67601023d in XRE_main build/toolkit/xre/nsAppRunner.cpp:4297 #11 0x48a2c7 in do_main build/browser/app/nsBrowserApp.cpp:282 #12 0x7fd67f1be76c in __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 #13 0x48972c in _start ??:0 0x613000284990 is located 144 bytes inside of 384-byte region [0x613000284900,0x613000284a80) freed by thread T0 here: #0 0x471b41 in __interceptor_free _asan_rtl_ #1 0x7fd6718b9a2c in Release build/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp:467 #2 0x7fd67061ad49 in ReleaseSliceNow build/xpcom/base/CycleCollectedJSRuntime.cpp:1066 #3 0x7fd67061b919 in ReleaseNow build/xpcom/base/CycleCollectedJSRuntime.cpp:1138 #4 0x7fd670619829 in OnGC build/xpcom/base/CycleCollectedJSRuntime.cpp:1238 previously allocated by thread T0 here: #0 0x471d41 in __interceptor_malloc _asan_rtl_ #1 0x7fd67b338bed in moz_xmalloc build/memory/mozalloc/mozalloc.cpp:52 #2 0x7fd6718e1509 in operator new build/obj-firefox/media/webrtc/signaling/signaling_ecc/../../../../dist/include/mozilla/mozalloc.h:201 #3 0x7fd6729af474 in _constructor build/obj-firefox/dom/bindings/./PeerConnectionImplBinding.cpp:1179 #4 0x7fd6734a5b78 in construct build/js/xpconnect/wrappers/XrayWrapper.cpp:1629
(In reply to Christian Holler (:decoder) from comment #27) > Here you go: Interesting, that looks more like bug 1018372. Glandium landed the fix for the ASAN issues (just waiting on merge to m-c), so we can wait at this point to see how things change once we start getting useful stacks and adjust this bug accordingly.
Summary: Intermittent 408431-1.html | Exited with code 1 during test run after "AddressSanitizer: heap-use-after-free ??:0 ??" error → Intermittent 408431-1.html | Exited with code 1 during test run (after "AddressSanitizer: heap-use-after-free PeerConnectionImpl.cpp:1832 IsClosed")
CC me again if needed.
Assignee: honzab.moz → nobody
Status: ASSIGNED → NEW
Assignee: honzab.moz → nobody
No longer blocks: 1011771
Status: ASSIGNED → NEW
Fixed by backout.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WORKSFORME
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Bill, it looks like maybe those WebRTC UAFs that started happening after your patch are back...
Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.