Closed Bug 1019934 Opened 9 years ago Closed 9 years ago

Intermittent 408431-1.html | Exited with code 1 during test run (after "AddressSanitizer: heap-use-after-free PeerConnectionImpl.cpp:1832 IsClosed")

Categories

(Core :: WebRTC, defect)

Product:
Core
Component:
WebRTC
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
firefox30 --- unaffected
firefox31 --- unaffected
firefox32 --- fixed
firefox33 --- affected
firefox-esr24 --- unaffected

People

(Reporter: RyanVM, Unassigned)

References

Details

(Keywords: crash, intermittent-failure)

Attachments

(2 files)

Manually symbolized trace
9.63 KB, text/plain
Details
v1
801 bytes, patch
michal
: review+
Details | Diff | Splinter Review 
This started on May 30, but was getting mis-starred under bug 1017068.

https://tbpl.mozilla.org/php/getParsedLog.php?id=40980021&tree=Mozilla-Central

Ubuntu ASAN VM 12.04 x64 mozilla-central opt test crashtest on 2014-06-03 13:23:28 PDT for push 298b39b50ff7
slave: tst-linux64-spot-1033

13:28:31     INFO -  ==1725==ERROR: AddressSanitizer: heap-use-after-free on address 0x6130002dc890 at pc 0x7fafa762851f bp 0x7fff41bf9670 sp 0x7fff41bf9668
13:28:31     INFO -  READ of size 4 at 0x6130002dc890 thread T0
13:28:31     INFO -      #0 0x7fafa762851e (/builds/slave/test/build/application/firefox/libxul.so+0x24d551e)
13:28:31     INFO -      #1 0x7fafa764e430 (/builds/slave/test/build/application/firefox/libxul.so+0x24fb430)
13:28:31     INFO -      #2 0x7fafa6474c95 (/builds/slave/test/build/application/firefox/libxul.so+0x1321c95)
13:28:31     INFO -      #3 0x7fafa6334bfa (/builds/slave/test/build/application/firefox/libxul.so+0x11e1bfa)
13:28:31     INFO -      #4 0x7fafa6c81989 (/builds/slave/test/build/application/firefox/libxul.so+0x1b2e989)
13:28:31     INFO -      #5 0x7fafa6c2b810 (/builds/slave/test/build/application/firefox/libxul.so+0x1ad8810)
13:28:31     INFO -      #6 0x7fafa8fdd537 (/builds/slave/test/build/application/firefox/libxul.so+0x3e8a537)
13:28:31     INFO -      #7 0x7fafabf03508 (/builds/slave/test/build/application/firefox/libxul.so+0x6db0508)
13:28:31     INFO -      #8 0x7fafabd72403 (/builds/slave/test/build/application/firefox/libxul.so+0x6c1f403)
13:28:31     INFO -      #9 0x7fafabd732e3 (/builds/slave/test/build/application/firefox/libxul.so+0x6c202e3)
13:28:31     INFO -      #10 0x7fafabd7412d (/builds/slave/test/build/application/firefox/libxul.so+0x6c2112d)
13:28:31     INFO -      #11 0x48a2c7 (/builds/slave/test/build/application/firefox/firefox+0x48a2c7)
13:28:31     INFO -      #12 0x7fafb4f2276c (/lib/x86_64-linux-gnu/libc.so.6+0x2176c)
13:28:31     INFO -      #13 0x48972c (/builds/slave/test/build/application/firefox/firefox+0x48972c)
13:28:31     INFO -  0x6130002dc890 is located 144 bytes inside of 384-byte region [0x6130002dc800,0x6130002dc980)
13:28:31     INFO -  freed by thread T0 here:
13:28:31     INFO -      #0 0x471b41 (/builds/slave/test/build/application/firefox/firefox+0x471b41)
13:28:31     INFO -      #1 0x7fafa761890c (/builds/slave/test/build/application/firefox/libxul.so+0x24c590c)
13:28:31     INFO -      #2 0x7fafa637d509 (/builds/slave/test/build/application/firefox/libxul.so+0x122a509)
13:28:31     INFO -      #3 0x7fafa637e0d9 (/builds/slave/test/build/application/firefox/libxul.so+0x122b0d9)
13:28:31     INFO -      #4 0x7fafa637c1c9 (/builds/slave/test/build/application/firefox/libxul.so+0x12291c9)
13:28:31     INFO -  previously allocated by thread T0 here:
13:28:31     INFO -      #0 0x471d41 (/builds/slave/test/build/application/firefox/firefox+0x471d41)
13:28:31     INFO -      #1 0x7fafb109cbed (/builds/slave/test/build/application/firefox/libmozalloc.so+0x1bed)
13:28:31     INFO -      #2 0x7fafa76403e9 (/builds/slave/test/build/application/firefox/libxul.so+0x24ed3e9)
13:28:31     INFO -      #3 0x7fafa871a5a4 (/builds/slave/test/build/application/firefox/libxul.so+0x35c75a4)
13:28:31     INFO -      #4 0x7fafa921c538 (/builds/slave/test/build/application/firefox/libxul.so+0x40c9538)
13:28:31     INFO -  SUMMARY: AddressSanitizer: heap-use-after-free ??:0 ??
13:28:31     INFO -  Shadow bytes around the buggy address:
13:28:31     INFO -    0x0c26800538c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
13:28:31     INFO -    0x0c26800538d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
13:28:31     INFO -    0x0c26800538e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
13:28:31     INFO -    0x0c26800538f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
13:28:31     INFO -    0x0c2680053900: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
13:28:31     INFO -  =>0x0c2680053910: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd
13:28:31     INFO -    0x0c2680053920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
13:28:31     INFO -    0x0c2680053930: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
13:28:31     INFO -    0x0c2680053940: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
13:28:31     INFO -    0x0c2680053950: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
13:28:31     INFO -    0x0c2680053960: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
13:28:31     INFO -  Shadow byte legend (one shadow byte represents 8 application bytes):
13:28:31     INFO -    Addressable:           00
13:28:31     INFO -    Partially addressable: 01 02 03 04 05 06 07
13:28:31     INFO -    Heap left redzone:       fa
13:28:31     INFO -    Heap right redzone:      fb
13:28:31     INFO -    Freed heap region:       fd
13:28:31     INFO -    Stack left redzone:      f1
13:28:31     INFO -    Stack mid redzone:       f2
13:28:31     INFO -    Stack right redzone:     f3
13:28:31     INFO -    Stack partial redzone:   f4
13:28:31     INFO -    Stack after return:      f5
13:28:31     INFO -    Stack use after scope:   f8
13:28:31     INFO -    Global redzone:          f9
13:28:31     INFO -    Global init order:       f6
13:28:31     INFO -    Poisoned by user:        f7
13:28:31     INFO -    Contiguous container OOB:fc
13:28:31     INFO -    ASan internal:           fe
13:28:31     INFO -  ==1725==ABORTING
13:28:32     INFO -  TEST-INFO | Main app process: killed by SIGHUP
13:28:32  WARNING -  TEST-UNEXPECTED-FAIL | file:///builds/slave/test/build/tests/reftest/tests/dom/src/offline/crashtests/408431-1.html | Exited with code 1 during test run
13:28:32     INFO -  INFO | automation.py | Application ran for: 0:02:12.725001
13:28:32     INFO -  INFO | zombiecheck | Reading PID log: /tmp/tmpEkaGO9pidlog
13:28:32     INFO -  WARNING | leakcheck | refcount logging is off, so leaks can't be detected!
13:28:32     INFO -  REFTEST INFO | runreftest.py | Running tests: end.
13:28:32    ERROR - Return code: 1
Stacks are bogus, which is bad.
Blocks: 1020584
Depends on: 1020590
Attached file Manually symbolized trace — 
Here's the manually symbolized trace, hope that helps until we fix the
symbolizer bug.
Looks cache-related.
Component: DOM → Networking: Cache
This seems like duplicate of bug 971980.  But here we have more info.  I will duplicate after checking on this bug.  Thanks!
(In reply to Honza Bambas (:mayhemer) from comment #10)
> This seems like duplicate of bug 971980.  But here we have more info.  I
> will duplicate after checking on this bug.  Thanks!

Err... overlook.  Not related...  This is new.
OK, more related to bug 1011771 landed on 2014-05-28.

However, the stack trace looks broken, doesn't make much sense, so hard to say for sure...
Blocks: 1011771
When looking at the stack trace at bug 1020584 I am more sure this is related to bug 1011771.  Seems like there still is a problem.

Michal, would you agree?
Flags: needinfo?(michal.novotny)
I don't see any similarity with bug 1011771 here. There is not much information in the stack, but it seems to me that it is more related to bug 971980.
Flags: needinfo?(michal.novotny)
(In reply to Michal Novotny (:michal) from comment #14)
> I don't see any similarity with bug 1011771 here. There is not much
> information in the stack, but it seems to me that it is more related to bug
> 971980.

That was my first thought too (overlook), but look at the stack at [1] mainly at:

 freed by thread T0 here:
     #0 0x471b41 in __interceptor_free _asan_rtl_
     #1 0x7fe135896c40 in Release /builds/slave/m-in-l64-asan-0000000000000000/build/netwerk/cache2/CacheFileChunk.cpp:77

Clearly double-delete of a CacheFileChunk object, the code we have touched in bug 1011771.

[1] https://bug1020584.bugzilla.mozilla.org/attachment.cgi?id=8434480
Ah!!!  I had once the same mistake...   I realize now.   We must not access mRefCnt in Release() after DispatchRelease() call since sometimes the object can already be freed...  

Sorry Michal, your patch v2 was correct with return a local copy of the ref counter in Release().  Will provide a patch - one line.
Assignee: nobody → honzab.moz
Status: NEW → ASSIGNED
Attached patch v1 — — Splinter Review 

        Attachment #8434838 -
        Flags: review?(michal.novotny)

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

        Attachment #8434838 -
        Flags: review?(michal.novotny) → review+

    
    

    
      
  

    
    

    
      
  

    
    

    
  
(In reply to TBPL Robot from comment #25)

This is on a run that definitely postdates comment 23 :(. Decoder, can we symbolize this one to see if anything looks different?
Flags: needinfo?(choller)
Keywords: leave-open

    
    

    
  
Here you go:


==1744==ERROR: AddressSanitizer: heap-use-after-free on address 0x613000284990 at pc 0x7fd6718c963f bp 0x7fff2d6e9950 sp 0x7fff2d6e9948
READ of size 4 at 0x613000284990 thread T0
    #0 0x7fd6718c963e in CheckApiState build/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp:1832
    #1 0x7fd6718ef550 in Run build/media/webrtc/signaling/../../../media/mtransport/runnable_utils_generated.h:122
    #2 0x7fd670712765 in ProcessNextEvent build/xpcom/threads/nsThread.cpp:766
    #3 0x7fd6705d225a in NS_ProcessNextEvent build/xpcom/glue/nsThreadUtils.cpp:263
    #4 0x7fd670f206d9 in Run build/ipc/glue/MessagePump.cpp:95
    #5 0x7fd670eca550 in RunInternal build/ipc/chromium/src/base/message_loop.cc:229
    #6 0x7fd673267df7 in Run build/widget/xpwidgets/nsBaseAppShell.cpp:164
    #7 0x7fd67619f618 in Run build/toolkit/components/startup/nsAppStartup.cpp:278
    #8 0x7fd67600e503 in XRE_mainRun build/toolkit/xre/nsAppRunner.cpp:4012
    #9 0x7fd67600f3e6 in XRE_main build/toolkit/xre/nsAppRunner.cpp:4083
    #10 0x7fd67601023d in XRE_main build/toolkit/xre/nsAppRunner.cpp:4297
    #11 0x48a2c7 in do_main build/browser/app/nsBrowserApp.cpp:282
    #12 0x7fd67f1be76c in __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226
    #13 0x48972c in _start ??:0
0x613000284990 is located 144 bytes inside of 384-byte region [0x613000284900,0x613000284a80)
freed by thread T0 here:
    #0 0x471b41 in __interceptor_free _asan_rtl_
    #1 0x7fd6718b9a2c in Release build/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp:467
    #2 0x7fd67061ad49 in ReleaseSliceNow build/xpcom/base/CycleCollectedJSRuntime.cpp:1066
    #3 0x7fd67061b919 in ReleaseNow build/xpcom/base/CycleCollectedJSRuntime.cpp:1138
    #4 0x7fd670619829 in OnGC build/xpcom/base/CycleCollectedJSRuntime.cpp:1238
previously allocated by thread T0 here:
    #0 0x471d41 in __interceptor_malloc _asan_rtl_
    #1 0x7fd67b338bed in moz_xmalloc build/memory/mozalloc/mozalloc.cpp:52
    #2 0x7fd6718e1509 in operator new build/obj-firefox/media/webrtc/signaling/signaling_ecc/../../../../dist/include/mozilla/mozalloc.h:201
    #3 0x7fd6729af474 in _constructor build/obj-firefox/dom/bindings/./PeerConnectionImplBinding.cpp:1179
    #4 0x7fd6734a5b78 in construct build/js/xpconnect/wrappers/XrayWrapper.cpp:1629
Flags: needinfo?(choller)

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  
(In reply to Christian Holler (:decoder) from comment #27)
> Here you go:

Interesting, that looks more like bug 1018372. Glandium landed the fix for the ASAN issues (just waiting on merge to m-c), so we can wait at this point to see how things change once we start getting useful stacks and adjust this bug accordingly.

    
    

    
      
  

    
    

    
      
  
Summary: Intermittent 408431-1.html | Exited with code 1 during test run after "AddressSanitizer: heap-use-after-free ??:0 ??" error → Intermittent 408431-1.html | Exited with code 1 during test run (after "AddressSanitizer: heap-use-after-free PeerConnectionImpl.cpp:1832 IsClosed")
Component: Networking: Cache → WebRTC

    
    

    
  
CC me again if needed.
Assignee: honzab.moz → nobody
Status: ASSIGNED → NEW

    
    

    
      
  

    
    

    
      
  
Depends on: 1021928

    
    

    
      
  
Assignee: nobody → honzab.moz
Status: NEW → ASSIGNED

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
  
See Also:  → 1018372

    
    

    
      
  
Assignee: honzab.moz → nobody
No longer blocks: 1011771
Status: ASSIGNED → NEW

    
    

    
      
  

    
    

    
  
Fixed by backout.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME

    
    

    
      
  
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---

    
    

    
  
Bill, it looks like maybe those WebRTC UAFs that started happening after your patch are back...
Flags: needinfo?(wmccloskey)
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Flags: needinfo?(wmccloskey)
Resolution: --- → WORKSFORME

          You need to log in
          before you can comment on or make changes to this bug.