Closed Bug 1291244 Opened 8 years ago Closed 8 years ago

WebRtc - Memory corruption in nr_socket_sendto()

Categories

(Core :: WebRTC, defect, P1)

Product:
Core
Component:
WebRTC
Version:
51 Branch
Platform:
x86
Windows
Type:
defect

Tracking

()

Status:
RESOLVED DUPLICATE of bug 1280443
Tracking Flags:
Tracking Status
firefox51 --- affected

People

(Reporter: loobenyang, Assigned: drno)

References

Details

(Keywords: crash, sec-critical, testcase)

Attachments

(1 file)

nr_socket_sendto.js
4.43 KB, text/plain
Details 
Steps to reproduce:
1. Run server side script nr_socket_sendto.js in Node.js (node nr_socket_sendto.js ).
2. Enter http://localhost:12345 in Firefox browser. 
(3. If it gets hang, restart Firefox and repeat step 2)
4. Firefox crashes by executing corrupted address in nr_socket_sendto():

OS: Windows 10
Firefox version: 51.0a1 (2016-08-01)

(3dbe0.3d434): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=0762e808 ebx=26048318 ecx=26534aac edx=261bc010 esi=2604800c edi=0b77f3c0
eip=0762e808 esp=0b77f1f4 ebp=0b77f20c iopl=0         nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202
0762e808 ??              ???

0:011> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


FAULTING_IP: 
+8b01e38c9f7f28a
0762e808 ??              ???

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0762e808
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000008
   Parameter[1]: 0762e808
Attempt to execute non-executable address 0762e808

CONTEXT:  00000000 -- (.cxr 0x0;r)
eax=0762e808 ebx=26048318 ecx=26534aac edx=261bc010 esi=2604800c edi=0b77f3c0
eip=0762e808 esp=0b77f1f4 ebp=0b77f20c iopl=0         nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202
0762e808 ??              ???

FAULTING_THREAD:  0003d434

PROCESS_NAME:  firefox.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_PARAMETER1:  00000008

EXCEPTION_PARAMETER2:  0762e808

WRITE_ADDRESS:  0762e808 

FOLLOWUP_IP: 
xul!nr_socket_sendto+20 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\media\mtransport\third_party\nicer\src\net\nr_socket.c @ 91]
11130d5d 83c414          add     esp,14h

FAILED_INSTRUCTION_ADDRESS: 
+e
0762e808 ??              ???

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

APP:  firefox.exe

ANALYSIS_VERSION: 6.3.9600.17029 (debuggers(dbg).140219-1702) x86fre

IP_ON_HEAP:  0762e808
The fault address in not in any loaded module, please check your build's rebase
log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may
contain the address if it were loaded.

IP_IN_RESERVED_BLOCK: 762e808

BUGCHECK_STR:  APPLICATION_FAULT_SOFTWARE_NX_FAULT_INVALID

PRIMARY_PROBLEM_CLASS:  SOFTWARE_NX_FAULT_INVALID

DEFAULT_BUCKET_ID:  SOFTWARE_NX_FAULT_INVALID

LAST_CONTROL_TRANSFER:  from 11130d5d to 0762e808

STACK_TEXT:  
WARNING: Frame IP not in any known module. Following frames may be wrong.
0b77f1f0 11130d5d 26151160 261bc010 00000064 0x762e808
0b77f20c 11132c28 00000064 00000000 2604809c xul!nr_socket_sendto+0x20
0b77f350 11132ebe 262764e8 0db1b0d8 0006ed70 xul!nr_stun_client_send_request+0x14a
0b77f36c 105c3aca 00000000 00000000 2604800c xul!nr_stun_client_timer_expired_cb+0xab
0b77f388 0fa36862 262764e8 26233740 1e6b2310 xul!mozilla::nrappkitTimerCallback::Notify+0x96
0b77f418 0fa366be 1e6b2310 02e0c348 02e08860 xul!nsTimerImpl::Fire+0x190
0b77f454 0f9e0d9f 1e6b2310 00000001 00000001 xul!nsTimerEvent::Run+0x41
0b77f4e4 0f9dfa2b 02e08860 00000001 0b77f4ff xul!nsThread::ProcessNextEvent+0x254
0b77f500 0f9e289b 02e0c34c 00000000 02e08860 xul!NS_ProcessNextEvent+0x16
0b77f6c4 0f9e0d9f 02e0c34c 02e73600 02ef5700 xul!mozilla::net::nsSocketTransportService::Run+0x27f
0b77f750 0f9dfa2b 02e08860 02ef5700 0b77f76b xul!nsThread::ProcessNextEvent+0x254
0b77f76c 0fc1f665 02e73600 02e73600 6b264140 xul!NS_ProcessNextEvent+0x16
0b77f78c 0fc1f59e 01e73600 2c75c159 02e73600 xul!mozilla::ipc::MessagePumpForNonMainThreads::Run+0x7f
0b77f7c4 0fc1f56d 02e08860 00000001 6b264100 xul!MessageLoop::RunHandler+0x20
0b77f7e4 0fc1fc28 02e0e350 02e0e350 02e18920 xul!MessageLoop::Run+0x19
0b77f80c 6b263465 02e0886c 072ee300 6b262ece xul!nsThread::ThreadFunc+0xb0
0b77f828 6b262edb 02e18920 0b77f870 6e7362a4 nss3!_PR_NativeRunThread+0xac
0b77f834 6e7362a4 02e18920 54a717aa 6e736250 nss3!pr_root+0xd
0b77f870 744238f4 072ee300 744238d0 4e963c62 ucrtbase!_crt_at_quick_exit+0x104
0b77f884 77935de3 072ee300 4d407891 00000000 KERNEL32!BaseThreadInitThunk+0x24
0b77f8cc 77935dae ffffffff 7795b7dc 00000000 ntdll!__RtlUserThreadStart+0x2f
0b77f8dc 00000000 6e736250 072ee300 00000000 ntdll!_RtlUserThreadStart+0x1b


STACK_COMMAND:  .cxr 0x0 ; kb

FAULTING_SOURCE_LINE:  c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\media\mtransport\third_party\nicer\src\net\nr_socket.c

FAULTING_SOURCE_FILE:  c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\media\mtransport\third_party\nicer\src\net\nr_socket.c

FAULTING_SOURCE_LINE_NUMBER:  91

SYMBOL_STACK_INDEX:  1

SYMBOL_NAME:  xul!nr_socket_sendto+20

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: xul

IMAGE_NAME:  xul.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  579f7f18

FAILURE_BUCKET_ID:  SOFTWARE_NX_FAULT_INVALID_c0000005_xul.dll!nr_socket_sendto

BUCKET_ID:  APPLICATION_FAULT_SOFTWARE_NX_FAULT_INVALID_BAD_IP_xul!nr_socket_sendto+20

ANALYSIS_SOURCE:  UM

FAILURE_ID_HASH_STRING:  um:software_nx_fault_invalid_c0000005_xul.dll!nr_socket_sendto

FAILURE_ID_HASH:  {6d7d7398-8caf-7374-4f84-190f193f56a1}

Followup: MachineOwner
---------
Attached file nr_socket_sendto.js — 
Flags: sec-bounty?

    
    

    
  
Looks like possible control of the IP marking as sec-critical for now. Please feel free to change if I am incorrect.
Group: core-security → media-core-security
Keywords: crash, 
          
            sec-critical, 
          
            testcase
Flags: needinfo?(rjesup)
Rank: 10
Priority: -- → P1

    
    

    
  
nils, byron - can you take a look?  Should be reproducible.
Flags: needinfo?(rjesup)
Flags: needinfo?(drno)
Flags: needinfo?(docfaraday)

    
    

    
  
I don't have a windows 10 system, and am having no luck reproducing on a windows 7 VM, linux ASAN, or OS X. Was this a standard nightly? How long does it typically take to crash?
Flags: needinfo?(docfaraday) → needinfo?(loobenyang)

    
  
See Also:  → 1293347

    
    

    
  
(In reply to Byron Campen [:bwc] from comment #4)
> I don't have a windows 10 system, and am having no luck reproducing on a
> windows 7 VM, linux ASAN, or OS X. Was this a standard nightly? How long
> does it typically take to crash?

Yes, it's official nighly daily update channel.
Typically it take a few seconds to crash if it's not hanged. If it's hanged, I restart the browser and retry and may need to try several times.
Flags: needinfo?(loobenyang)

    
    

    
  
Yeah, I've been running it continuously on my Win 7 VM all day, no luck. The stack is the same as one I've seen on crash-stats, which happens on Win 7, but it may be a different root cause. I'm getting a copy of win 10, maybe in the meantime drno or jesup can take a crack at it.

    
    

    
  
I'll try to repro tomorrow.
Flags: needinfo?(drno)

    
    

    
  
I have a crash almost instantly on win10 (Nightly).  I was unable to get it to crash in a local debug build; either due to timing (probably) or due to optimizer/code-gen.  Waiting for symbols to download to see what happened

    
    

    
  
nils - also try under linux ASAN

    
    

    
  
in nr_socket_sendto, sock is an e5e5 ptr, so loaded from freed memory.  One level up, ctx->sock shows having an 0xb691d158e5e5e5e5 object

    
    

    
  
No luck what so ever with lots of different ASAN builds on Linux.

Crashes locally for me with stock Nightly. Will try next with different variations of local builds.
Assignee: nobody → drno

    
    

    
  
I just had it repro with a debug build on Win 10. It takes a lot longer to hit the problem.
Never the less I guess logs are the only way forward. I'll try go these tomorrow.

    
    

    
  
(In reply to Randell Jesup [:jesup] from comment #10)
> in nr_socket_sendto, sock is an e5e5 ptr, so loaded from freed memory.  One
> level up, ctx->sock shows having an 0xb691d158e5e5e5e5 object

Thanks for the quick response guys. Glad both Randell and Nils reproduced it. I saw the e5e5 pattern too sometimes.

Using another test case, official Linux ASAN build did report a Use AFter Free in nr_socket_sendto():


Firefox version: 50.0a1 (2016-07-11)
=================================================================
==13533==ERROR: AddressSanitizer: heap-use-after-free on address 0x60300069e2a4 at pc 0x7f991319356d bp 0x7f98fdbffc60 sp 0x7f98fdbffc58
READ of size 8 at 0x60300069e2a4 thread T7 (Socket Thread)
    #0 0x7f991319356c in nr_socket_sendto /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/net/nr_socket.c:90:5
    #1 0x7f991319aa29 in nr_stun_client_send_request /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/stun/stun_client_ctx.c:387:10
    #2 0x7f991319cd34 in nr_stun_client_timer_expired_cb /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/stun/stun_client_ctx.c:255:5
    #3 0x7f990d22d16f in mozilla::nrappkitTimerCallback::Notify(nsITimer*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/nr_timer.cpp:123:3
    #4 0x7f990d22d27c in non-virtual thunk to mozilla::nrappkitTimerCallback::Notify(nsITimer*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/nr_timer.cpp:119:38
    #5 0x7f990b4b88d1 in nsTimerImpl::Fire() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/threads/nsTimerImpl.cpp:527:7
    #6 0x7f990b48fb7b in nsTimerEvent::Run() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/threads/TimerThread.cpp:286:3
    #7 0x7f990b49c996 in nsThread::ProcessNextEvent(bool, bool*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/threads/nsThread.cpp:1073:7
    #8 0x7f990b51ad3c in NS_ProcessNextEvent(nsIThread*, bool) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsThreadUtils.cpp:290:10
    #9 0x7f990b6e688e in mozilla::net::nsSocketTransportService::Run() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsSocketTransportService2.cpp:911:21
    #10 0x7f990b6e90bc in non-virtual thunk to mozilla::net::nsSocketTransportService::Run() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsSocketTransportService2.cpp:791:27
    #11 0x7f990b49c996 in nsThread::ProcessNextEvent(bool, bool*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/threads/nsThread.cpp:1073:7
    #12 0x7f990b51ad3c in NS_ProcessNextEvent(nsIThread*, bool) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsThreadUtils.cpp:290:10
    #13 0x7f990c262fba in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/ipc/glue/MessagePump.cpp:354:20
    #14 0x7f990c1d62e8 in RunInternal /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/ipc/chromium/src/base/message_loop.cc:235:3
    #15 0x7f990c1d62e8 in RunHandler /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/ipc/chromium/src/base/message_loop.cc:228
    #16 0x7f990c1d62e8 in MessageLoop::Run() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/ipc/chromium/src/base/message_loop.cc:208
    #17 0x7f990b497ca1 in nsThread::ThreadFunc(void*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/threads/nsThread.cpp:468:5
    #18 0x7f9922563378 in _pt_root /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/nsprpub/pr/src/pthreads/ptthread.c:216:5
    #19 0x7f9925ae4181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312
    #20 0x7f9924bd647c in clone /build/buildd/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111

0x60300069e2a4 is located 20 bytes inside of 32-byte region [0x60300069e290,0x60300069e2b0)
freed by thread T7 (Socket Thread) here:
    #0 0x4b215b in __interceptor_free /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:38:3
    #1 0x7f99131934ee in nr_socket_destroy /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/net/nr_socket.c:82:5
    #2 0x7f9913192cc4 in nr_ice_socket_close /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/ice/ice_socket.c:294:5
    #3 0x7f9913192cc4 in nr_ice_socket_destroy /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/ice/ice_socket.c:257
    #4 0x7f9913178147 in nr_ice_component_destroy /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/ice/ice_component.c:167:7
    #5 0x7f9913187407 in nr_ice_media_stream_destroy /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/ice/ice_media_stream.c:102:7
    #6 0x7f99131865cf in nr_ice_remove_media_stream /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/ice/ice_ctx.c:796:10
    #7 0x7f990d243d0f in mozilla::NrIceMediaStream::Close() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/nricemediastream.cpp:602:13
    #8 0x7f990d236e4a in mozilla::NrIceCtx::~NrIceCtx() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/nricectx.cpp:676:7
    #9 0x7f990d2374ed in mozilla::NrIceCtx::~NrIceCtx() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/nricectx.cpp:672:23
    #10 0x7f990d27d756 in Release /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/nricectx.h:333:3
    #11 0x7f990d27d756 in Release /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:39
    #12 0x7f990d27d756 in Release /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:377
    #13 0x7f990d27d756 in ~RefPtr /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:77
    #14 0x7f990d27d756 in mozilla::TransportLayerIce::~TransportLayerIce() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/transportlayerice.cpp:97
    #15 0x7f990d27d8ad in mozilla::TransportLayerIce::~TransportLayerIce() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/transportlayerice.cpp:95:41
    #16 0x7f990d25bc0c in ClearLayers /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/transportflow.cpp:57:5
    #17 0x7f990d25bc0c in mozilla::TransportFlow::DestroyFinal(nsAutoPtr<std::deque<mozilla::TransportLayer*, std::allocator<mozilla::TransportLayer*> > >) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/transportflow.cpp:45
    #18 0x7f990d25fb73 in apply<void (*)(nsAutoPtr<std::deque<mozilla::TransportLayer *, std::allocator<mozilla::TransportLayer *> > >), nsAutoPtr<std::deque<mozilla::TransportLayer *, std::allocator<mozilla::TransportLayer *> > > , 0> /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/runnable_utils.h:79:5
    #19 0x7f990d25fb73 in mozilla::runnable_args_func<void (*)(nsAutoPtr<std::deque<mozilla::TransportLayer*, std::allocator<mozilla::TransportLayer*> > >), nsAutoPtr<std::deque<mozilla::TransportLayer*, std::allocator<mozilla::TransportLayer*> > > >::Run() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/runnable_utils.h:118
    #20 0x7f990d25b593 in RunOnThreadInternal /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/runnable_utils.h:50:10
    #21 0x7f990d25b593 in RUN_ON_THREAD /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/runnable_utils.h:214
    #22 0x7f990d25b593 in mozilla::TransportFlow::~TransportFlow() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/transportflow.cpp:39
    #23 0x7f990d25bdcd in mozilla::TransportFlow::~TransportFlow() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/transportflow.cpp:23:33
    #24 0x7f990d25b202 in mozilla::TransportFlow::Release() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/transportflow.cpp:19:1
    #25 0x7f990b513abf in detail::ProxyReleaseEvent<nsISupports>::Run() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsProxyRelease.h:35:5
    #26 0x7f990b49c996 in nsThread::ProcessNextEvent(bool, bool*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/threads/nsThread.cpp:1073:7
    #27 0x7f990b51ad3c in NS_ProcessNextEvent(nsIThread*, bool) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsThreadUtils.cpp:290:10
    #28 0x7f990b6e688e in mozilla::net::nsSocketTransportService::Run() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsSocketTransportService2.cpp:911:21
    #29 0x7f990b6e90bc in non-virtual thunk to mozilla::net::nsSocketTransportService::Run() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsSocketTransportService2.cpp:791:27
    #30 0x7f990b49c996 in nsThread::ProcessNextEvent(bool, bool*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/threads/nsThread.cpp:1073:7
    #31 0x7f990b51ad3c in NS_ProcessNextEvent(nsIThread*, bool) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsThreadUtils.cpp:290:10
    #32 0x7f990c262fba in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/ipc/glue/MessagePump.cpp:354:20
    #33 0x7f990c1d62e8 in RunInternal /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/ipc/chromium/src/base/message_loop.cc:235:3
    #34 0x7f990c1d62e8 in RunHandler /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/ipc/chromium/src/base/message_loop.cc:228
    #35 0x7f990c1d62e8 in MessageLoop::Run() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/ipc/chromium/src/base/message_loop.cc:208
    #36 0x7f990b497ca1 in nsThread::ThreadFunc(void*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/threads/nsThread.cpp:468:5
    #37 0x7f9922563378 in _pt_root /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/nsprpub/pr/src/pthreads/ptthread.c:216:5
    #38 0x7f9925ae4181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312

previously allocated by thread T7 (Socket Thread) here:
    #0 0x4b247b in malloc /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:52:3
    #1 0x7f99131be596 in r_malloc /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nrappkit/src/util/libekr/r_memory.c:76:16
    #2 0x7f99131be596 in r_calloc /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nrappkit/src/util/libekr/r_memory.c:101
    #3 0x7f99131933c3 in nr_socket_create_int /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/net/nr_socket.c:48:15
    #4 0x7f990d2271f5 in nr_socket_local_create /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/nr_socket_prsock.cpp:2176:7
    #5 0x7f9913178b90 in nr_ice_component_initialize_udp /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/ice/ice_component.c:223:13
    #6 0x7f9913178b90 in nr_ice_component_initialize /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/ice/ice_component.c:636
    #7 0x7f99131877fa in nr_ice_media_stream_initialize /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/ice/ice_media_stream.c:136:12
    #8 0x7f9913185b73 in nr_ice_gather /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/ice/ice_ctx.c:750:12
    #9 0x7f990d239fe3 in mozilla::NrIceCtx::StartGathering() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/nricectx.cpp:843:11
    #10 0x7f990d16df87 in mozilla::PeerConnectionMedia::EnsureIceGathering_s() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/webrtc/signaling/src/peerconnection/PeerConnectionMedia.cpp:903:7
    #11 0x7f990d17ebc8 in apply<RefPtr<mozilla::PeerConnectionMedia>, void (mozilla::PeerConnectionMedia::*)()> /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/runnable_utils.h:102:5
    #12 0x7f990d17ebc8 in mozilla::runnable_args_memfn<RefPtr<mozilla::PeerConnectionMedia>, void (mozilla::PeerConnectionMedia::*)()>::Run() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/runnable_utils.h:169
    #13 0x7f990b49c996 in nsThread::ProcessNextEvent(bool, bool*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/threads/nsThread.cpp:1073:7
    #14 0x7f990b51ad3c in NS_ProcessNextEvent(nsIThread*, bool) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsThreadUtils.cpp:290:10
    #15 0x7f990b6e688e in mozilla::net::nsSocketTransportService::Run() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsSocketTransportService2.cpp:911:21
    #16 0x7f990b6e90bc in non-virtual thunk to mozilla::net::nsSocketTransportService::Run() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsSocketTransportService2.cpp:791:27
    #17 0x7f990b49c996 in nsThread::ProcessNextEvent(bool, bool*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/threads/nsThread.cpp:1073:7
    #18 0x7f990b51ad3c in NS_ProcessNextEvent(nsIThread*, bool) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsThreadUtils.cpp:290:10
    #19 0x7f990c262fba in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/ipc/glue/MessagePump.cpp:354:20
    #20 0x7f990c1d62e8 in RunInternal /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/ipc/chromium/src/base/message_loop.cc:235:3
    #21 0x7f990c1d62e8 in RunHandler /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/ipc/chromium/src/base/message_loop.cc:228
    #22 0x7f990c1d62e8 in MessageLoop::Run() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/ipc/chromium/src/base/message_loop.cc:208
    #23 0x7f990b497ca1 in nsThread::ThreadFunc(void*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/threads/nsThread.cpp:468:5
    #24 0x7f9922563378 in _pt_root /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/nsprpub/pr/src/pthreads/ptthread.c:216:5
    #25 0x7f9925ae4181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312

Thread T7 (Socket Thread) created by T0 here:
    #0 0x49a839 in __interceptor_pthread_create /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:238:3
    #1 0x7f992255ff3f in _PR_CreateThread /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/nsprpub/pr/src/pthreads/ptthread.c:457:14
    #2 0x7f992255fb4a in PR_CreateThread /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/nsprpub/pr/src/pthreads/ptthread.c:548:12
    #3 0x7f990b499413 in nsThread::Init() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/threads/nsThread.cpp:639:8
    #4 0x7f990b4a0b5f in nsThreadManager::NewThread(unsigned int, unsigned int, nsIThread**) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/threads/nsThreadManager.cpp:253:17
    #5 0x7f990b519d2e in NS_NewThread(nsIThread**, nsIRunnable*, unsigned int) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsThreadUtils.cpp:64:5
    #6 0x7f990b6e3d88 in nsresult NS_NewNamedThread<14ul>(char const (&) [14ul], nsIThread**, nsIRunnable*, unsigned int) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/obj-firefox/dist/include/nsThreadUtils.h:79:17
    #7 0x7f990b6e348f in mozilla::net::nsSocketTransportService::Init() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsSocketTransportService2.cpp:523:19
    #8 0x7f990c1ad1dc in nsSocketTransportServiceConstructor(nsISupports*, nsID const&, void**) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/build/nsNetModule.cpp:80:1
    #9 0x7f990b46d9f1 in nsComponentManagerImpl::CreateInstanceByContractID(char const*, nsISupports*, nsID const&, void**) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/components/nsComponentManager.cpp:1203:10
    #10 0x7f990b464e32 in nsComponentManagerImpl::GetServiceByContractID(char const*, nsID const&, void**) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/components/nsComponentManager.cpp:1559:10
    #11 0x7f990b50541a in CallGetService /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsComponentManagerUtils.cpp:67:10
    #12 0x7f990b50541a in nsGetServiceByContractIDWithError::operator()(nsID const&, void**) const /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsComponentManagerUtils.cpp:292
    #13 0x7f990b4fb870 in nsCOMPtr_base::assign_from_gs_contractid_with_error(nsGetServiceByContractIDWithError const&, nsID const&) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsCOMPtr.cpp:114:7
    #14 0x7f990b6496cf in operator= /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/obj-firefox/dist/include/nsCOMPtr.h:645:5
    #15 0x7f990b6496cf in InitializeSocketTransportService /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsIOService.cpp:297
    #16 0x7f990b6496cf in mozilla::net::nsIOService::SetOffline(bool) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsIOService.cpp:1070
    #17 0x7f990b64857d in mozilla::net::nsIOService::Init() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsIOService.cpp:264:5
    #18 0x7f990b64ae43 in mozilla::net::nsIOService::GetInstance() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsIOService.cpp:349:23
    #19 0x7f990c1acf47 in nsIOServiceConstructor(nsISupports*, nsID const&, void**) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/build/nsNetModule.cpp:62:1
    #20 0x7f990b46d9f1 in nsComponentManagerImpl::CreateInstanceByContractID(char const*, nsISupports*, nsID const&, void**) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/components/nsComponentManager.cpp:1203:10
    #21 0x7f990b464e32 in nsComponentManagerImpl::GetServiceByContractID(char const*, nsID const&, void**) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/components/nsComponentManager.cpp:1559:10
    #22 0x7f990b505381 in CallGetService /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsComponentManagerUtils.cpp:67:10
    #23 0x7f990b505381 in nsGetServiceByContractID::operator()(nsID const&, void**) const /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsComponentManagerUtils.cpp:280
    #24 0x7f990b4eab53 in assign_from_gs_contractid /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsCOMPtr.cpp:103:7
    #25 0x7f990b4eab53 in nsCOMPtr /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/glue/nsCOMPtr.h:540
    #26 0x7f990b4eab53 in mozilla::services::GetIOService() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/build/ServiceList.h:20
    #27 0x7f990b67cfa5 in do_GetIOService(nsresult*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsNetUtilInlines.h:46:33
    #28 0x7f990b67d5cf in net_EnsureIOService /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsNetUtilInlines.h:86:16
    #29 0x7f990b67d5cf in NS_NewURI(nsIURI**, nsACString_internal const&, char const*, nsIURI*, nsIIOService*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/netwerk/base/nsNetUtilInlines.h:113
    #30 0x7f990b4d5ce6 in GetManifestURI /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/chrome/nsChromeRegistryChrome.cpp:673:5
    #31 0x7f990b4d5ce6 in nsChromeRegistry::ManifestProcessingContext::ResolveURI(char const*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/chrome/nsChromeRegistryChrome.cpp:690
    #32 0x7f990b4d7263 in nsChromeRegistryChrome::ManifestLocale(nsChromeRegistry::ManifestProcessingContext&, int, char* const*, int) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/chrome/nsChromeRegistryChrome.cpp:771:31
    #33 0x7f990b477594 in ParseManifest(NSLocationType, mozilla::FileLocation&, char*, bool, bool) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/components/ManifestParser.cpp:798:7
    #34 0x7f990b468ac9 in DoRegisterManifest /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/components/nsComponentManager.cpp:625:5
    #35 0x7f990b468ac9 in nsComponentManagerImpl::RegisterManifest(NSLocationType, mozilla::FileLocation&, bool) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/components/nsComponentManager.cpp:638
    #36 0x7f990b468e43 in nsComponentManagerImpl::ManifestManifest(nsComponentManagerImpl::ManifestProcessingContext&, int, char* const*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/components/nsComponentManager.cpp:647:3
    #37 0x7f990b47785d in ParseManifest(NSLocationType, mozilla::FileLocation&, char*, bool, bool) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/components/ManifestParser.cpp:807:9
    #38 0x7f990b468ac9 in DoRegisterManifest /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/components/nsComponentManager.cpp:625:5
    #39 0x7f990b468ac9 in nsComponentManagerImpl::RegisterManifest(NSLocationType, mozilla::FileLocation&, bool) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/components/nsComponentManager.cpp:638
    #40 0x7f990b4667bc in RereadChromeManifests /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/components/nsComponentManager.cpp:833:5
    #41 0x7f990b4667bc in nsComponentManagerImpl::Init() /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/components/nsComponentManager.cpp:428
    #42 0x7f990b4ef457 in NS_InitXPCOM2 /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/xpcom/build/XPCOMInit.cpp:713:8
    #43 0x7f9913ca94c0 in Initialize /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/toolkit/xre/nsAppRunner.cpp:1411:8
    #44 0x7f9913ca94c0 in XREMain::XRE_main(int, char**, nsXREAppData const*) /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/toolkit/xre/nsAppRunner.cpp:4365
    #45 0x7f9913caa43a in XRE_main /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/toolkit/xre/nsAppRunner.cpp:4464:16
    #46 0x4dfb47 in do_main /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/browser/app/nsBrowserApp.cpp:251:10
    #47 0x4dfb47 in main /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/browser/app/nsBrowserApp.cpp:387
    #48 0x7f9924afdec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287

SUMMARY: AddressSanitizer: heap-use-after-free /builds/slave/m-cen-l64-asan-ntly-0000000000/build/src/media/mtransport/third_party/nICEr/src/net/nr_socket.c:90:5 in nr_socket_sendto
Shadow bytes around the buggy address:
  0x0c06800cbc00: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c06800cbc10: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c06800cbc20: fa fa fd fd fd fa fa fa fd fd fd fd fa fa fd fd
  0x0c06800cbc30: fd fd fa fa fd fd fd fd fa fa fd fd fd fa fa fa
  0x0c06800cbc40: fd fd fd fd fa fa fd fd fd fa fa fa fd fd fd fd
=>0x0c06800cbc50: fa fa fd fd[fd]fd fa fa fd fd fd fa fa fa fd fd
  0x0c06800cbc60: fd fa fa fa fd fd fd fd fa fa fd fd fd fd fa fa
  0x0c06800cbc70: fd fd fd fd fa fa fd fd fd fa fa fa fd fd fd fd
  0x0c06800cbc80: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd
  0x0c06800cbc90: fd fd fa fa fd fd fd fd fa fa fd fd fd fd fa fa
  0x0c06800cbca0: fd fd fd fa fa fa fd fd fd fd fa fa fd fd fd fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==13533==ABORTING

    
    

    
  
(In reply to Looben Yang from comment #13)
> (In reply to Randell Jesup [:jesup] from comment #10)
> > in nr_socket_sendto, sock is an e5e5 ptr, so loaded from freed memory.  One
> > level up, ctx->sock shows having an 0xb691d158e5e5e5e5 object
> 
> Thanks for the quick response guys. Glad both Randell and Nils reproduced
> it. I saw the e5e5 pattern too sometimes.
> 
> Using another test case, official Linux ASAN build did report a Use AFter
> Free in nr_socket_sendto():

Awesome. That looks exactly like the call stack we get for this bug. This helps a lot.
And I have an idea what might be going on here. I'll try some theories tomorrow.

    
    

    
  
Wow, I wonder how that callback is still alive after the entire ice_ctx is gone.

    
    

    
  
Actually, I might just know. This test case is also causing the check list queue to get into a loop sometimes. It might be that sometimes it is orphaning candidate pairs, which would prevent them from being destroyed when the ice_ctx is.

    
  
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → DUPLICATE

    
    

    
  
(In reply to Byron Campen [:bwc] from comment #18)
> 
> *** This bug has been marked as a duplicate of bug 1280443 ***

What is the point marking this as duplicate of an already closed bug? And the whole analysis and fix was based on this bug report? Seems more reasonable way is to mark the other bug as duplicate of this bug and check in fix with this bug.

https://bugzilla.mozilla.org/show_bug.cgi?id=1280443#c22
Flags: needinfo?(drno)

    
    

    
  
That bug is not closed, and is the older of the two. I mean, it's all arbitrary anyhow.

    
    

    
  
(In reply to Byron Campen [:bwc] from comment #20)
> That bug is not closed, and is the older of the two. I mean, it's all
> arbitrary anyhow.

If it's arbitrary then I see no reason not just working on this bug instead.
Developer develops fix based on this bug report.
Security team can assess the security severity based on this bug report.
And later QA also needs this bug report to verify the fix.
It just provide more context and trackable.

    
    

    
  
I'm not the one who closed it.
But I'm sure you contribution to fixing the issue will be considered in the bug bounty program, if that should be your concern.
Flags: needinfo?(drno)

    
    

    
  
FYI for sec-team -  I doubt we would have found the root cause of this problem anytime soon without this testcase.

    
    

    
  
fwiw the security team agrees with Looben in comment 19. There's no gain from slavishly following the "dupe to older bug" guideline when the older bug is essentially an empty crash-stats placeholder.
Flags: sec-bounty? → sec-bounty+

          
            Comment 23 is private:
          false
Group: media-core-security

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: