Closed Bug 1270752 Opened 4 years ago Closed 4 years ago

AddressSanitizer: heap-use-after-free /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:509 __interceptor_strlen

Categories

(Core :: IPC, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
mozilla49
Tracking Status
firefox49 --- fixed

People

(Reporter: jya, Assigned: mayhemer)

References

Details

(Keywords: csectype-uaf, Whiteboard: btpp-active)

Attachments

(1 file, 1 obsolete file)

https://treeherder.mozilla.org/logviewer.html#?job_id=20451930&repo=try#L2572

STR:
* Modify testing/mochitest/runtests.py so that some verbose mochitest are set:
+MOZ_LOG_MODULES = "MediaFormatReader:5"
* Run try with mochitest-media-e10s for linux64-asan

You get crashes:
 23:46:11     INFO -  ==10299==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d000020508 at pc 0x4609ce bp 0x7f0e167b6940 sp 0x7f0e167b6920
 23:46:11     INFO -  READ of size 123 at 0x60d000020508 thread T4 (Gecko_IOThread)
 23:46:11     INFO -      #0 0x4609cd in __interceptor_strlen /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:509
 23:46:11     INFO -      #1 0x7f0e3874b98c in PR_DuplicateEnvironment /builds/slave/try-l64-asan-00000000000000000/build/src/nsprpub/pr/src/misc/prenv.c:135
 23:46:12     INFO -      #2 0x7f0e1e0fc1d0 in EnvironmentEnvp /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/process_util_linux.cc:69
 23:46:12     INFO -      #3 0x7f0e1e0fc1d0 in Environment /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/process_util_linux.cc:127
 23:46:12     INFO -      #4 0x7f0e1e0fc1d0 in base::LaunchApp(std::vector<std::string, std::allocator<std::string> > const&, std::vector<std::pair<int, int>, std::allocator<std::pair<int, int> > > const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, base::ChildPrivileges, bool, int*, base::ProcessArchitecture) /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/process_util_linux.cc:234
 23:46:12     INFO -      #5 0x7f0e1e16d69d in mozilla::ipc::GeckoChildProcessHost::PerformAsyncLaunchInternal(std::vector<std::string, std::allocator<std::string> >&, base::ProcessArchitecture) /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/glue/GeckoChildProcessHost.cpp:873
 23:46:12     INFO -      #6 0x7f0e1e16be36 in mozilla::ipc::GeckoChildProcessHost::PerformAsyncLaunch(std::vector<std::string, std::allocator<std::string> >, base::ProcessArchitecture) /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/glue/GeckoChildProcessHost.cpp:545
 23:46:12     INFO -      #7 0x7f0e1e16a3d3 in mozilla::ipc::GeckoChildProcessHost::RunPerformAsyncLaunch(std::vector<std::string, std::allocator<std::string> >, base::ProcessArchitecture) /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/glue/GeckoChildProcessHost.cpp:563
 23:46:12     INFO -      #8 0x7f0e1e1712aa in _ZN25nsRunnableMethodArgumentsIJSt6vectorISsSaISsEEN4base19ProcessArchitectureEEE9applyImplIN7mozilla3ipc21GeckoChildProcessHostEMS9_FbS2_S4_EJ20StoreCopyPassByValueIS2_ESC_IS4_EEJLm0ELm1EEEEDTcldsdefp_fp0_spcldtclsr7mozillaE3GetIXT2_EEfp1_E15PassAsParameterEEEPT_T0_RNS7_5TupleIJDpT1_EEENS7_13IndexSequenceIJXspT2_EEEE /builds/slave/try-l64-asan-00000000000000000/build/src/obj-firefox/dist/include/nsThreadUtils.h:707
 23:46:12     INFO -      #9 0x7f0e1e170bbb in apply<mozilla::ipc::GeckoChildProcessHost, bool (mozilla::ipc::GeckoChildProcessHost::*)(std::vector<std::basic_string<char>, std::allocator<std::basic_string<char> > >, base::ProcessArchitecture)> /builds/slave/try-l64-asan-00000000000000000/build/src/obj-firefox/dist/include/nsThreadUtils.h:713
 23:46:12     INFO -      #10 0x7f0e1e170bbb in nsRunnableMethodImpl<bool (mozilla::ipc::GeckoChildProcessHost::*)(std::vector<std::string, std::allocator<std::string> >, base::ProcessArchitecture), false, false, std::vector<std::string, std::allocator<std::string> >, base::ProcessArchitecture>::Run() /builds/slave/try-l64-asan-00000000000000000/build/src/obj-firefox/dist/include/nsThreadUtils.h:741
 23:46:12     INFO -      #11 0x7f0e1e117cfc in forget /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/message_loop.cc:335
 23:46:12     INFO -      #12 0x7f0e1e117cfc in DeferOrRunPendingTask /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/message_loop.cc:343
 23:46:12     INFO -      #13 0x7f0e1e117cfc in MessageLoop::DoWork() /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/message_loop.cc:418
 23:46:12     INFO -      #14 0x7f0e1e11addc in base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/message_pump_libevent.cc:341
 23:46:12     INFO -      #15 0x7f0e1e1153dc in RunInternal /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/message_loop.cc:233
 23:46:12     INFO -      #16 0x7f0e1e1153dc in RunHandler /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/message_loop.cc:226
 23:46:12     INFO -      #17 0x7f0e1e1153dc in MessageLoop::Run() /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/message_loop.cc:206
 23:46:12     INFO -      #18 0x7f0e1e12d0b5 in base::Thread::ThreadMain() /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/thread.cc:178
 23:46:12     INFO -      #19 0x7f0e1e12e85c in ThreadFunc(void*) /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/platform_thread_posix.cc:36
 23:46:12     INFO -      #20 0x7f0e3bc92e99 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7e99)
 23:46:12     INFO -      #21 0x7f0e3ada238c (/lib/x86_64-linux-gnu/libc.so.6+0xf338c)
 23:46:12     INFO -  0x60d000020508 is located 8 bytes inside of 135-byte region [0x60d000020500,0x60d000020587)
 23:46:12     INFO -  freed by thread T4 (Gecko_IOThread) here:
 23:46:12     INFO -      #0 0x471fe1 in __interceptor_free /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:64
 23:46:12     INFO -      #1 0x7f0e1e16be23 in ~nsACString_internal /builds/slave/try-l64-asan-00000000000000000/build/src/obj-firefox/dist/include/nsTSubstring.h:95
 23:46:12     INFO -      #2 0x7f0e1e16be23 in SetChildLogName /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/glue/GeckoChildProcessHost.cpp:510
 23:46:12     INFO -      #3 0x7f0e1e16be23 in mozilla::ipc::GeckoChildProcessHost::PerformAsyncLaunch(std::vector<std::string, std::allocator<std::string> >, base::ProcessArchitecture) /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/glue/GeckoChildProcessHost.cpp:542
 23:46:12     INFO -      #4 0x7f0e1e16a3d3 in mozilla::ipc::GeckoChildProcessHost::RunPerformAsyncLaunch(std::vector<std::string, std::allocator<std::string> >, base::ProcessArchitecture) /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/glue/GeckoChildProcessHost.cpp:563
 23:46:12     INFO -      #5 0x7f0e1e1712aa in _ZN25nsRunnableMethodArgumentsIJSt6vectorISsSaISsEEN4base19ProcessArchitectureEEE9applyImplIN7mozilla3ipc21GeckoChildProcessHostEMS9_FbS2_S4_EJ20StoreCopyPassByValueIS2_ESC_IS4_EEJLm0ELm1EEEEDTcldsdefp_fp0_spcldtclsr7mozillaE3GetIXT2_EEfp1_E15PassAsParameterEEEPT_T0_RNS7_5TupleIJDpT1_EEENS7_13IndexSequenceIJXspT2_EEEE /builds/slave/try-l64-asan-00000000000000000/build/src/obj-firefox/dist/include/nsThreadUtils.h:707
 23:46:12     INFO -      #6 0x7f0e1e170bbb in apply<mozilla::ipc::GeckoChildProcessHost, bool (mozilla::ipc::GeckoChildProcessHost::*)(std::vector<std::basic_string<char>, std::allocator<std::basic_string<char> > >, base::ProcessArchitecture)> /builds/slave/try-l64-asan-00000000000000000/build/src/obj-firefox/dist/include/nsThreadUtils.h:713
 23:46:12     INFO -      #7 0x7f0e1e170bbb in nsRunnableMethodImpl<bool (mozilla::ipc::GeckoChildProcessHost::*)(std::vector<std::string, std::allocator<std::string> >, base::ProcessArchitecture), false, false, std::vector<std::string, std::allocator<std::string> >, base::ProcessArchitecture>::Run() /builds/slave/try-l64-asan-00000000000000000/build/src/obj-firefox/dist/include/nsThreadUtils.h:741
 23:46:12     INFO -  previously allocated by thread T4 (Gecko_IOThread) here:
 23:46:12     INFO -      #0 0x4721e1 in malloc /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:74
 23:46:12     INFO -      #1 0x7f0e1d2d0724 in Alloc /builds/slave/try-l64-asan-00000000000000000/build/src/xpcom/string/nsSubstring.cpp:217
 23:46:12     INFO -      #2 0x7f0e1d2d0724 in nsACString_internal::MutatePrep(unsigned int, char**, unsigned int*) /builds/slave/try-l64-asan-00000000000000000/build/src/xpcom/string/nsTSubstring.cpp:133
 23:46:12     INFO -      #3 0x7f0e1d2d99ee in nsACString_internal::ReplacePrepInternal(unsigned int, unsigned int, unsigned int, unsigned int) /builds/slave/try-l64-asan-00000000000000000/build/src/xpcom/string/nsTSubstring.cpp:195
 23:46:12     INFO -      #4 0x7f0e1d2d98d2 in nsACString_internal::ReplacePrep(unsigned int, unsigned int, unsigned int) /builds/slave/try-l64-asan-00000000000000000/build/src/xpcom/string/nsTSubstring.cpp:185
 23:46:13     INFO -      #5 0x7f0e1d2dc3b7 in nsACString_internal::Replace(unsigned int, unsigned int, char const*, unsigned int, mozilla::fallible_t const&) /builds/slave/try-l64-asan-00000000000000000/build/src/xpcom/string/nsTSubstring.cpp:562
 23:46:13     INFO -  Thread T4 (Gecko_IOThread) created by T0 here:
 23:46:13     INFO -      #0 0x45ea55 in __interceptor_pthread_create /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:175
 23:46:13     INFO -      #1 0x7f0e1e12cc74 in CreateThread /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/platform_thread_posix.cc:135
 23:46:13     INFO -      #2 0x7f0e1e12cc74 in Create /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/platform_thread_posix.cc:146
 23:46:13     INFO -      #3 0x7f0e1e12cc74 in base::Thread::StartWithOptions(base::Thread::Options const&) /builds/slave/try-l64-asan-00000000000000000/build/src/ipc/chromium/src/base/thread.cc:96
 23:46:13     INFO -      #4 0x7f0e1d475078 in NS_InitXPCOM2 /builds/slave/try-l64-asan-00000000000000000/build/src/xpcom/build/XPCOMInit.cpp:546
 23:46:13     INFO -      #5 0x7f0e256d2598 in Initialize /builds/slave/try-l64-asan-00000000000000000/build/src/toolkit/xre/nsAppRunner.cpp:1532
 23:46:13     INFO -      #6 0x7f0e256d2598 in XREMain::XRE_main(int, char**, nsXREAppData const*) /builds/slave/try-l64-asan-00000000000000000/build/src/toolkit/xre/nsAppRunner.cpp:4447
 23:46:13     INFO -      #7 0x7f0e256d351e in XRE_main /builds/slave/try-l64-asan-00000000000000000/build/src/toolkit/xre/nsAppRunner.cpp:4559
 23:46:13     INFO -      #8 0x48a793 in do_main /builds/slave/try-l64-asan-00000000000000000/build/src/browser/app/nsBrowserApp.cpp:220
 23:46:13     INFO -      #9 0x48a793 in main /builds/slave/try-l64-asan-00000000000000000/build/src/browser/app/nsBrowserApp.cpp:360
 23:46:13     INFO -      #10 0x7f0e3acd076c (/lib/x86_64-linux-gnu/libc.so.6+0x2176c)
 23:46:13     INFO -  SUMMARY: AddressSanitizer: heap-use-after-free /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:509 __interceptor_strlen
 23:46:13     INFO -  Shadow bytes around the buggy address:
 23:46:13     INFO -    0x0c1a7fffc050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 23:46:13     INFO -    0x0c1a7fffc060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 23:46:13     INFO -    0x0c1a7fffc070: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 23:46:13     INFO -    0x0c1a7fffc080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 23:46:13     INFO -    0x0c1a7fffc090: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 23:46:13     INFO -  =>0x0c1a7fffc0a0: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 23:46:13     INFO -    0x0c1a7fffc0b0: fd fa fa fa fa fa fa fa fa fa 00 00 00 00 00 00
 23:46:13     INFO -    0x0c1a7fffc0c0: 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa
 23:46:13     INFO -    0x0c1a7fffc0d0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
 23:46:13     INFO -    0x0c1a7fffc0e0: 00 00 00 00 00 fa fa fa fa fa fa fa fa fa 00 00
 23:46:13     INFO -    0x0c1a7fffc0f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa
 23:46:13     INFO -  Shadow byte legend (one shadow byte represents 8 application bytes):
 23:46:13     INFO -    Addressable:           00
 23:46:13     INFO -    Partially addressable: 01 02 03 04 05 06 07
 23:46:13     INFO -    Heap left redzone:       fa
 23:46:13     INFO -    Heap right redzone:      fb
 23:46:13     INFO -    Freed heap region:       fd
 23:46:13     INFO -    Stack left redzone:      f1
 23:46:13     INFO -    Stack mid redzone:       f2
 23:46:13     INFO -    Stack right redzone:     f3
 23:46:13     INFO -    Stack partial redzone:   f4
 23:46:13     INFO -    Stack after return:      f5
 23:46:13     INFO -    Stack use after scope:   f8
 23:46:13     INFO -    Global redzone:          f9
 23:46:13     INFO -    Global init order:       f6
 23:46:13     INFO -    Poisoned by user:        f7
 23:46:13     INFO -    Contiguous container OOB:fc
 23:46:13     INFO -    ASan internal:           fe
 23:46:13     INFO -  ==10299==ABORTING
Eric, could this be related to logging doing something incorrect? The stacks look like Chromium IPC stuff, but the STR seems to be just enabling logging. Thanks.
Flags: needinfo?(erahm)
The only think I can think of here is if something is essentially doing free(PR_GetEnv("foo")), I don't see anything obviously doing that in the logging code.
Flags: needinfo?(erahm)
(In reply to Eric Rahm [:erahm] from comment #2)
> The only think I can think of here is if something is essentially doing
> free(PR_GetEnv("foo")), I don't see anything obviously doing that in the
> logging code.

If you look at the link to the full log, you can see who is freeing it. Specifically, it is happening inside GeckoChildProcessHost::SetChildLogName(). I'm guessing the comment "Passing temporary to PR_SetEnv is ok here because env gets copied by exec, etc., to permanent storage in child when process launched." is not right...
Maybe this is a regression from bug 1248565, which last touched that code.
FWIW, here is an example of try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7428ec1bea4b83a60804d763d7954c4aaba61df8

where NSPR logging is enabled ; it makes troubleshooting e10s on asan extremely difficult for me at this stage (we have intermittent failures occurring)

example: https://treeherder.mozilla.org/logviewer.html#?job_id=21282469&repo=try
See Also: → 1275117
Seems like I screwed this up in bug 1248565.  Before my patch, the string passed to PR_SetEnv() was still alive (on stack) while PerformAsyncLaunchInternal was executed.  I thought that PR_SetEnv() does the copy immediately (badly translated the comment to myself).  But PR_DuplicateEnvironment() does that later.
Assignee: nobody → honzab.moz
Status: NEW → ASSIGNED
Blocks: 1248565
Attachment #8755866 - Flags: review?(jduell.mcbugs)
Comment on attachment 8755866 [details] [diff] [review]
v1 (keep buffer passed to PR_SetEnv() during process launch)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a7610f1a85f5
Whiteboard: btpp-active
Attachment #8755866 - Flags: review?(jduell.mcbugs) → review+
Comment on attachment 8755866 [details] [diff] [review]
v1 (keep buffer passed to PR_SetEnv() during process launch)

Review of attachment 8755866 [details] [diff] [review]:
-----------------------------------------------------------------

There's a small typo in the comment :   // Muest keep
(Fixed the typo)
Attachment #8755866 - Attachment is obsolete: true
Attachment #8756790 - Flags: review+
Keywords: checkin-needed
Blocks: 1275860
https://hg.mozilla.org/mozilla-central/rev/062491c62640
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla49
See Also: → 1298103
You need to log in before you can comment on or make changes to this bug.