968496 - Intermittent ASAN rlogringbuffer_unittest | test failed with return code 1 from a webrtc heap-buffer-overflow

Status: RESOLVED INCOMPLETE

(Core :: WebRTC, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=34155907&tree=Fx-Team

Ubuntu ASAN VM 12.04 x64 fx-team opt test cppunit on 2014-02-05 12:17:05 PST for push 0f1bc0a9caa4
slave: tst-linux64-spot-096

12:25:04     INFO -  cppunittests INFO | Running test rlogringbuffer_unittest
12:25:04     INFO -  =================================================================
12:25:04     INFO -  ==2529==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7fff7403a8f8 at pc 0x597242 bp 0x7fff7403a390 sp 0x7fff7403a388
12:25:04     INFO -  READ of size 8 at 0x7fff7403a8f8 thread T0
12:25:04     INFO -      #0 0x597241 in operator<<<char> /builds/slave/fx-team-l64-asan-0000000000000/build/media/webrtc/trunk/testing/gtest/include/gtest/gtest-message.h:140
12:25:04     INFO -      #1 0x597241 in testing::internal::String testing::internal::StreamableToString<char*>(char* const&) /builds/slave/fx-team-l64-asan-0000000000000/build/media/webrtc/trunk/testing/gtest/include/gtest/gtest.h:174
12:25:04     INFO -      #2 0x593d49 in void testing::internal::InitGoogleTestImpl<char>(int*, char**) /builds/slave/fx-team-l64-asan-0000000000000/build/media/webrtc/trunk/testing/gtest/src/gtest.cc:4911
12:25:04     INFO -      #3 0x46ab66 in main /builds/slave/fx-team-l64-asan-0000000000000/build/media/mtransport/test/rlogringbuffer_unittest.cpp:264
12:25:04     INFO -      #4 0x7fbbda24576c (/lib/x86_64-linux-gnu/libc.so.6+0x2176c)
12:25:04     INFO -      #5 0x45ec4c in _start (/builds/slave/test/build/tests/cppunittests/rlogringbuffer_unittest+0x45ec4c)
12:25:04     INFO -  Address 0x7fff7403a8f8 is located in stack of thread T0 at offset 408 in frame
12:25:04     INFO -      #0 0x46aaaf in main /builds/slave/fx-team-l64-asan-0000000000000/build/media/mtransport/test/rlogringbuffer_unittest.cpp:258
12:25:04     INFO -    This frame has 1 object(s):
12:25:04     INFO -      [32, 36) ''
12:25:04     INFO -  HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
12:25:04     INFO -        (longjmp and C++ exceptions *are* supported)
12:25:04     INFO -  SUMMARY: AddressSanitizer: heap-buffer-overflow /builds/slave/fx-team-l64-asan-0000000000000/build/media/webrtc/trunk/testing/gtest/include/gtest/gtest-message.h:140 operator<<<char>
12:25:04     INFO -  Shadow bytes around the buggy address:
12:25:04     INFO -    0x10006e7ff4c0: 01 f4 f4 f4 f2 f2 f2 f2 00 00 f4 f4 f2 f2 f2 f2
12:25:04     INFO -    0x10006e7ff4d0: 00 f4 f4 f4 f2 f2 f2 f2 00 00 f4 f4 f3 f3 f3 f3
12:25:04     INFO -    0x10006e7ff4e0: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
12:25:04     INFO -    0x10006e7ff4f0: 04 f4 f4 f4 f3 f3 f3 f3 00 00 00 00 00 00 00 00
12:25:04     INFO -    0x10006e7ff500: 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa
12:25:04     INFO -  =>0x10006e7ff510: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa[fa]
12:25:04     INFO -    0x10006e7ff520:fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
12:25:04     INFO -    0x10006e7ff530: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
12:25:04     INFO -    0x10006e7ff540: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
12:25:04     INFO -    0x10006e7ff550: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
12:25:04     INFO -    0x10006e7ff560: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
12:25:04     INFO -  Shadow byte legend (one shadow byte represents 8 application bytes):
12:25:04     INFO -    Addressable:           00
12:25:04     INFO -    Partially addressable: 01 02 03 04 05 06 07
12:25:04     INFO -    Heap left redzone:     fa
12:25:04     INFO -    Heap right redzone:    fb
12:25:04     INFO -    Freed heap region:     fd
12:25:04     INFO -    Stack left redzone:    f1
12:25:04     INFO -    Stack mid redzone:     f2
12:25:04     INFO -    Stack right redzone:   f3
12:25:04     INFO -    Stack partial redzone: f4
12:25:04     INFO -    Stack after return:    f5
12:25:04     INFO -    Stack use after scope: f8
12:25:04     INFO -    Global redzone:        f9
12:25:04     INFO -    Global init order:     f6
12:25:04     INFO -    Poisoned by user:      f7
12:25:04     INFO -    ASan internal:         fe
12:25:04     INFO -  ==2529==ABORTING
12:25:04     INFO -  cppunittests TEST-UNEXPECTED-FAIL | rlogringbuffer_unittest | test failed with return code 1

Comment 1

[Byron Campen [:bwc]]

So, this looks spurious. It complains that the address 0x7fff7403a8f8 cannot be read, but a few lines later says it is on the stack of the same thread.

Comment 2

[Byron Campen [:bwc]]

Hmm, but that address is clearly past the current stack pointer by quite a bit. Very odd. Will look some more.

Comment 3

[Byron Campen [:bwc]]

Actually, this appears to be well within the stack on a second look. At least, it is not past the top of the stack. It is somewhere below the stack frame for main, which is what I'd expect for argv (the offending address in this case). I'm guessing it is too far below.

Comment 4

[Byron Campen [:bwc]]

Running under tsan doesn't turn up anything. Has this ever happened before?

Comment 5

[Byron Campen [:bwc]]

I've looked through the code called via NR_reg_init and r_log_register, and I've found some minor problems, but nothing that would cause this kind of bug.

Comment 6

[Daniel Veditz [:dveditz]]

Why would the heap ('fa') be right next to the stack ('f3' and 'f4')? The ASAN output looks a little odd.

Comment 7

[Byron Campen [:bwc]]

(In reply to Daniel Veditz [:dveditz] from comment #6)
> Why would the heap ('fa') be right next to the stack ('f3' and 'f4')? The
> ASAN output looks a little odd.

Yes, it looks completely weird. I've tried to figure out what exactly "Heap left redzone" could mean when it is below the top of the stack, but have found no answers.

Comment 8

[Daniel Veditz [:dveditz]]

Should we just close this and see if it comes up again in the future? If it doesn't we can blame cosmic rays.

Comment 9

[Byron Campen [:bwc]]

Do we have anyone who is deeply familiar with the implementation of ASan that might be able to give this a look? For all we know, this could be a bug in ASan.

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

cdiehl?  Any comments given your use of ASAN?

Otherwise, I'm ok with assuming it was cosmic rays ;-)

Comment 11

[Christian Holler (:decoder)]

I'll let an ASan developer look at the output so we can at least figure out if the output makes any sense or not (and if this is possibly an ASan bug). NI on myself to do that later.

Comment 12

[Daniel Veditz [:dveditz]]

Reopen if this comes back