1482084 - Intermittent AddressSanitizer: global-buffer-overflow /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:23:3 in __asan_memcpy

Status: RESOLVED DUPLICATE of bug 1540759

(Core :: Networking, defect, P2)

Description

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&fromchange=cc99bd10a9ad36329858e730df04070674718e52&filter-classifiedState=unclassified&group_state=expanded&selectedJob=192977920

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=192977920&repo=autoland&lineNumber=2641

```
[task 2018-08-09T11:11:40.062Z] 11:11:40     INFO - TEST-START | browser/base/content/test/urlbar/browser_urlbar_blanking.js
[task 2018-08-09T11:11:41.448Z] 11:11:41     INFO - GECKO(1591) | =================================================================
[task 2018-08-09T11:11:41.449Z] 11:11:41    ERROR - GECKO(1591) | ==1666==ERROR: AddressSanitizer: global-buffer-overflow on address 0x7f12d93f9242 at pc 0x0000004c0f39 bp 0x7f12805a6550 sp 0x7f12805a5d00
[task 2018-08-09T11:11:41.450Z] 11:11:41     INFO - GECKO(1591) | READ of size 5430 at 0x7f12d93f9242 thread T10 (ImageIO)
[task 2018-08-09T11:11:41.880Z] 11:11:41     INFO - GECKO(1591) |     #0 0x4c0f38 in __asan_memcpy /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:23:3
[task 2018-08-09T11:11:42.630Z] 11:11:42     INFO - GECKO(1591) |     #1 0x7f12ce3fe358 in mozilla::image::SourceBuffer::Append(char const*, unsigned long) /builds/worker/workspace/build/src/image/SourceBuffer.cpp:435:3
[task 2018-08-09T11:11:42.630Z] 11:11:42     INFO - GECKO(1591) |     #2 0x7f12ce3ff61c in mozilla::image::AppendToSourceBuffer(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*) /builds/worker/workspace/build/src/image/SourceBuffer.cpp:492:31
[task 2018-08-09T11:11:42.631Z] 11:11:42     INFO - GECKO(1591) |     #3 0x7f12cb71969b in nsStringInputStream::ReadSegments(nsresult (*)(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*), void*, unsigned int, unsigned int*) /builds/worker/workspace/build/src/xpcom/io/nsStringStream.cpp:270:17
[task 2018-08-09T11:11:42.633Z] 11:11:42     INFO - GECKO(1591) |     #4 0x7f12cb6c9009 in mozilla::NonBlockingAsyncInputStream::ReadSegments(nsresult (*)(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*), void*, unsigned int, unsigned int*) /builds/worker/workspace/build/src/xpcom/io/NonBlockingAsyncInputStream.cpp:221:24
[task 2018-08-09T11:11:42.634Z] 11:11:42     INFO - GECKO(1591) |     #5 0x7f12ce3efb88 in AppendFromInputStream /builds/worker/workspace/build/src/image/SourceBuffer.cpp:508:31
[task 2018-08-09T11:11:42.635Z] 11:11:42     INFO - GECKO(1591) |     #6 0x7f12ce3efb88 in mozilla::image::RasterImage::OnImageDataAvailable(nsIRequest*, nsISupports*, nsIInputStream*, unsigned long, unsigned int) /builds/worker/workspace/build/src/image/RasterImage.cpp:987
[task 2018-08-09T11:11:42.644Z] 11:11:42     INFO - GECKO(1591) |     #7 0x7f12ce44392d in imgRequest::OnDataAvailable(nsIRequest*, nsISupports*, nsIInputStream*, unsigned long, unsigned int) /builds/worker/workspace/build/src/image/imgRequest.cpp:1235:14
[task 2018-08-09T11:11:42.649Z] 11:11:42     INFO - GECKO(1591) |     #8 0x7f12cb925c5c in nsBaseChannel::OnDataAvailable(nsIRequest*, nsISupports*, nsIInputStream*, unsigned long, unsigned int) /builds/worker/workspace/build/src/netwerk/base/nsBaseChannel.cpp:905:28
[task 2018-08-09T11:11:42.658Z] 11:11:42     INFO - GECKO(1591) |     #9 0x7f12cb97db77 in nsInputStreamPump::OnStateTransfer() /builds/worker/workspace/build/src/netwerk/base/nsInputStreamPump.cpp:597:29
[task 2018-08-09T11:11:42.659Z] 11:11:42     INFO - GECKO(1591) |     #10 0x7f12cb97c907 in nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) /builds/worker/workspace/build/src/netwerk/base/nsInputStreamPump.cpp:432:25
[task 2018-08-09T11:11:42.659Z] 11:11:42     INFO - GECKO(1591) |     #11 0x7f12cb6c8d66 in mozilla::NonBlockingAsyncInputStream::AsyncWaitRunnable::Run() /builds/worker/workspace/build/src/xpcom/io/NonBlockingAsyncInputStream.cpp:383:13
[task 2018-08-09T11:11:42.667Z] 11:11:42     INFO - GECKO(1591) |     #12 0x7f12cb78cddf in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:1235:14
[task 2018-08-09T11:11:42.668Z] 11:11:42     INFO - GECKO(1591) |     #13 0x7f12cb793fa8 in NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/workspace/build/src/xpcom/threads/nsThreadUtils.cpp:519:10
[task 2018-08-09T11:11:42.690Z] 11:11:42     INFO - GECKO(1591) |     #14 0x7f12cc6c339c in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /builds/worker/workspace/build/src/ipc/glue/MessagePump.cpp:364:5
[task 2018-08-09T11:11:42.690Z] 11:11:42     INFO - GECKO(1591) |     #15 0x7f12cc61674c in RunInternal /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:325:10
[task 2018-08-09T11:11:42.690Z] 11:11:42     INFO - GECKO(1591) |     #16 0x7f12cc61674c in RunHandler /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:318
[task 2018-08-09T11:11:42.690Z] 11:11:42     INFO - GECKO(1591) |     #17 0x7f12cc61674c in MessageLoop::Run() /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:298
[task 2018-08-09T11:11:42.690Z] 11:11:42     INFO - GECKO(1591) |     #18 0x7f12cb786734 in nsThread::ThreadFunc(void*) /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:467:11
[task 2018-08-09T11:11:42.692Z] 11:11:42     INFO - GECKO(1591) |     #19 0x7f12e85d6f08 in _pt_root /builds/worker/workspace/build/src/nsprpub/pr/src/pthreads/ptthread.c:201:5
[task 2018-08-09T11:11:42.699Z] 11:11:42     INFO - GECKO(1591) |     #20 0x7f12ec9736b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
[task 2018-08-09T11:11:42.781Z] 11:11:42     INFO - GECKO(1591) |     #21 0x7f12eb9fc41c in clone /build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109
[task 2018-08-09T11:11:42.781Z] 11:11:42     INFO - GECKO(1591) | 0x7f12d93f9242 is located 62 bytes to the left of global variable '<string literal>' defined in '/builds/worker/workspace/build/src/xpcom/string/nsSubstring.cpp:335:3' (0x7f12d93f9280) of size 59
[task 2018-08-09T11:11:42.781Z] 11:11:42     INFO - GECKO(1591) |   '<string literal>' is ascii string 'data[aLen] == char16_t(0) (data should be null terminated)'
[task 2018-08-09T11:11:42.781Z] 11:11:42     INFO - GECKO(1591) | 0x7f12d93f9242 is located 0 bytes to the right of global variable 'gNullChar' defined in '/builds/worker/workspace/build/src/xpcom/string/nsSubstring.cpp:56:23' (0x7f12d93f9240) of size 2
[task 2018-08-09T11:11:42.782Z] 11:11:42     INFO - GECKO(1591) | SUMMARY: AddressSanitizer: global-buffer-overflow /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:23:3 in __asan_memcpy
[task 2018-08-09T11:11:42.783Z] 11:11:42     INFO - GECKO(1591) | Shadow bytes around the buggy address:
[task 2018-08-09T11:11:42.784Z] 11:11:42     INFO - GECKO(1591) |   0x0fe2db2771f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2018-08-09T11:11:42.785Z] 11:11:42     INFO - GECKO(1591) |   0x0fe2db277200: 00 00 00 00 00 00 00 00 f9 f9 f9 f9 00 00 00 00
[task 2018-08-09T11:11:42.786Z] 11:11:42     INFO - GECKO(1591) |   0x0fe2db277210: 00 00 00 00 00 00 00 00 00 00 00 00 f9 f9 f9 f9
[task 2018-08-09T11:11:42.787Z] 11:11:42     INFO - GECKO(1591) |   0x0fe2db277220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2018-08-09T11:11:42.788Z] 11:11:42     INFO - GECKO(1591) |   0x0fe2db277230: f9 f9 f9 f9 00 00 00 00 00 00 00 00 00 00 00 00
[task 2018-08-09T11:11:42.789Z] 11:11:42     INFO - GECKO(1591) | =>0x0fe2db277240: 00 00 00 00 f9 f9 f9 f9[02]f9 f9 f9 f9 f9 f9 f9
[task 2018-08-09T11:11:42.791Z] 11:11:42     INFO - GECKO(1591) |   0x0fe2db277250: 00 00 00 00 00 00 00 03 f9 f9 f9 f9 00 00 00 00
[task 2018-08-09T11:11:42.793Z] 11:11:42     INFO - GECKO(1591) |   0x0fe2db277260: 00 00 00 00 f9 f9 f9 f9 00 00 00 00 00 00 00 00
[task 2018-08-09T11:11:42.794Z] 11:11:42     INFO - GECKO(1591) |   0x0fe2db277270: 00 07 f9 f9 f9 f9 f9 f9 00 00 00 00 00 00 07 f9
[task 2018-08-09T11:11:42.795Z] 11:11:42     INFO - GECKO(1591) |   0x0fe2db277280: f9 f9 f9 f9 00 00 00 00 00 00 00 00 00 03 f9 f9
[task 2018-08-09T11:11:42.796Z] 11:11:42     INFO - GECKO(1591) |   0x0fe2db277290: f9 f9 f9 f9 06 f9 f9 f9 f9 f9 f9 f9 00 06 f9 f9
[task 2018-08-09T11:11:42.796Z] 11:11:42     INFO - GECKO(1591) | Shadow byte legend (one shadow byte represents 8 application bytes):
[task 2018-08-09T11:11:42.797Z] 11:11:42     INFO - GECKO(1591) |   Addressable:           00
[task 2018-08-09T11:11:42.798Z] 11:11:42     INFO - GECKO(1591) |   Partially addressable: 01 02 03 04 05 06 07
[task 2018-08-09T11:11:42.798Z] 11:11:42     INFO - GECKO(1591) |   Heap left redzone:       fa
[task 2018-08-09T11:11:42.799Z] 11:11:42     INFO - GECKO(1591) |   Freed heap region:       fd
[task 2018-08-09T11:11:42.800Z] 11:11:42     INFO - GECKO(1591) |   Stack left redzone:      f1
[task 2018-08-09T11:11:42.800Z] 11:11:42     INFO - GECKO(1591) |   Stack mid redzone:       f2
[task 2018-08-09T11:11:42.801Z] 11:11:42     INFO - GECKO(1591) |   Stack right redzone:     f3
[task 2018-08-09T11:11:42.802Z] 11:11:42     INFO - GECKO(1591) |   Stack after return:      f5
[task 2018-08-09T11:11:42.803Z] 11:11:42     INFO - GECKO(1591) |   Stack use after scope:   f8
[task 2018-08-09T11:11:42.803Z] 11:11:42     INFO - GECKO(1591) |   Global redzone:          f9
[task 2018-08-09T11:11:42.804Z] 11:11:42     INFO - GECKO(1591) |   Global init order:       f6
[task 2018-08-09T11:11:42.805Z] 11:11:42     INFO - GECKO(1591) |   Poisoned by user:        f7
[task 2018-08-09T11:11:42.805Z] 11:11:42     INFO - GECKO(1591) |   Container overflow:      fc
[task 2018-08-09T11:11:42.806Z] 11:11:42     INFO - GECKO(1591) |   Array cookie:            ac
[task 2018-08-09T11:11:42.807Z] 11:11:42     INFO - GECKO(1591) |   Intra object redzone:    bb
[task 2018-08-09T11:11:42.807Z] 11:11:42     INFO - GECKO(1591) |   ASan internal:           fe
[task 2018-08-09T11:11:42.808Z] 11:11:42     INFO - GECKO(1591) |   Left alloca redzone:     ca
[task 2018-08-09T11:11:42.808Z] 11:11:42     INFO - GECKO(1591) |   Right alloca redzone:    cb
```

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Andrew, do you have cycles to take a look?

Comment 2

[Andrew Osmond [:aosmond] (he/him)]

I reviewed the SourceBuffer code, and there doesn't appear to be any obvious read overflow. It will write the segment data given to us by the input stream into a local buffer in either one (current chunk has enough room) or two pieces (current chunk has some room, and we need to allocate another chunk to fit the rest), but I don't believe it should ever exceed the segment length given to us. Perhaps the input stream given to us is somehow setup incorrectly? I quickly scanned for similar crashes originating from nsPipeInputStream::ReadSegments, and it looks like there are other components with a similar problem.

Comment 3

[Jason Duell]

cc-ing some necko folks.

I'm not clear on whether this is a necko bug or not.

The only thing I notice in the stack is nsBaseChannel::OnDataAvailable().  BaseChannel means we're /not/ using HTTP, so this is file:// or ftp:// or one of the other protocols that use basechannel.   So maybe there's a possibility that file I/O in particular delivers larger OnDataAvailable chunks, and those somehow cause an issue?   That's just a wild guess.

Honza knows streams stuff well, but is on PTO, so giving to Dragana for a first look.

Comment 4

[Dragana Damjanovic [:dragana]]

I had a look at stream code a bit, but I do not see any problems.
Honza, can you take a look?

The crashes are on 62 but not on 63. 62 was crashing when it was on beta. There are 3 crashes on 63 Nightly. Do you remember if we have change something?

Comment 5

[Selena Deckelmann :selenamarie :selena]

Assigning to :honza for investigation.

Comment 6

[Valentin Gosu [:valentin] (he/him)]

(In reply to Selena Deckelmann :selenamarie :selena use ni? pronoun: she from comment #5)
> Assigning to :honza for investigation.

I think you have the wrong Honza :)

Comment 7

[Honza Bambas (:mayhemer)]

Attachment: copy of the log from comment 0

..in case it disappears before I get to this

Comment 9

[Honza Bambas (:mayhemer)]

ni?me to retriage this one

Comment 10

[Honza Bambas (:mayhemer)]

This looks very similar (and possibly having the same solution) as bug 1523202.

Looking at the crash stats, this seems to be hit with any type of a stream, so the problem is more general.

Comment 11

[Honza Bambas (:mayhemer)]

I believe this is duplicate of bug 1540759.

Comment 12

[Honza Bambas (:mayhemer)]

To explain my findings: the bit

0x7f12d93f9242 is located 0 bytes to the right of global variable 'gNullChar'

means that (in the case of a string input stream) the string has already been released, probably here: nsTSubstring.h - mozsearch. The low crash rate means this happens at rare tight timing probability, but indicates the stream and its data is (being) destroyed.

Comment 13

[David Lawrence [:dkl]]

Removing employee no longer with company from CC list of private bugs.