1662200 - Assertion failure: rowPtr + rgbStride.value() == endOfRgbBuf, at src/image/decoders/nsAVIFDecoder.cpp:554

Status: NEW

(Core :: Graphics: ImageLib, defect, P3)

Description

[Tyson Smith [:tsmith]]

Attachment: testcase.avif

Found with m-c 20200830-a4b27ed1a68e

Assertion failure: rowPtr + rgbStride.value() == endOfRgbBuf, at src/image/decoders/nsAVIFDecoder.cpp:554

#0 0x7f0a8ddc7295 in AnnotateMozCrashReason /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:42:19
#1 0x7f0a8ddc7295 in mozilla::image::nsAVIFDecoder::DoDecode(mozilla::image::SourceBufferIterator&, mozilla::image::IResumable*) src/image/decoders/nsAVIFDecoder.cpp:554:7
#2 0x7f0a8dd06a7c in mozilla::image::Decoder::Decode(mozilla::image::IResumable*) src/image/Decoder.cpp:172:19
#3 0x7f0a8dd0ff0d in mozilla::image::DecodedSurfaceProvider::Run() src/image/DecodedSurfaceProvider.cpp:123:34
#4 0x7f0a8dd27c86 in mozilla::image::DecodePoolWorker::Run() src/image/DecodePool.cpp:276:23
#5 0x7f0a8c1cd368 in nsThread::ProcessNextEvent(bool, bool*) src/xpcom/threads/nsThread.cpp:1234:14
#6 0x7f0a8c1d2d3a in NS_ProcessNextEvent(nsIThread*, bool) src/xpcom/threads/nsThreadUtils.cpp:513:10
#7 0x7f0a8caff46c in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) src/ipc/glue/MessagePump.cpp:332:5
#8 0x7f0a8ca6e7b3 in MessageLoop::RunInternal() src/ipc/chromium/src/base/message_loop.cc:334:10
#9 0x7f0a8ca6e6cd in RunHandler src/ipc/chromium/src/base/message_loop.cc:327:3
#10 0x7f0a8ca6e6cd in MessageLoop::Run() src/ipc/chromium/src/base/message_loop.cc:309:3
#11 0x7f0a8c1c970d in nsThread::ThreadFunc(void*) src/xpcom/threads/nsThread.cpp:442:10
#12 0x7f0aa007667b in _pt_root src/nsprpub/pr/src/pthreads/ptthread.c:201:5
#13 0x7f0aa0de8608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)
#14 0x7f0aa09b1102 in clone /build/glibc-YYA7BZ/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment 1

[Jon Bauman [:jbauman:]]

I'm not able to locally reproduce this with any of the mozconfigs I've been using (debug, release, fuzzer-like). In all cases, dav1d_send_data fails (presumably because the encoded data is invalid) and that results in nsAVIFDecoder::DoDecode returning an error at line 475, well before this assertion on line 554.

Is there any chance this isn't the right testcase?

Comment 2

[Tyson Smith [:tsmith]]

Attachment: testcase.avif

Maybe? These AVIF testcases really seem to screw with Window Explorer (even with thumbnail & preview disabled!).

I triple checked this one but it seems to require a harness. I'll create and attach it once I figure out what is required.

Comment 3

[Tyson Smith [:tsmith]]

A Pernosco session is available here: https://pernos.co/debug/grLdb_xg9heaxuQB27qM-g/index.html

Comment 4

[Tyson Smith [:tsmith]]

Attachment: testcase.html

Here is the harness. This makes the assertion 100% reliable on my setup. Let me know if there are any issues.

Comment 5

[Jon Bauman [:jbauman:]]

That's really strange. I don't see why that harness would make any difference as the underlying AVIF is invalid and fails the decode step with both libaom and dav1d. I'll look more into the Pernosco session and see what I can figure out.

Comment 6

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Are there any prefs needed here? I'm having trouble reproing (on Windows).

Comment 7

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Looking at Pernosco.

• rgbStride.value() = 32 = 0x20, rowPtr = 0x0x55facf218130, endOfRgbBuf = 0x55facf225810.
• rowPtr + rgbStride.value() = 0x55facf218150 != 0x55facf225810. We're quite far short of the end of the buffer (0xd6c0 short).
• Seems like we're stopping early because the pipe is saying we've finished.
• The early stoppage is because the downscaling filter is hitting a case where it thinks we're going off the end of our output buffer. Pernosco recording at that point
• Looks like the values involved there come from OutputSize() being given to the pipe here.

I'm not 100% on what exactly is going wrong. Looking at the pipe documentation it seems like the config is valid -- output is smaller than input so should be downscaled. Is it possible that because we're downscaling so aggressively we should be ending early and not continue reading from our buffer?

Comment 8

[Tyson Smith [:tsmith]]

No extra prefs but we do use Xvfb in automation. I can confirm that I can only reproduce the issue with the attached test case under Xvfb.

Comment 9

[Tyson Smith [:tsmith]]

We use xvfbwrapper with a screen resolution of 1280x1024x24 (colour depth is default)

Comment 10

[Tyson Smith [:tsmith]]

Attachment: testcase_no_xvfb.avif

Here is another test case. I was able to trigger the issue without Xvfb.

Comment 11

[Tyson Smith [:tsmith]]

Attachment: testcase_windows.avif

This one seems to reproduce reliably on Windows using the harness (testcase.html)

Comment 12

[Jon Bauman [:jbauman:]]

I think I was mistaken in my previous comment (or things are somehow behaving differently despite no relevant changes I'm aware of), but what I'm seeing now on my local build (whether mozconfig is debug, release or fuzzing-like) is that the decode step succeeds with dav1d, but fails with libaom. I haven't dug into where the difference occurs, but that could be interesting, if it comes down to dav1d returning success when it should be failing.

That said, I'm still not able to reproduce the rowPtr + rgbStride.value() == endOfRgbBuf assertion failure, so I'm still at a loss.

Unfortunately I think I'll have to pause work on this for a bit, but I definitely want to come back and get to the bottom of it.