1853026 - WebP fuzzing target is running slowly

Status: RESOLVED FIXED

(Core :: Graphics: ImageLib, defect)

Description

[Tyson Smith [:tsmith]]

Attachment: testcase.webp

The ImageWebP targeted fuzzer is running slowly < 5 iterations a second and is hanging frequently.

To reproduce this issue run the attached test case with a fuzzing build.
FUZZER=ImageWebP <firefox> testcase.webm -detect_leaks=0 -malloc_limit_mb=12288 -rss_limit_mb=12288 -timeout=15

==2027845== ERROR: libFuzzer: timeout after 15 seconds
    #0 0x560f1af81b41 in __sanitizer_print_stack_trace /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:3
    #1 0x560f1b31c919 in fuzzer::PrintStackTrace() src/tools/fuzzing/libfuzzer/FuzzerUtil.cpp:210:5
    #2 0x560f1b2f474d in fuzzer::Fuzzer::AlarmCallback() src/tools/fuzzing/libfuzzer/FuzzerLoop.cpp:309:5
    #3 0x560f1b2f45ff in fuzzer::Fuzzer::StaticAlarmCallback() src/tools/fuzzing/libfuzzer/FuzzerLoop.cpp:199:6
    #4 0x560f1b31d577 in fuzzer::AlarmHandler(int, siginfo_t*, void*) src/tools/fuzzing/libfuzzer/FuzzerUtilPosix.cpp:33:3
    #5 0x7fe2c39a241f  (/lib/x86_64-linux-gnu/libpthread.so.0+0x1441f) (BuildId: 7b4536f41cdaa5888408e82d0836e33dcf436466)
    #6 0x7fe2c353499e in poll /build/glibc-SzIz7B/glibc-2.31/io/../sysdeps/unix/sysv/linux/poll.c:29:10 
    #7 0x560f1af20215 in poll /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:4177:13
    #8 0x7fe2999a6bbf in PollWrapper(_GPollFD*, unsigned int, int) src/widget/gtk/nsAppShell.cpp:77:16
    #9 0x7fe2bec5236d  (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x5236d) (BuildId: 5fdb313daf182a33a858ba2cc945211b11d34561)
    #10 0x7fe2bec524a2 in g_main_context_iteration (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x524a2) (BuildId: 5fdb313daf182a33a858ba2cc945211b11d34561)
    #11 0x7fe2999a72a0 in nsAppShell::ProcessNextNativeEvent(bool) src/widget/gtk/nsAppShell.cpp:418:26
    #12 0x7fe29973a681 in nsBaseAppShell::DoProcessNextNativeEvent(bool) src/widget/nsBaseAppShell.cpp:131:17
    #13 0x7fe29973b8a6 in nsBaseAppShell::OnProcessNextEvent(nsIThreadInternal*, bool) src/widget/nsBaseAppShell.cpp:267:10
    #14 0x7fe288decf29 in nsThread::ProcessNextEvent(bool, bool*) src/xpcom/threads/nsThread.cpp:1114:10
    #15 0x7fe288df9055 in NS_ProcessNextEvent(nsIThread*, bool) src/xpcom/threads/nsThreadUtils.cpp:480:10
    #16 0x7fe288e341c6 in bool mozilla::SpinEventLoopUntil<(mozilla::ProcessFailureBehavior)1, nsThreadSyncDispatch::SpinEventLoopUntilComplete(nsTSubstring<char> const&)::'lambda'()>(nsTSubstring<char> const&, nsThreadSyncDispatch::SpinEventLoopUntilComplete(nsTSubstring<char> const&)::'lambda'()&&, nsIThread*) src/objdir-ff-ubsan/dist/include/mozilla/SpinEventLoopUntil.h:176:25
    #17 0x7fe288e28f65 in nsThreadSyncDispatch::SpinEventLoopUntilComplete(nsTSubstring<char> const&) src/xpcom/threads/nsThreadSyncDispatch.h:32:5
    #18 0x7fe288e07fe8 in NS_DispatchAndSpinEventLoopUntilComplete(nsTSubstring<char> const&, nsIEventTarget*, already_AddRefed<nsIRunnable>) src/xpcom/threads/nsThreadUtils.cpp:767:12
    #19 0x7fe2863ba9af in RunDecodeToSurfaceFuzzing(nsCOMPtr<nsIInputStream>, char const*) src/image/test/fuzzing/TestDecoders.cpp:95:3
    #20 0x7fe2863bb0b6 in RunDecodeToSurfaceFuzzingWebP(nsCOMPtr<nsIInputStream>) src/image/test/fuzzing/TestDecoders.cpp:127:10
    #21 0x7fe2863b9391 in LibFuzzerTestImageWebP(unsigned char const*, unsigned long) src/image/test/fuzzing/TestDecoders.cpp:171:1
    #22 0x560f1b2f58cb in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) src/tools/fuzzing/libfuzzer/FuzzerLoop.cpp:570:11
    #23 0x560f1b2d929d in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) src/tools/fuzzing/libfuzzer/FuzzerDriver.cpp:301:6
    #24 0x560f1b2dc43f in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) src/tools/fuzzing/libfuzzer/FuzzerDriver.cpp:810:9
    #25 0x7fe2a57791bd in mozilla::FuzzerRunner::Run(int*, char***) src/tools/fuzzing/interface/harness/FuzzerRunner.cpp:75:13
    #26 0x7fe2a55f4c83 in XREMain::XRE_mainStartup(bool*) src/toolkit/xre/nsAppRunner.cpp:4673:35
    #27 0x7fe2a5604f6f in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:5872:12
    #28 0x7fe2a560577c in XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:5940:21
    #29 0x7fe2a563acc6 in mozilla::BootstrapImpl::XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/Bootstrap.cpp:45:12
    #30 0x560f1afb63cd in do_main(int, char**, char**) src/browser/app/nsBrowserApp.cpp:227:22
    #31 0x560f1afb500e in main src/browser/app/nsBrowserApp.cpp:445:16   
    #32 0x7fe2c3446082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
    #33 0x560f1aede6a8 in _start (src/objdir-ff-ubsan/dist/bin/firefox+0x1986a8) (BuildId: 67c952403a6e8964b7a61f7c56f92b80)

Comment 1

[Timothy Nikkel (:tnikkel)]

Almost all the time seems to be inside libwebp. The image is also quite large. Is it much faster to decode similar sized images in other formats?

Comment 2

[Tyson Smith [:tsmith]]

Tested 16384x11264:
Executed image-png-ffmpeg-solid-resolution-10.png in 3644 ms

FWIW I tried creating a jpeg that is 16384x11264 and ffmpeg and imagemagick would not let me.

Would it be reasonable to add an upper limit to the browser? Delegate working with huge images to dedicate image software? This would also help harden the browser.

Comment 3

[Timothy Nikkel (:tnikkel)]

We are actually wanting to increase the size of allowed images https://phabricator.services.mozilla.com/D162256

We could have a pref that quick rejects large sized images, but we'd probably want to also do some fuzzing without that pref to make sure large sized image issues are fuzzed too.

Comment 4

[Timothy Nikkel (:tnikkel)]

Attachment: Bug 1853026. Add a pref for fuzzers to limit image size so fuzzing can proceed more quickly. r?#gfx-reviewers

Large image sizes slow down fuzzing for webp and aren't likely to catch any more issues. We will still want to fuzz without this pref to catch any large image issues though.

Comment 5

[Timothy Nikkel (:tnikkel)]

Do you want to check that this patch works for you before I land it?

Comment 6

[Tyson Smith [:tsmith]]

Attachment: slow_1.webp

I found a couple files that really slow the fuzzer down with the patch applied and the limit set to 64000000 (4000 x 4000 x 4)

Comment 7

[Tyson Smith [:tsmith]]

Attachment: slow_2.webp

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

This bug prevents fuzzing from making progress; however, it has low severity. It is important for fuzz blocker bugs to be addressed in a timely manner (see here why?).
:tnikkel, could you consider increasing the severity?

For more information, please visit BugBot documentation.

Comment 9

[Tyson Smith [:tsmith]]

I ran for ~48h with the patch and it seems to be working as expected.

Regarding the slow_#.webp test cases, they take ~1.5s on a -O2 build and ~10s on -O0 build. If there is anything that can be done to improve the execution time of the new slow test cases we can open a new bug.

Comment 10

[Timothy Nikkel (:tnikkel)]

I updated the patch to interpret the pref value as in kb, so going forward 4k x 4k limit would use the numerical value 4000 x 4000 x 4/1024 = 62500.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1f76e350d909
Add a pref for fuzzers to limit image size so fuzzing can proceed more quickly. r=gfx-reviewers,lsalzman

Comment 12

[Timothy Nikkel (:tnikkel)]

I took a look at the two slow_n.webp images attached here. Nothing jumped out at me. The time in the webp decoding library was about equal to the time we spent downscaling the image, so they don't seem to take particularly long to decode. They are both almost 4000 x 4000 pixels, so perhaps they are just the largest sized images that fit within the size limit so they will take the longest.

Comment 13

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/1f76e350d909