1875000 - Corrupt JAR file causes fuzzing slowdown

Status: RESOLVED FIXED

(Core :: Networking: JAR, defect, P2)

Description

[Tyson Smith [:tsmith]]

Attachment: testcase.jar

Found with m-c 20240115-ae2d8becbc0b.

This testcase is tiny and has a major impact on performance. The fuzzer is able to find this fairly quickly which blocks fuzzing at scale. For reference the iteration rate is typically in the hundreds of iterations per second range.

Fuzzing interface docs can be found here: https://firefox-source-docs.mozilla.org/tools/fuzzing/fuzzing_interface.html
The command used to reproduce with patch from bug 1798631:

FUZZER=JARParser firefox testcase.jar -detect_leaks=0 -rss_limit_mb=3500 -timeout=2

==1542973== ERROR: libFuzzer: timeout after 6 seconds
    #0 0x5624aa5600a1 in __sanitizer_print_stack_trace /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:3
    #1 0x5624aa77bbf1 in fuzzer::PrintStackTrace() /builds/worker/checkouts/gecko/tools/fuzzing/libfuzzer/FuzzerUtil.cpp:210:5
    #2 0x5624aa764860 in fuzzer::Fuzzer::AlarmCallback() /builds/worker/checkouts/gecko/tools/fuzzing/libfuzzer/FuzzerLoop.cpp:309:5
    #3 0x7fd6c3e4541f  (/lib/x86_64-linux-gnu/libpthread.so.0+0x1441f) (BuildId: 0c044ba611aeeeaebb8374e660061f341ebc0bac)
    #4 0x5624aa56ea36 in internal_syscall<unsigned long, unsigned long> /builds/worker/fetches/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_syscall_linux_x86_64.inc:33:3
    #5 0x5624aa56ea36 in __sanitizer::internal_munmap(void*, unsigned long) /builds/worker/fetches/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp:223:10
    #6 0x5624aa57021b in __sanitizer::UnmapOrDie(void*, unsigned long) /builds/worker/fetches/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_posix.cpp:59:14
    #7 0x5624aa4c0b72 in RecyclePassThrough /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_allocator.cpp:261:5
    #8 0x5624aa4c0b72 in __sanitizer::Quarantine<__asan::QuarantineCallback, __asan::AsanChunk>::Put(__sanitizer::QuarantineCache<__asan::QuarantineCallback>*, __asan::QuarantineCallback, __asan::AsanChunk*, unsigned long) /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_quarantine.h:103:10
    #9 0x5624aa55546b in free /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:53:3
    #10 0x7fd69e14b04b in operator delete[] /builds/worker/workspace/obj-build/dist/include/mozilla/cxxalloc.h:60:10
    #11 0x7fd69e14b04b in operator() /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:471:5
    #12 0x7fd69e14b04b in reset /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:420:7
    #13 0x7fd69e14b04b in ~UniquePtr /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:381:18
    #14 0x7fd69e14b04b in ~nsZipItemPtr_base /builds/worker/checkouts/gecko/modules/libjar/nsZipArchive.h:297:7
    #15 0x7fd69e14b04b in operator() /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:459:5
    #16 0x7fd69e14b04b in reset /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:301:7
    #17 0x7fd69e14b04b in operator= /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:271:5
    #18 0x7fd69e14b04b in nsZipHandle::~nsZipHandle() /builds/worker/checkouts/gecko/modules/libjar/nsZipArchive.cpp:363:8
    #19 0x7fd69e112785 in Release /builds/worker/checkouts/gecko/modules/libjar/nsZipArchive.cpp:184:1
    #20 0x7fd69e112785 in Release /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:49:40
    #21 0x7fd69e112785 in Release /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:409:36
    #22 0x7fd69e112785 in ~RefPtr /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:80:7
    #23 0x7fd69e112785 in nsZipHandle::Init(nsZipArchive*, char const*, nsZipHandle**) /builds/worker/checkouts/gecko/modules/libjar/nsZipArchive.cpp:260:1
    #24 0x7fd69e111bdf in nsJAR::OpenInner(nsIZipReader*, nsTSubstring<char> const&) /builds/worker/checkouts/gecko/modules/libjar/nsJAR.cpp:138:10
    #25 0x7fd6985da042 in FuzzReader(char**, unsigned long*, nsIZipReader*) /builds/worker/checkouts/gecko/netwerk/test/fuzz/TestJARFuzzing.cpp:149:23
    #26 0x7fd6985db3c0 in FuzzingRunJARParser(unsigned char const*, unsigned long) /builds/worker/checkouts/gecko/netwerk/test/fuzz/TestJARFuzzing.cpp:180:10
    #27 0x5624aa765d0b in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /builds/worker/checkouts/gecko/tools/fuzzing/libfuzzer/FuzzerLoop.cpp:570:11
    #28 0x5624aa765791 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) /builds/worker/checkouts/gecko/tools/fuzzing/libfuzzer/FuzzerLoop.cpp:479:7
    #29 0x5624aa766bc7 in fuzzer::Fuzzer::MutateAndTestOne() /builds/worker/checkouts/gecko/tools/fuzzing/libfuzzer/FuzzerLoop.cpp:717:19
    #30 0x5624aa7675d5 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile>>&) /builds/worker/checkouts/gecko/tools/fuzzing/libfuzzer/FuzzerLoop.cpp:861:9
    #31 0x5624aa757f5b in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /builds/worker/checkouts/gecko/tools/fuzzing/libfuzzer/FuzzerDriver.cpp:864:14
    #32 0x7fd6ac368e0e in mozilla::FuzzerRunner::Run(int*, char***) /builds/worker/checkouts/gecko/tools/fuzzing/interface/harness/FuzzerRunner.cpp:75:13
    #33 0x7fd6ac273c56 in XREMain::XRE_mainStartup(bool*) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:4684:35
    #34 0x7fd6ac284dd6 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5940:12
    #35 0x7fd6ac286141 in XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:6009:21
    #36 0x5624aa595182 in do_main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:227:22
    #37 0x5624aa595182 in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:445:16
    #38 0x7fd6c38e9082 in __libc_start_main /build/glibc-wuryBv/glibc-2.31/csu/../csu/libc-start.c:308:16
    #39 0x5624aa4b9848 in _start (/home/twsmith/workspace/browsers/m-c-20240115163612-fuzzing-asan-opt/firefox+0xdc848) (BuildId: 0f090f4ee76d71f442fbdcf18308860fbb55d229)

Comment 1

[BugBot [:suhaib / :marco]]

This bug prevents fuzzing from making progress; however, it has low severity. It is important for fuzz blocker bugs to be addressed in a timely manner (see here why?).
:valentin, could you consider increasing the severity?

For more information, please visit BugBot documentation.

Comment 2

[Valentin Gosu [:valentin]]

Hi Tyson, are you sure this is the right testcase?
On my machine I've been unable to reproduce the timeout.

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

I tried with a local m-c build; no problems. is there anything else I need to do? That patch you linked landed last year

Comment 4

[Valentin Gosu [:valentin]]

Unassigning and removing from priority queue until we can get a reliable way to reproduce.

Comment 5

[Tyson Smith [:tsmith]]

Sorry for the delay. This appears to only affect ASan builds.

Comment 6

[Valentin Gosu [:valentin]]

I've tried it out on an ASAN-debug build, and couldn't get it to reproduce.
I can try it on ASAN-opt if that is indeed necessary. If you could provide a pernosco trace that would be even better.

Comment 7

[Tyson Smith [:tsmith]]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #6)

I've tried it out on an ASAN-debug build, and couldn't get it to reproduce.

Hmm I would assume that would have worked.

If you could provide a pernosco trace that would be even better.

Of course! Here are two, hopefully one of them highlights the issue.
OOM: https://pernos.co/debug/AA4IrmZm66OqOSibWnOEQA/index.html
Timeout: https://pernos.co/debug/tp0NCOnVMibJQKBX-rbzrA/index.html

Comment 8

[Valentin Gosu [:valentin]]

The pernosco trace doesn't have source or local variable information, but based on the stack trace I think the jar has a compressed entry whose real size is a really high number.
We then either OOM or hang allocating a large buffer.
I'm not exactly sure if this is something we encounter in the wild - I'll add a static pref for a max entry size so we return early instead of OOM.

Comment 9

[Valentin Gosu [:valentin]]

Attachment: Bug 1875000 - Avoid OOM in jar code r=#necko

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ca41c57249fc
Avoid OOM in jar code r=necko-reviewers,kershaw

Comment 11

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/ca41c57249fc

Comment 12

[BugBot [:suhaib / :marco]]

The patch landed in nightly and beta is affected.
:valentin, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox124 to wontfix.

For more information, please visit BugBot documentation.

Comment 13

[Tyson Smith [:tsmith]]

Success! This has unblocked the fuzzers. Thank you.

Comment 14

[Valentin Gosu [:valentin]]

I think this can ride the trains.
Thank you, Tyson! 🙏🏻