Closed Bug 1466021 Opened 7 years ago Closed 6 years ago

oss-fuzz firefox native fuzzing interface integration

Categories

(Core :: Fuzzing, defect)

Product:
Core
Component:
Fuzzing
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: u473386, Unassigned)

References

(Depends on 1 open bug)

Details

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36 Steps to reproduce: There are a few problems when I tried to integrate Firefox into oss-fuzz with the native fuzzing interface. Most importantly because oss-fuzz builds and runs the binaries in two separate docker images: base-builder and base-runner. You can read a bit about it below. https://github.com/google/oss-fuzz/blob/master/docs/fuzzer_environment.md base-builder is used to build the binaries, and can be modified freely. base-runner runs the binaries, but cannot be modified, and only comes with a small number of libraries installed. You can only copy files from base-builder to base-runner into a single directory, in which the binaries are then run. So I build Firefox in base-builder, and then copied obj-dir to base-runner. Running Firefox in base-runner yields this error. XPCOMGlueLoad error for file /out/obj-fuzz/dist/bin/libmozgtk.so: libgtk-3.so.0: cannot open shared object file: No such file or directory Couldn't load XPCOM. Which is because libgtk is not available on base-runner of course. I'm not sure if this can be solved, as Firefox would need to be build completely static, I think.
First of all you probably shouldn't just copy the objdir. A year ago, I ran this code to package it sufficiently without copying both libxul versions: mach build mach gtest donotrungtests make -C objdir package make -C objdir package-tests-common package-tests-gtest tar -xf objdir/dist/firefox*.tar.bz2 cd firefox unzip ../objdir/dist/firefox*.gtest.tests.zip rm libxul.so touch libxul.so mv gtest/dependentlibs.list.gtest . cd gtest rm *.jpg *.mp4 *.mp3 *.ivf *.bmp *.gif *.png *.webm *.ico *.icon *.mov *.py *.json mv gtest_bin/gtest/libxul.so . Afterwards, repack the firefox directory. This saves you a bunch of space because otherwise the build would contain two libxuls (one for regular running and the gtest libxul). However, I believe this doesn't solve your GTK problem. I am not sure at which stage we load that .so and you can try if we actually load it with MOZ_RUN_GTEST=1 but I assume so. If that is the case, then I think there is no way to do this with the Firefox build system. You probably need to update base-runner to include GTK.
Status: UNCONFIRMED → NEW
Component: Untriaged → Platform Fuzzing Team
Ever confirmed: true
Product: Firefox → Core
Yes, the error is with the environment variables set. Does firefox try to load the missing .so when it's adjacent to it? I can build GTK in base-builder, and then copy it with the other files. Ideally GTK would be built into a static archive, but then it needs to be linked into firefox somehow. If either of these work, it should also work for other possibly missing libraries. Of course I can ask Googlers to add the library, but I'm sure they will be very reluctant about it. And there may be more missing libraries. (I think it only reports the first.)
As far as I know, you can in general not link statically with GTK unless you have a version of GTK that was specifically build for static linking. That said, I don't think Firefox supports statically linking GTK *at all*. You really need to install GTK on the target machine where run (and other runtime dependencies). You should be able to figure out which these are by running ldd on the .so files produced during build (e.g. libmozgtk.so to see X/gtk dependencies). Maybe some runtime dependencies can be disabled at build-time, but GTK for example can't be disabled and I don't know from the top of my head if we can prevent it from trying to load that .so easily. The whole startup process of Firefox (and also the fuzzing interface) is very complex.
Well the good news is that libxul.so doesn't actually need libmozgtk.so to run libFuzzer. I used patchelf to patch the dependency out, and the fuzz targets I tried (SdpParser, StunParser, and Qcms I made myself) still work. The bad news is that there are still many more dependencies in libxul.so. libpthread.so.0 libnspr4.so libplc4.so libplds4.so liblgpllibs.so libnss3.so libnssutil3.so libsmime3.so libmozsqlite3.so libssl3.so librt.so.1 libm.so.6 libX11.so.6 libX11-xcb.so.1 libxcb.so.1 libXcomposite.so.1 libXcursor.so.1 libXdamage.so.1 libXext.so.6 libXfixes.so.3 libXi.so.6 libXrender.so.1 libdl.so.2 libc.so.6 ld-linux-x86-64.so.2 libfreetype.so.6 libfontconfig.so.1 libdbus-glib-1.so.2 libdbus-1.so.3 libgobject-2.0.so.0 libglib-2.0.so.0 libpangocairo-1.0.so.0 libpango-1.0.so.0 libatk-1.0.so.0 libcairo-gobject.so.2 libcairo.so.2 libgdk_pixbuf-2.0.so.0 libgio-2.0.so.0 libxcb-shm.so.0 libpangoft2-1.0.so.0 libXt.so.6 libgthread-2.0.so.0 libpulse.so.0 libstdc++.so.6 libgcc_s.so.1 I assume that libX* can be patched out without consequences. Maybe others as well. I'd really like to make this work, but Firefox may just be too complex for it to happen. oss-fuzz expects a self-contained package, more or less.
Well, it wasn't so easy. I deleted libmozgtk.so which makes it fail. So it's still loaded somewhere, and with it the dependencies.
Your list also contains .so files that we build and package ourselves (mostly on the top of your list). You should probably subtract those because they are going to be in your package, then check what is remaining and diff it to what is on the target operating system. There is no such thing as a "self-contained package", most packages have dependencies, and be it only libc or libstdc++ dependencies. Firefox has a few more, but no unusual ones that aren't built and packaged with Firefox itself. You should really check which of these are already available on your target system (and some *will* be) and which are not. Then we can ask OSS maintainers to provide an image that has these dependencies installed.
Also, if this doesn't work out and you cannot get a base image with the necessary dependencies, please let us know so we can talk to the respective people to get the necessary image in place. And thanks for working on this! :)
Ideally Google should make a separate image called fox-runner or so. Considering the relative importance of Firefox (and potential number of fuzz targets) I think it's justified. I still have 1-2 additional ideas how to make this work relatively easy even on base-runner. I'll report back.
As mentioned on the oss-fuzz tracker, I have this working with base-runner. The only remaining problem is that the fuzzer fails with recursion error, because of the ASAN_OPTIONS oss-fuzz sets. You can reproduce it normally (without oss-fuzz) like this. $ MOZ_RUN_GTEST=1 LIBFUZZER=1 FUZZER=StunParser ASAN_OPTIONS=detect_stack_use_after_return=1 obj-fuzz/dist/bin/firefox Running Fuzzer tests... too much recursion AddressSanitizer:DEADLYSIGNAL ================================================================= ==10==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fea497756be bp 0x7ffef4df64d0 sp 0x7ffef4df64c0 T0) ==10==The signal is caused by a WRITE memory access. ==10==Hint: address points to the zero page. SCARINESS: 10 (null-deref) #0 0x7fea497756bd in nsXPConnect::InitStatics() /src/mozilla-central/js/xpconnect/src/nsXPConnect.cpp:144:5 #1 0x7fea49709200 in xpcModuleCtor() /src/mozilla-central/js/xpconnect/src/XPCModule.cpp:13:5 #2 0x7fea4f15ce6d in Initialize() /src/mozilla-central/layout/build/nsLayoutModule.cpp:268:8 #4 0x7fea47d97ed9 in nsFactoryEntry::GetFactory() /src/mozilla-central/xpcom/components/nsComponentManager.cpp:1748:19 #5 0x7fea47d98e07 in nsComponentManagerImpl::CreateInstanceByContractID(char const*, nsISupports*, nsID const&, void**) /src/mozilla-central/xpcom/components/nsComponentManager.cpp:1046:41 #6 0x7fea47d93eb7 in nsComponentManagerImpl::GetServiceByContractID(char const*, nsID const&, void**) /src/mozilla-central/xpcom/components/nsComponentManager.cpp:1409:10 #7 0x7fea47d9c714 in nsGetServiceByContractID::operator()(nsID const&, void**) const /src/mozilla-central/xpcom/components/nsComponentManagerUtils.cpp:280:21 #8 0x7fea47c33a53 in nsCOMPtr_base::assign_from_gs_contractid(nsGetServiceByContractID, nsID const&) /src/mozilla-central/xpcom/base/nsCOMPtr.cpp:95:7 #9 0x7fea47e7423d in NS_InitXPCOM2 /src/mozilla-central/xpcom/build/XPCOMInit.cpp:719:5 #10 0x7fea5195f0f2 in (anonymous namespace)::ScopedXPCOM::ScopedXPCOM(char const*, nsIDirectoryServiceProvider*) /src/mozilla-central/tools/fuzzing/interface/harness/FuzzerTestHarness.h:73:21 #11 0x7fea5195edd8 in mozilla::FuzzerRunner::Run(int*, char***) /src/mozilla-central/tools/fuzzing/interface/harness/FuzzerRunner.cpp:28:15 #12 0x7fea5189d273 in XREMain::XRE_mainStartup(bool*) /src/mozilla-central/toolkit/xre/nsAppRunner.cpp:3932:35 #13 0x7fea518a902a in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /src/mozilla-central/toolkit/xre/nsAppRunner.cpp:4876:12 #14 0x7fea518a9e15 in XRE_main(int, char**, mozilla::BootstrapConfig const&) /src/mozilla-central/toolkit/xre/nsAppRunner.cpp:4983:21 #15 0x54816a in do_main(int, char**, char**) /src/mozilla-central/browser/app/nsBrowserApp.cpp:233:22 #16 0x5479ea in main /src/mozilla-central/browser/app/nsBrowserApp.cpp:311:16 From what I can tell, the error messages comes from JS. https://dxr.mozilla.org/mozilla-central/source/js/src/proxy/Proxy.cpp#575
We do not currently support detect_stack_use_after_return=1, I suggest omitting it for now. We should investigate the problems with that option at some point, but I don't have any spare cycles right now to do so.
Just want to write down two things, for reference or anyone who may also want to work on this. BinAST target doesn't work. Running Fuzzer tests... Fuzzing Interface: Error: No testing callback found Finished running Fuzzer tests. Image target does work, but only with detect_leaks=0, otherwise it reports many leaks immediately. This would be a rather maintenance-heavy target, as all image fuzzers very quickly produce images which cause OOM. Another problem is that it's very chatty with output like this. Expected: (abs(aColor.mBlue - data[i + 0])) <= (aFuzz), actual: 255 vs '\x1' (1) I assume it's not really meant to be used as a libfuzzer target.
(In reply to pdknsk from comment #11) > Just want to write down two things, for reference or anyone who may also > want to work on this. > > BinAST target doesn't work. > > Running Fuzzer tests... > Fuzzing Interface: Error: No testing callback found > Finished running Fuzzer tests. I have some local modifications for BinAST that I will try to land next week. > > Image target does work, but only with detect_leaks=0, otherwise it reports > many leaks immediately. This would be a rather maintenance-heavy target, as > all image fuzzers very quickly produce images which cause OOM. We need to look into this. Are these really leaks or just large allocations that we aren't protecting against? > Another problem is that it's very chatty with output like this. > > Expected: (abs(aColor.mBlue - data[i + 0])) <= (aFuzz), actual: 255 vs '\x1' > (1) > > I assume it's not really meant to be used as a libfuzzer target. We should be able to silence these. And the target is supposed to be used as a libfuzzer target. I am not running it though, :posidron is.
Flags: needinfo?(cdiehl)
We can always silence output from libFuzzer with |-close_fd_mask=3|.
True, it can produce quite quickly OOMs but there hasn't been time yet to look deeper into its origin but from what I have observed it also can happen by choosing a wrong MIME / corpus type combination. I'm not against -close_fd_mask=3 but believe it should only get added to specific targets i.e here for 'DecodeToSurface' - if we are talking about our cluster. There were some other modifications we needed to do regarding the Image target IIRC in a brief chat with :decoder but he needs to remember me again.
Flags: needinfo?(cdiehl)
Some problems with images and OOM I have observed (in general). Whoever consumes the image reads the image size from the header and pre-allocates a buffer to decode the image into. Immediate OOM with large values, and is produced by the fuzzers quickly. Simple to check on a higher (common) level. Sometimes this can't be checked on a higher level, particularly if the format supports animations. The common canvas for the animation can be checked, but internally the code may still allocate a large frame, which is then painted (cropped) on the small canvas. Another problem can be internal decompression-related allocations. So each format may require separate additional checks. That's what I meant with possibly maintenance-heavy. When I tried Image (only with image/gif so far), it produced an OOM in 1-2 minutes. On the leaks, those weren't in the target itself if I'm not mistaken, but before. I assume this might be because Image uses MOZ_FUZZING_INTERFACE_STREAM (rather than RAW)?
The problem with the leaks is sth. else. I've made a separate bug for it, and the other bug, for better tracking. https://bugzilla.mozilla.org/show_bug.cgi?id=1477844 https://bugzilla.mozilla.org/show_bug.cgi?id=1477846
It's fuzzing! :) There are still a few minor issues or topics to discuss, so I'll leave this bug open for now.
What do you think of moving/maintaining libfuzzer.content.blacklist.txt for MOZ_IPC_MESSAGE_FUZZ_BLACKLIST in-tree?
Flags: needinfo?(agaynor)
Passing off to :posidron.
Flags: needinfo?(agaynor) → needinfo?(cdiehl)
These kind of lists are usually too volatile to have them in-tree. It is easier to manage them from the outside and fetch/pass them to the respective fuzzing jobs. Unlike in-tree development/compile blacklists, like for the sanitizers.
Flags: needinfo?(cdiehl)
I've filed another related bug, which is probably caused by the fuzzing interface. I doubt anyone is interested in working on it though. Probably difficult to debug. https://bugzilla.mozilla.org/show_bug.cgi?id=1483130
libFuzzer stopped working in the past few days. Both on oss-fuzz and locally. All targets. $ MOZ_RUN_GTEST=1 LIBFUZZER=1 FUZZER=SdpParser obj-fuzz/dist/bin/firefox Running Fuzzer tests... INFO: Seed: 3954647428 INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes INFO: A corpus is not provided, starting from an empty corpus #2 INITED lim: 4 exec/s: 0 rss: 313Mb ERROR: no interesting inputs were found. Is the code instrumented for coverage? Exiting. Have there been any obvious changes recently which might've caused it? If not I'll try to bisect.
Flags: needinfo?(choller)
The only change I am remotely aware of that somewhat touches the fuzzing area is that we made changes to get a fuzzing build for android working (bug 1475573) but I'm not convinced that this could have caused this. Alex also hit the same problem. If you could bisect this, that would be really great. Unless Alex has already bisected, Alex?
Flags: needinfo?(choller) → needinfo?(agaynor)
I started bisecting earlier this afternoon. It's slow, but I'm hopeful I'll have pinned it down by tomorrow.
Flags: needinfo?(agaynor)
From which range have you started? Unless you can already exclude it, I'll just try to exclude the above Android-related change.
I started at f4aac816ce53.
It is the Android change. I tried f29562384b58 (430532) and d602a2f69ff8 (430533). Former works, latter doesn't.
:truber, can you look into this?
Flags: needinfo?(jschwartzentruber)
Yes, I had to explicitly link with -lfuzzer for the Android build to succeed, but apparently that's causing other problems. I'll make those changes conditional on Android until I can figure out how to fix it.
See Also: → 1475573
Reverting the Android change in bug 1475573 fixed this.
Flags: needinfo?(jschwartzentruber)
I added ContentParentIPC with the blacklist from fuzzdata, which seems to be working well. There were a few nullptr reports so far, no security bugs. Whoever owns the target should probably be added to oss-fuzz.
I think with the recent changes this bug can be closed. It was a bit more work than I expected, but then it also turned out better than I expected. ASAN and UBSAN are enabled, and coverage works too. Obviously there's still quite a lot of uncovered code. https://storage.googleapis.com/oss-fuzz-coverage/glib/reports/20180925/linux/report.html The only remaining obstacle is missing support for shared memory in the oss-fuzz runtime which affects two targets. That'll be solved on the oss-fuzz side in a few months or so. I think new targets should not be added until then. I'll unsubscribe myself from notifications in a few weeks or so, when everything is proven to be stable. I don't need access to found security bugs. PS. I think it might not be a bad idea to eventually move some of the files to mozilla-central or github/MozillaSecurity. I'm thinking of *options and mozconfig*, perhaps even target.c and build.sh.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
FF might use other tools as well, I'll have to ask. As for us, I'm probably not the person to ask if we should implement this. From reading the FF bug it looks like the setup and maintenance process is pretty involved and can get into more compiler options and fixing C++ code than I'm comfortable with. It also doesn't seem to be integrated with Taskcluster so I don't know that I would be involved much anyway. I think we should look into it, but maybe should make the call on if/when to implement.
You need to log in before you can comment on or make changes to this bug.