1463424 - Two divide-by-zero crashes in qcms

Status: RESOLVED FIXED

(Core :: Graphics: Color Management, defect)

Description

[Alex Gaynor [:Alex_Gaynor]]

Attachment: clusterfuzz-testcase-minimized-fuzz-5192209237278720

These were found by Google's OSS-Fuzz run on QCMS.

each of the attachments reproduces when run as an input to: https://github.com/google/oss-fuzz/blob/master/projects/qcms/fuzz.cc#L46

The two stacks are:

matrix.c:71:9: runtime error: division by zero
	#0 0x436219 in matrix_invert /src/firefox/gfx/qcms/matrix.c:71:9
	#1 0x439ed2 in qcms_transform_create /src/firefox/gfx/qcms/transform.c:1338:16
	#2 0x466272 in transform(_qcms_profile*, _qcms_profile*, int) /src/fuzz.cc:24:31
	#3 0x466163 in LLVMFuzzerTestOneInput /src/fuzz.cc:65:3

and

transform_util.c:423:56: runtime error: division by zero
	#0 0x43c404 in compute_precache /src/firefox/gfx/qcms/transform_util.c:423:56
	#1 0x437b92 in qcms_profile_precache_output_transform /src/firefox/gfx/qcms/transform.c:1139:6
	#2 0x46625e in transform(_qcms_profile*, _qcms_profile*, int) /src/fuzz.cc:22:3
	#3 0x466163 in LLVMFuzzerTestOneInput /src/fuzz.cc:65:3

Comment 1

[Alex Gaynor [:Alex_Gaynor]]

Attachment: clusterfuzz-testcase-minimized-fuzz-6212944353296384

Comment 2

[Alex Gaynor [:Alex_Gaynor]]

Attachment: clusterfuzz-testcase-minimized-fuzz-5721929230057472

Here's another divide by zero:

matrix.c:71:9: runtime error: division by zero
	#0 0x436219 in matrix_invert /src/firefox/gfx/qcms/matrix.c:71:9
	#1 0x43251a in qcms_modular_transform_create_output /src/firefox/gfx/qcms/chain.c:797:23
	#2 0x4309a5 in qcms_modular_transform_create /src/firefox/gfx/qcms/chain.c:942:16
	#3 0x4307c0 in qcms_chain_transform /src/firefox/gfx/qcms/chain.c:986:50
	#4 0x437f1c in qcms_transform_precacheLUT_float /src/firefox/gfx/qcms/transform.c:1182:9
	#5 0x439dc7 in qcms_transform_create /src/firefox/gfx/qcms/transform.c:1248:28
	#6 0x4662a2 in transform(_qcms_profile*, _qcms_profile*, int) /src/fuzz.cc:24:31
	#7 0x466193 in LLVMFuzzerTestOneInput /src/fuzz.cc:65:3
	#8 0x44a6d0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:526:13
	#9 0x43cc5c in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
	#10 0x44037e in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:706:9
	#11 0x43c968 in main /src/libfuzzer/FuzzerMain.cpp:20:10
	#12 0x7f7ee132e82f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
	#13 0x405c98 in _start

Comment 3

[Nicolas Silva [:nical]]

Attachment: Fix divide by zeroes and make sure callers andle invalid matrices.

Early return in matrix_invert if the determinant is null after setting the invalid flag, and make sure callers (and some times callers of the callers, properly deal with the invalid flag).

Comment 4

[Nicolas Silva [:nical]]

Comment on attachment 8980544 [details] [diff] [review]
Fix divide by zeroes and make sure callers andle invalid matrices.

(Adding Bas as potential reviewer in case Jeff is unavailable for a while)

Comment 5

[Bas Schouten (:bas.schouten)]

Comment on attachment 8980544 [details] [diff] [review]
Fix divide by zeroes and make sure callers andle invalid matrices.

Review of attachment 8980544 [details] [diff] [review]:
-----------------------------------------------------------------

Good job! (qcms coding style is gross)

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/39607a36ad6b
Fix divide by zeroes in qcms. r=Bas

Comment 7

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/39607a36ad6b