Two divide-by-zero crashes in qcms

RESOLVED FIXED in Firefox 62

Status

()

defect
RESOLVED FIXED
a year ago
a year ago

People

(Reporter: Alex_Gaynor, Assigned: nical)

Tracking

Trunk
mozilla62
Points:
---

Firefox Tracking Flags

(firefox62 fixed)

Details

(Whiteboard: gfx-noted)

Attachments

(4 attachments)

Reporter

Description

a year ago
These were found by Google's OSS-Fuzz run on QCMS.

each of the attachments reproduces when run as an input to: https://github.com/google/oss-fuzz/blob/master/projects/qcms/fuzz.cc#L46

The two stacks are:

matrix.c:71:9: runtime error: division by zero
	#0 0x436219 in matrix_invert /src/firefox/gfx/qcms/matrix.c:71:9
	#1 0x439ed2 in qcms_transform_create /src/firefox/gfx/qcms/transform.c:1338:16
	#2 0x466272 in transform(_qcms_profile*, _qcms_profile*, int) /src/fuzz.cc:24:31
	#3 0x466163 in LLVMFuzzerTestOneInput /src/fuzz.cc:65:3

and

transform_util.c:423:56: runtime error: division by zero
	#0 0x43c404 in compute_precache /src/firefox/gfx/qcms/transform_util.c:423:56
	#1 0x437b92 in qcms_profile_precache_output_transform /src/firefox/gfx/qcms/transform.c:1139:6
	#2 0x46625e in transform(_qcms_profile*, _qcms_profile*, int) /src/fuzz.cc:22:3
	#3 0x466163 in LLVMFuzzerTestOneInput /src/fuzz.cc:65:3
Reporter

Comment 2

a year ago
Here's another divide by zero:

matrix.c:71:9: runtime error: division by zero
	#0 0x436219 in matrix_invert /src/firefox/gfx/qcms/matrix.c:71:9
	#1 0x43251a in qcms_modular_transform_create_output /src/firefox/gfx/qcms/chain.c:797:23
	#2 0x4309a5 in qcms_modular_transform_create /src/firefox/gfx/qcms/chain.c:942:16
	#3 0x4307c0 in qcms_chain_transform /src/firefox/gfx/qcms/chain.c:986:50
	#4 0x437f1c in qcms_transform_precacheLUT_float /src/firefox/gfx/qcms/transform.c:1182:9
	#5 0x439dc7 in qcms_transform_create /src/firefox/gfx/qcms/transform.c:1248:28
	#6 0x4662a2 in transform(_qcms_profile*, _qcms_profile*, int) /src/fuzz.cc:24:31
	#7 0x466193 in LLVMFuzzerTestOneInput /src/fuzz.cc:65:3
	#8 0x44a6d0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:526:13
	#9 0x43cc5c in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
	#10 0x44037e in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:706:9
	#11 0x43c968 in main /src/libfuzzer/FuzzerMain.cpp:20:10
	#12 0x7f7ee132e82f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
	#13 0x405c98 in _start
Assignee

Comment 3

a year ago
Early return in matrix_invert if the determinant is null after setting the invalid flag, and make sure callers (and some times callers of the callers, properly deal with the invalid flag).
Assignee: nobody → nical.bugzilla
Attachment #8980544 - Flags: review?(jmuizelaar)
Assignee

Updated

a year ago
Whiteboard: gfx-noted
Assignee

Comment 4

a year ago
Comment on attachment 8980544 [details] [diff] [review]
Fix divide by zeroes and make sure callers andle invalid matrices.

(Adding Bas as potential reviewer in case Jeff is unavailable for a while)
Attachment #8980544 - Flags: review?(bas)
Comment on attachment 8980544 [details] [diff] [review]
Fix divide by zeroes and make sure callers andle invalid matrices.

Review of attachment 8980544 [details] [diff] [review]:
-----------------------------------------------------------------

Good job! (qcms coding style is gross)
Attachment #8980544 - Flags: review?(bas) → review+
Assignee

Updated

a year ago
Attachment #8980544 - Flags: review?(jmuizelaar)

Comment 7

a year ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/39607a36ad6b
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
Reporter

Updated

a year ago
See Also: → 1465075
You need to log in before you can comment on or make changes to this bug.