UBSan: left shift of negative value rec_mb.cpp:246:14

RESOLVED FIXED

Status

External Software Affecting Firefox
OpenH264
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: tsmith, Unassigned)

Tracking

(Blocks: 2 bugs, {sec-audit, testcase})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

3 years ago
This was found with fuzzing+UBSan build.

Output from UBSan:
codec/decoder/core/src/rec_mb.cpp:246:14: runtime error: left shift of negative value -30

Which is referring to:
iFullMVx = WELS_CLIP3 (iFullMVx, ((-PADDING_LENGTH + 2) << 2), ((pMCRefMem->iPicWidth + PADDING_LENGTH - 19) << 2));
(Reporter)

Updated

3 years ago
Blocks: 948160, 959432
(Reporter)

Comment 1

3 years ago
Created attachment 8665074 [details]
test_case.264

Updated

3 years ago
Depends on: 1170319

Comment 2

3 years ago
Hi Tyson, we actually know this but we don't think it is a problem we need to fixed, this left shift won't make a problem since the data range of this input is limited.

Updated

3 years ago
Flags: needinfo?(twsmith)
(Reporter)

Comment 3

3 years ago
(In reply to sijchen from comment #2)
> Hi Tyson, we actually know this but we don't think it is a problem we need
> to fixed, this left shift won't make a problem since the data range of this
> input is limited.

This is a runtime error reported by UBSan and the attached test case does actually trigger this issue. I am not exactly sure what the implications are that depends on how data resulting from this operation is used. I will attach a call stack to help debug this issue.
Flags: needinfo?(twsmith)
(Reporter)

Comment 4

3 years ago
Created attachment 8670979 [details]
call_stack.txt

Updated

3 years ago
Flags: needinfo?(haibozhu)

Comment 5

3 years ago
I think it is a common false alarm. The purpose of this implementation is to get the 4 times value of the motion vector or the padding offset. Here, the motion vector is at the range of -512 to 512 and the min value of the offset is -32. So the calculated results will not exceed the expected value. I think we can ignore this type of alarms.
Flags: needinfo?(haibozhu)
(Reporter)

Comment 6

3 years ago
Well it's not a false alarm, things just seem to be working at the moment. The problem is that we are treating a '<< 2' as if it were a '* 4'. The problem isn't the value range, yes it should be a concern, but in this case what is happening in undefined regardless. That means we are at the mercy of the compiler. Different things could happen with different compilers, different build types: x86, arm, 32 or 64-bit, optimized, debug, non-debug, combination of the build types or even different compiler version.

So TLDR; a left shift of a negative number by 2 is NOT equal to multiplying it by 4 although it may act that way some or most of the time (even when working within seemingly unaffected value ranges).

Comment 7

3 years ago
Although I think it is not a bug, I have added a fix to remove it at commit af6a9a8 at master branch and commit 9c88070 at openh264v1.5 branch. Please help to verify it.
(Reporter)

Comment 8

3 years ago
Thanks Haibo. Verified with commit: fb61733b2.

Comment 9

3 years ago
Talked to Sijia. Close it here since it is fixed and verified.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Group: media-core-security → core-security-release

Updated

3 years ago
Depends on: 1217114
No longer depends on: 1170319

Comment 10

3 years ago
Should be released with v1.5.
Depends on: 1170319
No longer depends on: 1217114
Group: core-security-release
You need to log in before you can comment on or make changes to this bug.