1215757 - OpenH264: UBSan signed integer overflow in [@WelsDec::ParseResidualBlockCabac]

Status: RESOLVED FIXED

(Core :: Audio/Video: GMP, defect)

Description

[Tyson Smith [:tsmith]]

Attachment: call_stack.txt

codec/decoder/core/src/parse_mb_syn_cabac.cpp:945:53: runtime error: signed integer overflow: 65978130 * 512 cannot be represented in type 'int'

Comment 1

[Tyson Smith [:tsmith]]

Attachment: test_case.264

Comment 2

[Haibo Zhu]

I have removed this undefined-behavior warning in the commit 3ee8784 at master branch, and the newest version of openh264v1.5 branch and v1.5-Firefox39 branch. Please help to verify it.

Comment 3

[Tyson Smith [:tsmith]]

Verified with commit: 3ee8784c0

Comment 5

[Tyson Smith [:tsmith]]

This was just uncovered again using this commit: https://github.com/cisco/openh264/commit/1a2606f45d36c7ae030826c4e0859052c5d9486b

Comment 6

[Tyson Smith [:tsmith]]

Attachment: test_case_2.264

Comment 7

[Tyson Smith [:tsmith]]

Attachment: call_stack_2.txt

Comment 8

[Tyson Smith [:tsmith]]

This was fixed ages ago but never marked. Another bug with the same stack was uncovered (bug 1260800).