1260800 - OpenH264: UBSan signed integer overflow in [@WelsDec::ParseResidualBlockCabac]

Status: RESOLVED FIXED

(Core :: Audio/Video: GMP, defect)

Description

[Tyson Smith [:tsmith]]

Attachment: test_case.264

Found while fuzzing openh264 revision c0641f40d91b8cb47f287bf26dc48d51a476c325

codec/decoder/core/src/parse_mb_syn_cabac.cpp:944:78: runtime error: signed integer overflow: -64886 * 43608 cannot be represented in type 'int'
    #0 0x693e9b in WelsDec::ParseResidualBlockCabac(WelsDec::TagNeighborAvail*, unsigned char*, WelsCommon::TagBitStringAux*, int, int, unsigned char const*, int, short*, unsigned char, WelsDec::TagWelsDecoderContext*) /home/user/code/openh264/codec/decoder/core/src/parse_mb_syn_cabac.cpp:944:78
    #1 0x6488d1 in WelsDec::WelsDecodeMbCabacPSliceBaseMode0(WelsDec::TagWelsDecoderContext*, WelsDec::TagNeighborAvail*, unsigned int&) /home/user/code/openh264/codec/decoder/core/src/decode_slice.cpp:973:15
    #2 0x64f871 in WelsDec::WelsDecodeMbCabacPSlice(WelsDec::TagWelsDecoderContext*, WelsDec::TagNalUnit*, unsigned int&) /home/user/code/openh264/codec/decoder/core/src/decode_slice.cpp:1101:3
    #3 0x6554bd in WelsDec::WelsDecodeSlice(WelsDec::TagWelsDecoderContext*, bool, WelsDec::TagNalUnit*) /home/user/code/openh264/codec/decoder/core/src/decode_slice.cpp:1229:12
    #4 0x55d45b in WelsDec::DecodeCurrentAccessUnit(WelsDec::TagWelsDecoderContext*, unsigned char**, TagBufferInfo*) /home/user/code/openh264/codec/decoder/core/src/decoder_core.cpp:2278:16
    #5 0x558506 in WelsDec::ConstructAccessUnit(WelsDec::TagWelsDecoderContext*, unsigned char**, TagBufferInfo*) /home/user/code/openh264/codec/decoder/core/src/decoder_core.cpp:2003:10
    #6 0x525401 in WelsDecodeBs /home/user/code/openh264/codec/decoder/core/src/decoder.cpp:788:7
    #7 0x5083c5 in WelsDec::CWelsDecoder::DecodeFrame2(unsigned char const*, int, unsigned char**, TagBufferInfo*) /home/user/code/openh264/codec/decoder/plus/src/welsDecoderExt.cpp:503:3
    #8 0x5067c5 in WelsDec::CWelsDecoder::DecodeFrameNoDelay(unsigned char const*, int, unsigned char**, TagBufferInfo*) /home/user/code/openh264/codec/decoder/plus/src/welsDecoderExt.cpp:432:11
    #9 0x4f9acf in H264DecodeInstance(ISVCDecoder*, char const*, char const*, int&, int&, char const*, char const*) /home/user/code/openh264/codec/console/dec/src/h264dec.cpp:208:5
    #10 0x4fb2dc in main /home/user/code/openh264/codec/console/dec/src/h264dec.cpp:347:5
    #11 0x7fe3b37a4ec4 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21ec4)
    #12 0x42e375 in _start (/home/ubuntu/build/build/h264dec_ub_asan+0x42e375)

Comment 1

[Tyson Smith [:tsmith]]

This is still reproducible as of openh264 commit 1eb735299a4606c01b7abbb2f561e7a904e08c1c

Comment 2

[Haibo Zhu]

This case has been fixed in the master branch, commit 46220cf, please help to verify it.