1486223 - OpenH264: index out of bound [@ WelsDec::FilteringEdgeLumaV]

Status: RESOLVED FIXED

(Core :: Audio/Video: GMP, defect, P2)

Description

[Tyson Smith [:tsmith]]

Attachment: testcase.264

Found while fuzzing openh264 revision 1b3980b3437e83f30001e9b7dfdf4a98e69b87bc

Build with "-fsanitize=undefined"

To reproduce:
./h264dec testcase.264 /dev/null

codec/decoder/core/src/deblocking.cpp:759:5: runtime error: index 18 out of bounds for type 'const int8_t [4]'
    #0 0x6683ca in WelsDec::FilteringEdgeLumaV(WelsDec::tagDeblockingFilter*, unsigned char*, int, unsigned char*) codec/decoder/core/src/deblocking.cpp:759:5
    #1 0x67014f in WelsDec::DeblockingInterMb(WelsDec::TagDqLayer*, WelsDec::tagDeblockingFilter*, unsigned char (*) [4][4], int) codec/decoder/core/src/deblocking.cpp:973:5
    #2 0x679f52 in WelsDec::WelsDeblockingMb(WelsDec::TagDqLayer*, WelsDec::tagDeblockingFilter*, int) codec/decoder/core/src/deblocking.cpp:1242:5
    #3 0x6950e5 in WelsDec::WelsDeblockingFilterSlice(WelsDec::TagWelsDecoderContext*, void (*)(WelsDec::TagDqLayer*, WelsDec::tagDeblockingFilter*, int)) codec/decoder/core/src/deblocking.cpp:1294:7
    #4 0x69ec90 in WelsDec::WelsTargetSliceConstruction(WelsDec::TagWelsDecoderContext*) codec/decoder/core/src/decode_slice.cpp:156:5
    #5 0x59b09e in WelsDec::WelsDecodeConstructSlice(WelsDec::TagWelsDecoderContext*, WelsDec::TagNalUnit*) codec/decoder/core/src/decoder_core.cpp:290:19
    #6 0x59b09e in WelsDec::DecodeCurrentAccessUnit(WelsDec::TagWelsDecoderContext*, unsigned char**, TagBufferInfo*) codec/decoder/core/src/decoder_core.cpp:2568
    #7 0x595e93 in WelsDec::ConstructAccessUnit(WelsDec::TagWelsDecoderContext*, unsigned char**, TagBufferInfo*) codec/decoder/core/src/decoder_core.cpp:2254:10
    #8 0x55a70e in WelsDecodeBs codec/decoder/core/src/decoder.cpp:798:7
    #9 0x52e3f2 in WelsDec::CWelsDecoder::DecodeFrame2(unsigned char const*, int, unsigned char**, TagBufferInfo*) codec/decoder/plus/src/welsDecoderExt.cpp:570:3
    #10 0x52c584 in WelsDec::CWelsDecoder::DecodeFrameNoDelay(unsigned char const*, int, unsigned char**, TagBufferInfo*) codec/decoder/plus/src/welsDecoderExt.cpp:495:11
    #11 0x516be9 in H264DecodeInstance(ISVCDecoder*, char const*, char const*, int&, int&, char const*, char const*, int, bool) codec/console/dec/src/h264dec.cpp:226:17
    #12 0x51c3df in main codec/console/dec/src/h264dec.cpp:510:3
    #13 0x7efc5555182f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:291
    #14 0x41d6d8 in _start (h264dec+0x41d6d8)

Comment 1

[xiaotian shi]

confirmed the problem and is being fixed.

Comment 2

[Nils Ohlmeier [:drno]]

Xiaotian is this still being worked on upstream?

Comment 3

[wayne]

This bug has been fixed in latest openh265 master branch. Please kindly have a look. Thanks.

Comment 4

[Tyson Smith [:tsmith]]

Verified with openh264 commit 70eeb783515dbfee3e0c781d6667838caba5113b