479770 - large |frame_width|, |frame_height| cause video crash

Status: RESOLVED DUPLICATE of bug 504843

(Core :: Audio/Video, defect)

Description

[georgi - hopefully not receiving bugspam]

Attachment: short crashing video

chronologically tried to reach the int overflows in |oc_calloc_2d|

http://mxr.mozilla.org/mozilla-central/source/media/libtheora/lib/dec/internal.c#320

accidentally random opportunistic values of |frame_width| and |frame_height| caused another crash when openning a video file (on both 64 and 32 bit linux).

the attached video was edited with hexeditor:
frame_width =0xffff
frame_height=0xffff
(both are transformed |<<4|)
checksum adjusted by hand

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x45280950 (LWP 7883)]
0x00007f72fbc36300 in memset () from /lib/libc.so.6
(gdb) bt
#0  0x00007f72fbc36300 in memset () from /lib/libc.so.6
#1  0x00007f72ee0d304b in oc_sb_create_plane_mapping (_sbs=0x0, _frag0=0, 
    _hfrags=131070, _vfrags=131070)
    at /opt/pub/firefox-central/src/media/libtheora/lib/dec/state.c:101
#2  0x00007f72ee0d3d95 in oc_state_frarray_init (_state=0x7f72e6245000)
    at /opt/pub/firefox-central/src/media/libtheora/lib/dec/state.c:442
#3  0x00007f72ee0d43d7 in oc_state_init (_state=0x7f72e6245000, 
    _info=0x4527fa30)
    at /opt/pub/firefox-central/src/media/libtheora/lib/dec/state.c:580
#4  0x00007f72ee0d6d0c in oc_dec_init (_dec=0x7f72e6245000, _info=0x4527fa30, 
    _setup=0x7f72e63be800)
    at /opt/pub/firefox-central/src/media/libtheora/lib/dec/decode.c:164
#5  0x00007f72ee0db800 in th_decode_alloc (_info=0x4527fa30, 
    _setup=0x7f72e63be800)
    at /opt/pub/firefox-central/src/media/libtheora/lib/dec/decode.c:1795
#6  0x00007f72ee0d21fb in theora_decode_init (_td=0x7f72e66582c8,


(gdb) frame 1
#1  0x00007f72ee0d304b in oc_sb_create_plane_mapping (_sbs=0x0, _frag0=0, 
    _hfrags=131070, _vfrags=131070)
    at /opt/pub/firefox-central/src/media/libtheora/lib/dec/state.c:101
101           memset(sb->map[0],0xFF,sizeof(sb->map));
Current language:  auto; currently c
(gdb) p &sb->map[0]
$1 = (int (*)[4]) 0x4
(gdb) frame 0
#0  0x00007f72fbc36300 in memset () from /lib/libc.so.6
(gdb) x/i $pc
0x7f72fbc36300 <memset+48>:     mov    %sil,(%rcx)
(gdb) p/x $rcx
$2 = 0x4

Comment 1

[georgi - hopefully not receiving bugspam]

This is xine (X11 gui) - a free video player v0.99.6cvs.
(c) 2000-2007 The xine Team.
xiTK received SIGSEGV signal, RIP.

Comment 2

[georgi - hopefully not receiving bugspam]

MPlayer interrupted by signal 11 in module: init_video_codec
- MPlayer crashed by bad usage of CPU/FPU/RAM.
  Recompile MPlayer with --enable-debug and make a 'gdb' backtrace and
  disassembly. Details in DOCS/HTML/en/bugreports_what.html#bugreports_crash.

Comment 3

[georgi - hopefully not receiving bugspam]

since this affects xine and mplayer probably someone mainstream should be cc'ed

Comment 4

[Chris Pearce [:cpearce (Not reading bugmail)]]

Attachment: Smaller testcase; width=height=2^15

Testcase, care of Greg Maxwell. Width and height of video are 2^15, results in failing malloc and crash.

Comment 5

[cajbir (:cajbir)]

Might also want to see bug 504843 and bug 505811 for discussion on large sizes with malloc, etc.

Comment 6

[Chris Pearce [:cpearce (Not reading bugmail)]]

Fixed on trunk and 192, probably by bug 504843.

191 is still crashing with this test file.

I'll dupe this as bug 504843, since that fixes this.