<a class="header-button" href="https://bugzilla.mozilla.org/home" title="Go to home page"> Bugzilla

BugBot [:suhaib / :marco/ :calixte]

Assignee

Comment 3

•

9 months ago

:padenot, do you have any thoughts about what might be causing this?

Flags: needinfo?(padenot)

Comment 4

•

9 months ago

Set release status flags based on info from the regressing bug 1940449

status-firefox134: --- → unaffected

status-firefox135: --- → unaffected

status-firefox-esr128: --- → unaffected

Alastor Wu [:alwu]

Updated

•

9 months ago

Severity: -- → S3

Priority: -- → P2

Assignee

Comment 5

•

9 months ago

I have this reproing consistently locally with a tsan build, dom/media/test/test_background_video_resume_after_end_show_last_frame.html does a good job triggering it on my machine. With any luck I'll be able to get it reproing with rr and get something up on Pernosco.

Assignee

Comment 6

•

9 months ago

Actually, scratch that last message - was running into a different issue. Will try a few more things and see if I can get a capture going.

Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)

Assignee

Comment 7

•

9 months ago

Update - I have this reproing locally after all, no luck getting a capture yet though. Will run it on a loop in the background, with any luck it'll repro under rr.

23:22.82 GECKO(2296694) ==================
23:22.82 GECKO(2296694) WARNING: ThreadSanitizer: data race (pid=2318339)
23:22.82 GECKO(2296694)   Read of size 1 at 0x72b40002f848 by thread T3026 (mutexes: write M0):
23:22.82 GECKO(2296694)     #0 find_ref_mvs /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9mvs.c:179:13 (libmozavcodec.so+0x275038) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.82 GECKO(2296694)     #1 ff_vp9_fill_mv /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9mvs.c:303:9 (libmozavcodec.so+0x270932) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.82 GECKO(2296694)     #2 decode_mode /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9block.c:670:13 (libmozavcodec.so+0x1e55fb) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.82 GECKO(2296694)     #3 ff_vp9_decode_block /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9block.c:1290:9 (libmozavcodec.so+0x1e55fb)
23:22.82 GECKO(2296694)     #4 decode_sb /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9.c:1116:17 (libmozavcodec.so+0x1d43ca) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.82 GECKO(2296694)     #5 decode_tiles /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9.c:1345:25 (libmozavcodec.so+0x1c7ced) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.82 GECKO(2296694)     #6 vp9_decode_frame /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9.c:1724:19 (libmozavcodec.so+0x1c7ced)
23:22.82 GECKO(2296694)     #7 decode_simple_internal /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/decode.c:443:16 (libmozavcodec.so+0xe694e) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.82 GECKO(2296694)     #8 decode_simple_receive_frame /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/decode.c:613:15 (libmozavcodec.so+0xe694e)
23:22.82 GECKO(2296694)     #9 ff_decode_receive_frame_internal /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/decode.c:649:15 (libmozavcodec.so+0xe694e)
23:22.82 GECKO(2296694)     #10 frame_worker_thread /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread_frame.c:295:19 (libmozavcodec.so+0x187c6e) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.82 GECKO(2296694)   Previous write of size 1 at 0x72b40002f848 by thread T3025 (mutexes: write M1):
23:22.82 GECKO(2296694)     #0 decode_mode /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9block.c:796:30 (libmozavcodec.so+0x1da33d) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.82 GECKO(2296694)     #1 ff_vp9_decode_block /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9block.c:1290:9 (libmozavcodec.so+0x1da33d)
23:22.82 GECKO(2296694)     #2 decode_sb /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9.c:1116:17 (libmozavcodec.so+0x1d43ca) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.82 GECKO(2296694)     #3 decode_sb /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9.c:1131:17 (libmozavcodec.so+0x1d3fe0) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.82 GECKO(2296694)     #4 decode_tiles /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9.c:1345:25 (libmozavcodec.so+0x1c7ced) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.82 GECKO(2296694)     #5 vp9_decode_frame /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9.c:1724:19 (libmozavcodec.so+0x1c7ced)
23:22.82 GECKO(2296694)     #6 decode_simple_internal /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/decode.c:443:16 (libmozavcodec.so+0xe694e) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.82 GECKO(2296694)     #7 decode_simple_receive_frame /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/decode.c:613:15 (libmozavcodec.so+0xe694e)
23:22.82 GECKO(2296694)     #8 ff_decode_receive_frame_internal /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/decode.c:649:15 (libmozavcodec.so+0xe694e)
23:22.82 GECKO(2296694)     #9 frame_worker_thread /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread_frame.c:295:19 (libmozavcodec.so+0x187c6e) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.82 GECKO(2296694)   Location is heap block of size 37504 at 0x72b400028000 allocated by thread T3026:
23:22.82 GECKO(2296694)     #0 posix_memalign /builds/worker/fetches/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:859:3 (firefox+0xce18a) (BuildId: 44825889e7e9b8b86517b2a8796a6eed)
23:22.83 GECKO(2296694)     #1 av_malloc /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavutil/mem.c:107:9 (libmozavutil.so+0x5a672) (BuildId: f9a91cb40d35d6223bf5782758f03fba)
23:22.83 GECKO(2296694)     #2 av_refstruct_alloc_ext_c /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavutil/refstruct.c:109:11 (libmozavutil.so+0x6a1ac) (BuildId: f9a91cb40d35d6223bf5782758f03fba)
23:22.83 GECKO(2296694)     #3 av_refstruct_alloc_ext /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavutil/refstruct.h:97:12 (libmozavutil.so+0x6a1ac)
23:22.83 GECKO(2296694)     #4 refstruct_pool_get_ext /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavutil/refstruct.c:269:15 (libmozavutil.so+0x6a1ac)
23:22.83 GECKO(2296694)     #5 av_refstruct_pool_get /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavutil/refstruct.c:300:5 (libmozavutil.so+0x6a1ac)
23:22.83 GECKO(2296694)     #6 vp9_frame_alloc /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9.c:126:20 (libmozavcodec.so+0x1c5b55) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.83 GECKO(2296694)     #7 vp9_decode_frame /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9.c:1592:16 (libmozavcodec.so+0x1c5b55)
23:22.83 GECKO(2296694)     #8 decode_simple_internal /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/decode.c:443:16 (libmozavcodec.so+0xe694e) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.83 GECKO(2296694)     #9 decode_simple_receive_frame /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/decode.c:613:15 (libmozavcodec.so+0xe694e)
23:22.83 GECKO(2296694)     #10 ff_decode_receive_frame_internal /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/decode.c:649:15 (libmozavcodec.so+0xe694e)
23:22.83 GECKO(2296694)     #11 frame_worker_thread /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread_frame.c:295:19 (libmozavcodec.so+0x187c6e) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.83 GECKO(2296694)   Mutex M0 (0x725800047ef0) created at:
23:22.83 GECKO(2296694)     #0 pthread_mutex_init /builds/worker/fetches/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1327:3 (firefox+0xd0203) (BuildId: 44825889e7e9b8b86517b2a8796a6eed)
23:22.83 GECKO(2296694)     #1 ff_pthread_init /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread.c:120:5 (libmozavcodec.so+0x184d6b) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.83 GECKO(2296694)     #2 init_thread /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread_frame.c:873:11 (libmozavcodec.so+0x18814a) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.83 GECKO(2296694)     #3 ff_frame_thread_init /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread_frame.c:976:15 (libmozavcodec.so+0x18704b) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.83 GECKO(2296694)     #4 ff_thread_init /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread.c:79:16 (libmozavcodec.so+0x184bd1) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.83 GECKO(2296694)     #5 avcodec_open2 /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/avcodec.c:323:15 (libmozavcodec.so+0xbf622) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.83 GECKO(2296694)     #6 mozilla::FFmpegDataDecoder<46465650>::InitSWDecoder(AVDictionary**) /home/ashley/git/mozilla-unified-linux/dom/media/platforms/ffmpeg/FFmpegDataDecoder.cpp:136:7 (libxul.so+0x9fe92c6) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #7 mozilla::FFmpegVideoDecoder<46465650>::Init() /home/ashley/git/mozilla-unified-linux/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp:648:20 (libxul.so+0x9ffec26) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #8 operator() /home/ashley/git/mozilla-unified-linux/dom/media/platforms/wrappers/MediaDataDecoderProxy.cpp:18:33 (libxul.so+0x9f6d033) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #9 mozilla::detail::ProxyFunctionRunnable<mozilla::MediaDataDecoderProxy::Init()::$_0, mozilla::MozPromise<mozilla::TrackInfo::TrackType, mozilla::MediaResult, true>>::Run() /home/ashley/git/mozilla-unified-linux/objdir-ff-tsan/dist/include/mozilla/MozPromise.h:1816:29 (libxul.so+0x9f6d033)
23:22.83 GECKO(2296694)     #10 mozilla::TaskQueue::Runner::Run() /home/ashley/git/mozilla-unified-linux/xpcom/threads/TaskQueue.cpp:261:20 (libxul.so+0x5207ed4) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #11 nsThreadPool::Run() /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThreadPool.cpp:456:14 (libxul.so+0x523a96f) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #12 nsThread::ProcessNextEvent(bool, bool*) /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThread.cpp:1153:16 (libxul.so+0x5232594) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #13 NS_ProcessNextEvent(nsIThread*, bool) /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThreadUtils.cpp:480:10 (libxul.so+0x52380d4) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #14 mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /home/ashley/git/mozilla-unified-linux/ipc/glue/MessagePump.cpp:299:20 (libxul.so+0x5ff4c0e) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #15 RunInternal /home/ashley/git/mozilla-unified-linux/ipc/chromium/src/base/message_loop.cc:369:10 (libxul.so+0x5f008e8) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #16 RunHandler /home/ashley/git/mozilla-unified-linux/ipc/chromium/src/base/message_loop.cc:362:3 (libxul.so+0x5f008e8)
23:22.83 GECKO(2296694)     #17 MessageLoop::Run() /home/ashley/git/mozilla-unified-linux/ipc/chromium/src/base/message_loop.cc:344:3 (libxul.so+0x5f008e8)
23:22.83 GECKO(2296694)     #18 nsThread::ThreadFunc(void*) /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThread.cpp:366:10 (libxul.so+0x522dd43) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #19 _pt_root /home/ashley/git/mozilla-unified-linux/nsprpub/pr/src/pthreads/ptthread.c:191:3 (libnspr4.so+0x501e8) (BuildId: 2caf66d365269d1f312e2517b537c84e)
23:22.83 GECKO(2296694)   Mutex M1 (0x725800047da8) created at:
23:22.83 GECKO(2296694)     #0 pthread_mutex_init /builds/worker/fetches/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1327:3 (firefox+0xd0203) (BuildId: 44825889e7e9b8b86517b2a8796a6eed)
23:22.83 GECKO(2296694)     #1 ff_pthread_init /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread.c:120:5 (libmozavcodec.so+0x184d6b) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.83 GECKO(2296694)     #2 init_thread /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread_frame.c:873:11 (libmozavcodec.so+0x18814a) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.83 GECKO(2296694)     #3 ff_frame_thread_init /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread_frame.c:976:15 (libmozavcodec.so+0x18704b) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.83 GECKO(2296694)     #4 ff_thread_init /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread.c:79:16 (libmozavcodec.so+0x184bd1) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.83 GECKO(2296694)     #5 avcodec_open2 /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/avcodec.c:323:15 (libmozavcodec.so+0xbf622) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.83 GECKO(2296694)     #6 mozilla::FFmpegDataDecoder<46465650>::InitSWDecoder(AVDictionary**) /home/ashley/git/mozilla-unified-linux/dom/media/platforms/ffmpeg/FFmpegDataDecoder.cpp:136:7 (libxul.so+0x9fe92c6) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #7 mozilla::FFmpegVideoDecoder<46465650>::Init() /home/ashley/git/mozilla-unified-linux/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp:648:20 (libxul.so+0x9ffec26) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #8 operator() /home/ashley/git/mozilla-unified-linux/dom/media/platforms/wrappers/MediaDataDecoderProxy.cpp:18:33 (libxul.so+0x9f6d033) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #9 mozilla::detail::ProxyFunctionRunnable<mozilla::MediaDataDecoderProxy::Init()::$_0, mozilla::MozPromise<mozilla::TrackInfo::TrackType, mozilla::MediaResult, true>>::Run() /home/ashley/git/mozilla-unified-linux/objdir-ff-tsan/dist/include/mozilla/MozPromise.h:1816:29 (libxul.so+0x9f6d033)
23:22.83 GECKO(2296694)     #10 mozilla::TaskQueue::Runner::Run() /home/ashley/git/mozilla-unified-linux/xpcom/threads/TaskQueue.cpp:261:20 (libxul.so+0x5207ed4) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.83 GECKO(2296694)     #11 nsThreadPool::Run() /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThreadPool.cpp:456:14 (libxul.so+0x523a96f) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #12 nsThread::ProcessNextEvent(bool, bool*) /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThread.cpp:1153:16 (libxul.so+0x5232594) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #13 NS_ProcessNextEvent(nsIThread*, bool) /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThreadUtils.cpp:480:10 (libxul.so+0x52380d4) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #14 mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /home/ashley/git/mozilla-unified-linux/ipc/glue/MessagePump.cpp:299:20 (libxul.so+0x5ff4c0e) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #15 RunInternal /home/ashley/git/mozilla-unified-linux/ipc/chromium/src/base/message_loop.cc:369:10 (libxul.so+0x5f008e8) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #16 RunHandler /home/ashley/git/mozilla-unified-linux/ipc/chromium/src/base/message_loop.cc:362:3 (libxul.so+0x5f008e8)
23:22.84 GECKO(2296694)     #17 MessageLoop::Run() /home/ashley/git/mozilla-unified-linux/ipc/chromium/src/base/message_loop.cc:344:3 (libxul.so+0x5f008e8)
23:22.84 GECKO(2296694)     #18 nsThread::ThreadFunc(void*) /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThread.cpp:366:10 (libxul.so+0x522dd43) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #19 _pt_root /home/ashley/git/mozilla-unified-linux/nsprpub/pr/src/pthreads/ptthread.c:191:3 (libnspr4.so+0x501e8) (BuildId: 2caf66d365269d1f312e2517b537c84e)
23:22.84 GECKO(2296694)   Thread T3026 (tid=2621888, running) created by thread T3024 at:
23:22.84 GECKO(2296694)     #0 pthread_create /builds/worker/fetches/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1023:3 (firefox+0xce831) (BuildId: 44825889e7e9b8b86517b2a8796a6eed)
23:22.84 GECKO(2296694)     #1 init_thread /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread_frame.c:917:11 (libmozavcodec.so+0x18836f) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.84 GECKO(2296694)     #2 ff_frame_thread_init /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread_frame.c:976:15 (libmozavcodec.so+0x18704b) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.84 GECKO(2296694)     #3 ff_thread_init /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread.c:79:16 (libmozavcodec.so+0x184bd1) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.84 GECKO(2296694)     #4 avcodec_open2 /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/avcodec.c:323:15 (libmozavcodec.so+0xbf622) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.84 GECKO(2296694)     #5 mozilla::FFmpegDataDecoder<46465650>::InitSWDecoder(AVDictionary**) /home/ashley/git/mozilla-unified-linux/dom/media/platforms/ffmpeg/FFmpegDataDecoder.cpp:136:7 (libxul.so+0x9fe92c6) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #6 mozilla::FFmpegVideoDecoder<46465650>::Init() /home/ashley/git/mozilla-unified-linux/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp:648:20 (libxul.so+0x9ffec26) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #7 operator() /home/ashley/git/mozilla-unified-linux/dom/media/platforms/wrappers/MediaDataDecoderProxy.cpp:18:33 (libxul.so+0x9f6d033) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #8 mozilla::detail::ProxyFunctionRunnable<mozilla::MediaDataDecoderProxy::Init()::$_0, mozilla::MozPromise<mozilla::TrackInfo::TrackType, mozilla::MediaResult, true>>::Run() /home/ashley/git/mozilla-unified-linux/objdir-ff-tsan/dist/include/mozilla/MozPromise.h:1816:29 (libxul.so+0x9f6d033)
23:22.84 GECKO(2296694)     #9 mozilla::TaskQueue::Runner::Run() /home/ashley/git/mozilla-unified-linux/xpcom/threads/TaskQueue.cpp:261:20 (libxul.so+0x5207ed4) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #10 nsThreadPool::Run() /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThreadPool.cpp:456:14 (libxul.so+0x523a96f) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #11 nsThread::ProcessNextEvent(bool, bool*) /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThread.cpp:1153:16 (libxul.so+0x5232594) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #12 NS_ProcessNextEvent(nsIThread*, bool) /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThreadUtils.cpp:480:10 (libxul.so+0x52380d4) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #13 mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /home/ashley/git/mozilla-unified-linux/ipc/glue/MessagePump.cpp:299:20 (libxul.so+0x5ff4c0e) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #14 RunInternal /home/ashley/git/mozilla-unified-linux/ipc/chromium/src/base/message_loop.cc:369:10 (libxul.so+0x5f008e8) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #15 RunHandler /home/ashley/git/mozilla-unified-linux/ipc/chromium/src/base/message_loop.cc:362:3 (libxul.so+0x5f008e8)
23:22.84 GECKO(2296694)     #16 MessageLoop::Run() /home/ashley/git/mozilla-unified-linux/ipc/chromium/src/base/message_loop.cc:344:3 (libxul.so+0x5f008e8)
23:22.84 GECKO(2296694)     #17 nsThread::ThreadFunc(void*) /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThread.cpp:366:10 (libxul.so+0x522dd43) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #18 _pt_root /home/ashley/git/mozilla-unified-linux/nsprpub/pr/src/pthreads/ptthread.c:191:3 (libnspr4.so+0x501e8) (BuildId: 2caf66d365269d1f312e2517b537c84e)
23:22.84 GECKO(2296694)   Thread T3025 (tid=2621887, running) created by thread T3024 at:
23:22.84 GECKO(2296694)     #0 pthread_create /builds/worker/fetches/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1023:3 (firefox+0xce831) (BuildId: 44825889e7e9b8b86517b2a8796a6eed)
23:22.84 GECKO(2296694)     #1 init_thread /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread_frame.c:917:11 (libmozavcodec.so+0x18836f) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.84 GECKO(2296694)     #2 ff_frame_thread_init /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread_frame.c:976:15 (libmozavcodec.so+0x18704b) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.84 GECKO(2296694)     #3 ff_thread_init /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/pthread.c:79:16 (libmozavcodec.so+0x184bd1) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.84 GECKO(2296694)     #4 avcodec_open2 /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/avcodec.c:323:15 (libmozavcodec.so+0xbf622) (BuildId: 716d215059d2fa779f5f449e2d669c98)
23:22.84 GECKO(2296694)     #5 mozilla::FFmpegDataDecoder<46465650>::InitSWDecoder(AVDictionary**) /home/ashley/git/mozilla-unified-linux/dom/media/platforms/ffmpeg/FFmpegDataDecoder.cpp:136:7 (libxul.so+0x9fe92c6) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #6 mozilla::FFmpegVideoDecoder<46465650>::Init() /home/ashley/git/mozilla-unified-linux/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp:648:20 (libxul.so+0x9ffec26) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #7 operator() /home/ashley/git/mozilla-unified-linux/dom/media/platforms/wrappers/MediaDataDecoderProxy.cpp:18:33 (libxul.so+0x9f6d033) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #8 mozilla::detail::ProxyFunctionRunnable<mozilla::MediaDataDecoderProxy::Init()::$_0, mozilla::MozPromise<mozilla::TrackInfo::TrackType, mozilla::MediaResult, true>>::Run() /home/ashley/git/mozilla-unified-linux/objdir-ff-tsan/dist/include/mozilla/MozPromise.h:1816:29 (libxul.so+0x9f6d033)
23:22.84 GECKO(2296694)     #9 mozilla::TaskQueue::Runner::Run() /home/ashley/git/mozilla-unified-linux/xpcom/threads/TaskQueue.cpp:261:20 (libxul.so+0x5207ed4) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #10 nsThreadPool::Run() /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThreadPool.cpp:456:14 (libxul.so+0x523a96f) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #11 nsThread::ProcessNextEvent(bool, bool*) /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThread.cpp:1153:16 (libxul.so+0x5232594) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #12 NS_ProcessNextEvent(nsIThread*, bool) /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThreadUtils.cpp:480:10 (libxul.so+0x52380d4) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #13 mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /home/ashley/git/mozilla-unified-linux/ipc/glue/MessagePump.cpp:299:20 (libxul.so+0x5ff4c0e) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #14 RunInternal /home/ashley/git/mozilla-unified-linux/ipc/chromium/src/base/message_loop.cc:369:10 (libxul.so+0x5f008e8) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #15 RunHandler /home/ashley/git/mozilla-unified-linux/ipc/chromium/src/base/message_loop.cc:362:3 (libxul.so+0x5f008e8)
23:22.84 GECKO(2296694)     #16 MessageLoop::Run() /home/ashley/git/mozilla-unified-linux/ipc/chromium/src/base/message_loop.cc:344:3 (libxul.so+0x5f008e8)
23:22.84 GECKO(2296694)     #17 nsThread::ThreadFunc(void*) /home/ashley/git/mozilla-unified-linux/xpcom/threads/nsThread.cpp:366:10 (libxul.so+0x522dd43) (BuildId: 65698b6aa08affe4a78bb9543ec0aeaf)
23:22.84 GECKO(2296694)     #18 _pt_root /home/ashley/git/mozilla-unified-linux/nsprpub/pr/src/pthreads/ptthread.c:191:3 (libnspr4.so+0x501e8) (BuildId: 2caf66d365269d1f312e2517b537c84e)
23:22.85 GECKO(2296694) SUMMARY: ThreadSanitizer: data race /home/ashley/git/mozilla-unified-linux/media/ffvpx/libavcodec/vp9mvs.c:179:13 in find_ref_mvs
23:22.85 GECKO(2296694) ==================
23:22.85 GECKO(2296694) [Child 2349132, MediaDecoderStateMachine #1] WARNING: Decoder=725800057900 Decode error: NS_ERROR_DOM_MEDIA_FATAL_ERR (0x806e0005): file /home/ashley/git/mozilla-unified-linux/dom/media/MediaDecoderStateMachineBase.cpp:168
23:22.86 GECKO(2296694) Crash Annotation GraphicsCriticalError: |[0][GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt (t=5.82243) |[1][GFX1-]: VideoBridgeParent receives IPC close with reason=AbnormalShutdown (t=1400.39) [GFX1-]: VideoBridgeParent receives IPC close with reason=AbnormalShutdown

Jim Mathies [:jimm]

Comment 8

•

9 months ago

•

Edited

Severity: S3 → S2

Jim Mathies [:jimm]

Updated

•

9 months ago

Flags: needinfo?(padenot)

Updated

•

9 months ago

Duplicate of this bug: 1943549

Andrew McCreight [:mccr8]

Assignee

Comment 10

•

9 months ago

Attached file repro_with_logging.txt.gz — Details

Repro with more logging information included - doesn't seem to repro in rr

Updated

•

8 months ago

Duplicate of this bug: 1945025

Assignee

Comment 12

•

8 months ago

•

Edited

Did some more work on this and was able to collect a Pernosco trace with full debug + some additional info here. Should be able to get to the bottom of this soon.

Assignee

Comment 13

•

8 months ago

I'm adding Ronald to this race condition bug as I've heard his expertise with the code could be really helpful here. The race condition appears to originate in FFmpeg and pops up during VP9 decoding. We took a look and captured a Pernsoco trace. I have notes in the notebook panel which will bring you to the offending read/writes when clicked on. Scrolling down a bit in the stdout window will show the full TSan output. It's not obvious where reads/writes are synchronized, and we weren't able to find the source of the race condition. TSan detects it a few times a day on our Linux CI and I'm able to repro locally on a TSan build by running:

/mach mochitest --headless --run-until-failure --repeat 1000  dom/media/test/test_background_video_resume_after_end_show_last_frame.html

It takes ~30 minutes for a repro to occur on my machine. Our last FFmpeg update happened last April, and the race condition is new since then. If you have any insights or suggestions, I'd really appreciate it! Let me know if you have any questions - I'd be happy to provide more details about Pernosco or anything else that might help. Thank you for your time!

Flags: needinfo?(rsbultje)

BugBot [:suhaib / :marco/ :calixte]

Comment 14

•

8 months ago

hi :) first, rr isn't particularly useful for threading issues, it's single-threaded (unfortunately - we've tried using it).

you're mentioning this bug is new since the april pull, do you know what revision you pulled in, and what revision you used before that? I'd like to look at the commit log in that range to see if anything changed in vp9 or the generic frame-threading code. Thanks!

Flags: needinfo?(rsbultje)

Comment 15

•

8 months ago

Set release status flags based on info from the regressing bug 1940449

status-firefox137: --- → affected

Assignee

Comment 16

•

8 months ago

Hi Ronald, thanks for checking in!

FFmpeg version information:

Current FFmpeg:           252fc2e047297697dea78e63aa908377b47c2136
Previous FFmpeg (approx): 240fd04db231efa4abad3abbd808140b4112324a

I wrote (approx) since we don't have the exact hash of the last FFmpeg update stored, but I put together a script to run through the repos and compare commits. Comparing the diffs, 240fd04db231efa4abad3abbd808140b4112324a is the most similar to what we previously had in our the tree and is dated 2024-04-04. The corresponding patch on our side was submitted 2024-04-08 so the timeline lines up.

Let me know if there's any other info that I can provide to help.

Thanks!

Flags: needinfo?(rsbultje)

Comment 17

•

8 months ago

The commits that stand out to me are:
5acbdd2264
1165c14444
7bd3b73716

Do you have a link to the VP9 file being loaded in your unit test? I'd like to develop a simpler repro that doesn't require Firefox.

Flags: needinfo?(rsbultje)

Assignee

Comment 18

•

8 months ago

Hi Ronald! The two files used by the test are gizmo.webm and gizmo-noaudio.webm. For some additional info, the test itself in question is test_background_video_resume_after_end_show_last_frame.html.

The failure after the race condition seems to usually look like this:

FAIL gizmo.mp4-0 video1 and video2 have different content.
FAIL gizmo.mp4-0-muted video1 and video2 have different content.

Hope this helps - please let me know if there's anything more we can provide from our side. Thanks!

Flags: needinfo?(rsbultje)

Comment 19

•

8 months ago

Btw I don't have access to the pernosco trace, can I be added there?

Comment 20

•

8 months ago

How do I CC someone new? I'd like to add andreas.rheinhardt@outlook.com to the bug report, he was the last one to touch the frame-threading code and refactor some of the VP9 code. I suspect he would be able to pinpoint this faster than me.

Flags: needinfo?(rsbultje)

Paul Adenot (:padenot)

Comment 21

•

8 months ago

Ronald, Andreas would need an account here, and then we can add him. It's possible to log in here using GitHub, but otherwise it's not too complicated to create an account. If that's too much hassle we can also move this conversation wherever is best for you, just let us know.

We're also sorting out the pernosco trace access at the same time.

Assignee

Comment 22

•

8 months ago

Ronald, you should have access to the Pernosco trace now. If it gives you any difficulty please let me know. Thanks!

Flags: needinfo?(rsbultje)

Comment 23

•

8 months ago

I can see it, thanks. The trace pretty clearly shows a race where we're not waiting for the "prev" frame at the current sbrow. This was a default wait historically, but there has been a lot of refactoring in the generic framework so I don't know the details of how this works nowadays. I'm hoping Andreas can provide some information here. I tried running gizmo.webm locally under tsan but I can't immediately get it to fail. I see above it takes you guys 30min to run also, so perhaps I'm just too impatient. I'm hoping to find a better (quicker / more reliable) repro so we can bisect between the commit range of your previous and current ffmpeg syncpoint. Andreas, have you have some time to look into this?

Paul Adenot (:padenot)

Comment 24

•

8 months ago

Ronald, :az spent a significant time and effort reproducing this under rr (to provide the pernosco recording), but afaik she's got to a point she's able to reproduce it reliably, so maybe she can explain her setup. Reliably doesn't mean quickly, we all know how TSAN stuff goes.

As far as our previous ffmpeg update point, I'm afraid we (and by that I mean I) were quite sloppy and didn't record the hash as we usually do. We'll eventually find it to shorten the bisection range based on the file content, iirc :az has been brute forcing it, not sure if she's had any result yet. :az, maybe you can confirm this, but it looks like my previous update said:

changeset:   722555:02ab0b6c0713
user:        Paul Adenot <paul@paul.cx>
date:        Fri Nov 04 12:36:38 2022 +0000
summary:     Bug 1796532 - Update vendored ffmpeg to revision 0957d62f. r=alwu

hopefully this is somewhat correct?

The next pull was ffmpeg 252fc2, and we pull this on 2025-01-09.

The first TSAN failure we had for this was on 2025-01-11 (we run lot of TSAN, this took about 2 days to be triggered). All in all, that might indicate us an unfortunate ~1 year range, that might be perfectly useless depending on the timing of Andreas' refactoring merge.

Let us know if we can provide any more information or assistance, we're more than happy to help the ffmpeg project.

Comment 25

•

8 months ago

1yr range is not so bad, if we can reproduce this reliably. We're talking ~3.6k commits, so a simple bisect would be ceil(log2(3600))=12 steps.

Comment 26

•

8 months ago

Actually I can reproduce myself also on git/master upstream:

$ while true; do ./ffmpeg -threads 4 -i ~/Downloads/gizmo.webm -an -f md5 -v error -nostats -y /tmp/out.log || break; x=`cat /tmp/out.log`; if [ $x != "MD5=a08e3e064e3589e5610bc2ccc2a4ab4c" ]; then break; fi; done
ffmpeg(7971,0x1144a1600) malloc: nano zone abandoned due to inability to preallocate reserved vm space.
[opus @ 0x7b5c00000700] Error parsing Opus packet header.
[this repeats ~20 times]
==================
WARNING: ThreadSanitizer: data race (pid=8053)
  Read of size 1 at 0x7bb40003cb88 by thread T4 (mutexes: write M257):
    #0 <null> <null>:2 (ffmpeg:x86_64+0x10187043f)
    #1 ff_vp9_fill_mv <null>:2 (ffmpeg:x86_64+0x10186bcae)
    #2 ff_vp9_decode_block <null>:2 (ffmpeg:x86_64+0x1017ac8b9)
    #3 <null> <null>:2 (ffmpeg:x86_64+0x1017a21f9)
    #4 <null> <null>:2 (ffmpeg:x86_64+0x10178f210)
    #5 ff_decode_receive_frame_internal <null>:2 (ffmpeg:x86_64+0x100b75044)
    #6 <null> <null>:2 (ffmpeg:x86_64+0x101429909)

  Previous write of size 1 at 0x7bb40003cb88 by thread T3 (mutexes: write M255):
    #0 ff_vp9_decode_block <null>:2 (ffmpeg:x86_64+0x1017a8305)
    #1 <null> <null>:2 (ffmpeg:x86_64+0x1017a21f9)
    #2 <null> <null>:2 (ffmpeg:x86_64+0x10178f210)
    #3 ff_decode_receive_frame_internal <null>:2 (ffmpeg:x86_64+0x100b75044)
    #4 <null> <null>:2 (ffmpeg:x86_64+0x101429909)

I can run the above with a threshold of 1000 before accepting "success" and see if that works.

Comment 27

•

8 months ago

I've opened a bug report upstream also: https://trac.ffmpeg.org/ticket/11456#ticket

Comment 28

•

8 months ago

James bisected this and found it to be caused by 7bd3b7371632250a2fbc38bee6597b0aa189346a. We believe the following patch silences tsan:

$ git diff
diff --git a/libavcodec/threadprogress.c b/libavcodec/threadprogress.c
index 62c4fd898b..aa72ff80e7 100644
--- a/libavcodec/threadprogress.c
+++ b/libavcodec/threadprogress.c
@@ -55,9 +55,8 @@ void ff_thread_progress_report(ThreadProgress *pro, int n)
     if (atomic_load_explicit(&pro->progress, memory_order_relaxed) >= n)
         return;
 
-    atomic_store_explicit(&pro->progress, n, memory_order_release);
-
     ff_mutex_lock(&pro->progress_mutex);
+    atomic_store_explicit(&pro->progress, n, memory_order_release);
     ff_cond_broadcast(&pro->progress_cond);
     ff_mutex_unlock(&pro->progress_mutex);
 }

I'll submit this upstream.

Assignee

Comment 29

•

8 months ago

Thank you Ronald and James! That's great news - I'll get a patch up on our side.

Assignee

Comment 30

•

8 months ago

Attached file Add ffmpeg patch from upstream — Details

Updated

•

8 months ago

Assignee: nobody → azebrowski

Status: NEW → ASSIGNED

Assignee

Comment 31

•

8 months ago

Attached file Add ffmpeg patch from upstream — Details

Original Revision: https://phabricator.services.mozilla.com/D237218

Updated

•

8 months ago

Attachment #9464716 - Flags: approval-mozilla-beta?

Updated

•

8 months ago

status-firefox-esr115: --- → unaffected

tracking-firefox136: --- → +

tracking-firefox137: --- → +

Updated

•

8 months ago

Attachment #9464716 - Attachment description: Bug 1941137 - Fix race condition in ffmpeg libavcodec/threadprogress.c → Bug 1941137 - Add ffmpeg patch from upstream

Updated

•

8 months ago

Attachment #9464662 - Attachment description: Bug 1941137 - Fix race condition in ffmpeg libavcodec/threadprogress.c → Add ffmpeg patch from upstream

Updated

•

8 months ago

Attachment #9464716 - Attachment description: Bug 1941137 - Add ffmpeg patch from upstream → Add ffmpeg patch from upstream

Assignee

Comment 32

•

8 months ago

•

Edited

Removed commit bug number/message to remove reference to race condition as per the docs. This is my first time landing a fix for a secbug, so please let me know if there's anything additional I need to do / am missing. Thanks!

Comment 33

•

8 months ago

beta Uplift Approval Request

User impact if declined: Secbug/continuing crashes due to race condition
Code covered by automated testing: yes
Fix verified in Nightly: no
Needs manual QE test: no
Steps to reproduce for manual QE testing: On a TSAN build, ./mach mochitest --headless --run-until-failure --repeat 1000 dom/media/test/test_background_video_resume_after_end_show_last_frame.html will repro but may take 30+ minutes.
Risk associated with taking this patch: Low
Explanation of risk level: Patch from ffmpeg devs + being pushed upstream on their side as well
String changes made/needed: N/A
Is Android affected?: no

Comment 34

•

8 months ago

:az thank for the beta uplift request, are you planning on landing this to autoland soon?

Flags: needinfo?(azebrowski)

Assignee

Comment 35

•

8 months ago

Thanks for the NI, I thought that it had already gone to autoland when I submitted it on Phab. I just queued it for landing.

Flags: needinfo?(azebrowski)

Pulsebot

Comment 36

•

8 months ago

Pushed by azebrowski@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/d7b6dc7bee51 Add ffmpeg patch from upstream r=padenot

Assignee

Updated

•

8 months ago

Flags: needinfo?(dmeehan)

Updated

•

8 months ago

Flags: needinfo?(dmeehan)

https://hg.mozilla.org/mozilla-central/rev/d7b6dc7bee51

Comment 37

•

8 months ago

Group: media-core-security → core-security-release

Status: ASSIGNED → RESOLVED

Closed: 8 months ago

status-firefox137: affected → fixed

Resolution: --- → FIXED

Target Milestone: --- → 137 Branch