Closed Bug 1034957 Opened 6 years ago Closed 5 years ago

Media Decode thread seems to hang

Categories

(Core :: Audio/Video, defect)

x86_64
Linux
defect
Not set

Tracking

()

RESOLVED FIXED
mozilla34
Tracking Status
firefox32 --- unaffected
firefox33 + fixed
firefox34 + fixed

People

(Reporter: evilpie, Assigned: jwwang)

References

Details

(Keywords: regression)

Attachments

(4 files, 3 obsolete files)

Attached file backtrace
I have observed this multiple times now and this time I actually got a backtrace. While this happens there isn't any video playing, but I think I might have closed a tab with video recently.
If you can reproduce it again, it would be very handy to also see what the other threads are doing (`thread apply all bt full`).
I can see this problem as well, and it's quite easily reproducible.
The thread hangs immediately after I watch an HTML5 youtube video.
I will try to reproduce it with a debug build, and post the thread  backtrace.
Attached file debug_backtrace.txt (all threads) (obsolete) —
On debug I was only able to get it to hang at shutdown.
Possibly related to bug 929385
Attaching backtrace of all threads (PS this is using a clean profile, with no addons)
Attaching backtrace of all threads just after watching a video, when the CPU goes to 100%.
Attachment #8457955 - Attachment is obsolete: true
It doesn't look like a deadlock. If so, the CPU wouldn't have been 100%.
(In reply to JW Wang [:jwwang] from comment #5)
> It doesn't look like a deadlock. If so, the CPU wouldn't have been 100%.
Not a deadlock, but it seems to be stuck in a loop.
Looking at the stack trace, it seems to be here: http://dxr.mozilla.org/mozilla-central/source/content/media/MediaDecoderReader.cpp#247
Can you describe the situation when the problem happened? Did audio or video freeze at the moment?
I was watching a youtube video on Google+ (HTML5 player). Just as the video ended, the cpu went to 100%.
The process doesn't terminate if I close all windows, so I have to use kill -9.
After the thread hangs I am able to watch other videos. At one point, I had 2 threads that were at 100%.
Let me know if you need other debugging info.

I'm using firefox-trunk from the nightly ppa - version 33.0a1 (2014-07-07) on ubuntu-gnome
I watched several video clips at
http://www.google.com/+/learnmore/resources/videos.html.

I didn't find the problem at the end of each playback.

How can I know if HTML5 player is in action? Can you detail the web page and which video clip you were watching?
I watched a few videos on this page: https://plus.google.com/+BrianKinger/posts
When right clicking on the video, it usually tells you which player you're using - flash/html5. I expect this problem wouldn't appear with the flash one, since it runs in plugin-container.

If you don't manage to reproduce it, I can provide ssh access into my machine. I seem to hit it fairly easily.
Still can't repro it...

I wonder is it related to alsa or pulse-audio backend. Which audio backend is in use on your computer?
I'm using pulse-audio.
Finally I can repro it with firefox built with gstreamer 1.0. Is that also your configuration?
I can repro the problem by opening this page: https://www.youtube.com/watch?v=oko6TzPQE6Y

At end of playback, GST_EVENT_GAP is received in GStreamerReader::EventProbe(). I called gst_event_parse_gap() to parse the gap and find out:
timestamp = 177748752834 <-- 177sec, this is the end time of the clip which is 2:57
duration = 18446744073709551615 <-- this is an insanely large value

I guess the gap is causing the problem.

Hi Alessandro,
Do you have any idea about what this gap mean?
Flags: needinfo?(alessandro.d)
(In reply to JW Wang [:jwwang] from comment #13)
> Finally I can repro it with firefox built with gstreamer 1.0. Is that also
> your configuration?
Yes, it's built with --enable-gstreamer=1.0
Blocks: 947287
I can repro it by playing content/media/test/gizmo.mp4.

The events received near the end of playback on firefox with gstreamer 0.1:
2014-07-24 06:37:48.206803 UTC - -1682020608[7f84a37cc6e0]: event probe for audio: eos
2014-07-24 06:37:48.206866 UTC - -1682020608[7f84a37cc6e0]: GStreamerReader::Eos sink=audio
2014-07-24 06:37:49.169703 UTC - -1658849536[7f84a37cf080]: event probe for video: eos
2014-07-24 06:37:49.169744 UTC - -1658849536[7f84a37cf080]: GStreamerReader::Eos sink=video

Events with gstreamer 1.0:
2014-07-24 06:44:31.747290 UTC - 1072162560[7f6346483860]: event probe for audio: gap
2014-07-24 06:44:31.747301 UTC - 1072162560[7f6346483860]: audio gap: t=5567999999, duration=18446744073709551615

It is curious that GST_EVENT_TAG is received instead of GST_EVENT_EOS on gstreamer 1.0 at the end of  playback.
Sorry, typo...
s/GST_EVENT_TAG/GST_EVENT_GAP/
I found a workaround for this on Ubuntu.

> sudo apt-get remove gstreamer1.0-plugins-*

This seems to fix the thread hanging, while the youtube player is still HTML5.
Is it possible that we're loading a different library than we're expecting?
Do you still build with --enable-gstreamer=1.0?
I just checked, and it seems that removing the plugins has removed h264 capability. The HTML5 player I get is WebM. I'll check if the cisco plugin is any better.

(In reply to JW Wang [:jwwang] from comment #19)
> Do you still build with --enable-gstreamer=1.0?
Yes, I'm using the default build provided by the mozilla-daily ppa.

target
x86_64-pc-linux-gnu
Build tools
Compiler 	Version 	Compiler flags
gcc 	4.8.2 	-Wall -Wpointer-arith -Wdeclaration-after-statement -Werror=return-type -Werror=int-to-pointer-cast -Wtype-limits -Wempty-body -Wsign-compare -Wno-unused -Wcast-align -std=gnu99 -fgnu89-inline -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-math-errno -pthread -pipe
c++ 	4.8.2 	-Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Werror=int-to-pointer-cast -Wtype-limits -Wempty-body -Wsign-compare -Wno-invalid-offsetof -Wcast-align -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -std=gnu++0x -pthread -pipe -DNDEBUG -DTRIMMED -g -Os -freorder-blocks -fno-omit-frame-pointer
Configure arguments

--host=x86_64-linux-gnu --prefix=/usr --libexecdir=/usr/lib/firefox-trunk '--with-l10n-base=/build/buildd/firefox-trunk-34.0~a1~hg20140723r195564/./l10n' '--srcdir=/build/buildd/firefox-trunk-34.0~a1~hg20140723r195564/.' --enable-release --disable-install-strip --disable-updater --enable-application=browser --enable-startup-notification --with-distribution-id=com.ubuntu --enable-optimize --enable-tests --disable-crashreporter --with-branding=browser/branding/nightly --disable-gnomevfs --enable-gio --enable-update-channel=nightly --disable-debug --disable-elf-hack --with-app-basename=Firefox-Trunk --with-app-name=firefox-trunk --enable-profiling --enable-gstreamer=1.0 '--with-google-api-keyfile=/build/buildd/firefox-trunk-34.0~a1~hg20140723r195564/debian/g'
After I removed the plugins, I couldn't play gizmo.mp4 either.

As bug 947287 said, GStreamer 1.x is the way to go and we still have to fix this bug though...
I hit this as well. A trunk compile with --enable-gstreamer=0.10 doesn't have this problem.
Attached patch 1034957_fix_gstreamer_hang.patch (obsolete) — Splinter Review
Hi Valentin,
Can you try this patch? It works for me with gstreamer=1.0. Thanks.
Assignee: nobody → jwwang
Status: NEW → ASSIGNED
Flags: needinfo?(valentin.gosu)
Blocks: 1044263
Hi JW, I can confirm that the patch fixes the issue for me.
Without the patch the thread hangs, but doesn't after I apply the patch. I tested with a debug build on rev 196360:0698e2cbadf9.
Great job! Thanks!
Flags: needinfo?(valentin.gosu)
Ok, I will have more tests before the patch is ready for review. Thanks.
[Tracking Requested - why for this release]: Affects <video> playback on Linux.
Don't spin calling DecodeAudioData() in MediaDecoderReader::RequestAudioData for it somehow prevents audio EOS from coming in gstreamer 1.x when there is still video buffer waiting to be consumed. (|mVideoSinkBufferCount| > 0 in GStreamerReader.cpp)

I think this issue could also be solved by using different MediaTaskQueues for video/audio decoding and there is a bug planning for that. However we still need to move forward and come back later to remove the code not needed anymore when the bug is fixed.

Also fix a race in GStreamerReader.cpp.

Try on desktop platforms: https://tbpl.mozilla.org/?tree=Try&rev=eb4cf13d7c72
Try on Android: https://tbpl.mozilla.org/?tree=Try&rev=9154102bbf4c
Try on B2G: https://tbpl.mozilla.org/?tree=Try&rev=ec9312f81d9f
Try on B2G desktop: https://tbpl.mozilla.org/?tree=Try&rev=36bdd92886a0

Btw, I can't enable gstreamer=1.0 on Try. So the Try logs above only provide food for no regression.
Attachment #8463281 - Attachment is obsolete: true
Attachment #8465180 - Flags: review?(cpearce)
Comment on attachment 8465180 [details] [diff] [review]
1034957_fix_gstreamer_hang-v2.patch

Review of attachment 8465180 [details] [diff] [review]:
-----------------------------------------------------------------

Until we can switch over to using MP4Reader on Linux, sure.
Attachment #8465180 - Flags: review?(cpearce) → review+
https://hg.mozilla.org/mozilla-central/rev/eb9afaa0614d
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla34
Will this patch get added to firefox33, or do I have to wait for aurora to get bumped to version 34?
I'd love to see this in aurora as well, I'm getting this bug all the time.
Once this bug has been on nightly for a few days the developer can request Aurora uplift. Release management is currently tracking this work and will expect the Aurora uplift. Verifying this fix on a nightly build at https://nightly.mozilla.org would be helpful to this process.
Aren't nightly builds still using the old gstreamer 0.10 ? This bug only affects gstreamer 1.0.
Either way, nightly is working fine with youtube on my machine.
im not sure if its due to this implementation, but every time i open any YouTube video, the system volume is automatically set high. 
Any body else have this problem?
(In reply to bull500 from comment #37)
> im not sure if its due to this implementation, but every time i open any
> YouTube video, the system volume is automatically set high. 
> Any body else have this problem?

I'm guessing you use pulseaudio as a sound server. By default, pulseaudio uses flat volumes. You can see here[1] both an explanation of what that means and how you can disable this (just search for flat-volumes).

[1] https://wiki.archlinux.org/index.php/PulseAudio
Depends on: 1046837
@Adrian - well yes my system Fedora 20 makes use of Pluseaudio, but this automatic setting high of volume never happened before.
All my other apps can play at their own volumes without affected the general system volume.

I've disabled the flat-volumes thing for now. Thanks for the tip :)
Just pointing out, I use heftig's builds of aurora on Arch Linux, and he now applies a patch for this:

http://pkgbuild.com/~heftig/packages/aurora/gstreamer-hang.patch

which fixes this bug.
JW, could you fill an uplift request for 33? thanks
Flags: needinfo?(jwwang)
Per bug 1046837 comment 177, I will merge the patches of this bug and bug 1046837 for an uplift to Aurora.
Flags: needinfo?(jwwang)
Approval Request Comment
[Feature/regressing bug #]:unknown
[User impact if declined]:playback will get stuck at the end of the stream when gstreamer 1.0 is turned on
[Describe test coverage new/current, TBPL]:
Try: https://tbpl.mozilla.org/?tree=Try&rev=69c479d002fd
It has been running in Central for several weeks.
[Risks and why]: low. The change is well tested on Central for weeks.
[String/UUID change made/needed]:none
Attachment #8483197 - Flags: review+
Attachment #8483197 - Flags: approval-mozilla-aurora?
(In reply to JW Wang [:jwwang] from comment #42)
> Per bug 1046837 comment 177, I will merge the patches of this bug and bug
> 1046837 for an uplift to Aurora.

This fix landed on 34 (now Aurora). 33 is now beta. Did you mean to submit a beta approval request?
Flags: needinfo?(jwwang)
Oops, I just missed the train...
I need to test Beta on Try again before I can request uplift to Beta.
Flags: needinfo?(jwwang)
Attachment #8483197 - Flags: approval-mozilla-aurora?
Approval Request Comment
[Feature/regressing bug #]:unknown
[User impact if declined]:playback will get stuck at the end of the stream when gstreamer 1.0 is turned on
[Describe test coverage new/current, TBPL]:
Try: https://tbpl.mozilla.org/?tree=Try&rev=aaf194683e39
[Risks and why]: low. The change is well tested on Central for weeks.
[String/UUID change made/needed]:none
Attachment #8483197 - Attachment is obsolete: true
Attachment #8483985 - Flags: review+
Attachment #8483985 - Flags: approval-mozilla-beta?
Attachment #8483985 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Depends on: 1100701
You need to log in before you can comment on or make changes to this bug.