Closed Bug 841239 Opened 11 years ago Closed 11 years ago

Pandora.com stops playing music due to jplayer error frequently, using nightly

Categories

(Core :: Audio/Video, defect)

Product:
Core
Component:
Audio/Video
Platform:
x86_64
Windows 7
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla22
Tracking Flags:
Tracking Status
firefox20 --- unaffected
firefox21 + verified
firefox22 --- verified

People

(Reporter: stephend, Assigned: cpearce)

References

(
URL
)

Details

(Keywords: regression)

Attachments

(5 files)

Screenshot (error message disappeared when I took the screenshot)
868.72 KB, image/png
Details
Error on Mac OS X, after crashing in Flash/Open GL
2.03 MB, image/png
Details
Still getting the same error, using the try build
854.43 KB, image/png
Details
NSPR log of failure
241.83 KB, application/octet-stream
Details
Patch: Make our canPlayType() codecs=mp3 and audio/x-m4a behave the same as other browsers
4.64 KB, patch
padenot
: review+
bajaj
: approval-mozilla-aurora+
Details | Diff | Splinter Review 
This might be DOM stuff, dunno.

Pretty often, on Pandora.com, using nightly on Windows 7, I get an error saying "Sorry, it's our fault, please reload to continue".

STR:

1. Using nightly on Windows 7, at least (Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20130213 Firefox/21.0), load Pandora.com
2. Log in
3. Play a station
4. I think it helps if you either skip tracks or do the "I'm tired of this track" action

Expected:

Keeps playing

Actual:

See above, and throws this in the Web Console:

[17:00:49.813] POST http://www.pandora.com/radio/xmlrpc/v35?rid=3096942P&lid=5635016&method=proxyAdRequest&arg1=aHR0cDovL2FkLmRvdWJsZWNsaWNrLm5ldC9wZmFkeC9wYW5kLmRlZmF1bHQvcHJvZC50dW5lcjtzZWdtZW50PTY7aW5kZXg9MTtmYW09MTthcnRpc3Q9UjE2Nzk4MTtnY2F0PW5vbmU7Z2VucmU9ZWxlY3Ryb25pY2E7YWc9MzY7Z25kPTE7emlwPTk0MDg2O2hvdXJzPTExO2NvbXBlZD0wO2V4cD0wO2ZiPTA7ZG1hPTgwNztjbGVhbj0wO21zYT0wMDk7bXNhPTIxNTtzdD1DQTtjbz0wNjA4NTtldD0yO3Bpbj0wO2FhPTA7aGlzcD0wO2hoaT0wO2FuPTE7aXA9MTtjNmI9MjA3O2M4ZD0xMzI7Yzc9MTgzO2M2YT0yMDQ7YzEyPTE3NDtjMTM9MjEzO2M1ZD0xMTE7YzhjPTEzNTtjNT0xMTc7YzQ9MTY1O2MxMD0yMTA7YzNhPTE5MjtjMT0xNDc7dT1sKjdwZXdwZ2p6YzdtdXEhc2VnbWVudCo2IWluZGV4KjEhZmFtKjEhYWcqMzYhZ25kKjEhemlwKjk0MDg2IWRtYSo4MDchY2xlYW4qMCFtc2EqMDA5IW1zYSoyMTUhc3QqQ0EhY28qMDYwODUhZXQqMiFwaW4qMCFhYSowIWhpc3AqMCFoaGkqMCFhbioxIWlwKjEhc3oqMTM0eDE4NSwyMDAweDEyIWM2YioyMDchYzhkKjEzMiFjNyoxODMhYzZhKjIwNCFjMTIqMTc0IWMxMyoyMTMhYzVkKjExMSFjOGMqMTM1IWM1KjExNyFjNCoxNjUhYzEwKjIxMCFjM2EqMTkyIWMxKjE0NyFnY2F0Km5vbmUhZmIqMCFnZW5yZSplbGVjdHJvbmljYSFhcnRpc3QqUjE2Nzk4MSFjb21wZWQqMCFleHAqMCFob3Vycyox [HTTP/1.1 200 OK 262ms]
--
[17:01:01.001] jPlayer error (count: 0): It is not possible to play any media format provided in setMedia() on this browser using your current options.
[17:01:01.002] jPlayer error (count: 1): Attempt to issue media playback commands, while no media url is set.
So, this fails for me using:

Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20130212 Firefox/21.0:
* Built from http://hg.mozilla.org/mozilla-central/rev/36525224b14e
Keywords: regression
Last working build is:

Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20130211 Firefox/21.0
* Built from http://hg.mozilla.org/mozilla-central/rev/5835bc763be7
That would be http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=5835bc763be7&tochange=36525224b14e

Are you sure that they use a plugin to play audio and not html5 audio ?
There are a few checkins for html5 audio in that range: bug 837842, bug 805261, bug 837430, bug 839319
Component: Plug-ins → Video/Audio
Can somebody please inspect the page using DOMI and figure out whether they are using Flash or <audio> ?
Keywords: qawanted
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #5)
> Can somebody please inspect the page using DOMI and figure out whether they
> are using Flash or <audio> ?

They are embedding resources like:

http://audio-dc6-t3-1.pandora.com/access/?version=4&lid=5635016&token=mOdkF461zLzG7gamfZA0u4JZ%2BqQLWFQVgrDye5jZLUkg4VIqkoYzdqen2kWZCroE2Gz7QB2WhYizQ2XDy2ALpC2xqXS81MHmagi5ZY1zAGCgaJUlNnoVURIhqPnTdfazUvE6F6KsnQDzPtBgXmW97L%2BYbk6GleJGqcYh66GHTzX%2BcOBlcog%2Fd%2FP2qOLwbBgZVbANczJl2NTVF7KREHOr4Xxy4XoVCMdKv1afJPMAaq3n%2FqafzHqz63XdOxp3p2%2Fw5OFJyoxPhozY7WpsCYcefdBjnwTxDW9V9q0pVy9yzUCb5fSLHf69VQOGy6OshtK0V0fjxv9fRDv1wk9wkQn7JlBn6CORwv4d&lo=5 which has a MIME type of audio/mp4

Matti had me toggle media.window-media-foundation.enabled to |false|, and I get SWF resources, instead, in that case.
When you say embedding do you mean <embed> or <audio> ?
Looks like <audio> but there is some <object> flash in there too.

script 
http://player.ooyala.com/v3/4a71bfa5c2bf45e9b11ee25cb6092f15

<div style="width: 0px; height: 0px;" id="jPlayer1">
<img style="width: 0px; height: 0px; display: none;" id="jp_poster_0">
<audio src="http://audio-ch1-t1-2.pandora.com/access/?version=4&amp;lid=...&amp;token=..." preload="metadata" id="jp_audio_0"></audio>
</div>
<object data="/js/libs/flXHR-1.0.6/flXHR.swf" class="flXHRhideSwf" name="flXHR_swf_1" id="flXHR_swf_1" type="application/x-shockwave-flash" height="1" width="1"><param value="always" name="allowScriptAccess"></object>

<object data="/js/libs/flXHR-1.0.6/flXHR.swf" class="flXHRhideSwf" name="flXHR_swf_2" id="flXHR_swf_2" type="application/x-shockwave-flash" height="1" width="1"><param value="always" name="allowScriptAccess"></object>

<div style="width: 0px; height: 0px;" id="jPlayer2">
<img style="width: 0px; height: 0px; display: none;" id="jp_poster_1">
<audio preload="metadata" id="jp_audio_1"></audio>
</div>
from https://groups.google.com/forum/m/#!msg/jplayer/5nukqWoeC8o/RJcd6pzgGhgJ
>since our audio streams are encoded as AAC-HE, it plays natively in Chrome, Safari, >and IE9 but still requires the Flash fallback for IE7/8, and Firefox
the nightly supports playing AAC and it should use the html5 audio tag unless you disable AAC support with media.window-media-foundation.enabled to false.

from comment#4
"and I get SWF resources, instead, in that case."
Assuming that the four bugs mentioned in comment 3 can be built independently, can we try to narrow the regression range to a single one? I certainly experience this and pandora is basically unusable because of it.

Is this windows-only?
status-firefox21: --- → affected
tracking-firefox21: --- → ?
Keywords: regressionwindow-wanted
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #10)
> Assuming that the four bugs mentioned in comment 3 can be built
> independently, can we try to narrow the regression range to a single one? I
> certainly experience this and pandora is basically unusable because of it.
> 
> Is this windows-only?

I haven't yet checked Linux, but it doesn't happen on Mac:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:21.0) Gecko/20130214 Firefox/21.0 is fine.
If the error is when they're serving MP4 files to Nightly it won't be seen on Mac or Linux since neither of those have MP4 support yet (unless you --enable-gstreamer on Linux)
(In reply to Chris Double (:doublec) from comment #12)
> If the error is when they're serving MP4 files to Nightly it won't be seen
> on Mac or Linux since neither of those have MP4 support yet (unless you
> --enable-gstreamer on Linux)

Interestingly, I just recently had a crash (https://crash-stats.mozilla.com/report/index/bp-2d65a44b-41d9-4a69-ab42-b92c92130214) and the following screenshot showed the same behavior (caught the error message this time) as in comment 0, on Windows.
Looks like this was indeed a Flash crash:

plugin Shockwave Flash Version:11.6.602.167 Filename: Flash Player.plugin
(In reply to Stephen Donner [:stephend] from comment #13)
> (In reply to Chris Double (:doublec) from comment #12)
> > If the error is when they're serving MP4 files to Nightly it won't be seen
> > on Mac or Linux since neither of those have MP4 support yet (unless you
> > --enable-gstreamer on Linux)
> 
> Interestingly, I just recently had a crash
> (https://crash-stats.mozilla.com/report/index/bp-2d65a44b-41d9-4a69-ab42-
> b92c92130214) and the following screenshot showed the same behavior (caught
> the error message this time) as in comment 0, on Windows.

This should have a different bug on file, since it's unlikely this crash is the same issue as comment 0.
Chris/Matt - can either of you own this bug and create try builds for QA/Stephen/Benjamin to test against?
tracking-firefox21: ? → +
e:/builds/moz2_slave/try-w32-0000000000000000000000/build/media/libcubeb/src/cubeb_winmm.c(339) : error C2065: 'stream_parms' : undeclared identifier

e:/builds/moz2_slave/try-w32-0000000000000000000000/build/media/libcubeb/src/cubeb_winmm.c(339) : error C2224: left of '.format' must have struct/union type
Actually, ignore that, sorry.
(In reply to Matthew Gregan [:kinetik] from comment #17)
> There's a try build that backs out part of bug 839319

There's now a confirmed regression from bug 839319 that is being tracked in bug 842176.  If the build linked here solves the Pandora regression, please make this bug depend on 842176.
I can confirm that on Windows we're using HTML5 audio and the new Windows Media Foundation backend to play the resource.

I am testing now. How long does it usually take for playback to fail?
I'm not certain, but ISTM every couple of songs.
I am still experiencing this error with today's nightly which has the fix from bug 842176.
Stephen can you please try to find a regression range for this bug?
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #26)
> Stephen can you please try to find a regression range for this bug?

I have -- see comment 1 and comment 2; it's between builds:

Working:
Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20130211 Firefox/21.0
* Built from http://hg.mozilla.org/mozilla-central/rev/5835bc763be7

Failing:
Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20130212 Firefox/21.0:
* Built from http://hg.mozilla.org/mozilla-central/rev/36525224b14e
Sorry I missed that Stephen. Here is the push log for that regression range:
> http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=5835bc763be7&tochange=36525224b14e

I see a few "media" related changes. Can someone with more expertise please look at the pushlog for potential regressors?
Keywords: qawanted, regressionwindow-wanted
(In reply to Matthew Gregan [:kinetik] from comment #29)
> I pushed a try build with bug 837842 reverted:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mgregan@mozilla.
> com-c1c5fd3d838e/

Build is now available for testing.
Stephen, can you please test the Try build in comment 30?
I hit the error using the try build from comment 29.
tracking-firefox21: + → ?
Assignee: nobody → kinetik
tracking-firefox21: ? → +
I finally figured out how to reproduce this locally; I need to use an SSH-tunnel/proxy through my people.mozilla.com account in order for the bug to appear.

I have built changeset f0422702a077 which is the changeset just before the first changeset in the regression range, and the bug reproduces in that build, so the regression range is definitely wrong and the changesets suspected in comment 3 cannot possibly be the cause of this bug.
Severity: major → critical
I observed the bug in Nightly build from 2013-02-05, which is the first Nightly to have the Windows Media Foundation backend for HTML5 <audio>/<video> enabled. I'll test the Nightly before that, but chances are the bug is in the new WMF backend.
Assignee: kinetik → cpearce
Attached file NSPR log of failure — 
I captured a media NSPR log of the failure. It looks like this is a bug on their side.

On line 71570 is the error:
JavaScript error: http://www.pandora.com/script.head.js?v=944595814, line 7: too much recursion

The script then proceeds to try to load "about:blank" in an audio element, and that load understandably fails.

script.head.js is obfuscated, so I can't easily tell where the actual error is.

Do we have any contacts at Pandora to help with this?
Chris: Can you try this without the JS jit ?
The too much recursion could be a jit bug.
The Firefox safemode is the easiest way to disable the jit but also disabled the hardware acceleration:
The probably better alternative is to disable javascript.options.methodjit.content + javascript.options.ion.content
I set the prefs javascript.options.methodjit.content and javascript.options.ion.content to false, restarted and still observed this error.

I imagine running in safe mode would prevent Flash loading? Pandora refuses to load without Flash enabled.
Safe mode does not affect Flash loading. But I'm surprised that Pandora refuses to load without Flash enabled; it WFM with click-to-play on at least.

Note that script.head.js line 7 is basically jquery minimized.

I have a pandora contact;  Is there an easy way to see what the JS stack of "too much recursion" is using our dev tools? I know Pandora has internal builds with unminized. What exactly should we be asking them?

Should we try and recreate the testcase using only jPlayer, which is open-source?
Ok, so with some help from the guys at Pandora we've determined that the problem is caused by our HTMLMediaElement.canPlayType behaviour. We're not returning affirmatively for canPlayType('audio/mpeg; codecs="mp3"'), and jPlayer is getting confused by this. When I change canPlayType to respond affirmatively to codecs="mp3", the bug goes away.

I've argued in bug 838331 that we should do so, even though it appears to not be in accordance with the spec, all the other browsers that support MP3 do this, so we should too.
We should also handle canplayType('audio/x-m4a; codecs="mp4a.40.2"'), as other browsers handle this too.
* Make us report that we can play "audio/{mp3,mpeg}; codecs="mp3" and "audio/x-m4a; codecs=mp4a.40.2" (which some sites use to detect AAC).

https://tbpl.mozilla.org/?tree=Try&rev=a9591d5eb5c3
Attachment #718180 - Flags: review?(paul)
(In reply to Chris Pearce (:cpearce) from comment #40)
> I've argued in bug 838331 that we should do so, even though it appears to
> not be in accordance with the spec, all the other browsers that support MP3
> do this, so we should too.

And we should reflect this to the spec so that new browser vendors (if at all present) don't have to inspect other browsers again.
Comment on attachment 718180 [details] [diff] [review]
Patch: Make our canPlayType() codecs=mp3 and audio/x-m4a behave the same as other browsers

Review of attachment 718180 [details] [diff] [review]:
-----------------------------------------------------------------

::: content/html/content/src/nsHTMLMediaElement.cpp
@@ +2186,5 @@
>    case CANPLAY_MAYBE:
>      aResult.AssignLiteral("maybe");
>      break;
>    }
> +  

nit: white spaces

::: content/media/test/can_play_type_mpeg.js
@@ +34,5 @@
> +  check("audio/mpeg; codecs=mp3", "probably");
> +
> +  check("audio/mp3; codecs=\"mp3\"", "probably");
> +  check("audio/mp3; codecs=mp3", "probably");
> +  

nit: trailing spaces.

::: content/media/wmf/WMFDecoder.cpp
@@ -26,5 @@
>        NS_FAILED(LoadDLLs()))
>      return false;
>  
> -  // MP3 is specified to have no codecs in its "type" param:
> -  // http://wiki.whatwg.org/wiki/Video_type_parameters#MPEG

How do we go about changing an RFC? At the very least, we could change this page so people know.
Attachment #718180 - Flags: review?(paul) → review+
(In reply to Paul Adenot (:padenot) from comment #44)
> How do we go about changing an RFC? At the very least, we could change this
> page so people know.

RFCs are immutable. You can update them by submitting a new Internet-Draft to the appropriate WG at the IETF and getting it published as an RFC. For example, RFC 6381 <http://tools.ietf.org/html/rfc6381>, which currently defines the codecs parameter, updates RFCs 3839, 4337, and 4393 (but not 3003).
https://hg.mozilla.org/mozilla-central/rev/a212819b76b7
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla22
Once this has proved stable in Nightly for a few days we'll need to get this patch on Aurora too.
Very happy to report that I haven't had this happen for a couple hours steady, now, using:

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20130228 Firefox/22.0 Built from http://hg.mozilla.org/mozilla-central/rev/b0e08db3bc2a

Verified FIXED on trunk/nightly!
Status: RESOLVED → VERIFIED
Comment on attachment 718180 [details] [diff] [review]
Patch: Make our canPlayType() codecs=mp3 and audio/x-m4a behave the same as other browsers

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 837859

User impact if declined: Some sites will fail to recognise that they can use MP3/AAC HTML5 <audio>. For example Pandora.com was failing randomly on music track changes.

Testing completed (on m-c, etc.): Been on m-c for a few days.

Risk to taking this patch (and alternatives if risky): Pretty low.

String or UUID changes made by this patch: None.
Attachment #718180 - Flags: approval-mozilla-aurora?
Comment on attachment 718180 [details] [diff] [review]
Patch: Make our canPlayType() codecs=mp3 and audio/x-m4a behave the same as other browsers

low risk patch for a FF21 regression.well Baked on m-c , the issue's reported on this bug & Bug 844711 have been verified fixed on nightly.Approving for uplift.
Attachment #718180 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
(In reply to Stephen Donner [:stephend] from comment #49)
> Verified FIXED on trunk/nightly!

Thanks Stephen. Would you mind repeating this with tomorrow's Firefox Aurora 21.0a2 builds?
status-firefox22: fixed → verified
Verified with Build identifier: Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20130307 Firefox/21.0, too -- what keyword or whiteboard magic do I use to denote that?
Sorry, forgot to check the flag, but I verified this in Build identifier: Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20130307 Firefox/21.0 (see comment 54, above).
status-firefox21: fixed → verified
Thanks Stephen!
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #56)
> Thanks Stephen!

<3.  Anytime.
Blocks: 865827
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: