1265815 - Spike in Windows GMP crashes in Firefox 46

Status: RESOLVED FIXED

(Core :: Audio/Video: GMP, defect, P1)

Description

[Ryan VanderMeulen [:RyanVM]]

[Tracking Requested - why for this release]: Significant spike in GMP crashes.

It was brought to our attention that the Telemetry crash data shows a significant spike in GMP crashes starting on Beta with the last uplift in March. Anthony Zhang and myself looked at the data on Nightly and the spike appears to have started on the Feb. 25 nightly. Looking at the pushlog range, bug 1250766 stands out as a possible culprit since it landed on Nightly47 and was uplifted to Aurora46 shortly afterward. This theory is especially bolstered because there was a similar drop-off in crash rate in early January when bug 1237145 was landed.

Chris, can you please look into this soon?

Comment 1

[Anthony Zhang [:azhang] (last day at Mozilla: 2016-04-29)]

Crash spike: https://sql.telemetry.mozilla.org/queries/195#321

Comment 2

[Liz Henry (:lizzard) (relman/hg->git project)]

Can you include the crash data in a way that we can see in the bug? The telemetry server isn't accepting my login. But we also want to put data into the bug, so that it will persist while a query may have different results a month from now. 

Too late for 46 fix, but we should be aware of this issue in 47.   If the crash volume is such that we should reopen this in 46 please let me know or change the flag back to affected. Thanks!

Comment 3

[Robert Kaiser]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #2)
> Can you include the crash data in a way that we can see in the bug? The
> telemetry server isn't accepting my login.

Try your @mozilla.com Google account for login, that should work. I hope we'll be able to make publicly visible dashboards out of this in the future but the system there seems to not support that at this time.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

KaiRo, my understanding is that crash-stats data for GMP crashes is pretty low, but is there any discernible pattern in GMP crash reports starting around 25-Feb on Nightly/Aurora or on Beta after 46b1 was released?

Comment 5

[Anthony Zhang [:azhang] (last day at Mozilla: 2016-04-29)]

Attachment: GMPlugin Crashes by BuildID - Breakdown by Windows versions

I've attached the exported CSV for reference (though we don't really expect that data to change).

Note that @mozilla.com users can log into re:dash using their Google Apps accounts - LDAP credentials will not work.

Comment 6

[Anthony Zhang [:azhang] (last day at Mozilla: 2016-04-29)]

Attachment: GMPlugin Crash Rate vs BuildID by Windows Version.png

Here's a picture as well, for convenience.

Comment 7

[Benjamin Smedberg]

Liz, I don't think we should say "too late for 46" without understanding this better. This is a pretty massive regression and could easily be a sign that we broke video playback for some users. I suggest that we consider this a blocker until we understand it better.

Comment 8

[Liz Henry (:lizzard) (relman/hg->git project)]

"Possible release blocker" sounds a lot different than "significant spike".  Who is investigating this? I notice it's assigned to nobody.   

I did not realize this telemetry data was finalized and well understood enough to affect our decisions for release. Is that the case?  My impression was that you all are still testing this system. 

KaiRo: Naturally, when I first looked at this bug this morning I tried my LDAP login, since that is the most incredibly obvious thing to do. However, it did not work.

Comment 9

[Chris Pearce [:cpearce (Not reading bugmail)]]

RyanVM: I can't access the data you're referring to. I don't see a crash spike in crash-stats.mozilla.com.

Is this crash still happening? We've fixed a crash that was affecting Windows XP users caused by Adobe's GMP not working on machines without SSE2.

Comment 10

[Chris Pearce [:cpearce (Not reading bugmail)]]

The crashes I can see for GMP plugins are these:

https://crash-stats.mozilla.com/search/?gmp_plugin=!__null__&date=%3E%3D2016-01-01&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Where is the spike?

Comment 11

[Ryan VanderMeulen [:RyanVM]]

My understanding is that very few GMP crashes get reported to crash-stats because it requires user opt-in and that submission rate is incredibly low.

Comment 12

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Chris Pearce (:cpearce) from comment #9)
> Is this crash still happening? We've fixed a crash that was affecting
> Windows XP users caused by Adobe's GMP not working on machines without SSE2.

This fix I'm referring to is bug 1258220, so which landed on beta on 2016-03-28 and Nightly on 2016-03-25.

Comment 13

[Anthony Zhang [:azhang] (last day at Mozilla: 2016-04-29)]

I think it's important to mention here that on the graph, the purple part is Windows XP - the crash spike *only seems to affect XP users*.

:cpearce, are you using the "Log In With Google" button mentioned earlier in the bug? That is currently the correct way to log into re:dash.

Although the GMPlugin crash rate lowered quite a bit on March 28, the crash rate was still higher than when the plugin wasn't enabled.

Comment 14

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Anthony Zhang [:azhang] from comment #13)
> I think it's important to mention here that on the graph, the purple part is
> Windows XP - the crash spike *only seems to affect XP users*.
> 
> :cpearce, are you using the "Log In With Google" button mentioned earlier in
> the bug? That is currently the correct way to log into re:dash.

Ah, that works. Thanks for pointing that out.

 
> Although the GMPlugin crash rate lowered quite a bit on March 28, the crash
> rate was still higher than when the plugin wasn't enabled.

So based on my reading of your CSV file, this shows there were crashes on all windows version in build ids up until about April, and in about April the crashes switched to being pretty much only on Windows XP.

How do we map build ids to Firefox versions? That is, are we seeing crashes in Beta46, or in Nightly 48?

Comment 15

[Chris Pearce [:cpearce (Not reading bugmail)]]

Based on reading the source of your query, isn't this data for the nightly channel only?

Comment 16

[Chris Pearce [:cpearce (Not reading bugmail)]]

If I fork the query and modify it to run for beta, the number of crashes goes up significantly:

https://sql.telemetry.mozilla.org/queries/204/source#table

Which isn't surprising, since beta has more users.

Comment 17

[Anthony Zhang [:azhang] (last day at Mozilla: 2016-04-29)]

(In reply to Chris Pearce (:cpearce) from comment #15)
> Based on reading the source of your query, isn't this data for the nightly
> channel only?

That's correct, but this also occurs on Beta as well [1].

(In reply to Chris Pearce (:cpearce) from comment #14)
> So based on my reading of your CSV file, this shows there were crashes on
> all windows version in build ids up until about April, and in about April
> the crashes switched to being pretty much only on Windows XP.
> 
> How do we map build ids to Firefox versions? That is, are we seeing crashes
> in Beta46, or in Nightly 48?

The buildIDs map to versions as listed in the release calendar [2]. It does seem to be the case that the crashes used to be for all Windows versions, basically stopped after disabling the Adobe thing, and then started up again for Windows XP after we re-enabled it.

So on [1] you're looking at Nightly 48 for the February 25 spike, and Beta 46 for the March 7 spike (Beta 46 was released on March 7).

Feel free to needinfo me again if you have any questions.

[1]: https://sql.telemetry.mozilla.org/queries/197/source#334
[2]: https://wiki.mozilla.org/RapidRelease/Calendar

Comment 18

[Liz Henry (:lizzard) (relman/hg->git project)]

That is a pretty big spike. We should probably undo whatever it was we did for Adobe then. Unless that makes things worse somehow, like crashing the entire browser.

Comment 19

[Chris Pearce [:cpearce (Not reading bugmail)]]

We've decided to disable unencrypted decoding via the Adobe GMP in 46. That should drop the crashes, but also means users on Windows XP and Windows without platform codecs won't get MP4 video playback for another release. :(

bsmedberg: how can we get crash reports for these crashes?

Comment 20

[Ryan VanderMeulen [:RyanVM]]

Chris, I'm happy to have my Softvision team do some exploratory GMP testing on WinXP if you can confirm which prefs we'll need to make sure are set.

Comment 21

[Chris Pearce [:cpearce (Not reading bugmail)]]

Attachment: [Beta46] Disable unencrypted Adobe GMP decoding due to crashes

We detected a spike in crashes in GMP plugin processes that coincide with
turning on unencrypted decoding via GMP.

This happens on all Windows versions, but particularly on Windows XP.

We also need to revert bug 1234099 so that we don't show the plugin in
the add-on manager.

Comment 22

[Chris Pearce [:cpearce (Not reading bugmail)]]

Comment on attachment 8743044 [details] [diff] [review]
[Beta46] Disable unencrypted Adobe GMP decoding due to crashes

r? spohl for firefox bits.

Comment 23

[Chris Pearce [:cpearce (Not reading bugmail)]]

I will split out hiding the Adobe GMP on WinXP into bug 1265815 per liz's request so that it's easier to track uplifts.

Comment 24

[Chris Pearce [:cpearce (Not reading bugmail)]]

Attachment: [Beta46] Disable unencrypted Adobe GMP decoding due to crashes

We detected a spike in crashes in GMP plugin processes that coincide with
turning on unencrypted decoding via GMP.

This happens on all Windows versions, but particularly on Windows XP.

Comment 25

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Chris Pearce (:cpearce) from comment #23)
> I will split out hiding the Adobe GMP on WinXP into bug 1265815 per liz's
> request so that it's easier to track uplifts.

Copy paste error, I meant bug 1265928.

Comment 26

[Chris Pearce [:cpearce (Not reading bugmail)]]

Comment on attachment 8743058 [details] [diff] [review]
[Beta46] Disable unencrypted Adobe GMP decoding due to crashes

Approval Request Comment
[Feature/regressing bug #]: Unencrypted MP4 playback using Adobe Gecko Media Plugin
[User impact if declined]: GMP child process crashes while trying to play MP4 video. Note: Firefox itself won't crash, the user visible effect is video will just fail to play.
[Describe test coverage new/current, TreeHerder]: We have a test covering unencrypted decoding.
[Risks and why]: Fairly low; this is a revert to previous behaviour
[String/UUID change made/needed]: None

Comment 27

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8743058 [details] [diff] [review]
[Beta46] Disable unencrypted Adobe GMP decoding due to crashes

No longer allowing this as it has caused many crashes on 46 beta and we need to look into why. We are leaving this enabled in nightly though.

Comment 28

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8743058 [details] [diff] [review]
[Beta46] Disable unencrypted Adobe GMP decoding due to crashes

Also land this on m-r since we are building RC2 from there.

Comment 29

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/releases/mozilla-release/rev/a70211859e3a157c3d243ded003301c4d0b8de29
https://hg.mozilla.org/releases/mozilla-beta/rev/c5df1b572b8e80ff8d9fafb61c01dc3cb1e5fc77
https://hg.mozilla.org/releases/mozilla-aurora/rev/75d615fb8e1db3d263427b6684bf95fb8c17ed12

Comment 30

[Robert Kaiser]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #4)
> KaiRo, my understanding is that crash-stats data for GMP crashes is pretty
> low, but is there any discernible pattern in GMP crash reports starting
> around 25-Feb on Nightly/Aurora or on Beta after 46b1 was released?

As comment #10 shows, we have almost no reports at all in crash-stats, and no discernible pattern for sure. I'm pretty sure that the crash prompt for GMP doesn't work usefully at least in this case, the few reports we get in may very well be only from people who go to about:crashes and click on reports there.

Comment 31

[Robert Kaiser]

(In reply to Anthony Zhang [:azhang] from comment #17)
> (In reply to Chris Pearce (:cpearce) from comment #14)
> > How do we map build ids to Firefox versions? That is, are we seeing crashes
> > in Beta46, or in Nightly 48?
> 
> The buildIDs map to versions as listed in the release calendar [2].

We may want to work on a database that we can join data with that actually maps build IDs to released versions (there are some thoughts about building a DB like that in general, no project for it yet though), but right now we have to manually find that out via release calendar or other tools.

Comment 32

[Liviu Cirdei]

I reproduced the crash on Nightly 48.0a1 (2016-04-24) on a Windows XP machine with Intel Pentium 4. When I tried to play a mp4 video (e.g. http://techslides.com/demos/sample-videos/small.mp4), the video failed to play, and I could see crashes in about:crashes, even if the Firefox itself did not crash.
This is reproducible both with preferences "media.wmf.enabled" and "media.webm.enabled" being set to true or false.

All crashes have the same signature: [@ mozilla::gmp::GMPChild::ProcessingError]. The same signature was reported in bug 1262377, where Vimeo videos fail to play on Windows XP Intel Pentium 4.

Comment 33

[Chris Pearce [:cpearce (Not reading bugmail)]]

Let's disable unencrypted GMP decoding everywhere until we've got crash reporting hooked up. We can't debug this without crash reporting hooked up. That's happening in bug 1267918.

Comment 34

[Chris Pearce [:cpearce (Not reading bugmail)]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/d0dae4dc43e25b6ea5e94ca5ae90f0bef1164f6d
Bug 1265815 - Disable unencrypted Adobe GMP decoding due to crashes. r=kentuckyfriedtakehe

Comment 35

[Chris Pearce [:cpearce (Not reading bugmail)]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/40994e384c13

Comment 36

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Chris Pearce (:cpearce) from comment #35)
> https://hg.mozilla.org/releases/mozilla-aurora/rev/40994e384c13

I also took the liberty of disabling this on Firefox 48 (Aurora) since this had slipped through in the last uplift.

Unencrypted decoding via the Adobe Primetime GMP is now disabled everywhere. EME is still working.

Comment 37

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/d0dae4dc43e2