1582222 - Increasing YouTube player speed causes tab crashes without any debug trace

Status: VERIFIED FIXED

(Core :: Audio/Video: cubeb, defect, P1)

Description

[gwarser]

Attachment: about-support.txt

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0

Steps to reproduce:

• open fresh Firefox profile
• load https://www.youtube.com/watch?v=IKsQsxubuAA
• wait when page fully loads
• click on player and then immediately tap three times on Shift+>

If nothing unexpected happens for 5 seconds, tap left arrow key to rewind to beginning.

If still nothing, try from beginning few more times.

Actual results:

I cannot reproduce this 100% of the time, but:

• there may be small freeze when cube rotates and then:
• tab crash after up to 5 seconds (when cube rotates); OR
• tab crash just after rewinding to the start
• no crash dump in about:crashes
• no info in browser console
• no info in shell console (besides of info about connection with sub process lost unexpectedly)

Expected results:

Nothing should crash, video plays with increased speed.

This happens on random videos.

At first I thought this is related to vp9 decoder, so I started using enhanced-h264ify to disable vp9 and blamed bug 1570046 Today I discovered it still happens with vp9 blocked. Maybe blocking vp9 only decreases probability of this crash? Or it's caused by sound codec? Looks similar to bug 1572878 but still not fixed for me.

I tried bisecting, and today it clearly shows me https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=d6571848ba1b4d80fac14c7fdba19caf10049b2e&tochange=f058271e55d9e9dcd472b310d2bc92048545ff72

But I tried bisecting many times already, few weeks ago, and at best I can narrow it to the second part of June, because mozregression always stopped in different pushlog range - crashes was too random. I will attach few logs.

Maybe there is some way to produce more debug output?

Comment 1

[gwarser]

Attachment: new_2019-09-18_bisection.txt

Comment 2

[gwarser]

Attachment: old_2019-08-21a_bisection.txt

Comment 3

[gwarser]

Attachment: old_2019-08-21b_bisection_interrupted.txt

Comment 4

[Paul Adenot (:padenot)]

Will be fixed by bug 1575883.

Something happens when changing the rate, that blows the real-time budget. It would be good to have measurements of that.

Comment 6

[Paul Adenot (:padenot)]

Attachment: Bug 1582222 - Disable rt threads on beta for Linux for now. r?kinetik

Comment 7

[Paul Adenot (:padenot)]

Attachment: Bug 1582222 - Disable rt threads for Linux for now. r?kinetik

Comment 8

[Paul Adenot (:padenot)]

Attachment: Bug 1582222 - Disable rt threads on release on Linux for now. r?kinetik

Comment 9

[Paul Adenot (:padenot)]

Comment on attachment 9094108 [details]
Bug 1582222 - Disable rt threads on beta for Linux for now. r?kinetik

Beta/Release Uplift Approval Request

• User impact if declined: Potential crash without crash report when changing the playbackRate of a media element. This is common because the feature is exposed in the UI in, e.g., YouTube.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Open a YouTube video. Change the playbackRate, in the cog icon, verify that it does not crash, on multiple video, because this is intermittent.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is just stopping the promotion of a thread at the system level.
• String changes made/needed: none

Comment 10

[gwarser]

I want to add - if you find video where it crashed once, then you can reproduce again in similar playback position with high probability.
Few videos which are crashing for me:

• https://www.youtube.com/watch?v=BV-WEb2oxLk&t=2m15s - crashes around 2:35s
• https://www.youtube.com/watch?v=F3lJwV7ZIIk&t=2m15s - around 2:35s
• https://www.youtube.com/watch?v=0RiwQWqZQf8 - crash in the first few seconds

All tested with speed 1.75 (three taps on shift+>)

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/86f456156bd0
Disable rt threads for Linux for now. r=kinetik

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

Changing the priority to p1 as the bug is tracked by a release manager for the current beta.
See What Do You Triage for more information

Comment 13

[Anca Soncutean, Desktop QA]

I’ve reproduced the crash with Fx 71.0a1 (2019-09-18) on Ubuntu 18.04 x64.
After a thorough test session, I could no longer reproduce it with Fx 71.0a1 (20190924112407); I've tried with different Youtube videos, different resolutions / speed / fullscreen scenarios.

Comment 14

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/86f456156bd0

Comment 15

[Paul Adenot (:padenot)]

(In reply to Anca Soncutean [:Anca], Desktop Release QA from comment #13)

I’ve reproduced the crash with Fx 71.0a1 (2019-09-18) on Ubuntu 18.04 x64.
After a thorough test session, I could no longer reproduce it with Fx 71.0a1 (20190924112407); I've tried with different Youtube videos, different resolutions / speed / fullscreen scenarios.

This is unexpected, except if you had a bunch of Firefox running, in which the thread promotion could have failed, because there is a maximum number of threads that can be promoted by a given user.

Can you try again with tomorrow's Nightly (with the patch merged), with only this Firefox Nightly running?

Comment 16

[u636583]

fixed for me in firefox-hg r494624+.79b08a0a1c3f+-1
confirmed by first reproducing it in firefox-hg r493411+.22ced1a079e0+-1
reproduction steps from Comment 10's last vid like this:

1. visit https://www.youtube.com/watch?v=0RiwQWqZQf8
2. click on video to play
3. shift+> three times to get 1.75x
4. alt+d to focus address bar
5. press Enter to reload
6. crashed (or click play on the video to start playing which then will insta crash)

Comment 17

[Anca Soncutean, Desktop QA]

(In reply to Paul Adenot (:padenot) from comment #15)

This is unexpected, except if you had a bunch of Firefox running, in which the thread promotion could have failed, because there is a maximum number of threads that can be promoted by a given user.

Can you try again with tomorrow's Nightly (with the patch merged), with only this Firefox Nightly running?

I forgot to mention in my previous comment, that I used the treeherder build (from comment 11) for the initial verification.

I’ve retested with the official Nightly 71.0a1 (2019-09-25) on Ubuntu 18.04 x64. Since this issue is intermittent, I’ve double checked with one of my colleague to make sure this is indeed proper verified. NO crash occurred for both of us. Also ran a quick check on Windows 10 x64 and macOS 10.13.

Comment 18

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 9094108 [details]
Bug 1582222 - Disable rt threads on beta for Linux for now. r?kinetik

Crash fix, verified in Nightly (thanks!!)
Let's uplift for beta 10.

Comment 19

[Liz Henry (:lizzard) (relman/hg->git project)]

Paul, just double checking that I have this correct. You only mean to uplift one of these attachments. Could you check that I have the correct one and mark the others obsolete if that's your intent?

Comment 20

[Paul Adenot (:padenot)]

Yes, sorry. The one that is labelled "on release" for release and the one you just a+ed on beta. Thanks!

Comment 21

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/c1868de79e07

Comment 22

[Anca Soncutean, Desktop QA]

The issue is verified fixed with the treeherder build 70.0b10 (20190926150335) on Ubuntu 18.04 x64, Windows 10 x64 and macOS 10.13.

Comment 24

[Dr. David Alan Gilbert]

Just duped my 1584838 here; if you turn on Linux's print_fatal_signals you get:

[15713.188681] CPU Watchdog Timeout (hard): AudioIPC1[8234]

Comment 25

[Ryan VanderMeulen [:RyanVM][PTO June 24-28]]

Comment on attachment 9094110 [details]
Bug 1582222 - Disable rt threads on release on Linux for now. r?kinetik

Fixes a Linux crash when changing the playback rate of YouTube videos by reverted to previous known-good behavior. Approved for 69.0.2.

Comment 26

[Ryan VanderMeulen [:RyanVM][PTO June 24-28]]

https://hg.mozilla.org/releases/mozilla-release/rev/1343ac01ae76

Comment 27

[Anca Soncutean, Desktop QA]

The issue is verified fixed with 69.0.2 (20191001234643) on Ubuntu 18.04 x64, Windows 10 x64 and macOS 10.14.

Comment 28

[Ryan VanderMeulen [:RyanVM][PTO June 24-28]]

Added to the Firefox 69.0.2 release notes:

Fixed a Linux-only crash when changing the playback speed while watching YouTube videos

Comment 29

[Dr. David Alan Gilbert]

This is back in 73.0.1 - filed as https://bugzilla.mozilla.org/show_bug.cgi?id=1617422