1510340 - Opening nytimes.com HLS video crashes the browser - [@ libart.so@0x31b6df ]

Status: VERIFIED FIXED

(Firefox for Android Graveyard :: Audio/Video, defect, P3)

Description

[csheany]

User Agent: Mozilla/5.0 (Android 7.1.1; Tablet; rv:65.0) Gecko/65.0 Firefox/65.0

Steps to reproduce:

1. Open nytimes.com/video


Actual results:

After a few seconds the browser crashes.


Expected results:

The browser shouldn't crash.

Comment 1

[csheany]

https://crash-stats.mozilla.com/report/index/5e7f91ea-5a95-4dc4-9093-12d360181127

Comment 2

[csheany]

The desktop site doesn't seem to be affected.

Comment 3

[csheany]

What does this mean from a browser standpoint?

Is there a fix?

Comment 4

[csheany]

Also,

Crash in @ java.lang.OutOfMemoryError: at java.lang.Long.valueOf(Long.java)

https://crash-stats.mozilla.org/report/index/b496a419-545d-4e78-9270-c95080181226

Crash in @ java.lang.OutOfMemoryError

https://crash-stats.mozilla.org/report/index/c998b98a-8eeb-4be4-a48f-208060181226

Comment 5

[csheany]

Liz, would you mind taking a look?

Comment 6

[Liz Henry (:lizzard) (privacy/fingerprinting team)]

I'm not hitting the crash. Petru, want to see if it's reproducible for you?

Comment 7

[Kevin Brosnan [Ex-Mozilla]]

This crash is specific to a device. This crash is on the Samsung Galaxy Tab A 8.0 (2015) SM-T350.

Comment 8

[csheany]

With only this in one tab and Nightly Home in another it doesn't seem to crash but does cause Bug 1494748.

Comment 9

[csheany]

What does the first crash signature indicate?

Comment 10

[csheany]

Crash in @ java.lang.OutOfMemoryError: at java.util.concurrent.ConcurrentLinkedQueue.iterator(ConcurrentLinkedQueue.java)

https://crash-stats.mozilla.org/report/index/42776b2b-5799-4e46-9a99-aa6cb0181227

Comment 11

[csheany]

Crash in @ java.lang.OutOfMemoryError: at org.mozilla.gecko.media.GeckoHlsVideoRenderer.createInputBuffer(GeckoHlsVideoRenderer.java)

https://crash-stats.mozilla.org/report/index/a4ada562-c967-4bbe-862a-aff5e0181227

Comment 12

[csheany]

Crash in @ java.lang.OutOfMemoryError: at org.mozilla.gecko.mozglue.GeckoLoader.nativeRun(Native Method)

https://crash-stats.mozilla.org/report/index/f56198d9-254f-4d8a-a341-57bb90181228

Comment 13

[csheany]

Crash in @ java.lang.OutOfMemoryError

https://crash-stats.mozilla.org/report/index/460f0c0d-6eab-42d6-b8a6-68f610181228

Comment 14

[csheany]

Jan, please see comment 8

Comment 15

[Jan Henning [:JanH]]

OOMs/near-OOMs/resource exhaustion can cause all sorts of weird behaviour.

Comment 16

[csheany]

Crash in @ java.lang.OutOfMemoryError

https://crash-stats.mozilla.org/report/index/178d530f-b5b5-46c3-b56c-79f7c0190102

Comment 17

[csheany]

With media.hls.enabled set to false the page didn't cause any issues.

Comment 18

[csheany]

Crash in @ java.lang.OutOfMemoryError: at com.android.okhttp.okio.Segment.<init>(Segment.java)

https://crash-stats.mozilla.org/report/index/63959b72-cbdf-41f5-9efd-0820d0190108

Comment 19

[csheany]

Would you mind investigating this with Focus+GV?

After some time about:blank opens and I wonder if there are similiarites.

Comment 20

[Chris Peterson [:cpeterson]]

(In reply to csheany from comment #17)

With media.hls.enabled set to false the page didn't cause any issues.

Interesting! Some of your OutOfMemoryError stack traces all point to video codec code, so perhaps there is a bug in Gecko and/or the codec that causes video data to keep growing.

Can the video still play video when media.hls.enabled is set to false?

(In reply to csheany from comment #19)

Would you mind investigating this with Focus+GV?

Sorry, I don't have a Samsung Galaxy Tab A to test.

After some time about:blank opens and I wonder if there are similiarites.

Perhaps the about:blank indicates the tab (content process) crashed? There is a different bug open about about:blank opening in tabs.

Comment 21

[csheany]

Thank you for your response.

Yes, the videos do still play. I forgot to mention that.

I didn't necessarily mean with the samme device but the app in general.

Comment 22

[csheany]

Also, it doesn't require playing video just scroling the page or having it in the background.

Comment 23

[Petru-Mugurel Lingurar [:petru]]

Removing my NI as it is now on geckoview team's list.

Comment 24

[csheany]

David, would you mind taking a look?

Comment 25

[David Bolter [:davidb] (NeedInfo me for attention)]

Andreas what is the priority for this bug? (This should affect GV as well)

Comment 26

[csheany]

Alastor, would you mind taking a look?

Comment 27

[Alastor Wu [:alwu]]

It looks like the crash happened while site was calling MediaCapabilities.decodingInfo() which would create a HLS decoder and caused the crash.

I will forward this NI to John, who know more Android video decoding than me, to see if he has any idea.

Comment 28

[John Lin [:jhlin][:jolin]]

The OutOfMemoryError cases were fixed in bug 1517252. Mark this as dup for now. If the crash still happens when playing HLS video in 66, please reopen it.

Comment 29

[csheany]

Thank you both for your response.

It doesn't seem to crash but does give error messages.

Comment 30

[csheany]

Attachment: Screenshot_20190207-183538.jpg

Comment 31

[Chris Peterson [:cpeterson]]

John, are the "This video is currently unavailable" and "Video Data Failed to Load" errors in csheany's screenshot just a side effect of Fennec gracefully handling the OOM instead of crashing (bug 1517252)? If so, why are we OOM'ing in the first place?

Comment 32

[csheany]

After further testing, it still crashes.

Comment 33

[John Lin [:jhlin][:jolin]]

(In reply to Chris Peterson [:cpeterson] from comment #31)

John, are the "This video is currently unavailable" and "Video Data Failed to Load" errors in csheany's screenshot just a side effect of Fennec gracefully handling the OOM instead of crashing (bug 1517252)? If so, why are we OOM'ing in the first place?

Yes, I think the screenshot shows the fix in bug 1517252 is effective.

I've managed to reproduce the OOM once by letting NYT video site play video for a couple for hours. From what I observed, it looks quite similar to what James Cheng found in bug 1424168 comment 2: the NYT video page creates and loads 10 video elements and Exoplayer allocates more than 100MB in the Java heap just to buffer contents for them. Although bug 1424168 has reduced the size significantly, it is still not enough for sites that preloads many videos.

Reopen this bug and will investigate if there is other way to further reduce the memory required for this sort of sites.

Comment 35

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1510340 - p1: reduce ExoPlayer default buffer size. r?snorp

Comment 36

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1510340 - p2: limit the concurrently created HLSDecoder instances. r?jya

Each instance has an instance of Java ExoPlayer that consumes memory in the
limited JVM heap. Too many concurrent players will cause OutOfMemoryError.

Depends on D20419

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/da7bf6175838
p1: reduce ExoPlayer default buffer size. r=snorp
https://hg.mozilla.org/integration/autoland/rev/63b009fbe2bc
p2: limit the concurrently created HLSDecoder instances. r=jya

Comment 38

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/da7bf6175838
https://hg.mozilla.org/mozilla-central/rev/63b009fbe2bc

Comment 39

[Liz Henry (:lizzard) (privacy/fingerprinting team)]

This doesn't look like a frequently occuring crash (at least not with this particular signature).
Do you think letting this go to release with 67 is OK, or do you want to try beta uplift?
Keeping in mind we only have 2 weeks of beta 66 left.

Comment 40

[John Lin [:jhlin][:jolin]]

(In reply to Liz Henry (:lizzard) (use needinfo) from comment #39)

This doesn't look like a frequently occuring crash (at least not with this particular signature).
Do you think letting this go to release with 67 is OK, or do you want to try beta uplift?
Keeping in mind we only have 2 weeks of beta 66 left.

I prefer not to uplift to beta.

Comment 41

[csheany]

I think the first crash is irrelevant at this point hence all the others.

The time depends on the amount of other tabs open.

In the middle of writing/ testing the browser crashed and didn't trip the reporter.

I have a feeling Bug 1508641 may be of further assistance.

Comment 42

[csheany]

It's still not a great experience.

There are a lot of empty elements.

More videos can be loaded than the minimum and I'm not sure it adapts.

The progress bar takes a while to complete. The page just feels heavy.

FWIW, Focus with WebView doesn't seem to have any issues.

Comment 43

[csheany]

Please see Comment 12

Comment 44

[csheany]

If it helps, based on Comment 28 and Comment 33...

I have been testing further with Reference Browser 1.0.1910 (Build #1063145 Gecko 67.0a1-20190227104426) to see which Crash Signatures it triggers.

Comment 45

[csheany]

... how it performs and

Comment 46

[Eitan Isaacson [:eeejay]]

I am not assigned to this bug and I don't think I can help. If this needs anymore attention you can needinfo the assignee.

Comment 47

[csheany]

Thank you for your response.

The reason I brought this to your attention was to provide a use case for Bug 1508641 which I noticed has not been fixed yet.

Comment 48

[Eitan Isaacson [:eeejay]]

If there is anything in that bug that needs clarification you can needinfo me there. It seems to not be frequent anymore, and the cases where it still happens are not accessibility related.

Comment 49

[csheany]

Marcia, does this site give you any problems?

Comment 50

[csheany]

Have you had any issues?

Comment 51

[Marcia Knous [:marcia]]

(In reply to csheany from comment #49)

Marcia, does this site give you any problems?

I will test over the weekend on all my devices. A cursory test on Pixel 3 running Android Pie showed no issues playing videos on that site.

Comment 52

[Sorina Florean [:sflorean]]

(In reply to csheany from comment #50)

Have you had any issues?

Tested on the latest Nightly build following your steps and the browser did not crash. The videos played with no interruptions and no error was displayed.
Device: Nexus 9(Android 7.1.1).
Note that we don't have the Samsung Galaxy Tab A 8.0 (2015) SM-T350 device.

Comment 53

[csheany]

Thank you for your response.

Would you mind testing Nightly with other devices as well as Release?

Also, how much time did you allow for and how many tabs did you have open?

Were videos loaded after scrolling?

Comment 54

[Sorina Florean [:sflorean]]

Tested on 67.0b3 with the following devices:

• Xiaomi Mi4i(Android 5.0.2)
• Samsung Galaxy Tab S3 (Android 8.0)
• Google Pixel (Android 9)
• Samsung Galaxy Note 8 (Android 9.0)

Didn't encounter any issues, the video played without interruptions. I had more than 7 tabs opened and also just one, watched 4-5 videos in fullscreen mode without interacting with the screen and also played them individually. In all cases, no issues were displayed.

If you have other issues, and no crash displayed, please make a video to understand the problem. Thanks!

Comment 55

[Sorina Florean [:sflorean]]

Based on my last comment I will update the flag to VERIFIED.

Comment 56

[csheany]

The problem isn't necessarily playing video but keeping the browser open.

It still crashes but not sure why.

Comment 57

[Sorina Florean [:sflorean]]

(In reply to csheany from comment #56)

The problem isn't necessarily playing video but keeping the browser open.

It still crashes but not sure why.

The bug was filed for the crashes and this issue is fixed, if you find another problem, file a new bug. I asked in comment 54 to make a video with the new issue but with no answer.

Comment 58

[csheany]

The issue is still the same. It might be fixed for some but not all.

Even without these patches other devices might not have a problem in Release.

The only thing a video would display is the browser crashing.

Without a report attached that is not very actionable (Bug 1522107)

Given the different signatures it would have been ideal for this to be a meta and spread out but that is neither here nor there.

I'm still trying to understand the approach (Bug 1517785)

There are a few others (Bug 1517985 - Bug 1512147 - Bug 1544888)