1521760 - HTML5 camera capture and image processing methods cause crash since ver 64.0.2 released

Status: RESOLVED INCOMPLETE

(Firefox for Android Graveyard :: General, defect, P2)

Description

[John Calder]

Attachment: Bugzilla report image capture camera mitcitmm.docx

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0

Steps to reproduce:

My web app accesses the device camera with code like in this MDN article:
https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia

I run my simplified test app created for this bug report:
https://manukautech.github.io/xmrt
Video feed appears. Click [Start] button. The counter increments on motion detection events .....

Actual results:

If keeping screen on, Firefox-for-Android crashes after 4 minutes.
If locking the screen, leave for 20 min and then unlock to find the "crash" has happened.
This "crash" consists of the screen returning to the Android system start display.
Firefox is still running but it has started a new session with a reloaded page.
Firefox-for-Android 65 beta has the same behaviour as 64.

Expected results:

Firefox-for-Android should continue indefinitely. It was working well before ver 64.0.2
Firefox is the only practically useful browser for a remote monitoring field camera because Firefox can continue running while the screen is locked.

More details in the attached document.

Comment 1

[Stefan Deiac]

Hi and thanks for your report. I tried with all your information a few times on Release version 64.0.2 and Beta 65.0b13 but without any success. Do you have any specific settings? Also, if you can catch a crash report it will be very helpful for us. You can find it by searching about:crashes on your URL bar. I'll wait for your updates, thanks.

Comment 2

[John Calder]

My 2 android phone devices with this "crash" behaviour are:

• Vodafone Smart E8 VFD-513 rebadged ZTE, Android 7.1.1, 1.0 G RAM
• Vodafone Smart Mini 7 VFD-300, Android 6.0, 0.5 G RAM

Settings:

• screen timeout change from 0.5 min to 1.0 min but that is only to make testing easier and no difference observed
• developer option "Stay Awake" turned on, again only to make testing easier and no difference observed
• app "Keep Screen On" is only to make testing easier and no difference observed
• VFD-513 has "Settings -- Power Manager -- App power-saver manager -- Firefox -- Allow deep sleep" which I have set to "Off".

I have tried "about:crashes".
Both devices respond: "no crash reports have been submitted".

Firefox on Windows 10 PC works as expected as in it does not "crash".

I am now trying to borrow other Android devices from family and friends.

Comment 3

[John Calder]

More testing on the same and more devices. I am focussing on running long tests with the screen locked. The use of keep-screen-on methods speeds up the crashes but is not something I can do with borrowed devices. Also the long test with screen locked is a better simulation of "real world" use of these functions.

There is more complicated pattern emerging where the low cost low resource devices experience the crashes. However these devices could be playing the useful role of "canary down the mineshaft" in that they are highlighting a memory management issue that could be more difficult to investigate on a phone with more memory where this could show up as a lurking rare intermittent problem.

What I want to do now is run comparative trials with the previous version of Firefox. 63?
How and where can I download ".apk format" installer files for that?

Run - Samsung J3 Pro. Android 8.0.0. 60 minutes run OK. Did NOT crash but I could only borrow it for 60 min.

Run - Samsung Tablet Galaxy Tab A SM-T350, Android 7.1.1, 1.5 G RAM. 85 min run OK.

Run - Alps X01S SmartWatch. Android 5.1, 1 G RAM, 82 minutes run OK.

Run - Vodafone VFD-300 0.5 G RAM. 40 minutes to crash.

Run - Vodafone VFD-513 1 G RAM. 150 minutes to crash.
That last result from late in the day, casts a cloud over the previous "reliable" results of less than 150 min and suggests they need longer runs. A little difficult to organise with the device owners. The J3 owner temporarily installed Firefox and uninstalled it immediately after the trial, such are the practical issues of this kind of testing.

Run x 2 -- comparison test using app "Keep Screen On" with the Vodafone VFD-300:
Firefox 2 minutes to crash. Google Chrome 112 minutes run OK.

My current hypothesis is that Firefox 64 has a change in image handling system functions that causes a small slow memory leak or transient high memory demands.

Comment 4

[John Calder]

Minor correction to the above based on closer observation as to what a "crash" looks like.
On unlocking the screen to check status, I have been seeing a frozen image web page in the cascade view of the current app windows. This web page then reloads when I select it. I thought at first I was seeing a "crash" happen. However trials of the full app give more data so I now know that this is the aftermath of a previous "crash".

Therefore from above:
"VFD-300 ... 40 minutes to crash" should be "crash between 0 and 40 minutes"
"VFD-513 ... 150 minutes to crash" should be "crash between 70 and 150 minutes"

New result:
Run - Samsung Tablet Galaxy Tab A SM-T350, Android 7.1.1, 1.5 G RAM.
Running the full app with the screen locked. As I write that run is still going OK after 3 and a half hours (210 minutes).

Comment 5

[Stefan Deiac]

Hi, thanks for your updates. I tried to reproduce this issue after all your information using Nexus 5 (Android 6.0.1) and Motorola Nexus 6 (Android 7.1.1). on the Release version (64.0.2) but still without any success.

(In reply to John Calder from comment #3)

What I want to do now is run comparative trials with the previous version of Firefox. 63?
How and where can I download ".apk format" installer files for that?

You can install any version of Firefox Beta and Release from here: http://archive.mozilla.org/pub/mobile/candidates/ and for Nightly version from here: http://archive.mozilla.org/pub/mobile/nightly/

Comment 6

[John Calder]

Thanks Stefan - that was interesting.

NEW HYPOTHESIS
In customising Android to make low cost phone packages, some developers have made use of old deprecated APIs. Firefox 64 has dropped support for these APIs.

EVIDENCE
(1) An interesting result from a test with a different purpose. I have an 8 year old Sony Experia with Android 4.0.4 that I would like to use. I discover that that no browser running on Android 4.0.4 is capable of supporting my "getUserMedia" code. I assembled a test website including "adapter.js" which is a "polyfill" aiming to modify the interfaces of some older APIs to be consistent with current standards. This did not work for the Sony Experia. Android 4.0.4 is too old for that limited rescue attempt. But before going to bed last night I set up a run of this new test website for the VFD-513 and Firefox 64 expecting it to add to my crash data. However checking on it the next morning it was still running well after 9 hours (540 min). More impressive is that this website is the "full" web app rather than the lightweight test version. In other words, a fix intended for old APIs has solved my immediate problem. It raises the interesting question as to why an Android 7.1.1 phone bought new in Dec 2018 responds well to a polyfill fix for old APIs? Other phones do not need this. Hence the above new hypothesis.

(2) The Vodafone VFD-513 with Firefox 63 appears to be long term stable with the screen locked. Longest run so far is 290 minutes all stable. The same test conditions with Firefox 64 did give crash results. That does show a difference between Firefox 63 and Firefox 64. However "different" does not necessarily mean "bug".

---

There is another related but different issue that I have observed as a result of my tests.
Both Firefox 63 and Firefox 64 are unstable when running my "getUserMedia" apps under screen "WAKE" conditions.
The behaviour is the same for the developer option "Stay Awake"(with USB connected) and the app "Keep Screen On".
That is very unstable as in crash after only 2 to 4 minutes.
Chrome, Opera and Microsoft Edge are stable on the same test on the same phones.
I have however only done this on the Vodafone devices and it is therefore possible that Chrome, Opera and Edge are holding on longer to deprecated APIs. Have you tried running my test app under "WAKE" conditions?

Comment 7

[Stefan Deiac]

Hi, I tried all your scenarios but without any success. Devices: Motorola Nexus 6 (Android 7.1.1) and Samsung Galaxy Tab S3 (Android 8.0).
I'll continue to investigate this issue due to your findings and I'll come with an update if I found something new. Also if you found something new please let us know, thanks.

Comment 8

[John Calder]

Thanks for testing. I too will keep this investigation open, probably with reduced attention as I catch up with my other projects. Those other projects may deliver more information. For example, yesterday I coded a camera local app rather than a web app. Testing on 2 phones it did good runs on an old Sony Experia. But on the Vodafone VFD-513 it became less and less responsive over a 10 minute period followed by freeze and crash. I then simplified the code and it now works on both phones. Different coding platform, but again the low cost and low resource device is the one that gives more development challenges.

Comment 9

[Stefan Deiac]

Attachment: logcat.txt

Hi, I checked it again with Prestigio Grace X5 (Android 4.4.2) thinking probably it's a specific issue based on the RAM of the phone and I was able to reproduce this issue but just 2/5 attempts. Tested on the Nightly version 66.0a1 (2019-01-28) and Release 65.0.1 build 2.
I tried to catch a logcat with this problem and I found this. I will let the developers investigate my findings and come with updates.

Comment 10

[Marcia Knous [:marcia]]

Marking as a P2 based on discussion today at Fennec triage. It was mentioned that it would be helpful to have a regression range if possible, and to check whether 65 is affected.

Comment 11

[Kevin Brosnan [Ex-Mozilla]]

Guide to using Mozregression GUI on Windows. This process will download 10 - 20 builds and find the regressing bug.

• Set your phone to accept USB debugging connections via Android's "Developer
  options" in the settings
• download and install the android platform tools for your operating system
  https://developer.android.com/studio/releases/platform-tools
• Using the command line confirm that "adb devices" returns your phone's information

Once you have the phone connected via adb then install mozregression

• download mozregression-gui from https://github.com/mozilla/mozregression/releases

• From the File menu choose Run a new bisection

• Basic configuration: Application: select fennec and press next

• Profile selection: Do nothing on this page and press next

• Last known good build: 2018-05-07 no change for First known bad build

• answer good or bad for the build

• the next build to test will be downloaded automatically

• once the bisection is done you should be down to a single checkin. Paste the
  last 5 or so lines of the bisection that shows the hg.mozilla.org link to the regressing bug.

If you run into issues email me directly at my user name at mozilla com

Comment 12

[Stefan Deiac]

Last good revision: 3acee2d5cd9d94252bbffba40799b09aee9daa64
First bad revision: 9b4b53a145386e6be47b9acce399a02f427285f4
INFO: Pushlog:https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=3acee2d5cd9d94252bbffba40799b09aee9daa64&tochange=9b4b53a145386e6be47b9acce399a02f427285f4

I hope this will help the developers.
Device: Prestigio Grace X5 (Android 4.4.2);

Comment 13

[Andrew Overholt [:overholt]]

(In reply to Stefan Deiac from comment #12)

Last good revision: 3acee2d5cd9d94252bbffba40799b09aee9daa64
First bad revision: 9b4b53a145386e6be47b9acce399a02f427285f4
INFO: Pushlog:https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=3acee2d5cd9d94252bbffba40799b09aee9daa64&tochange=9b4b53a145386e6be47b9acce399a02f427285f4

I hope this will help the developers.
Device: Prestigio Grace X5 (Android 4.4.2);

Maybe Chris has thoughts about this since his work is in that range.

Comment 14

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Andrew Overholt [:overholt] from comment #13)

(In reply to Stefan Deiac from comment #12)

Last good revision: 3acee2d5cd9d94252bbffba40799b09aee9daa64
First bad revision: 9b4b53a145386e6be47b9acce399a02f427285f4
INFO: Pushlog:https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=3acee2d5cd9d94252bbffba40799b09aee9daa64&tochange=9b4b53a145386e6be47b9acce399a02f427285f4

This range seems unlikely to be the cause; the patches in this range affect our autoplay blocking behaviour; if they caused a crash, the crash would be immediate, rather than after some time as the bug description describes.

John Calder: If you open about:crashes on the device that crashes, are you able to view one of the crash reports and retrieve its URL and post that URL into this bug report please? That should tell us exactly what code is crashing, helping us narrow down the problem.

Comment 15

[Andrew Overholt [:overholt]]

(for comment 14)

Comment 16

[Liz Henry (:lizzard) (relman/hg->git project)]

John Calder - can you share a crash report url as described in comment 14? Thanks very much!

Comment 17

[John Calder]

I have only seen this today because this account belongs to a project with reduced activity. I have now installed FireFox into a VFD-300 Android phone with a history of this behaviour and it is now doing a test run. In all past events "about:crashes" has returned no result. More recent experience suggests that all apps with visual elements have issues running on Android when the screen is locked and I have changed to the alternative "pinned app" mode of operation. I will therefore try "pinned app" as well.

Comment 18

[John Calder]

As before. "about:crashes" gives result "No crash reports have been submitted". Behaviour as before is that website returns to its home page with its login session forgotten, like a "refresh". I will now repeat using "pinned" as I now do for other apps.

Comment 19

[John Calder]

"Pinned" gives the same result, with the "crash" happening after about 5 min.

"Pinned" gives me the opportunity to run the same website in the same way with Google Chrome.
And for the first time, I see a similar "Chrome crash".
Chrome freezes the JavaScript-programmed functionality on this website with this message:
"This page uses too much memory so Chrome removed some content".
There is also a link to attempt to continue. Clicking that causes a "crash" similar to that observed with FireFox.

The VFD-300 has only 500Meg of RAM so these operations may be stressing it.

---

The above is with the full version of my app.

I now repeat with the smaller test web app.
https://manukautech.github.io/xmrt
This time Google Chrome is stable and Firefox crashes.

---

Following the hypothesis that this is about the phone running out of memory,
I think that Google built into the phone software design may have give Chrome an advantage in memory use.
For this trial I have done "Force Stop" and "Disable" to all Google-related apps that have these options.
This makes no difference.
Firefox "crashes" even when running 'normally' rather than 'pinned'.

---

I do not have much left in the way of possible tests.
The change for this one is to turn off an app "Keep Screen On" that I have been using.
I now run Firefox and visit the small test web app.
https://manukautech.github.io/xmrt
I touch the screen in blank whitespace every few seconds to keep it visible.
After about 1 minute the "crash" happens.
The screen goes black for about 1 second then the phone shows the home screen.
Using the "Overview" button - I hope I have the name correct for the square icon on the lower right that shows me the running apps - Firefox shows as still running. I touch its top bar to activate and I see it reload the website.
"about:crashes" gives the same result as previously: "No crash reports have been submitted".

---

Just my opinion but I think the most significant result here is similar crashes for both Chrome and Firefox when running a high complexity version of my web app including the high demand JavaScript method "getUserMedia".
Maybe Firefox hits memory limits more than Chrome for this use case which stresses Firefox more on low memory devices?

---

This Firefox is a new install tonight from Google Play. ver 66.0.2

Comment 20

[Marcia Knous [:marcia]]

John - Is this issue still happening with the current release, Firefox for Android 67.0?

Comment 21

[BMO Automation]

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.