Closed Bug 1324932 Opened 5 years ago Closed 2 years ago

Intermittent dom/media/tests/mochitest/test_getUserMedia_mediaStreamClone.html | application timed out after 330/370 seconds with no output

Categories

(Core :: WebRTC: Audio/Video, defect, P4)

ARM
Android
defect

Tracking

()

RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell unknown])

Rank: 35
Component: Audio/Video → WebRTC: Audio/Video
Priority: -- → P3
I did some retriggers here:
https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=android%20debug%20mda3&tochange=af78a23cda9ce82be8b2a3fd0a6b9dc3a537d08c&fromchange=11a4e3ddcccacb51247f78bb53f0af886765dd91

it looks like this went to a perma fail status when bug 1366672 landed, although it showed up much less frequently before hand.

:walkingice, can you look at this almost perma failing test and fix it?
Depends on: 1394132
Flags: needinfo?(walkingice0204)
Whiteboard: [stockwell needswork]
No longer depends on: 1394132
My changes is for an Android widget(progress bar) which should not effect media stream at all. Ran the test on emulator, and it passed. Maybe I should understand the test to know which part cause this problem.
I think your new progress bar caused an increase in load, which on the slow VMs caused underruns in the audio thread, and so our test doesn't finish (we check for a certain audio sample pattern, and with an underrun a bunch of samples will be 0).

That, or timing, or both.
Mass change P3->P4 to align with new Mozilla triage process.
Priority: P3 → P4
Flags: needinfo?(walkingice0204)
Summary: Intermittent dom/media/tests/mochitest/test_getUserMedia_mediaStreamClone.html | application timed out after 330 seconds with no output → Intermittent dom/media/tests/mochitest/test_getUserMedia_mediaStreamClone.html | application timed out after 330/370 seconds with no output
Seeing this in the logging:

"Browser unexpectedly found running. Killing..."

Looks like a timeout to me, but for some reason the code in remoteautomation.py is not printing a TEST-UNEXPECTED-FAIL for the timeout itself.
There have been 31 failures in the past week, according to orange factor:

Ocurrences per plaform:

21 on android-api-16-gradle
10 on android-4-3-armv7-api16

This is failing only on the opt build type.

Here a relevant log file and a snippet with the failure:
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=155265409&lineNumber=3531

[task 2018-01-10T08:00:26.055Z] 08:00:26     INFO -  TEST-INFO | screentopng: exit 0
3528
[task 2018-01-10T08:00:29.856Z] 08:00:29     INFO -  org.mozilla.fennec_aurora still alive after SIGABRT: waiting...
3529
[task 2018-01-10T08:00:35.078Z] 08:00:35     INFO -  org.mozilla.fennec_aurora still alive after SIGABRT: waiting...
3530
[task 2018-01-10T08:00:40.302Z] 08:00:40     INFO -  org.mozilla.fennec_aurora still alive after SIGABRT: waiting...
3531
[task 2018-01-10T08:00:45.886Z] 08:00:45  WARNING -  TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_getUserMedia_mediaStreamClone.html | application timed out after 370 seconds with no output
3532
[task 2018-01-10T08:00:45.886Z] 08:00:45     INFO -  INFO | automation.py | Application ran for: 0:18:07.840265
Flags: needinfo?(jib)
Whiteboard: [stockwell unknown] → [stockwell needswork]
Comment 19 is still our best guess here (overloaded VM). This test is already disabled in debug [1].

Are there any prefs we can use to reduce the load on android (like turning off the progress bar?)

Andreas is on PTO, leaving another needinfo to not drop it.

[1] https://searchfox.org/mozilla-central/rev/88439dc6c5b02e167032374071cdc697a056efa1/dom/media/tests/mochitest/mochitest.ini#78
Flags: needinfo?(jib) → needinfo?(apehrson)
OS: Unspecified → Android
Hardware: Unspecified → ARM
(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #37)
> Comment 19 is still our best guess here (overloaded VM). This test is
> already disabled in debug [1].
> 
> Are there any prefs we can use to reduce the load on android (like turning
> off the progress bar?)

walkingice are you aware of way to disable animations while running tests?
Flags: needinfo?(walkingice0204)
Currently I am not working on Fennec so not have much time on this. All-hands is around the corner, if anyone could guide me how to run the test on my laptop, I am willing to make a patch to disable animated-progress bar while running test.
(In reply to Julian Chu [:walkingice] from comment #50)
> Currently I am not working on Fennec so not have much time on this.
> All-hands is around the corner, if anyone could guide me how to run the test
> on my laptop, I am willing to make a patch to disable animated-progress bar
> while running test.

Hey Juian,
To run mochitests you can follow - https://wiki.mozilla.org/Mobile/Fennec/Android/Testing#mochitest_.28plain_and_chrome.29
Basically, what I've done is:
- mach build && mach package && mach install
- have an emulator started (with Google APIs, not Google Play - needed for root)
- "./mach mochitest dom/media/tests/mochitest/test_getUserMedia_mediaStreamClone.html" to run this specific test

A recent push to try of mine also failed because of this but when I tried to test it locally it went fine.
According to Andreas
> I think your new progress bar caused an increase in load, which on the slow VMs caused underruns in the audio thread
So maybe you could try running the test on an emulator with very low heap and ram size in the hope of catching it fail.
Also you could push to try for this test alone with "mach try -b o -p android-x86,android-api-16 -u mochitest-media-3 -t none --artifact"

This was disabled in bug 1282262. Not very actionable atm, though this type of entire-process timeout is more severe than the simpler test timeout in bug 1282262. I'd still go with the hunch that the emulator being slow was behind this, unless we see more evidence of something else.

Flags: needinfo?(apehrson)

Do we still care about emulator tests on arm, now that we have pixels and x86 emulator? Is the test solid there?

I don't think we should care, but I also don't know how much of those tests are running well on real phones and fast emulators.

I think there are higher priorities.

Status: NEW → RESOLVED
Closed: 2 years ago
Flags: needinfo?(walkingice0204)
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.