1247027 - DOM Media tests do not consistently handle errors

Status: NEW

(Core :: Audio/Video: Playback, defect, P3)

Description

[Bob Clary [:bc] (inactive)]

Attachment: videoplayback-webm-enabled.log

The DOM Media tests do not provide a fallback in the event that the device running the test can not load the media being tested. Since these are asynchronous tests, this results in the test not calling SimpleTest.finish() which causes the test to time out.

On Desktop, this is less of an issue since a) the media types are generally supported and b) the timeout value is relatively short. I believe 300 seconds is the current timeout for desktop.

On Android, this is a much bigger problem since the timeout is set to 5400 seconds (90 minutes): https://dxr.mozilla.org/mozilla-central/source/build/mobile/remoteautomation.py#289

This extra long timeout prevents Autophone from enabling the tests on any device which raises an error during a test which prevents SimpleTest.finish() from finishing.

There are two issues here: 1) remoteautomation's use of such a long timeout and 2) asynchronous mochitest tests failing to ensure that SimpleTest.finish() is called regardless of errors.

This problem of not ensuring SimpleTest.finish() is called is probably not limited to the DOM Media tests but the scope of this bug is limited to them.

Attaching normal desktop test output for dom/media/test/test_VideoPlaybackQuality.html with webm enabled.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: videoplayback-webm-disabled.log

Disabled webm via prefs:
mach mochitest --setpref media.encoder.webm.enabled=false --setpref media.mediasource.webm.audio.enabled=false --setpref media.mediasource.webm.enabled=false --setpref media.webm.enabled=false --keep-open false dom/media/test/test_VideoPlaybackQuality.html

Comment 2

[Bob Clary [:bc] (inactive)]

Attachment: mdm-content-errors.patch

I started working on a patch before realizing the number of tests I would have to update. Attaching here as an example of the types of things that need to be done.

Comment 3

[Geoff Brown [:gbrown]]

(In reply to Bob Clary [:bc:] from comment #0)
> On Android, this is a much bigger problem since the timeout is set to 5400
> seconds (90 minutes):
> https://dxr.mozilla.org/mozilla-central/source/build/mobile/remoteautomation.
> py#289

That timeout is the longest that the harness will wait for a job to complete. Some jobs currently take longer than 75 minutes, so there is little room for reduction. I would like to split some jobs (more chunks) to reduce run time (and enable a reduction in the timeout), but I think buildbot is still unable to run many more jobs.

There is also a no-output timeout, which should be 330 seconds for Android mochitests: If a test is hung, waiting for SimpleTest.finish(), I would expect no new output and the harness should fail the job after 330 seconds.

Comment 4

[Jean-Yves Avenard [:jya]]

Id be more interested in knowing why those tests are failing in the first place and what capabilities is missing in Android. 

AFAIK, there's nothing on android that isn't supported and features should be identical to desktops.

Comment 5

[Jean-Yves Avenard [:jya]]

Disabling webm for the test is silly. We've even disabled the compilation option to disable webm. All devices support webm and they can't not support it. As such, you're now testing something that just can't happen on a real device. 

So what's the point?

Comment 6

[Bob Clary [:bc] (inactive)]

I disabled webm as an example that the test would time out since it could not complete loading the media files rather than run to completion and report errors. I didn't show it as an example of the way it should be expected to run.

I have seen examples of devices which did not load the media. If I recall correctly, at least some of these were AOSP builds. Other may have been older devices. I doubt "All devices support webm and they can't not support it." is a absolutely true statement.

The point is I have seen these problems for real and they prevent me from running the dom media tests on Android.

Comment 7

[Bob Clary [:bc] (inactive)]

The other issue is that the tests generally do not handle errors at all and if any error causes the script to fail to complete and run SimpleTest.finish() then the test will time out. On Android this is currently 90 minutes which is not supportable with the number of devices at our disposal.

Comment 8

[Jean-Yves Avenard [:jya]]

(In reply to Bob Clary [:bc:] from comment #6)

> I have seen examples of devices which did not load the media. If I recall
> correctly, at least some of these were AOSP builds. Other may have been
> older devices. I doubt "All devices support webm and they can't not support
> it." is a absolutely true statement.
> 

It is..

We ship our own software decoder for webm. A device can not, not support webm.

> The point is I have seen these problems for real and they prevent me from
> running the dom media tests on Android.
Not for the reasons you stated. And I would much prefer to fix the actual cause of the error than simply make the tests get around the errors more easily.

Or at least create a blocking bug tracking the error you're seeing.

Comment 9

[Bob Clary [:bc] (inactive)]

I am not talking about ignoring the errors. I would expect that we would report any such errors as failures which can be diagnosed and fixed. What I am talking about is not failing to terminate a test due a failure. As an example of how that might be useful is if I were able to run these tests regularly we would already know what the failures were.

Mark this wontfix if you like.

Comment 10

[Jean-Yves Avenard [:jya]]

Generally, all the mochitests only finish if they succeed. We treat timeout as failure.

Note that we have most media mochitest enabled on try and treeherder runs. If they were timing out under normal conditions, it would be known.

Comment 11

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #3)
> There is also a no-output timeout, which should be 330 seconds for Android
> mochitests: If a test is hung, waiting for SimpleTest.finish(), I would
> expect no new output and the harness should fail the job after 330 seconds.

Maybe the no-output timeout is disabled somehow. I see, for example,

https://autophone.s3.amazonaws.com/pub/mobile/try-builds/esawin%40mozilla.com-3e127d0647362152e5206f936e4c98b437d8a985/try-android-api-15/mochitest-dom-media-mochitests-dom-media-settings.ini-1-nexus-6-2-a54f067b-9604-4204-8036-a10b58dba024.log

0 INFO SimpleTest START
1 INFO TEST-START | dom/media/test/test_VideoPlaybackQuality.html
file=[xpconnect wrapped nsILocalFile]
file=[xpconnect wrapped nsILocalFile]
file=[xpconnect wrapped nsILocalFile]
file=[xpconnect wrapped nsILocalFile]
file=[xpconnect wrapped nsILocalFile]
file=[xpconnect wrapped nsILocalFile]
file=[xpconnect wrapped nsILocalFile]
2 INFO TEST-PASS | dom/media/test/test_VideoPlaybackQuality.html | getVideoPlaybackQuality should be exposed with pref set 
3 INFO TEST-PASS | dom/media/test/test_VideoPlaybackQuality.html | getVideoPlaybackQuality should return an object 
4 INFO TEST-PASS | dom/media/test/test_VideoPlaybackQuality.html | creationTime should be in the past 
5 INFO TEST-PASS | dom/media/test/test_VideoPlaybackQuality.html | totalVideoFrames should be 0 
6 INFO TEST-PASS | dom/media/test/test_VideoPlaybackQuality.html | droppedVideoFrames should be 0 
7 INFO TEST-PASS | dom/media/test/test_VideoPlaybackQuality.html | corruptedVideoFrames should be 0 
8 INFO TEST-PASS | dom/media/test/test_VideoPlaybackQuality.html | getVideoPlaybackQuality should return a new object 
9 INFO TEST-PASS | dom/media/test/test_VideoPlaybackQuality.html | VideoPlaybackQuality objects should have increasing creationTime 
10 INFO TEST-PASS | dom/media/test/test_VideoPlaybackQuality.html | getVideoPlaybackQuality should not be available on Audio elements 
11 INFO TEST-UNEXPECTED-FAIL | dom/media/test/test_VideoPlaybackQuality.html | Test timed out. 
    reportError@SimpleTest/TestRunner.js:114:7
    TestRunner._checkForHangs@SimpleTest/TestRunner.js:134:7
12 INFO TEST-OK | dom/media/test/test_VideoPlaybackQuality.html | took 4801301ms

Comment 12

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Mass change P2 -> P3