1352050 - MSE mochitests are disabled on Android

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, task, P3)

Description

[Jean-Yves Avenard [:jya]]

MSE mochitests are disabled on all Android versions just because they can't work with API 15.

They should be enabled for API >= 16.

Having those mochitests enabled would have prevented a fair amount of regressions over time.

Comment 1

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1352050: P1. Enable all MSE mochitests on API >= 16.

They should have never been disabled to start with in such a broad manner.

Review commit: https://reviewboard.mozilla.org/r/125404/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125404/

Comment 2

[Bob Clary [:bc] (inactive)]

Comment on attachment 8853317 [details]
Bug 1352050: P1. Enable all MSE mochitests on API >= 16.

https://reviewboard.mozilla.org/r/125404/#review127970

::: dom/media/mediasource/test/mochitest.ini:50
(Diff revision 1)
>    flac/IS.mp4 flac/IS.mp4^headers^ flac/00001.m4s flac/00001.m4s^headers^
>    flac/00002.m4s flac/00002.m4s^headers^ flac/00003.m4s flac/00003.m4s^headers^
>    avc3/init.mp4 avc3/init.mp4^headers^ avc3/segment1.m4s avc3/segment1.m4s^headers^
>  
>  [test_AVC3_mp4.html]
> -skip-if = ((os == "win" && os_version == "5.1") || (toolkit == 'android')) # Not supported on xp and android 2.3
> +skip-if = ((os == "win" && os_version == "5.1") || (toolkit == 'android' && android_version <= '15')) # Not supported on xp and android 2.3

We don't support anything below Android 2.3/sdk 15. If you want to skip Android 2.3/sdk 15 just use:

android_version == '15'

Comment 3

[Bob Clary [:bc] (inactive)]

Actually, since you will still want these disabled on the emulators, the test should be:

android_version == '15' || android_version == '18'

Autophone's devices currently cover '17', '19', '21' (limited), '22' (limited), 23', and '25'.

Comment 4

[Jean-Yves Avenard [:jya]]

Well. Technically we support only if >= 16. It felt clearer for me to use that test, rather than be based on the assumption that our tree herder are using a given version known in advance.  

But all emulators are 18 or 15?

Comment 5

[Bob Clary [:bc] (inactive)]

In terms of which devices we can run on, 15 is the lowest. I guess media's minimum is different. The emulators are all 18 though gbrown has been thinking about improving that.

gbrown: We currently distinguish between the emulators and the real devices by the sdk level. What are your plans for adding emulators with newer sdks? If you add newer emulators some day how will we continue to distinguish them?

Comment 6

[Geoff Brown [:gbrown]]

The emulator tests that show on treeherder as "Android 4.3 API15+ opt/debug" are sdk level 18. If a test needs to be skipped because of failures in that environment, we try to use "android_version == 18" (but sometimes we get lazy).

The emulator tests that show on treeherder as "Android 4.2 x86 opt" are sdk level 17. Note that we only run a limited set of tests there: xpcshell and some mochitest-chrome. If a test needs to be skipped because of failures in that environment, we try to use "os == 'android' && processor == 'x86'". "android_version == 17" would work as well, of course.

I don't have specific plans for adding emulator tests with newer sdks. I think it would be a good idea to run on 6.0 or 7.0, but the mobile team hasn't requested it and I have other projects requiring attention.

I am a little concerned about distinguishing "this test fails on Android x.x on the emulator" from "this test fails on Android x.x on a phone (on Autophone?)" or just "this test fails on Android x.x". There may be other factors too: Are we running the emulator on aws or elsewhere, is KVM or other acceleration available, etc. 

At one time I thought we should have some mechanism to identify the test environment. Perhaps an environment variable, MOZ_TEST_ENVIRONMENT = "Android 5.0 armv7 emulator on aws" and then we could:

  skip-if = test_env == "Android 5.0 armv7 emulator on aws"

or get fancy and say things like

  skip-if = test_env == "Android * * emulator on *"

But we have been okay without this. And I hate to think of how complicated our manifests might look if we tried to track problematic tests at that level.

So my tentative "plan" is to add emulator tests with newer sdks when time allows or conditions dictate, and when that happens, use "android_version == xx" to skip whatever new tests need to be skipped for that environment. If we find tests that have different results on Autophone vs emulator for the same sdk level, we should look at those cases carefully, but at the end of the day, I think we need to skip tests if they frequently fail in either test environment (post best efforts at investigation, do what we need to do to get consistent results).

Comment 7

[Jean-Yves Avenard [:jya]]

I hope you won't add new emulators of any kinds. They serve no purpose as they behave so differently to real hardware. They are so slow that they are unable to play any media in real time: Causing all the media tests or webrtc tests to fail.


If there's no way to distinguish the emulator from the real hardware, then all tests will have to be disabled regardless of the platform and that would be very disappointing (and that's how things are right now)

At the very least, we must be able to distinguish an emulator from the autophone.

Comment 8

[Geoff Brown [:gbrown]]

(In reply to Jean-Yves Avenard [:jya] from comment #7)
> I hope you won't add new emulators of any kinds. They serve no purpose as
> they behave so differently to real hardware. They are so slow that they are
> unable to play any media in real time: Causing all the media tests or webrtc
> tests to fail.

I understand that media tests rarely work on our emulators and I've put a lot of effort into alerting people to that limitation. This situation is frustrating and it is "dangerous" in that people either don't know or forget that certain tests are skipped. We should be able to mitigate the limitation by using Autophone to run tests that don't run on the emulator.

Lots of other tests work just fine on emulators and we use those tests to catch regressions every day. The emulator + aws combination scales, is easy to maintain, and is cost effective.

The current armv7/Android 4.3 emulator has two particular performance limitations - a hard limit on memory and poor multi-processor support - which may be addressed by newer emulators running newer Android versions.

> If there's no way to distinguish the emulator from the real hardware, then
> all tests will have to be disabled regardless of the platform and that would
> be very disappointing (and that's how things are right now)

It's not "all tests", but media tests, right? It's not regardless of platform: skip-if = android_version == '18' will effectively skip the test on the emulators and allow the tests to run on all devices that are not sdk 18. That solution is not ideal because we lose coverage on real devices running that android version, but it has been good enough for all our other tests for years now.

> At the very least, we must be able to distinguish an emulator from the
> autophone.

If that is needed, then we need to invent a way to do that. The mechanism used to determine "android_version" could be modified to get some other information from the device and add that to mozinfo. Or we could feed an environment variable into mozinfo, or ... something else. This might be a good unit of work to split off into a separate bug.

Comment 9

[Blake Wu [:bwu][:blakewu]]

Comment on attachment 8853317 [details]
Bug 1352050: P1. Enable all MSE mochitests on API >= 16.

https://reviewboard.mozilla.org/r/125404/#review129450

Per comment 5 to 8, it looks like we need to first figure out which SDK version we should enable it on. IIUC, it would be better to enable media test cases on autophone only, like comment 3.

Comment 10

[Jean-Yves Avenard [:jya]]

John, I enabled all the MSE mochitests on Android, and most of them failed:

https://treeherder.mozilla.org/logviewer.html#?job_id=198371839&repo=try&lineNumber=2001

I can understand why they would play smoothly, but I'm confused on why it wouldn't even fire loadeddata

any ideas?

Comment 11

[Jean-Yves Avenard [:jya]]

it works locally in the arm emulator...

Comment 12

[John Lin [:jhlin][:jolin]]

The specific failure in comment 10 can be traced back to an error returned by sw decoder:

09-10 01:51:35.771   911   926 E SoftAVC : Decoder failed: -2
09-10 01:51:35.781   911   925 E ACodec  : [OMX.google.h264.decoder] ERROR(0x80001001)
09-10 01:51:35.781   911   925 E MediaCodec: Codec reported an error. (omx error 0x80001001, internalError -2147483648)

What API version does your local emulator use? If it is API 23+, then it's possible that the change Google made in [1] added in-band SPS/PPS support. (Test servers use API 18, AFAICT.)

As for other test failures, I couldn't find any obvious decoder error in the log and need to try reproducing them locally for further investigation.

[1] https://android.googlesource.com/platform/frameworks/av/+/e5b9b3db87a55d3430f881ba59ba95f7d6ae41d0

Comment 13

[Alastor Wu [:alwu]]

As now we only run tests on Android 7.0, I tried to enable all of them on Android again. Let's see if they work well.
https://treeherder.mozilla.org/jobs?repo=try&revision=4e95f63297d35d4f7b8fda1a47827bb1db661c48

Comment 14

[Alastor Wu [:alwu]]

Attachment: Bug 1352050 - re-enable MSE tests on Android.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/549deb422138
re-enable MSE tests on Android. r=bryce

Comment 16

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/549deb422138