1299884 - determine what environment to run mochitest-media tests on

Status: NEW

(Testing :: General, defect, P3)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

right now for windows7 we run the 'mda' (mochitest-media) jobs for non-e10s on a VM and the e10s version on hardware.  Do we need to run these on hardware all the time?  Are there some tests which we can run on a VM?

I would like to find the right balance and shift things accordingly.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:jesup, we had a good discussion on IRC about this, I would like to follow up and change things (or at least get started on changing things) to meet the testing needs of the media suite.

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Joel Maher ( :jmaher) from comment #1)
> :jesup, we had a good discussion on IRC about this, I would like to follow
> up and change things (or at least get started on changing things) to meet
> the testing needs of the media suite.

Sure - I'd love to see what our options are, and how it affects the spurious intermittent rates

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

right now I see our options as:
1) AWS VM
2) AWS VM with dedicated GPU
3) HW machine in scl3 (what we ran on a month ago and currently do for e10s mda jobs)

if there are other needs not met by either of the 3 options, we should outline them and see how realistic it is to achieve it.

Comment 4

[Nils Ohlmeier [:drno]]

If you say 'AWS VM' are these only the spot instances or are there different sizes we can require for the mochitest-media tests?

My concern here is that some of the mochitest-media tests are pretty heavy and the single CPU spot instances can easily cause intermittents because the machines would be too slow.
If that helps we could also annotate the mochitest-media tests in dom/media/tests/mochitest/mochitest.ini in some way to indicate which one have higher demands on the machine.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Regarding AWS VM- we currently have two types:
https://aws.amazon.com/ec2/instance-types/

standard (c3.2xl):
* 8 VCPU (Intel Xeon E5-2680 v2)
* 15 GB RAM

dedicated gpu (g2.2xl):
* 8 VCPU - Intel Xeon E5-2670
* NVIDIA GPUs, each with 1,536 CUDA cores and 4GB
* 15 GB RAM


that is for windows7, we have different instance types for linux that we use:

desktop-test (mochitest-plain/browser-chrome, originally all jobs):
* m1.medium
* 1 vCPU
* 3.5 GB RAM

desktop-large (mochitest-media!, and many other jobs)
* m3.large
* Intel Xeon E5-2670 v2
* 2 vCPU
* 7.5 GB RAM

desktop-xlarge (web-platform-tests, android emulators, a few other jobs):
* m3.xlarge
* Intel Xeon E5-2670 v2
* 4 vCPU
* 15 GB RAM


as you can see our VM solutions are multi core and plenty of RAM- possibly there are other needs or concerns.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I would like to come to a conclusion in this bug and get this resolved this month.  Right now we are rebuilding our windows 7/10 environments and if we determine the cloud is good, then lets focus our efforts there.  If we lose a lot of test coverage without real devices, then we need to call that out as a requirement and possibly restrict these tests to hardware.

:drno, can you help make the call here or ni others who could?

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

for example on windows [1], I see this in the log file:
TEST DEVICES: No test devices found (in media.{audio,video}_loopback_dev, using fake streams.

but on linux [2], I see this:
[task 2016-12-06T15:19:07.087037Z] 15:19:07     INFO - TEST DEVICES: Using media devices:
[task 2016-12-06T15:19:07.087923Z] 15:19:07     INFO - audio: Sine source at 440 Hz
[task 2016-12-06T15:19:07.087994Z] 15:19:07     INFO - video: Dummy video device (0x0000)


I do not know if there is a v4l2loopback equivalent for windows- right now are current windows 7 mochitest-media tests are run on hardware

[1] - https://public-artifacts.taskcluster.net/Skt-nmrSSPS1DJB8nqnhAA/0/public/logs/live_backing.log
[2] - https://public-artifacts.taskcluster.net/FhmTBCMGRRuuMy1M1NlyNQ/0/public/logs/live_backing.log

Comment 8

[Dan Minor [:dminor]]

Bug 1019102 is for implementing fake streams for Windows.

Comment 9

[Nils Ohlmeier [:drno]]

So from the WebRTC's perspective I think we are fine with running the tests on the standard (c3.2xl) VM without the GPU.

But since the media tests also include tests outside of WebRTC I think Anthony should sign this off.

Comment 10

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

We will loose some coverage without GPUs but I'm OK with that. I'm going to punt this to Chris to see if he is has any concerns.

Comment 11

[Chris Pearce [:cpearce (Not reading bugmail)]]

I do not think it's a good idea to lose test coverage of hardware accelerated video decoding. Playing back video well is a key browser feature, and hardware acceleration is a key way we make it nice. We want regression tests to ensure we don't regress that.

So either an AWS VM with dedicated GPU (assuming our code actually uses hardware acceleration for video decoding in this environment, we need to verify that, and we should have add a test that explicitly ensures it's actually used!) or physical hardware is desirable.

I'd guess of those two, the AWS VM with GPU is the easiest option.

We could split the media playback tests out from the WebRTC tests if that's desirable, to reduce the financial cost, as I assume a VM with a GPU costs more than without. I have often thought it would be nice to split the WebRTC tests from the Playback tests anyway, since they have different error profiles.