1352333 - Autophone - add complete dom/media tests

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, enhancement)

Description

[Jean-Yves Avenard [:jya]]

As far as I can tell, none of the dom/media/mediasource/test mochitests are being run.

Here is a log where autophone tests are run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=da2bee1a39dfdc1fdfda2981b68982bc2ef980b0&filter-tier=1&filter-tier=2&filter-tier=3

logs for Mdm:
https://autophone.s3.amazonaws.com/v1/task/P5_AQvldThe4IPmDAVS8LQ/runs/0/artifacts/public/build/mochitest-dom-media-mochitests-dom-media-settings.ini-1-nexus-6p-11-70222d20-4664-4eec-931b-7d38508ae58d.log

logs for Mm:
https://autophone.s3.amazonaws.com/v1/task/P5_AQvldThe4IPmDAVS8LQ/runs/0/artifacts/public/build/mochitest-media-mochitests-media.ini-1-nexus-6p-11-52a1a5c2-2ef0-40b0-94d7-d05a690c2567.log

in there we can see the various dom/media/test being run.

But nothing about dom/media/mediasource

So apparently, we have no coverage of the mediasource code on android (they are disabled on android 4.3 simulator due to the slowness of the thing)

Comment 1

[Bob Clary [:bc] (inactive)]

The media tests which are currently defined for Autophone are:

Mdm: dom/media/test/mochitest.ini
Mm:  testing/mochitest/manifests/autophone-media.ini
Mw:  testing/mochitest/manifests/autophone-webrtc.ini
Cw:  testing/crashtest/autophone-crashtest-webrtc.list

We could add an additional test for dom/media/mediasource/test/mochitest.ini

Comment 2

[Bob Clary [:bc] (inactive)]

I looked into including the option of running all of the tests defined under dom/media. This test run includes an example for all of the mochitest manifests.

https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=891981e67948aaebf7a63bba5181ef0a538ce163&filter-searchStr=autophone

Mdm  dom/media/test/mochitest.ini
Mdm1 dom/media/tests/mochitest/mochitest.ini
Mdm2 dom/media/mediasource/test/mochitest.ini
Mdm3 dom/media/tests/mochitest/identity/mochitest.ini
Mdm4 dom/media/webaudio/test/mochitest.ini
Mdm5 dom/media/webaudio/test/blink/mochitest.ini
Mdm6 dom/media/webspeech/recognition/test/mochitest.ini
Mdm7 dom/media/webspeech/synth/test/mochitest.ini
Mdm8 dom/media/webspeech/synth/test/startup/mochitest.ini

It is still lacking the crash tests but that can be added pretty easily. I'm thinking of:

Cdm  dom/media/test/crashtests/crashtests.list
Cdm1 dom/media/tests/crashtests/crashtests.list
Cdm2 dom/media/mediasource/test/crashtests/crashtests.list
Cdm3 dom/media/webspeech/synth/crashtests/crashtests.list

dminor: We could eliminate the autophone only Cw and Mw webrtc tests if we ran the default manifests. What do you think? We'd have to make sure to annotate the default manifests with the appropriate skips, but I think that would be the more appropriate route. Is there an advantage of keeping Cw, Mw?

jya: This would fit your current request for mediasource. Any other thoughts about adding the additional tests?

snorp: Thoughts?

jmaher, gbrown: Thoughts on naming, etc?

Comment 3

[Dan Minor [:dminor]]

We run more tests on Autophone than we do on the emulators (at least the last time I checked), so we'd need to be able to distinguish between an Autophone run and an emulator run in the manifests. Assuming that's not a problem, I don't see any reason to keep separate manifests.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I don't like |Cdm Cdm1 Cdm2 Cdm3|, I would prefer if we have numbered jobs that all jobs with the same letters have a corresponding number, so |Cdm1 Cdm2 Cdm3 Cdm4|

Comment 5

[Bob Clary [:bc] (inactive)]

I did that to match the already existing Mdm job. Would you be ok with changing all of the Mdm and Cdm to numbered jobs as well? Or should we just leave Mdm alone?

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

change might be confusing, but leaves less questions.  Happy to take the route of least resistance :)

Comment 7

[Jean-Yves Avenard [:jya]]

So long that the content of what is being run doesn't change across run and between debug/release like it does for the web platform tests.

That made the wpt choice totally useless as you can't ever predict where one particular test will run, could be 1, 2, 3 or 5 etc

Otherwise, sounds good, though why mdm vs mdm1/mdm2 etc? why not number them all?

Comment 8

[Bob Clary [:bc] (inactive)]

I had existing bugs with Mdm etc. But I think I'll go ahead and bite the bullet and do them all.

Comment 9

[Bob Clary [:bc] (inactive)]

Morph this for full coverage of dom/media

Comment 10

[James Willcox (:snorp) ([e-mail]) (he/him)]

Great!

Comment 11

[Bob Clary [:bc] (inactive)]

Attachment: bug-1352333-autophone-v1.patch

Example runs of the full set of tests using patches for the test manifests in inbound/55, aurora/54, beta/54 and release/53. The autophone test manifests should only run the green only tests modulo the intermittent failures.

https://treeherder.allizom.org/#/jobs?repo=try&author=[e-mail]&fromchange=74120ddfd810582322c688881d9aded924baa28e&exclusion_profile=false&tochange=5877457eef7730567fb0a49fe62aad1cd92a0440

I'll attach the in tree manifest changes next.

Comment 12

[Bob Clary [:bc] (inactive)]

Attachment: bug-1352333-sync-autophone-webrtc-manifests-inbound.patch

This patch syncs the skips etc from the autophone webrtc tests into the normal manifests.

Try run look ok in that I don't seen any issues with the manifests.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=dbc3415780480c0c2065cf86f11d6a10ad55ed89

Comment 13

[Bob Clary [:bc] (inactive)]

Attachment: bug-1352333-remove-autophone-only-webrtc-media-inbound.patch

The autophone webrtc and media tests are no longer needed now that we have full coverage.

Comment 14

[Bob Clary [:bc] (inactive)]

Attachment: bug-1352333-sync-autophone-webrtc-manifests-beta.patch

beta version of the patch.

Comment 15

[Bob Clary [:bc] (inactive)]

Attachment: bug-1352333-remove-autophone-only-webrtc-media-beta.patch

beta version of the patch.

Comment 16

[Bob Clary [:bc] (inactive)]

Attachment: bug-1352333-sync-autophone-webrtc-manifests-release.patch

release version of the patch.

Comment 17

[Bob Clary [:bc] (inactive)]

Attachment: bug-1352333-remove-autophone-only-webrtc-media-release.patch

release version of the patch.

Comment 18

[Bob Clary [:bc] (inactive)]

Attachment: assignments-master.txt

current test assignments.

Comment 19

[Bob Clary [:bc] (inactive)]

Attachment: assignments-development.txt

assignments after.

Comment 20

[Bob Clary [:bc] (inactive)]

PS. I do have aurora versions of the patches (they are now identical to the beta versions), but I don't think they are necessary as aurora is doa.

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8860014 [details] [diff] [review]
bug-1352333-autophone-v1.patch

Review of attachment 8860014 [details] [diff] [review]:
-----------------------------------------------------------------

lots of rubber stamping here.   As I see Aurora coming out of the configs, do we get any real value from mozilla-release?  I know there is not much volume there, but it could save a little bit as well.

Comment 22

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8860017 [details] [diff] [review]
bug-1352333-sync-autophone-webrtc-manifests-inbound.patch

Review of attachment 8860017 [details] [diff] [review]:
-----------------------------------------------------------------

there is a TODO for a bug filed, please sort that out so we don't need to land more pushes to update a comment; likewise ensure a try push for the emulator runs.

Comment 23

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8860025 [details] [diff] [review]
bug-1352333-sync-autophone-webrtc-manifests-beta.patch

Review of attachment 8860025 [details] [diff] [review]:
-----------------------------------------------------------------

sort out the TODO item :)

Comment 24

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8860027 [details] [diff] [review]
bug-1352333-sync-autophone-webrtc-manifests-release.patch

Review of attachment 8860027 [details] [diff] [review]:
-----------------------------------------------------------------

and this has the same TODO message as the other branches manifest changes :)

Comment 25

[Dan Minor [:dminor]]

Comment on attachment 8860017 [details] [diff] [review]
bug-1352333-sync-autophone-webrtc-manifests-inbound.patch

Review of attachment 8860017 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 26

[Dan Minor [:dminor]]

Comment on attachment 8860021 [details] [diff] [review]
bug-1352333-remove-autophone-only-webrtc-media-inbound.patch

Review of attachment 8860021 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 27

[Dan Minor [:dminor]]

Comment on attachment 8860025 [details] [diff] [review]
bug-1352333-sync-autophone-webrtc-manifests-beta.patch

Review of attachment 8860025 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 28

[Dan Minor [:dminor]]

Comment on attachment 8860027 [details] [diff] [review]
bug-1352333-sync-autophone-webrtc-manifests-release.patch

Review of attachment 8860027 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 29

[Dan Minor [:dminor]]

Comment on attachment 8860028 [details] [diff] [review]
bug-1352333-remove-autophone-only-webrtc-media-release.patch

Review of attachment 8860028 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 30

[Bob Clary [:bc] (inactive)]

(In reply to Joel Maher ( :jmaher) from comment #21)
> Comment on attachment 8860014 [details] [diff] [review]
> bug-1352333-autophone-v1.patch
> 
> Review of attachment 8860014 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> lots of rubber stamping here.   As I see Aurora coming out of the configs,
> do we get any real value from mozilla-release?  I know there is not much
> volume there, but it could save a little bit as well.

I consider it a safe guard to not let anything get by us before it is released. I would hate to miss a regression.(In reply to Joel Maher ( :jmaher) from comment #22)
> Comment on attachment 8860017 [details] [diff] [review]
> bug-1352333-sync-autophone-webrtc-manifests-inbound.patch
> 
> Review of attachment 8860017 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> there is a TODO for a bug filed, please sort that out so we don't need to
> land more pushes to update a comment; likewise ensure a try push for the
> emulator runs.

Thanks. try run linked in comment 12.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=dbc3415780480c0c2065cf86f11d6a10ad55ed89&filter-searchStr=android

Is that not sufficient?

Comment 31

[Joel Maher ( :jmaher ) (UTC -8)]

missed that, all looks good!

Comment 32

[Bob Clary [:bc] (inactive)]

Filed Bug 1358271

Comment 33

[Bob Clary [:bc] (inactive)]

Attachment: bug-1352333-sync-autophone-webrtc-manifests-beta.patch

updated beta version of patch.

Comment 34

[Bob Clary [:bc] (inactive)]

Attachment: bug-1352333-sync-autophone-webrtc-manifests-release.patch

updated release version of patch.

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/68842e58a066
sync autophone webrtc test manifests with normal webrtc manifests, r=jmaher,dminor.
https://hg.mozilla.org/integration/mozilla-inbound/rev/0215b10785da
remove autophone webrtc test manifests, r=dminor.

Comment 36

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/b2c0b2fe0ff2405ddc469281dda993b0d93be7ee

Comment 37

[Bob Clary [:bc] (inactive)]

Attachment: https://github.com/mozilla-releng/services/pull/269

Update trychooser.

We have a bit of a race between landing this on inbound, merging to autoland and mozilla-central then uplifting to beta, mozila-release and landing the corresponding changes on Autophone.

We can probably live with the breakage if this goes live before everything is completed. I'll just deal with any try pushes that use the old syntax.

Comment 38

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8860174 [details] [review]
https://github.com/mozilla-releng/services/pull/269

I don't think I'm a great person to review that.  Reading through the bug I don't really know what's going on.

Github suggest garbas - I'm not sure he'd have an opinion either, but he can at least rubber-stamp.

Comment 39

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/68842e58a066
https://hg.mozilla.org/mozilla-central/rev/0215b10785da

Comment 40

[Bob Clary [:bc] (inactive)]

Tomcat, can you take care of uplifting the appropriate beta and release versions of the patches today? There shouldn't be any merge conflicts with these patches unlike the trunk versions.

I'm intentionally skipping aurora though the beta patch applies there.

Comment 41

[Carsten Book [:Tomcat]]

hey bc, seems this needs rebasing for beta. could you take a look ? thanks!

Comment 42

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-beta/rev/ca38cf3eba178b7d1a25a9e1e8ba72dd466fed2e
https://hg.mozilla.org/releases/mozilla-release/rev/8e81b5b7a28a5b91a78aed79a9362703b76921f8

Comment 43

[Carsten Book [:Tomcat]]

 https://hg.mozilla.org/releases/mozilla-beta/rev/7e0110d709641ac572bef90185052f37c9ea45c0
https://hg.mozilla.org/releases/mozilla-release/rev/d4686a554950979e0f34c5d4b86f2bfc04ae793b

Comment 44

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/b2c0b2fe0ff2405ddc469281dda993b0d93be7ee

I made two errors that required patches

https://github.com/mozilla/autophone/commit/ebec9c29cda6da6ec80c1db188aabf6b06301b17
Bug 1352333 - Autophone - fix duplicate devices in tests/production-autophone-2.ini, r=self

https://github.com/mozilla/autophone/commit/9703f5ed3f6ae73c796e8111269fdf83af208abe
Bug 1352333 - Autophone - mark nexus-6p-13 as a spare, r=self.

Comment 45

[Intermittent Failures Robot]

58 failures in 162 pushes (0.358 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 49
* mozilla-central: 9

Platform breakdown:
* android-7-1-armv8-api15: 22
* android-4-4-armv7-api15: 16
* android-6-0-armv8-api15: 13
* android-5-1-armv7-api15: 5
* android-5-0-armv8-api15: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1352333&startday=2017-04-21&endday=2017-04-21&tree=all

Comment 46

[Intermittent Failures Robot]

71 failures in 817 pushes (0.087 failures/push) were associated with this bug in the last 7 days. 

This is the #7 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-inbound: 62
* mozilla-central: 9

Platform breakdown:
* android-7-1-armv8-api15: 28
* android-6-0-armv8-api15: 18
* android-4-4-armv7-api15: 18
* android-5-1-armv7-api15: 5
* android-5-0-armv8-api15: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1352333&startday=2017-04-17&endday=2017-04-23&tree=all

Comment 47

[Bob Clary [:bc] (inactive)]

(In reply to OrangeFactor Robot from comment #46)
> 71 failures in 817 pushes (0.087 failures/push) were associated with this
> bug in the last 7 days. 

This high failure rate was due to the failure on my part of coordinating the changes to autophone and the various repositories. I pointed the failures here to track them but this should not reoccur.

Comment 48

[Bob Clary [:bc] (inactive)]

Comment on attachment 8860174 [details] [review]
https://github.com/mozilla-releng/services/pull/269

garbas merged this over the weekend.

Comment 49

[Bob Clary [:bc] (inactive)]

Attachment: https://github.com/mozilla-releng/services/pull/273

This is a follow up to fix a copy/paste error on my part in the trychooser.

Comment 50

[Bob Clary [:bc] (inactive)]

https://mozilla-releng.net/trychooser/ is live.

Thanks garbas and everyone!

Comment 51

[Intermittent Failures Robot]

6 failures in 883 pushes (0.007 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 6

Platform breakdown:
* android-5-1-armv7-api15: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1352333&startday=2017-04-24&endday=2017-04-30&tree=all