1151439 - Autophone - enable Mochitest Media, Reftest ogg and webm video tests

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

Bug 1138942 added a new autophone-media (Mm) manifest and k17e asked for reftest video tests as well. I found the ogg-video (Rov) and webm-video (Rwv) reftests manifests, which I assume k17e was referring.

I ran tests using my local devices (nexus one, galaxy s3) on treeherder staging with the old mochitests I have been running as well as the new ones:

https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=644662d12dfa&exclusion_profile=false&filter-searchStr=autophone

https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=f8b8ad4abc7d&exclusion_profile=false&filter-searchStr=autophone

As you can see they are pretty orange on my local devices. The orange is due to unexpected failures, unexpected passes, and some crashes.

I would like to try to know if there is a way to get these to run green other than skipping the individual failing tests. For example one of the nexus ones is green on Rov but the other two are not but I don't know why.

Let me know if the the ogg-video and webm-video tests were what was desired and if the job symbols (Mm, Rov, Rwv) are ok and I will file a bug to add these to try chooser.

Comment 1

[Bob Clary [:bc] (inactive)]

k17e: Are ogg-video, webm-video what you wanted?

gbrown, snorp, k17e: Any ideas on making these green?

Comment 2

[Geoff Brown [:gbrown]]

Sorry, I can't offer much insight here.

Comment 3

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

We probably want something simpler for the device specific reftests.

Comment 4

[Bob Clary [:bc] (inactive)]

I'm calling this fixed with the configuration https://github.com/mozilla/autophone/blob/master/tests/production-manifest.ini 

We can revisit when we have more concrete details and more capacity.