1230079 - b-m permafail on m-i since bug 1211489 landed

Status: RESOLVED FIXED

(Core :: Audio/Video, defect)

Description

[Nigel Babu [:nigelb]]

Media tests have been perma-failing since this landed, I've retriggered tests in a job that was previously green to confirm this theory.

Links to failure: https://treeherder.mozilla.org/logviewer.html#?job_id=18157425&repo=mozilla-inbound

Comment 1

[Andreas Tolfsen ❲:ato❳]

It looks to me that the firefox-media-tests repository that is checked out is pinned to an old revision:

    09:54:34     INFO -  'firefox_media_rev': 'b11d6c3d7f6af166be314d2ac6673e78c1edb566',

which defines the marionette-driver dependency to be 0.13:

    https://github.com/mjzffr/firefox-media-tests/blob/b11d6c3d7f6af166be314d2ac6673e78c1edb566/setup.py

Because I did not know the b-m job was backed by Marionette, it was not included in my initial try run.  But it looks to me that this bustage can be resolved rather easily by updating the firefox_media_rev SHA to one that includes marionette-driver == 1.1.1.

Comment 2

[Andreas Tolfsen ❲:ato❳]

The references are defined here:

https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/mediatests/buildbot_windows_config.py#58
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/mediatests/buildbot_posix_config.py#44

I notice that firefox-ui-tests are also pulled in and installed.  The logs seem to indicate that they compete with firefox-media-tests in which Python package versions are installed, and that whichever of the two is installed last decides on the ultimate versions.

A couple of observations:

1. The b-m job depends on Github availability.

2. Since the tests live out-of-tree, they do not automatically pick up the in-tree Marionette client.

3. It is impossible to test if a change to Marionette breaks the b-m tests because of a circular dependency: They depend on a pinned version of Marionette, but because marionette-driver lives in-tree a new version can only be released by a commit to central, and since firefox-media-tests is pinned there is bound to be a constant discrepancy between the two rendering it impossible to apply a protocol upgrade like this cleanly.

Comment 3

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

My bad, I forgot these jobs were fetching a revision with too-old Marionette dependencies. I'll update the revision now. 

Fortunately, the 'fetching-from-Github' is a temporary measure until all the tests move in-tree, which is happening soon in bug 1197234.

Comment 4

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

CC'ing Syd Polk, who's transitioning toward being the main contact for media-tests this quarter.

Comment 5

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7464aa372d19

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/7464aa372d19