Media tests have been perma-failing since this landed, I've retriggered tests in a job that was previously green to confirm this theory. Links to failure: https://treeherder.mozilla.org/logviewer.html#?job_id=18157425&repo=mozilla-inbound
It looks to me that the firefox-media-tests repository that is checked out is pinned to an old revision: 09:54:34 INFO - 'firefox_media_rev': 'b11d6c3d7f6af166be314d2ac6673e78c1edb566', which defines the marionette-driver dependency to be 0.13: https://github.com/mjzffr/firefox-media-tests/blob/b11d6c3d7f6af166be314d2ac6673e78c1edb566/setup.py Because I did not know the b-m job was backed by Marionette, it was not included in my initial try run. But it looks to me that this bustage can be resolved rather easily by updating the firefox_media_rev SHA to one that includes marionette-driver == 1.1.1.
The references are defined here: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/mediatests/buildbot_windows_config.py#58 https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/mediatests/buildbot_posix_config.py#44 I notice that firefox-ui-tests are also pulled in and installed. The logs seem to indicate that they compete with firefox-media-tests in which Python package versions are installed, and that whichever of the two is installed last decides on the ultimate versions. A couple of observations: 1. The b-m job depends on Github availability. 2. Since the tests live out-of-tree, they do not automatically pick up the in-tree Marionette client. 3. It is impossible to test if a change to Marionette breaks the b-m tests because of a circular dependency: They depend on a pinned version of Marionette, but because marionette-driver lives in-tree a new version can only be released by a commit to central, and since firefox-media-tests is pinned there is bound to be a constant discrepancy between the two rendering it impossible to apply a protocol upgrade like this cleanly.
My bad, I forgot these jobs were fetching a revision with too-old Marionette dependencies. I'll update the revision now. Fortunately, the 'fetching-from-Github' is a temporary measure until all the tests move in-tree, which is happening soon in bug 1197234.
CC'ing Syd Polk, who's transitioning toward being the main contact for media-tests this quarter.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox45: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla45
You need to log in before you can comment on or make changes to this bug.