b-m permafail on m-i since bug 1211489 landed

RESOLVED FIXED in Firefox 45

Status

()

Core
Audio/Video
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: nigelb, Unassigned)

Tracking

unspecified
mozilla45
Points:
---

Firefox Tracking Flags

(firefox45 fixed)

Details

(URL)

(Reporter)

Description

2 years ago
Media tests have been perma-failing since this landed, I've retriggered tests in a job that was previously green to confirm this theory.

Links to failure: https://treeherder.mozilla.org/logviewer.html#?job_id=18157425&repo=mozilla-inbound
Depends on: 1211489
It looks to me that the firefox-media-tests repository that is checked out is pinned to an old revision:

    09:54:34     INFO -  'firefox_media_rev': 'b11d6c3d7f6af166be314d2ac6673e78c1edb566',

which defines the marionette-driver dependency to be 0.13:

    https://github.com/mjzffr/firefox-media-tests/blob/b11d6c3d7f6af166be314d2ac6673e78c1edb566/setup.py

Because I did not know the b-m job was backed by Marionette, it was not included in my initial try run.  But it looks to me that this bustage can be resolved rather easily by updating the firefox_media_rev SHA to one that includes marionette-driver == 1.1.1.
Flags: needinfo?(mjzffr)
The references are defined here:

https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/mediatests/buildbot_windows_config.py#58
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/mediatests/buildbot_posix_config.py#44

I notice that firefox-ui-tests are also pulled in and installed.  The logs seem to indicate that they compete with firefox-media-tests in which Python package versions are installed, and that whichever of the two is installed last decides on the ultimate versions.

A couple of observations:

1. The b-m job depends on Github availability.

2. Since the tests live out-of-tree, they do not automatically pick up the in-tree Marionette client.

3. It is impossible to test if a change to Marionette breaks the b-m tests because of a circular dependency: They depend on a pinned version of Marionette, but because marionette-driver lives in-tree a new version can only be released by a commit to central, and since firefox-media-tests is pinned there is bound to be a constant discrepancy between the two rendering it impossible to apply a protocol upgrade like this cleanly.
My bad, I forgot these jobs were fetching a revision with too-old Marionette dependencies. I'll update the revision now. 

Fortunately, the 'fetching-from-Github' is a temporary measure until all the tests move in-tree, which is happening soon in bug 1197234.
Flags: needinfo?(mjzffr)
CC'ing Syd Polk, who's transitioning toward being the main contact for media-tests this quarter.

Updated

2 years ago
Duplicate of this bug: 1230242

Comment 7

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/7464aa372d19
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox45: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla45
You need to log in before you can comment on or make changes to this bug.