Closed Bug 1163131 Opened 10 years ago Closed 10 years ago

Get qualifying subset of Platform QA media automation (MSE/EME/WebRTC Steeplechase) reporting to Treeherder

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: impossibus, Assigned: impossibus)

References

Details

Attachments

(1 file)

PR for reporting WebRTC Steeplechase jobs
58 bytes, text/x-github-pull-request
sydpolk
: review+
Details | Review
There is a collection of media test jobs deployed or soon-to-be-deployed on "pf-jenkins" [1] and we want to have them report to Treeherder. Many moons ago, Syd started working on this for WebRTC jobs so we already have: - Staging credentials for Treeherder - A script to submit WebRTC test results to Treeherder [1]; tested on Treeherder staging. - S3 buckets for storing pf-jenkins logs The MSE/EME jobs [2] work a little differently than the WebRTC jobs: they are run by a mozharness script triggered on pf-jenkins. [1] http://pf-jenkins.qa.mtv2.mozilla.com:8080 [2] https://github.com/sydvicious/mozplatformqa-jenkins/blob/master/jenkinsherder.py [3] https://github.com/mjzffr/firefox-media-tests/tree/pf-jenkin
Two pf-jenkins jobs (on Mac 10.10, Win 7) are submitting to staging. After a few tweaks, latest iterations look good. https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=4700d1cdf489&filter-job_group_symbol=VP
And now the log is parsed so the Failure Summary shows up properly. I don't keep track of line numbers in the log, though. https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=e10e2e8d8bf2&filter-job_group_symbol=VP
All pf-jenkins jobs for firefox-media-tests (MSE/EME) now report to Treeherder staging (mozilla-central, mozilla-aurora, mozilla-beta), using job group symbol "VP" (as in "Video Puppeteer"). Should have pointed to these earlier: https://github.com/mjzffr/firefox-media-tests/commit/b79dc1001468574aaa7d168177671a08a4a2b046 https://github.com/mjzffr/firefox-media-tests/commit/d1989812dda449322813c55c4f57d29d30e1ebbd https://github.com/mjzffr/firefox-media-tests/commit/1fe8d725aee306a0eec126702f6531604fc0d2c1
Hi Ryan, do you have any feedback on the following job/group symbols to be used on Treeherder? (Tier 2 jobs.) MSE/EME video tests, which use Marionette - Group: VideoPuppeteer (VP) - Jobs: -- 'm' for 'MSE Video Playback' -- 'e' for 'EME Video Playback' We could make these more granular - e.g. indicate whether they're short or long. (ms, ml, es, el?) WebRTC tests, which run on pairs of machines, using Steeplechase. - Group: Paired WebRTC Steeplechase Tests (PW) - Jobs: --'p' for 'WebRTC pair' --'t' for 'WebRTC TURN pair' IRC discussion with sydpolk, drno, jgriffin concluded that each two-machine test run will be reported as (up to) two Treeherder jobs, one for each machine platform.
Flags: needinfo?(ryanvm)
As noticed on IRC right now, we have an overlap with implementing such a feature for firefox-ui-tests based tests. Please also see bug 1166683 for the changes which will directly land in the above harness. Maybe you can benefit from those and only have to adjust settings. Shall we mark the dependency?
(In reply to Henrik Skupin (:whimboo) from comment #6) > Shall we mark the dependency? The work for this bug also applies to WebRTC Steeplechase tests, so I'm not sure a dependency is appropriate. The Treeherder-specific code here can probably be reused in firefox-ui-tests, especially treeherding.py and s3.py, which are based on the implementation in Autophone. See Comment 4.
(In reply to Maja Frydrychowicz (:maja_zf) from comment #7) > sure a dependency is appropriate. The Treeherder-specific code here can > probably be reused in firefox-ui-tests, especially treeherding.py and s3.py, > which are based on the implementation in Autophone. See Comment 4. I will have a look, but given that 3 different tools are doing a copy and paste of the code doesn't sound scalable. Maybe it would be good to get at least parts of it into treeherder client core. CC'ing William for information.
(In reply to Maja Frydrychowicz (:maja_zf) from comment #5) > Hi Ryan, do you have any feedback on the following job/group symbols to be > used on Treeherder? (Tier 2 jobs.) I don't have a strong opinion on this. What you've got looks OK to me.
Flags: needinfo?(ryanvm)
Attachment #8623150 - Flags: review?(spolk) → review+
More detail about the WebRTC symbols - here's an example: https://treeherder.allizom.org/#/jobs?repo=mozilla-aurora&revision=38801423063b&filter-job_group_symbol=PW Each WebRTC test involves two machines, so each 'p' symbol is the same job run on a different combination of Firefox version and platform, ex: aurora on win8-x64 vs nightly on linux-32. The 'p' repeats many times because there are many combinations. We're using the same symbol for each combination because the build/platform info is partly conveyed by where the symbol appears in the Treeherder UI. To see the other "half" of the combination, go to Job Details and check for an artifact like "jenkins-webrtc-nightly-linux64-aurora-linux32-623". This is the Jenkins build tag and it describes the full combination used for the job. Still ok with this, Ryan?
Flags: needinfo?(ryanvm)
I'm not horribly crazy about it, no. Can we discuss in-person next week?
Flags: needinfo?(ryanvm)
Since all WebRTC and MSE/EME jobs on pf-jenkins are now reporting to Treeherder staging, but require different steps before they report to production, I'm closing this bug. I will open new bugs (one each for WebRTC, MSE/EME) to track the progress toward reporting to Treeherder prod: agreeing on job symbols, meeting visibility criteria for Tier-2, etc.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Summary: Get qualifying subset of Jenkins-based media automation (MSE/EME/WebRTC) reporting to Treeherder → Get qualifying subset of Platform QA media automation (MSE/EME/WebRTC Steeplechase) reporting to Treeherder
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: