Get qualifying subset of Platform QA media automation (MSE/EME/WebRTC Steeplechase) reporting to Treeherder

RESOLVED FIXED

Status

RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: maja_zf, Assigned: maja_zf)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

There is a collection of media test jobs deployed or soon-to-be-deployed on "pf-jenkins" [1] and we want to have them report to Treeherder. 

Many moons ago, Syd started working on this for WebRTC jobs so we already have:
- Staging credentials for Treeherder 
- A script to submit WebRTC test results to Treeherder [1]; tested on Treeherder staging.
- S3 buckets for storing pf-jenkins logs

The MSE/EME jobs [2] work a little differently than the WebRTC jobs: they are run by a mozharness script triggered on pf-jenkins. 

[1] http://pf-jenkins.qa.mtv2.mozilla.com:8080
[2] https://github.com/sydvicious/mozplatformqa-jenkins/blob/master/jenkinsherder.py
[3] https://github.com/mjzffr/firefox-media-tests/tree/pf-jenkin
(Assignee)

Comment 2

3 years ago
Two pf-jenkins jobs (on Mac 10.10, Win 7) are submitting to staging. After a few tweaks, latest iterations look good. 

https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=4700d1cdf489&filter-job_group_symbol=VP
(Assignee)

Comment 3

3 years ago
And now the log is parsed so the Failure Summary shows up properly. I don't keep track of line numbers in the log, though.

https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=e10e2e8d8bf2&filter-job_group_symbol=VP
(Assignee)

Comment 4

3 years ago
All pf-jenkins jobs for firefox-media-tests (MSE/EME) now report to Treeherder staging (mozilla-central, mozilla-aurora, mozilla-beta), using job group symbol "VP" (as in "Video Puppeteer").

Should have pointed to these earlier: 
https://github.com/mjzffr/firefox-media-tests/commit/b79dc1001468574aaa7d168177671a08a4a2b046

https://github.com/mjzffr/firefox-media-tests/commit/d1989812dda449322813c55c4f57d29d30e1ebbd

https://github.com/mjzffr/firefox-media-tests/commit/1fe8d725aee306a0eec126702f6531604fc0d2c1
(Assignee)

Updated

3 years ago
Depends on: 1173817
(Assignee)

Comment 5

3 years ago
Hi Ryan, do you have any feedback on the following job/group symbols to be used on Treeherder? (Tier 2 jobs.)

MSE/EME video tests, which use Marionette
- Group: VideoPuppeteer (VP)
- Jobs: 
-- 'm' for 'MSE Video Playback' 
-- 'e' for 'EME Video Playback' 
We could make these more granular - e.g. indicate whether they're short or long. (ms, ml, es, el?)

WebRTC tests, which run on pairs of machines, using Steeplechase. 
- Group: Paired WebRTC Steeplechase Tests (PW)
- Jobs: 
--'p' for 'WebRTC pair'
--'t' for 'WebRTC TURN pair'
IRC discussion with sydpolk, drno, jgriffin concluded that each two-machine test run will be reported as (up to) two Treeherder jobs, one for each machine platform.
Flags: needinfo?(ryanvm)
As noticed on IRC right now, we have an overlap with implementing such a feature for firefox-ui-tests based tests. Please also see bug 1166683 for the changes which will directly land in the above harness. Maybe you can benefit from those and only have to adjust settings. Shall we mark the dependency?
(Assignee)

Comment 7

3 years ago
(In reply to Henrik Skupin (:whimboo) from comment #6)
> Shall we mark the dependency?
The work for this bug also applies to WebRTC Steeplechase tests, so I'm not sure a dependency is appropriate. The Treeherder-specific code here can probably be reused in firefox-ui-tests, especially treeherding.py and s3.py, which are based on the implementation in Autophone. See Comment 4.
(In reply to Maja Frydrychowicz (:maja_zf) from comment #7)
> sure a dependency is appropriate. The Treeherder-specific code here can
> probably be reused in firefox-ui-tests, especially treeherding.py and s3.py,
> which are based on the implementation in Autophone. See Comment 4.

I will have a look, but given that 3 different tools are doing a copy and paste of the code doesn't sound scalable. Maybe it would be good to get at least parts of it into treeherder client core. CC'ing William for information.
(In reply to Maja Frydrychowicz (:maja_zf) from comment #5)
> Hi Ryan, do you have any feedback on the following job/group symbols to be
> used on Treeherder? (Tier 2 jobs.)

I don't have a strong opinion on this. What you've got looks OK to me.
Flags: needinfo?(ryanvm)
Created attachment 8623150 [details] [review]
PR for reporting WebRTC Steeplechase jobs
Attachment #8623150 - Flags: review?(spolk)

Updated

3 years ago
Attachment #8623150 - Flags: review?(spolk) → review+
More detail about the WebRTC symbols - here's an example: https://treeherder.allizom.org/#/jobs?repo=mozilla-aurora&revision=38801423063b&filter-job_group_symbol=PW

Each WebRTC test involves two machines, so each 'p' symbol is the same job run on a different combination of Firefox version and platform, ex: aurora on win8-x64 vs nightly on linux-32. The 'p' repeats many times because there are many combinations. 

We're using the same symbol for each combination because the build/platform info is partly conveyed by where the symbol appears in the Treeherder UI. To see the other "half" of the combination, go to Job Details and check for an artifact like "jenkins-webrtc-nightly-linux64-aurora-linux32-623". This is the Jenkins build tag and it describes the full combination used for the job.

Still ok with this, Ryan?
Flags: needinfo?(ryanvm)
I'm not horribly crazy about it, no. Can we discuss in-person next week?
Flags: needinfo?(ryanvm)
Since all WebRTC and MSE/EME jobs on pf-jenkins are now reporting to Treeherder staging, but require different steps before they report to production, I'm closing this bug. I will open new bugs (one each for WebRTC, MSE/EME) to track the progress toward reporting to Treeherder prod: agreeing on job symbols, meeting visibility criteria for Tier-2, etc.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Summary: Get qualifying subset of Jenkins-based media automation (MSE/EME/WebRTC) reporting to Treeherder → Get qualifying subset of Platform QA media automation (MSE/EME/WebRTC Steeplechase) reporting to Treeherder
(Assignee)

Updated

3 years ago
Blocks: 1176420
You need to log in before you can comment on or make changes to this bug.