There is a collection of media test jobs deployed or soon-to-be-deployed on "pf-jenkins"  and we want to have them report to Treeherder.
Many moons ago, Syd started working on this for WebRTC jobs so we already have:
- Staging credentials for Treeherder
- A script to submit WebRTC test results to Treeherder ; tested on Treeherder staging.
- S3 buckets for storing pf-jenkins logs
The MSE/EME jobs  work a little differently than the WebRTC jobs: they are run by a mozharness script triggered on pf-jenkins.
Whoops, broken link.  should be https://github.com/mjzffr/firefox-media-tests/tree/pf-jenkins
Two pf-jenkins jobs (on Mac 10.10, Win 7) are submitting to staging. After a few tweaks, latest iterations look good.
And now the log is parsed so the Failure Summary shows up properly. I don't keep track of line numbers in the log, though.
All pf-jenkins jobs for firefox-media-tests (MSE/EME) now report to Treeherder staging (mozilla-central, mozilla-aurora, mozilla-beta), using job group symbol "VP" (as in "Video Puppeteer").
Should have pointed to these earlier:
Hi Ryan, do you have any feedback on the following job/group symbols to be used on Treeherder? (Tier 2 jobs.)
MSE/EME video tests, which use Marionette
- Group: VideoPuppeteer (VP)
-- 'm' for 'MSE Video Playback'
-- 'e' for 'EME Video Playback'
We could make these more granular - e.g. indicate whether they're short or long. (ms, ml, es, el?)
WebRTC tests, which run on pairs of machines, using Steeplechase.
- Group: Paired WebRTC Steeplechase Tests (PW)
--'p' for 'WebRTC pair'
--'t' for 'WebRTC TURN pair'
IRC discussion with sydpolk, drno, jgriffin concluded that each two-machine test run will be reported as (up to) two Treeherder jobs, one for each machine platform.
As noticed on IRC right now, we have an overlap with implementing such a feature for firefox-ui-tests based tests. Please also see bug 1166683 for the changes which will directly land in the above harness. Maybe you can benefit from those and only have to adjust settings. Shall we mark the dependency?
(In reply to Henrik Skupin (:whimboo) from comment #6)
> Shall we mark the dependency?
The work for this bug also applies to WebRTC Steeplechase tests, so I'm not sure a dependency is appropriate. The Treeherder-specific code here can probably be reused in firefox-ui-tests, especially treeherding.py and s3.py, which are based on the implementation in Autophone. See Comment 4.
Fixed display of build, platform, job sig. ex: https://treeherder.allizom.org/#/jobs?repo=mozilla-aurora&filter-job_group_symbol=VP&filter-platform=Windows%208%20x64
(In reply to Maja Frydrychowicz (:maja_zf) from comment #7)
> sure a dependency is appropriate. The Treeherder-specific code here can
> probably be reused in firefox-ui-tests, especially treeherding.py and s3.py,
> which are based on the implementation in Autophone. See Comment 4.
I will have a look, but given that 3 different tools are doing a copy and paste of the code doesn't sound scalable. Maybe it would be good to get at least parts of it into treeherder client core. CC'ing William for information.
(In reply to Maja Frydrychowicz (:maja_zf) from comment #5)
> Hi Ryan, do you have any feedback on the following job/group symbols to be
> used on Treeherder? (Tier 2 jobs.)
I don't have a strong opinion on this. What you've got looks OK to me.
Created attachment 8623150 [details] [review]
PR for reporting WebRTC Steeplechase jobs
More detail about the WebRTC symbols - here's an example: https://treeherder.allizom.org/#/jobs?repo=mozilla-aurora&revision=38801423063b&filter-job_group_symbol=PW
Each WebRTC test involves two machines, so each 'p' symbol is the same job run on a different combination of Firefox version and platform, ex: aurora on win8-x64 vs nightly on linux-32. The 'p' repeats many times because there are many combinations.
We're using the same symbol for each combination because the build/platform info is partly conveyed by where the symbol appears in the Treeherder UI. To see the other "half" of the combination, go to Job Details and check for an artifact like "jenkins-webrtc-nightly-linux64-aurora-linux32-623". This is the Jenkins build tag and it describes the full combination used for the job.
Still ok with this, Ryan?
I'm not horribly crazy about it, no. Can we discuss in-person next week?
Since all WebRTC and MSE/EME jobs on pf-jenkins are now reporting to Treeherder staging, but require different steps before they report to production, I'm closing this bug. I will open new bugs (one each for WebRTC, MSE/EME) to track the progress toward reporting to Treeherder prod: agreeing on job symbols, meeting visibility criteria for Tier-2, etc.