Last Comment Bug 1163131 - Get qualifying subset of Platform QA media automation (MSE/EME/WebRTC Steeplechase) reporting to Treeherder
: Get qualifying subset of Platform QA media automation (MSE/EME/WebRTC Steeple...
Status: RESOLVED FIXED
:
Product: Testing
Classification: Components
Component: General (show other bugs)
: unspecified
: Unspecified Unspecified
-- normal (vote)
: ---
Assigned To: Maja Frydrychowicz (:maja_zf) (away)
:
: Geoff Brown [:gbrown]
Mentors:
Depends on: 1151620 1173817
Blocks: 1176420
  Show dependency treegraph
 
Reported: 2015-05-08 12:58 PDT by Maja Frydrychowicz (:maja_zf) (away)
Modified: 2015-06-19 14:55 PDT (History)
7 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
PR for reporting WebRTC Steeplechase jobs (58 bytes, text/x-github-pull-request)
2015-06-16 10:32 PDT, Maja Frydrychowicz (:maja_zf) (away)
sydpolk: review+
Details | Review | Splinter Review

Description User image Maja Frydrychowicz (:maja_zf) (away) 2015-05-08 12:58:53 PDT
There is a collection of media test jobs deployed or soon-to-be-deployed on "pf-jenkins" [1] and we want to have them report to Treeherder. 

Many moons ago, Syd started working on this for WebRTC jobs so we already have:
- Staging credentials for Treeherder 
- A script to submit WebRTC test results to Treeherder [1]; tested on Treeherder staging.
- S3 buckets for storing pf-jenkins logs

The MSE/EME jobs [2] work a little differently than the WebRTC jobs: they are run by a mozharness script triggered on pf-jenkins. 

[1] http://pf-jenkins.qa.mtv2.mozilla.com:8080
[2] https://github.com/sydvicious/mozplatformqa-jenkins/blob/master/jenkinsherder.py
[3] https://github.com/mjzffr/firefox-media-tests/tree/pf-jenkin
Comment 1 User image Maja Frydrychowicz (:maja_zf) (away) 2015-05-08 13:00:49 PDT
Whoops, broken link. [3] should be https://github.com/mjzffr/firefox-media-tests/tree/pf-jenkins
Comment 2 User image Maja Frydrychowicz (:maja_zf) (away) 2015-06-08 11:22:14 PDT
Two pf-jenkins jobs (on Mac 10.10, Win 7) are submitting to staging. After a few tweaks, latest iterations look good. 

https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=4700d1cdf489&filter-job_group_symbol=VP
Comment 3 User image Maja Frydrychowicz (:maja_zf) (away) 2015-06-09 08:46:55 PDT
And now the log is parsed so the Failure Summary shows up properly. I don't keep track of line numbers in the log, though.

https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=e10e2e8d8bf2&filter-job_group_symbol=VP
Comment 4 User image Maja Frydrychowicz (:maja_zf) (away) 2015-06-11 08:06:35 PDT
All pf-jenkins jobs for firefox-media-tests (MSE/EME) now report to Treeherder staging (mozilla-central, mozilla-aurora, mozilla-beta), using job group symbol "VP" (as in "Video Puppeteer").

Should have pointed to these earlier: 
https://github.com/mjzffr/firefox-media-tests/commit/b79dc1001468574aaa7d168177671a08a4a2b046

https://github.com/mjzffr/firefox-media-tests/commit/d1989812dda449322813c55c4f57d29d30e1ebbd

https://github.com/mjzffr/firefox-media-tests/commit/1fe8d725aee306a0eec126702f6531604fc0d2c1
Comment 5 User image Maja Frydrychowicz (:maja_zf) (away) 2015-06-12 08:03:20 PDT
Hi Ryan, do you have any feedback on the following job/group symbols to be used on Treeherder? (Tier 2 jobs.)

MSE/EME video tests, which use Marionette
- Group: VideoPuppeteer (VP)
- Jobs: 
-- 'm' for 'MSE Video Playback' 
-- 'e' for 'EME Video Playback' 
We could make these more granular - e.g. indicate whether they're short or long. (ms, ml, es, el?)

WebRTC tests, which run on pairs of machines, using Steeplechase. 
- Group: Paired WebRTC Steeplechase Tests (PW)
- Jobs: 
--'p' for 'WebRTC pair'
--'t' for 'WebRTC TURN pair'
IRC discussion with sydpolk, drno, jgriffin concluded that each two-machine test run will be reported as (up to) two Treeherder jobs, one for each machine platform.
Comment 6 User image Henrik Skupin (:whimboo) 2015-06-12 09:58:34 PDT
As noticed on IRC right now, we have an overlap with implementing such a feature for firefox-ui-tests based tests. Please also see bug 1166683 for the changes which will directly land in the above harness. Maybe you can benefit from those and only have to adjust settings. Shall we mark the dependency?
Comment 7 User image Maja Frydrychowicz (:maja_zf) (away) 2015-06-12 11:41:19 PDT
(In reply to Henrik Skupin (:whimboo) from comment #6)
> Shall we mark the dependency?
The work for this bug also applies to WebRTC Steeplechase tests, so I'm not sure a dependency is appropriate. The Treeherder-specific code here can probably be reused in firefox-ui-tests, especially treeherding.py and s3.py, which are based on the implementation in Autophone. See Comment 4.
Comment 9 User image Henrik Skupin (:whimboo) 2015-06-15 01:13:07 PDT
(In reply to Maja Frydrychowicz (:maja_zf) from comment #7)
> sure a dependency is appropriate. The Treeherder-specific code here can
> probably be reused in firefox-ui-tests, especially treeherding.py and s3.py,
> which are based on the implementation in Autophone. See Comment 4.

I will have a look, but given that 3 different tools are doing a copy and paste of the code doesn't sound scalable. Maybe it would be good to get at least parts of it into treeherder client core. CC'ing William for information.
Comment 10 User image Ryan VanderMeulen [:RyanVM] 2015-06-15 12:09:01 PDT
(In reply to Maja Frydrychowicz (:maja_zf) from comment #5)
> Hi Ryan, do you have any feedback on the following job/group symbols to be
> used on Treeherder? (Tier 2 jobs.)

I don't have a strong opinion on this. What you've got looks OK to me.
Comment 11 User image Maja Frydrychowicz (:maja_zf) (away) 2015-06-16 10:32:21 PDT
Created attachment 8623150 [details] [review]
PR for reporting WebRTC Steeplechase jobs
Comment 12 User image Maja Frydrychowicz (:maja_zf) (away) 2015-06-18 14:42:06 PDT
More detail about the WebRTC symbols - here's an example: https://treeherder.allizom.org/#/jobs?repo=mozilla-aurora&revision=38801423063b&filter-job_group_symbol=PW

Each WebRTC test involves two machines, so each 'p' symbol is the same job run on a different combination of Firefox version and platform, ex: aurora on win8-x64 vs nightly on linux-32. The 'p' repeats many times because there are many combinations. 

We're using the same symbol for each combination because the build/platform info is partly conveyed by where the symbol appears in the Treeherder UI. To see the other "half" of the combination, go to Job Details and check for an artifact like "jenkins-webrtc-nightly-linux64-aurora-linux32-623". This is the Jenkins build tag and it describes the full combination used for the job.

Still ok with this, Ryan?
Comment 13 User image Ryan VanderMeulen [:RyanVM] 2015-06-19 07:39:10 PDT
I'm not horribly crazy about it, no. Can we discuss in-person next week?
Comment 14 User image Maja Frydrychowicz (:maja_zf) (away) 2015-06-19 09:05:05 PDT
Since all WebRTC and MSE/EME jobs on pf-jenkins are now reporting to Treeherder staging, but require different steps before they report to production, I'm closing this bug. I will open new bugs (one each for WebRTC, MSE/EME) to track the progress toward reporting to Treeherder prod: agreeing on job symbols, meeting visibility criteria for Tier-2, etc.

Note You need to log in before you can comment on or make changes to this bug.