1213542 - [jenkins] reporting: treeherder integration

Status: RESOLVED WONTFIX

(Cloud Services :: QA: Test Automation, defect)

Description

[Richard Pappalardo [:rpapa][:rpappalardo]]

Create a tool for integrating test results with treeherder

Comment 1

[Richard Pappalardo [:rpapa][:rpappalardo]]

take a look at coversheet for treeherder integration:
https://github.com/mozilla/coversheet

Comment 2

[Richard Pappalardo [:rpapa][:rpappalardo]]

https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy

Comment 3

[Richard Pappalardo [:rpapa][:rpappalardo]]

NOTES for rshenkar for getting started:

---------------------------------------------------
DOCUMENTATION
---------------------------------------------------

EXAMPLE #0 - treeherder sandbox example

- https://github.com/mjzffr/treeherder-submission-example
- it would be a good exercise update so it works with the latest versions of everything


EXAMPLE #1 - used by firefox-ui-tests 

https://github.com/mozilla/mozmill-ci/blob/production/jenkins-master/jobs/scripts/workspace/submission.py
- The treeherder API changes from time to time, but the main difference is that we've 
  gradually narrowed down on some best practices on how to submit to treeherder from outside of buildbot.


EXAMPLE #2 - soon-to-be-deprecated (but works for now)

- https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/treeherding.py
- used by this mozharness script: 
  https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/run_media_tests.py
  i would not mimic this, but it shows how things work
  it was an early project for doing so (aka: submission still works, but syd and henrik have undertaken 
  to refactor and consolidate, hence lots of changes.


---------------------------------------------------
CONTACTS
---------------------------------------------------

IRC #treeherder 
:maja_zf - general questions about how treeherder works

treeherder documentation
http://treeherder.readthedocs.org/en/latest/installation.html


---------------------------------------------------
TIER 2 - GETTING CONNECTED
---------------------------------------------------

Example bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1176420

Sheriffing policy:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy

NOTES from :maja_zf:

first you're expected to get submission working and the tests mostly green on treeherder staging
to submit to treeherder staging or production you need to set up credentials. the best person to ask about that is emorley
or take a look at this comment:
 https://bugzilla.mozilla.org/show_bug.cgi?id=1212936#c1

Comment 4

[Richard Pappalardo [:rpapa][:rpappalardo]]

:sphilp,

Can we get Rithesh a VPN key so he can access Jenkins?

Comment 5

[Richard Pappalardo [:rpapa][:rpappalardo]]

treeherder submission code will eventually reside here:
https://github.com/mozilla-services/services-test/tree/dev/_shared

Comment 6

[Stuart Philp :sphilp]

bug 1246986 is tracking the account stuff

Comment 8

[Richard Pappalardo [:rpapa][:rpappalardo]]

UPDATE

:rshenkar has VPN/LDAP access and can now login to Jenkins.
also has treeherder running locally. 

Apparently, treeherder not only reports 'end-run' status: PASS/FAIL along with trace logging, but you can also notify treeherder at the beginning of a run.  Speaking with :rshenkar, our plan is to add pre/post build steps to update treeherder accordingly.  For this to happen, it will probably be easier to use a run.py script for our tests (rather than a run.sh) so we can invoke treeherder updates/submissions more directly.

Comment 9

[Richard Pappalardo [:rpapa][:rpappalardo]]

We should look at using either python or node to do treeherder calls:
see:
 https://www.npmjs.com/package/mozilla-treeherder for node

Comment 10

[Richard Pappalardo [:rpapa][:rpappalardo]]

REFERENCE:

Rithesh's repo:
https://github.com/RitheshShenthar/treeherder-demo-client

To add dropdown menu w/ services project:
Bug 1268958 - Add the autopush repository to Treeherder

other notes:
https://public.etherpad-mozilla.org/p/treeherder-demo

Comment 11

[Rithesh Shenthar]

Until this lands : Bug 1264074 - Handle autocreation of resultsets for github repos
we will need to provide a dataset before submitting data:
eg:-
https://github.com/shinglyu/servo-perf/blob/master/submit_to_perfherder.py#L60

Comment 12

[Richard Pappalardo [:rpapa][:rpappalardo]]

see also notes in:
Bug 1108259 - Create treeherder help docs for services (e2e) tests

Comment 14

[Richard Pappalardo [:rpapa][:rpappalardo]]

This is outdated.  Reporting will be handled via ActiveData