Closed Bug 1213542 Opened 5 years ago Closed 4 years ago

[jenkins] reporting: treeherder integration

Categories

(Cloud Services :: QA: Test Automation, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: rpapa, Assigned: rpapa, Mentored)

References

Details

Create a tool for integrating test results with treeherder
take a look at coversheet for treeherder integration:
https://github.com/mozilla/coversheet
NOTES for rshenkar for getting started:

---------------------------------------------------
DOCUMENTATION
---------------------------------------------------

EXAMPLE #0 - treeherder sandbox example

- https://github.com/mjzffr/treeherder-submission-example
- it would be a good exercise update so it works with the latest versions of everything


EXAMPLE #1 - used by firefox-ui-tests 

https://github.com/mozilla/mozmill-ci/blob/production/jenkins-master/jobs/scripts/workspace/submission.py
- The treeherder API changes from time to time, but the main difference is that we've 
  gradually narrowed down on some best practices on how to submit to treeherder from outside of buildbot.


EXAMPLE #2 - soon-to-be-deprecated (but works for now)

- https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/treeherding.py
- used by this mozharness script: 
  https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/run_media_tests.py
  i would not mimic this, but it shows how things work
  it was an early project for doing so (aka: submission still works, but syd and henrik have undertaken 
  to refactor and consolidate, hence lots of changes.


---------------------------------------------------
CONTACTS
---------------------------------------------------

IRC #treeherder 
:maja_zf - general questions about how treeherder works

treeherder documentation
http://treeherder.readthedocs.org/en/latest/installation.html


---------------------------------------------------
TIER 2 - GETTING CONNECTED
---------------------------------------------------

Example bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1176420

Sheriffing policy:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy

NOTES from :maja_zf:

first you're expected to get submission working and the tests mostly green on treeherder staging
to submit to treeherder staging or production you need to set up credentials. the best person to ask about that is emorley
or take a look at this comment:
 https://bugzilla.mozilla.org/show_bug.cgi?id=1212936#c1
Status: NEW → ASSIGNED
QA Contact: rpappalardo
:sphilp,

Can we get Rithesh a VPN key so he can access Jenkins?
Assignee: sphilp → rshenthar
Mentor: rpappalardo
Flags: needinfo?(sphilp)
treeherder submission code will eventually reside here:
https://github.com/mozilla-services/services-test/tree/dev/_shared
bug 1246986 is tracking the account stuff
Flags: needinfo?(sphilp)
Duplicate of this bug: 1205984
UPDATE

:rshenkar has VPN/LDAP access and can now login to Jenkins.
also has treeherder running locally. 

Apparently, treeherder not only reports 'end-run' status: PASS/FAIL along with trace logging, but you can also notify treeherder at the beginning of a run.  Speaking with :rshenkar, our plan is to add pre/post build steps to update treeherder accordingly.  For this to happen, it will probably be easier to use a run.py script for our tests (rather than a run.sh) so we can invoke treeherder updates/submissions more directly.
We should look at using either python or node to do treeherder calls:
see:
 https://www.npmjs.com/package/mozilla-treeherder for node
Depends on: 1263644
Depends on: 1266133
REFERENCE:

Rithesh's repo:
https://github.com/RitheshShenthar/treeherder-demo-client

To add dropdown menu w/ services project:
Bug 1268958 - Add the autopush repository to Treeherder

other notes:
https://public.etherpad-mozilla.org/p/treeherder-demo
Until this lands : Bug 1264074 - Handle autocreation of resultsets for github repos
we will need to provide a dataset before submitting data:
eg:-
https://github.com/shinglyu/servo-perf/blob/master/submit_to_perfherder.py#L60
see also notes in:
Bug 1108259 - Create treeherder help docs for services (e2e) tests
Assignee: rshenthar → rpappalardo
Duplicate of this bug: 1108259
This is outdated.  Reporting will be handled via ActiveData
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.