1053086 - Allow Mozmill to send reports to Treeherder

Status: RESOLVED WONTFIX

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

We want to allow Mozmill to send test reports to Treeherder instead of CouchDB.

That should give us the following features:

* Better visualization of test results

* Better tracking of open issues (auto-suggestion) across platforms and branches

* Better stability, and in-house support by the A-Team (compared to IrisCouch)

* No more those crazy mapping code 


Things to figure out:

* How does OAuth work

* What would we have to change in our JSON reports to be compatible

* Who can send reports (also community members?)

* Is there a way to see an overview of all the tests of a specific build including all the locales and testruns (functional, remote, update...)

* Would Treeherder be able to handle our update tests, where at least two versions of Firefox are involved


There might be indeed more things to check, but as I can see there are still blockers left to fix. So for us it would be good to get started in testing treeherder, and to be able to give our feedback.

For more info and we may base on bug 1043718, which is about sending gaia test results to Treeherder.

Jonathan, could we get some information about the open questions? Especially I wonder where we would have to get this implemented. If results are only for our internal CI tests, it may be better to also do it via Jenkins and not Mozmill proper.

Comment 1

[Jonathan French (:jfrench)]

I am pretty sure work could potentially involve any all of the the Github repos, I've listed them here for clarity.

treeherder-ui
treeherder-service
treeherder-client
treeherder-node

Beyond that, I'd say that jeads would be best positioned to answer the open questions you've posed. Plus he is awesome! :) Also adding Cameron and Mauro to the cc. Any related Treeherder bugs can be found at the link below, and any new Treeherder specific-issues which impact a mozmill transition, I would think can be entered against Treeherder, if jeads agrees that makes sense.

https://wiki.mozilla.org/Auto-tools/Projects/Treeherder

Comment 2

[Mauro Doglio [:mdoglio]]

(In reply to Henrik Skupin (:whimboo) [away 08/16 - 08/23] from comment #0)
> We want to allow Mozmill to send test reports to Treeherder instead of
> CouchDB.
> 
> That should give us the following features:
> 
> * Better visualization of test results
> 
> * Better tracking of open issues (auto-suggestion) across platforms and
> branches
> 
> * Better stability, and in-house support by the A-Team (compared to
> IrisCouch)
> 
> * No more those crazy mapping code 
> 
> 
> Things to figure out:
> 
> * How does OAuth work
Treeherder uses a 2-legged oauth authentication with different key/secret pairs for each repository. 
> 
> * What would we have to change in our JSON reports to be compatible
See https://github.com/mozilla/treeherder-client/blob/master/README.md
 
> 
> * Who can send reports (also community members?)
Whoever has the key/secret pair.
> * Is there a way to see an overview of all the tests of a specific build
> including all the locales and testruns (functional, remote, update...)
Not sure what you mean here. If you are referring to all the builds/tests ingested for a specific platform,
you can fetch the main resultset api with a filter on the revision and then iterate over all the platforms to find the one you are interested in. See https://treeherder.mozilla.org/api/project/mozilla-central/resultset/?full=true&revision=7fc96293ada8&with_jobs=true
> 
> * Would Treeherder be able to handle our update tests, where at least two
> versions of Firefox are involved
> 
As long as you specify a revision hash, an identifier for your job and the platform and test properties, treeherder can handle it. 
> 
> There might be indeed more things to check, but as I can see there are still
> blockers left to fix. So for us it would be good to get started in testing
> treeherder, and to be able to give our feedback.
I highly suggest to start hacking on treeherder using the vagrant environment shipped with the service repo
https://github.com/mozilla/treeherder-service
> 
> For more info and we may base on bug 1043718, which is about sending gaia
> test results to Treeherder.
> 
> Jonathan, could we get some information about the open questions? Especially
> I wonder where we would have to get this implemented. If results are only
> for our internal CI tests, it may be better to also do it via Jenkins and
> not Mozmill proper.
If you are targeting a python environment https://github.com/mozilla/treeherder-client is probably the way to go (it's on pypi now thanks to :davehunt). There is also a nodejs implementation written by the folks working on taskcluster https://github.com/lightsofapollo/treeherder-node.

:whimboo feel free to ping me or jeads on #treeherder if you want to have a chat about it

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

Given that we transition to Marionette with Mozmill tests, this is not necessary anymore.