1043718 - Make gaia-ui-tests on Flames report to Treeherder

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Jonathan Griffin (:jgriffin)]

We'd like to report the results of gaia-ui-tests to Treeherder, so people without Jenkins access can see the results.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Here's some more information from Jeads about how to do this:

Take a look at https://github.com/mozilla/treeherder-client. Basically, you just need to build json that treeherder accepts and POST it, with the proper OAuth credentials, to the associated repository/branch.

Most of the work required to submit data to treeherder will be adapting the data you have available in Jenkins to the json structures that treeherder accepts. There are two basic data structures you will need to submit. A resultset which is analogous to a push https://github.com/mozilla/treeherder-client#resultset-collection, and a job which can be any kind of build or test data https://github.com/mozilla/treeherder-client#job-collection. There's also a job info artifact that will allow you to control what get's populated in the job detail panel when you select a job in the UI (https://github.com/mozilla/treeherder-client#job-artifacts-format).

Once, you have the json structures ready, we can provide you with the OAuth credentials for b2g-inbound and you can start submitting structures to our stage environment https://treeherder.allizom.org or development environment treeherder-dev.allizom.org. Once the data looks good to you, we will provide you with the production credentials for https://treeherder.mozilla.org and you're all set.

You can also try all of this locally at any time if you want https://treeherder-service.readthedocs.org/en/latest/installation.html

If you're submitting the resultset/push from jenkins this is straight forward. If you're trying to associate jenkins test output with a push that was submitted by buildbot there's a bit more to it, just need to make sure we map revision's correctly. Let us know if you need help, feel free to ask questions in #treeherder. We can also chat on vidyo if needed.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Our test runner currently supports JUnit XML and HTML output formats, and I think we can also output structured logging. Would we expect to add reporting to Treeherder into the runner, or would it be a separate process? I personally think it would make more sense to build this into the test runner.

As I understand it we'll need to submit a job data structure, which is linked to a resultset that was submitted by buildbot. If so, we should probably also try to ensure that the build under test matches the resultset and has not been modified locally (such as resetting Gaia).

Comment 3

[Jonathan Griffin (:jgriffin)]

> As I understand it we'll need to submit a job data structure, which is linked to a resultset that was 
> submitted by buildbot.

Yes.

I agree it makes sense to do this in the runner, and I agree that for jobs reporting to Treeherder we should be careful not to change the Gaia version.

A couple of other things we'll need for true Treeherder integration (but which are outside the scope of this bug):

- publicly accessible logs
- a way to retrigger runs against a given gecko commit

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I've set up a local instance of Treeherder to look into this without affecting any of our hosted instances.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Add support for submitting device results to Treeherder. v1.0

This patch enables support for submitting to Treeherder. I've tested it successfully with a local Treeherder instance. I'm going to raise a couple of blocking bugs such as releasing treeherder-client to PyPI and exposing suite start/end times from the Marionette test runner. There will also be followup bugs for attaching the logs/reports as artifacts.

Comment 6

[Mauro Doglio [:mdoglio]]

Comment on attachment 8468581 [details] [diff] [review]
Add support for submitting device results to Treeherder. v1.0

Review of attachment 8468581 [details] [diff] [review]:
-----------------------------------------------------------------

::: tests/python/gaia-ui-tests/gaiatest/mixins/treeherder.py
@@ +112,5 @@
> +            job.add_result('success')
> +
> +        job.add_start_timestamp(int(time.time()) - 300)  # TODO: use start time
> +        job.add_end_timestamp(int(time.time()))  # TODO: use end time
> +        job.add_submit_timestamp(int(time.time()))

submit_timestamp should be the submission time to the build system, so submit_timestamp < start_timestamp < end_timestamp

Comment 7

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8468581 [details] [diff] [review]
Add support for submitting device results to Treeherder. v1.0

Review of attachment 8468581 [details] [diff] [review]:
-----------------------------------------------------------------

::: tests/python/gaia-ui-tests/gaiatest/mixins/treeherder.py
@@ +71,5 @@
> +
> +        device = version.get('device_id')
> +        if not device:
> +            sys.exit('Submitting to Treeherder is currently limited to '
> +                     'devices.')

Instead of sys.exit, we should either throw an exception or print a warning and return.

@@ +82,5 @@
> +            job.add_job_symbol('Gip')
> +        except KeyError:
> +            sys.exit('Unknown device id: %s, unable to determine '
> +                     'Treeherder group. Supported device ids: %s' % (
> +                         device, DEVICE_GROUP_MAP.keys()))

same as above

::: tests/python/gaia-ui-tests/gaiatest/runtests.py
@@ +75,5 @@
>  
> +    def run_tests(self, tests):
> +        super(GaiaTestRunner, self).run_tests(tests)
> +        if self.treeherder:
> +            self.post_to_treeherder()

In order to avoid overriding run_tests, could we use mixin_run_tests like we do for html reporting?  i.e., http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/runner/mixins/reporting.py#32

::: tests/python/gaia-ui-tests/requirements.txt
@@ +2,4 @@
>  mozdevice>=0.34
>  moztest>=0.6
>  requests
> +treeherder-client>=1.0

We will need to upload treeherder-client to internal pypi before we can land this, otherwise it will cause problems in buildbot.

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Add support for submitting device results to Treeherder. v1.1

(In reply to Mauro Doglio [:mdoglio] from comment #6)
> Comment on attachment 8468581 [details] [diff] [review]
> Add support for submitting device results to Treeherder. v1.0
> 
> Review of attachment 8468581 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: tests/python/gaia-ui-tests/gaiatest/mixins/treeherder.py
> @@ +112,5 @@
> > +            job.add_result('success')
> > +
> > +        job.add_start_timestamp(int(time.time()) - 300)  # TODO: use start time
> > +        job.add_end_timestamp(int(time.time()))  # TODO: use end time
> > +        job.add_submit_timestamp(int(time.time()))
> 
> submit_timestamp should be the submission time to the build system, so
> submit_timestamp < start_timestamp < end_timestamp

Thanks Mauro. As discussed on IRC I don't think it's going to be easy to determine when the job entered the queue, so submit time will be equal to start time. These changes are dependent on bug 1049717.

(In reply to Jonathan Griffin (:jgriffin) from comment #7)
> Comment on attachment 8468581 [details] [diff] [review]
> Add support for submitting device results to Treeherder. v1.0
> 
> Review of attachment 8468581 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: tests/python/gaia-ui-tests/gaiatest/mixins/treeherder.py
> @@ +71,5 @@
> > +
> > +        device = version.get('device_id')
> > +        if not device:
> > +            sys.exit('Submitting to Treeherder is currently limited to '
> > +                     'devices.')
> 
> Instead of sys.exit, we should either throw an exception or print a warning
> and return.

Done.

> @@ +82,5 @@
> > +            job.add_job_symbol('Gip')
> > +        except KeyError:
> > +            sys.exit('Unknown device id: %s, unable to determine '
> > +                     'Treeherder group. Supported device ids: %s' % (
> > +                         device, DEVICE_GROUP_MAP.keys()))
> 
> same as above

Done.

> ::: tests/python/gaia-ui-tests/gaiatest/runtests.py
> @@ +75,5 @@
> >  
> > +    def run_tests(self, tests):
> > +        super(GaiaTestRunner, self).run_tests(tests)
> > +        if self.treeherder:
> > +            self.post_to_treeherder()
> 
> In order to avoid overriding run_tests, could we use mixin_run_tests like we
> do for html reporting?  i.e.,
> http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/
> marionette/runner/mixins/reporting.py#32

Good call. Done.

> ::: tests/python/gaia-ui-tests/requirements.txt
> @@ +2,4 @@
> >  mozdevice>=0.34
> >  moztest>=0.6
> >  requests
> > +treeherder-client>=1.0
> 
> We will need to upload treeherder-client to internal pypi before we can land
> this, otherwise it will cause problems in buildbot.

Yep, this was taken care of by bug 1049711.

Comment 9

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8469974 [details] [diff] [review]
Add support for submitting device results to Treeherder. v1.1

Review of attachment 8469974 [details] [diff] [review]:
-----------------------------------------------------------------

::: tests/python/gaia-ui-tests/gaiatest/mixins/treeherder.py
@@ +57,5 @@
> +        self.treeherder = treeherder
> +        self.treeherder_url = treeherder_url
> +        self.treeherder_key = treeherder_key
> +        self.treeherder_secret = treeherder_secret
> +        self.mixin_run_tests.append(self.post_to_treeherder)

We probably want add another line before this one:

if self.treeherder:

otherwise this will happen on every run, AFAICT.

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22724

(In reply to Jonathan Griffin (:jgriffin) from comment #9)
> Comment on attachment 8469974 [details] [diff] [review]
> Add support for submitting device results to Treeherder. v1.1
> 
> Review of attachment 8469974 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: tests/python/gaia-ui-tests/gaiatest/mixins/treeherder.py
> @@ +57,5 @@
> > +        self.treeherder = treeherder
> > +        self.treeherder_url = treeherder_url
> > +        self.treeherder_key = treeherder_key
> > +        self.treeherder_secret = treeherder_secret
> > +        self.mixin_run_tests.append(self.post_to_treeherder)
> 
> We probably want add another line before this one:
> 
> if self.treeherder:
> 
> otherwise this will happen on every run, AFAICT.

Good catch. Fixed in this patch. Note that we should hold off landing this until we have released marionette-client 0.8.2 and bumped the dependency. I've updated the dependencies.

Comment 11

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8470754 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22724

lgtm

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Landed in:
https://github.com/mozilla-b2g/gaia/commit/93de36c0da489fe6d5dc16b7c28377c24e1932d6

To submit results to Treeherder you will need the OAuth key/secret for the instance and project you want to submit to. You can request these from :jeads, and I would suggest first submitting to the treeherder.allizom.org instance, most likely for the b2g-inbound project.

Once you have these, you will need to add the following command line options when running tests:

--treeherder
--treeherder-url
--treeherder-key
--treeherder-secret

You should also include the --ci-url command line option if running tests from Jenkins. This would be --ci-url=${JENKINS_URL} and will provide a link from Treeherder back to the originating Jenkins build.

If you have any issues or questions, let me know.