1166683 - [15Q2] Implement publishing of test results for firefox-ui-tests to treeherder

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This quarter we want to get publishing of test results to treeherder implemented. With that we will have a dashboard for our Firefox UI Tests.

I will have a look at the code for Gaia UI Tests and how they implemented it. It may help me get our code faster done.

https://github.com/mozilla-b2g/gaia/blob/f34ce82a840ad3c0aed3bfff18517b3f6a0eb37f/tests/python/gaia-ui-tests/gaiatest/mixins/treeherder.py

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

CC'ing some folks to let them know about the current activity in getting our test results published in treeherder.

For now I will start to setup a local instance of treeherder, and read through the document as given in the URL field.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I talked with Ed on IRC about grouping our tests given that we have such a large testing matrix. As he mentioned we can have as many as groups as we want. So that would help in terms of having the locales as job names then. So far I haven't seen that 'F' is used by another framework yet. Here my proposal:

> Group Name              Group Symbol     Job Name      Job Symbol
> ==============================================================================
> FirefoxUI functional    Ff               en-US         en-US
> [..]
> FirefoxUI remote        Fr               en-US         en-US
> [..]
> FirefoxUI update        Fu               en-US         en-US

This would look like the following in treeherder:

> Linux opt     Fu( en-US de ru ) Ff( en-US de zh-TW) Fu ( en-US fa)

Keep in mind that we are testing up to nearly 100 locales. The number of jobs in each group will be very high! I think it would be nice to have the possibility to group by locale, but I don't see that this would work right now. So you will have to check each group for results of the same locale.

Ed and Mauro, how does the above sounds to you? Is that ok, or do you have any suggestions?

Comment 3

[Mauro Doglio [:mdoglio]]

Two questions:
- Are those jobs gonna be sheriffed?
- On how many platforms are them running? We currently have more than 30 platforms. If we want these jobs to run on each platform it will be +9k jobs per push. Resulting in ~5 times the number of jobs we currently have per push. If that's the case I would be a bit worried about performance.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

No, those tests are not Tier 1 and run externally, so not via buildbot and per commit. We run it once a day for the Nightly builds with a limit set of locales, and with all locales for beta and release candidate builds.

Those tests only run on desktop, so OS X (10.6, 10.8, 10.9, 10.10), Linux (Ubuntu 14.04 32/64, Ubuntu 14.10 32/64), and Windows (XP, Vista, 7 32/64, 8 32/64, 8.1 32/64)

Comment 5

[Mauro Doglio [:mdoglio]]

Ok so no reason to worry about perf. Maybe it's a stupid idea, but what about using unicode flag characters for the job symbol? For example 

Comment 6

[Mauro Doglio [:mdoglio]]

Bugzilla clearly doesn't like the unicode character representing the american flag :)

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, so I will start playing with a local treeherder VM until the dependency bugs have been fixed and I have the keys for the staging treeherder server.

Comment 8

[Ed Morley [:emorley]]

(In reply to Henrik Skupin (:whimboo) from comment #2)
> I talked with Ed on IRC about grouping our tests given that we have such a
> large testing matrix. As he mentioned we can have as many as groups as we
> want. So that would help in terms of having the locales as job names then.
> So far I haven't seen that 'F' is used by another framework yet. Here my
> proposal:
> 
> > Group Name              Group Symbol     Job Name      Job Symbol
> > ==============================================================================
> > FirefoxUI functional    Ff               en-US         en-US
> > [..]
> > FirefoxUI remote        Fr               en-US         en-US
> > [..]
> > FirefoxUI update        Fu               en-US         en-US
> 
> This would look like the following in treeherder:
> 
> > Linux opt     Fu( en-US de ru ) Ff( en-US de zh-TW) Fu ( en-US fa)
> 
> Keep in mind that we are testing up to nearly 100 locales. The number of
> jobs in each group will be very high! I think it would be nice to have the
> possibility to group by locale, but I don't see that this would work right
> now. So you will have to check each group for results of the same locale.
> 
> Ed and Mauro, how does the above sounds to you? Is that ok, or do you have
> any suggestions?

CCing the sheriffs, since they tend to decide re naming. (Though the perf issue Mauro mentioned is something that may affect it)

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

A quick update here. By today I'm able to send a job collection to a treeherder instance. It doesn't contain any data but at least all the platform bits have been figured out. The biggest problem here is a working mapping between the platform details we get from mozinfo and what treeherder expects. For possible values see https://github.com/mozilla/treeherder/blob/master/ui/js/values.js#L7

Today I would like to fix the this mapping in our repository so mozinfo would work directly. For the treeherder client we have bug 1174186. If my mapping solution works fine, I might even add this feature over there.

Beside that there is one thing I noticed today. Given that we run multiple versions of Ubuntu we might have to tweak the group names even more. At least for Linux and Linux64. It would look like:

> Linux opt     Ff-ub14.04 ( en-US ) Ff-ub15.04 ( en-US )
> Linux x64 opt Ff-ub14.04 ( en-US ) Ff-ub15.04 ( en-US )

That way we can separate the different results of both platforms, without having to kill the 15.04 VMs yet. We might have to in the future, but lets see.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Here an update of the current state. With yesterdays observations I will no longer continue to integrate the treeherder feature directly into the firefox-ui-tests. Instead report submission will become part of our Mozmill CI system. The reasons you can find below:

1. None of our contributors would be able to send reports to treeherder given the auth restrictions. I don't think that we want to hand out official keys and secrets. Maybe we could setup our own instance at some point but even then its unclear how to handle authentication. That means only Mozmill CI can submit test results to treeherder.

2. Submission of results to treeherder can fail if Marionette exists early and we do not hit the code path for submission. That's why our wrapper run_tests.py script will handle it all. With that we will also have access to data like build url, locale, type of build (opt, debug, ...), and some others which might need a little update to the Jenkins jobs. Further we can at least send reports for running and completed. I even wanna see pending at some point.

3. Having the treeherder submission code outside of firefox-ui-tests gives us the opportunity to replace our wrapper scripts very easily with mozharness, and then completely let our tests run via taskcluster. Treeherder code in the test harness would have to be removed then.

All in all I got the basics of report submission implemented and will cover code implementation related work over on the following github issue: https://github.com/mozilla/mozmill-ci/issues/595

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We are done here. All jobs which depend on firefox-ui-tests now report to treeherder staging:

https://treeherder.allizom.org/#/jobs?repo=mozilla-central&filter-job_group_symbol=fu&filter-job_group_symbol=ff&filter-job_group_symbol=fr

https://treeherder.allizom.org/#/jobs?repo=mozilla-aurora&filter-job_group_symbol=fu&filter-job_group_symbol=ff&filter-job_group_symbol=fr

https://treeherder.allizom.org/#/jobs?repo=mozilla-beta&filter-job_group_symbol=fu&filter-job_group_symbol=ff&filter-job_group_symbol=fr

https://treeherder.allizom.org/#/jobs?repo=mozilla-release&filter-job_group_symbol=fu&filter-job_group_symbol=ff&filter-job_group_symbol=fr

https://treeherder.allizom.org/#/jobs?repo=mozilla-esr38&filter-job_group_symbol=fu&filter-job_group_symbol=ff&filter-job_group_symbol=fr

We are now a Tier 3, but with the move to mozharness next quarter I want to hopefully see us reaching Tier 2 level.