1272228 - [16Q2] Make Firefox-ui-tests results from TaskCluster Tier-1 or Tier-2 on Treeherder

Status: RESOLVED FIXED

(Testing :: Firefox UI Tests, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As of now the firefox-ui-test results are reported as tier-3 level to Treeherder, which means they are hidden by default and no-one takes care of them. After being a month away in April, most of it was busted beginning of May. It's something which we do not want anymore. So we would like to bump the level to Tier-1 and/or Tier-2 - all dependent on the remote test data usage (eg. different SSL subdomains on mozqa.com).

As I found on the Wiki here the requirements for both levels:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#Requirements_for_jobs_shown_in_the_default_Treeherder_view

The following points are stuff we have to add or at least document somewhere:

Necessary documentation outstanding and to be added to https://developer.mozilla.org/en-US/docs/Mozilla/QA/firefox-ui-tests:
* How to disable an individual failing test
* The current owner/who to contact for help
* The Bugzilla product/component where bugs should be filed

Here a special Tier-1 requirement:
* "Ensure MOZ_DISABLE_NONLOCAL_CONNECTIONS is defined in the automation environment" (see bug 995417). This is interesting in the case of Marionette, because it seems that its not getting used at all in that test framework. I have to do more investigation today and raise a bug if that is really true.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Just to add, this only applies to our tests as executed via Taskcluster but not mozmill-ci. What we need is better focus of developers on inbound and other integration branches but not nightly or release builds.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Henrik Skupin (:whimboo) from comment #0)
> * How to disable an individual failing test

This has been added under https://developer.mozilla.org/en-US/docs/Mozilla/QA/Firefox_UI_tests#How_to_disable_failing_tests.

> * The current owner/who to contact for help
> * The Bugzilla product/component where bugs should be filed

I'm not sure where to put this content. I cannot see similar information for other test suites on MDN. Is there a special page for sheriffs where this has to be added to?

> Here a special Tier-1 requirement:
> * "Ensure MOZ_DISABLE_NONLOCAL_CONNECTIONS is defined in the automation
> environment" (see bug 995417). This is interesting in the case of
> Marionette, because it seems that its not getting used at all in that test
> framework. I have to do more investigation today and raise a bug if that is
> really true.

This is actually bug 1272255 now and all Marionette related test suites which report to Treeherder as Tier-1 are affected. So I don't think it should be a showstopper for us now.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I updated our project page for firefox-ui-tests (https://wiki.mozilla.org/Auto-tools/Projects/Firefox_UI_Tests) in regards of contacts and the primary owner. It also lists the Bugzilla component for bugs to be filed. 

Ryan, not sure if any sheriff right now could look at this, or if maybe you could have a quick look at the following two documents. Is that enough to become Tier-2?

https://wiki.mozilla.org/Auto-tools/Projects/Firefox_UI_Tests
https://developer.mozilla.org/en-US/docs/Mozilla/QA/Firefox_UI_tests

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Talked to Carsten on IRC and putting ni? on the sheriff alias might not work. So adjusting it to Carsten and Wes directly.

Comment 5

[Carsten Book [:Tomcat]]

Henrik do you know if most tests pass currently like don't make sense to have a tier 2 component that has tests that are perma failure etc

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Carsten Book [:Tomcat] from comment #5)
> Henrik do you know if most tests pass currently like don't make sense to
> have a tier 2 component that has tests that are perma failure etc

Our tests are perma green except if developers land changes which cause complete bustage for us as what happened yesterday. Also there is an orange crasher on and off but that's all. Here you can see a list for inbound:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=Firefox%20UI&filter-tier=2&filter-tier=3

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=Firefox%20UI&filter-tier=2&filter-tier=3

In terms of mozilla-central we only speak about the tc-Fxfn jobs as noted in the bug summary, so listed test failures for Nightly builds don't play in here, and is another bug.

My open question is still about all the other documentation related items.

Comment 7

[Carsten Book [:Tomcat]]

i'm ok with having this as tier 2 - ryan since you did this i guess before is there any more "paperwork" needed regarding the "documentation related items"

Comment 8

[Ryan VanderMeulen [:RyanVM]]

I'm OK with the wiki and MDN pages.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks to you both! I'm going to file a new bug for bumping the tier level to 2 for now. When we have this running for a while, I will check again to make the local tests tier 1.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

In the last two weeks we had perma test failures twice for at least two days. This was because code in Firefox or the test harness has been changed. In both cases we didn't backout the causing patch, which actually added a big burden on the sheriffs to have to star each changeset for that particular failure.

I would say it is time to actually bump most of our fx-ui tests to Tier-1, so we have a clear backout strategy in case of regressions. So I will try to come up with a patch today which will divide our tests into two chunks. One for local test data and another one for remote (like SSL tests).

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Firefox ui tests with local test data are now reporting as tier-1. The check-in for the patch on bug 1283596 was delayed due to issue on autoland.

Everything we needed for this bug is done.