Last Comment Bug 1272228 - [16Q2] Make Firefox-ui-tests results from TaskCluster Tier-1 or Tier-2 on Treeherder
: [16Q2] Make Firefox-ui-tests results from TaskCluster Tier-1 or Tier-2 on Tre...
Status: RESOLVED FIXED
:
Product: Testing
Classification: Components
Component: Firefox UI Tests (show other bugs)
: 49 Branch
: Unspecified Unspecified
-- normal (vote)
: ---
Assigned To: Henrik Skupin (:whimboo) [away 02/18 - 02/27]
: Henrik Skupin (:whimboo) [away 02/18 - 02/27]
:
Mentors:
Depends on: 1272255 1275249 1276175 1277055 1277209 1283596
Blocks:
  Show dependency treegraph
 
Reported: 2016-05-12 00:35 PDT by Henrik Skupin (:whimboo) [away 02/18 - 02/27]
Modified: 2016-07-05 08:01 PDT (History)
8 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-05-12 00:35:25 PDT
As of now the firefox-ui-test results are reported as tier-3 level to Treeherder, which means they are hidden by default and no-one takes care of them. After being a month away in April, most of it was busted beginning of May. It's something which we do not want anymore. So we would like to bump the level to Tier-1 and/or Tier-2 - all dependent on the remote test data usage (eg. different SSL subdomains on mozqa.com).

As I found on the Wiki here the requirements for both levels:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#Requirements_for_jobs_shown_in_the_default_Treeherder_view

The following points are stuff we have to add or at least document somewhere:

Necessary documentation outstanding and to be added to https://developer.mozilla.org/en-US/docs/Mozilla/QA/firefox-ui-tests:
* How to disable an individual failing test
* The current owner/who to contact for help
* The Bugzilla product/component where bugs should be filed

Here a special Tier-1 requirement:
* "Ensure MOZ_DISABLE_NONLOCAL_CONNECTIONS is defined in the automation environment" (see bug 995417). This is interesting in the case of Marionette, because it seems that its not getting used at all in that test framework. I have to do more investigation today and raise a bug if that is really true.
Comment 1 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-05-12 00:44:01 PDT
Just to add, this only applies to our tests as executed via Taskcluster but not mozmill-ci. What we need is better focus of developers on inbound and other integration branches but not nightly or release builds.
Comment 2 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-05-19 05:58:16 PDT
(In reply to Henrik Skupin (:whimboo) from comment #0)
> * How to disable an individual failing test

This has been added under https://developer.mozilla.org/en-US/docs/Mozilla/QA/Firefox_UI_tests#How_to_disable_failing_tests.

> * The current owner/who to contact for help
> * The Bugzilla product/component where bugs should be filed

I'm not sure where to put this content. I cannot see similar information for other test suites on MDN. Is there a special page for sheriffs where this has to be added to?

> Here a special Tier-1 requirement:
> * "Ensure MOZ_DISABLE_NONLOCAL_CONNECTIONS is defined in the automation
> environment" (see bug 995417). This is interesting in the case of
> Marionette, because it seems that its not getting used at all in that test
> framework. I have to do more investigation today and raise a bug if that is
> really true.

This is actually bug 1272255 now and all Marionette related test suites which report to Treeherder as Tier-1 are affected. So I don't think it should be a showstopper for us now.
Comment 3 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-05-20 09:34:29 PDT
I updated our project page for firefox-ui-tests (https://wiki.mozilla.org/Auto-tools/Projects/Firefox_UI_Tests) in regards of contacts and the primary owner. It also lists the Bugzilla component for bugs to be filed. 

Ryan, not sure if any sheriff right now could look at this, or if maybe you could have a quick look at the following two documents. Is that enough to become Tier-2?

https://wiki.mozilla.org/Auto-tools/Projects/Firefox_UI_Tests
https://developer.mozilla.org/en-US/docs/Mozilla/QA/Firefox_UI_tests
Comment 4 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-05-24 08:46:05 PDT
Talked to Carsten on IRC and putting ni? on the sheriff alias might not work. So adjusting it to Carsten and Wes directly.
Comment 5 User image Carsten Book [:Tomcat] 2016-05-25 06:46:09 PDT
Henrik do you know if most tests pass currently like don't make sense to have a tier 2 component that has tests that are perma failure etc
Comment 6 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-05-25 07:35:40 PDT
(In reply to Carsten Book [:Tomcat] from comment #5)
> Henrik do you know if most tests pass currently like don't make sense to
> have a tier 2 component that has tests that are perma failure etc

Our tests are perma green except if developers land changes which cause complete bustage for us as what happened yesterday. Also there is an orange crasher on and off but that's all. Here you can see a list for inbound:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=Firefox%20UI&filter-tier=2&filter-tier=3

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=Firefox%20UI&filter-tier=2&filter-tier=3

In terms of mozilla-central we only speak about the tc-Fxfn jobs as noted in the bug summary, so listed test failures for Nightly builds don't play in here, and is another bug.

My open question is still about all the other documentation related items.
Comment 7 User image Carsten Book [:Tomcat] 2016-05-27 04:26:01 PDT
i'm ok with having this as tier 2 - ryan since you did this i guess before is there any more "paperwork" needed regarding the "documentation related items"
Comment 8 User image Ryan VanderMeulen [:RyanVM] 2016-05-31 12:14:07 PDT
I'm OK with the wiki and MDN pages.
Comment 9 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-05-31 12:50:52 PDT
Thanks to you both! I'm going to file a new bug for bumping the tier level to 2 for now. When we have this running for a while, I will check again to make the local tests tier 1.
Comment 10 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-06-29 00:21:05 PDT
In the last two weeks we had perma test failures twice for at least two days. This was because code in Firefox or the test harness has been changed. In both cases we didn't backout the causing patch, which actually added a big burden on the sheriffs to have to star each changeset for that particular failure.

I would say it is time to actually bump most of our fx-ui tests to Tier-1, so we have a clear backout strategy in case of regressions. So I will try to come up with a patch today which will divide our tests into two chunks. One for local test data and another one for remote (like SSL tests).
Comment 11 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-07-05 08:01:38 PDT
Firefox ui tests with local test data are now reporting as tier-1. The check-in for the patch on bug 1283596 was delayed due to issue on autoland.

Everything we needed for this bug is done.

Note You need to log in before you can comment on or make changes to this bug.