Last Comment Bug 1237550 - [16Q1] Execute firefox-ui-tests in TaskCluster on Linux and run those tests for each check-in on integration branches and try
: [16Q1] Execute firefox-ui-tests in TaskCluster on Linux and run those tests f...
Status: RESOLVED FIXED
: meta
Product: Testing
Classification: Components
Component: Firefox UI Tests (show other bugs)
: unspecified
: Unspecified Unspecified
-- normal (vote)
: mozilla47
Assigned To: Henrik Skupin (:whimboo) [away 02/18 - 02/27]
: Henrik Skupin (:whimboo) [away 02/18 - 02/27]
:
Mentors:
Depends on: 1246283 1252229 1252242
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-07 02:24 PST by Henrik Skupin (:whimboo) [away 02/18 - 02/27]
Modified: 2016-03-03 08:50 PST (History)
5 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
affected
affected
fixed
affected

MozReview Requests
Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:
Show discarded requests

Attachments
MozReview Request: Bug 1237550 - Add Taskcluster tasks for Firefox UI functional tests. r?ahal (58 bytes, text/x-review-board-request)
2016-02-26 07:04 PST, Henrik Skupin (:whimboo) [away 02/18 - 02/27]
ahalberstadt: review+
Details | Review

Description User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-01-07 02:24:05 PST
This quarter we want to get our firefox-ui-tests executed in TaskCluster. For now they only support Linux, so we get started with this platform. Adding more platforms will be done once support gets added.

Also the execution of tests has to happen for each check-in across our integration branches and try
Comment 1 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-02-05 13:56:52 PST
We might be blocked on bug 1246283 here. So adding it as dependency for now.
Comment 2 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-02-24 02:47:10 PST
I got my first firefox-ui-tests executed by TaskCluster for non-e10s and e10s mode. Both tasks passed without a failure:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=abef13e53bd7&filter-tier=1&filter-tier=2&filter-tier=3

I have to make tweaks to the group and job names so we do not end-up with such ultra-long descriptions in Treeherder. Also e10s tests are not correctly put into their own group yet. That's behavior of Treeherder if you only update the group symbol but not the group description. I will work with Cameron to get this fixed.

I might have a patch later today.
Comment 3 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-02-26 07:04:32 PST
Created attachment 8724073 [details]
MozReview Request: Bug 1237550 - Add Taskcluster tasks for Firefox UI functional tests. r?ahal

Review commit: https://reviewboard.mozilla.org/r/36841/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/36841/
Comment 4 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-02-26 07:08:06 PST
There is already a successful try run here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a5b2a748fc3c&filter-tier=1&filter-tier=2&filter-tier=3

The one I started via mozreview is just for sanity.
Comment 5 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-02-26 08:53:23 PST
Note to myself: Once the patch is in the tree I will also have to update http://trychooser.pub.build.mozilla.org/ to get our tests listed.
Comment 6 User image Andrew Halberstadt [:ahal] 2016-02-26 13:04:08 PST
Comment on attachment 8724073 [details]
MozReview Request: Bug 1237550 - Add Taskcluster tasks for Firefox UI functional tests. r?ahal

https://reviewboard.mozilla.org/r/36841/#review33519

Looks good, just a nit about the symbol being used.

::: testing/taskcluster/tasks/tests/fx_linux64_firefox_ui.yml:25
(Diff revision 1)
> +      groupSymbol: tc-Fxfn
> +      symbol: en-US

I don't think this should be 'en-US'. It should be a unique symbol that identifies the job.

What about something like:
groupName: tc-FxUI
symbol: fn
Comment 7 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-02-26 14:00:56 PST
So we run tests for a lot of locales in case of nightly and release builds. That's why the job name has to be the locale of the build. If we would introduce a new group now, I feel that would cause confusion because there is a disconnect then. Also someone could think `fn` is a locale. Please have a look at this example:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=7557d8e62bd9&filter-searchStr=Firefox%20UI&filter-tier=2&filter-tier=3

Keep in mind that in-tree we can only cover the functional but not the update tests. So I only added tasks for Fxfn to taskcluster.
Comment 8 User image Andrew Halberstadt [:ahal] 2016-02-26 14:39:34 PST
Comment on attachment 8724073 [details]
MozReview Request: Bug 1237550 - Add Taskcluster tasks for Firefox UI functional tests. r?ahal

https://reviewboard.mozilla.org/r/36841/#review33557

Ok, those are good arguments.
Comment 10 User image Carsten Book [:Tomcat] 2016-02-29 02:39:21 PST
https://hg.mozilla.org/mozilla-central/rev/607da9c6ab37
Comment 11 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-02-29 02:50:34 PST
Missed to mark it as keep-open. I still have to add the tests to the try choose webpage.
Comment 12 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-02-29 12:58:51 PST
Actually there is a mistake in the landed patch. I'm using firefox-ui as name of the test jobs which would not scale. This should have been firefox-ui-functional instead. I noticed that while trying to update the trychooser web page and thinking about other fx ui test types. I will come up with a follow-up patch.
Comment 13 User image Henrik Skupin (:whimboo) [away 02/18 - 02/27] 2016-03-03 08:50:36 PST
All dependencies have been fixed. So this bug is done now.

Note You need to log in before you can comment on or make changes to this bug.