1237550 - [16Q1] Execute firefox-ui-tests in TaskCluster on Linux and run those tests for each check-in on integration branches and try

Status: RESOLVED FIXED

(Testing :: Firefox UI Tests, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This quarter we want to get our firefox-ui-tests executed in TaskCluster. For now they only support Linux, so we get started with this platform. Adding more platforms will be done once support gets added.

Also the execution of tests has to happen for each check-in across our integration branches and try

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We might be blocked on bug 1246283 here. So adding it as dependency for now.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I got my first firefox-ui-tests executed by TaskCluster for non-e10s and e10s mode. Both tasks passed without a failure:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=abef13e53bd7&filter-tier=1&filter-tier=2&filter-tier=3

I have to make tweaks to the group and job names so we do not end-up with such ultra-long descriptions in Treeherder. Also e10s tests are not correctly put into their own group yet. That's behavior of Treeherder if you only update the group symbol but not the group description. I will work with Cameron to get this fixed.

I might have a patch later today.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: MozReview Request: Bug 1237550 - Add Taskcluster tasks for Firefox UI functional tests. r?ahal

Review commit: https://reviewboard.mozilla.org/r/36841/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/36841/

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

There is already a successful try run here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a5b2a748fc3c&filter-tier=1&filter-tier=2&filter-tier=3

The one I started via mozreview is just for sanity.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Note to myself: Once the patch is in the tree I will also have to update http://trychooser.pub.build.mozilla.org/ to get our tests listed.

Comment 6

[Andrew Halberstadt [:ahal]]

Comment on attachment 8724073 [details]
MozReview Request: Bug 1237550 - Add Taskcluster tasks for Firefox UI functional tests. r?ahal

https://reviewboard.mozilla.org/r/36841/#review33519

Looks good, just a nit about the symbol being used.

::: testing/taskcluster/tasks/tests/fx_linux64_firefox_ui.yml:25
(Diff revision 1)
> +      groupSymbol: tc-Fxfn
> +      symbol: en-US

I don't think this should be 'en-US'. It should be a unique symbol that identifies the job.

What about something like:
groupName: tc-FxUI
symbol: fn

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So we run tests for a lot of locales in case of nightly and release builds. That's why the job name has to be the locale of the build. If we would introduce a new group now, I feel that would cause confusion because there is a disconnect then. Also someone could think `fn` is a locale. Please have a look at this example:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=7557d8e62bd9&filter-searchStr=Firefox%20UI&filter-tier=2&filter-tier=3

Keep in mind that in-tree we can only cover the functional but not the update tests. So I only added tasks for Fxfn to taskcluster.

Comment 8

[Andrew Halberstadt [:ahal]]

Comment on attachment 8724073 [details]
MozReview Request: Bug 1237550 - Add Taskcluster tasks for Firefox UI functional tests. r?ahal

https://reviewboard.mozilla.org/r/36841/#review33557

Ok, those are good arguments.

Comment 9

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/607da9c6ab37

Comment 10

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/607da9c6ab37

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Missed to mark it as keep-open. I still have to add the tests to the try choose webpage.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Actually there is a mistake in the landed patch. I'm using firefox-ui as name of the test jobs which would not scale. This should have been firefox-ui-functional instead. I noticed that while trying to update the trychooser web page and thinking about other fx ui test types. I will come up with a follow-up patch.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

All dependencies have been fixed. So this bug is done now.