1283596 - Bump tier level of Firefox UI tests with local test data on Treeherder to Tier-1

Status: RESOLVED FIXED

(Testing :: Firefox UI Tests, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As discussed on bug 1272228 we want to bump the Firefox UI test results to tier level 1 for the tests with local test data only. The tests with remote test data like lots of the security and safebrowsing tests will remain at tier-2 level.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 1283596 - Add --tag option to firefox-ui-tests mozharness scripts.

Review commit: https://reviewboard.mozilla.org/r/61646/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/61646/

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 1283596 - Update firefox-ui-tests task template files for local and remote tests.

Review commit: https://reviewboard.mozilla.org/r/61648/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/61648/

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8766927 [details]
Bug 1283596 - Add --tag option to firefox-ui-tests mozharness scripts.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61646/diff/1-2/

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8766928 [details]
Bug 1283596 - Update firefox-ui-tests task template files for local and remote tests.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61648/diff/1-2/

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

https://reviewboard.mozilla.org/r/61648/#review58488

::: taskcluster/ci/legacy/tasks/tests/fx_linux64_fxui_functional_local.yml:16
(Diff revision 2)
> +  extra:
> +    suite:
> +      name: firefox-ui
> +      flavor: functional local
> +    treeherder:
> +      symbol: l-en-US

Given that both types will definitely stay in different tiers I wonder if it would make more sense to use a different group symbol here.

Right now we have:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e6aad7127de7&selectedJob=23185271

It might be confusing to see two entries for tc-Fxfn() with different tiers. Maybe the following might be better?

tc-Fxfn-l(en-US) and tc-Fxfn-r[Tier-2](en-US)

Comment 6

[David Burns :automatedtester]

Comment on attachment 8766927 [details]
Bug 1283596 - Add --tag option to firefox-ui-tests mozharness scripts.

https://reviewboard.mozilla.org/r/61646/#review58490

Comment 7

[Greg Arndt [:garndt]]

Comment on attachment 8766928 [details]
Bug 1283596 - Update firefox-ui-tests task template files for local and remote tests.

https://reviewboard.mozilla.org/r/61648/#review58524

I don't see any obvious problems with this, so if the task graph renders how you are expecting this should be good

Comment 8

[Greg Arndt [:garndt]]

(In reply to Henrik Skupin (:whimboo) from comment #5)
> https://reviewboard.mozilla.org/r/61648/#review58488
> 
> ::: taskcluster/ci/legacy/tasks/tests/fx_linux64_fxui_functional_local.yml:16
> (Diff revision 2)
> > +  extra:
> > +    suite:
> > +      name: firefox-ui
> > +      flavor: functional local
> > +    treeherder:
> > +      symbol: l-en-US
> 
> Given that both types will definitely stay in different tiers I wonder if it
> would make more sense to use a different group symbol here.
> 
> Right now we have:
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=e6aad7127de7&selectedJob=23185271
> 
> It might be confusing to see two entries for tc-Fxfn() with different tiers.
> Maybe the following might be better?
> 
> tc-Fxfn-l(en-US) and tc-Fxfn-r[Tier-2](en-US)

I do think that differentiating between local and remote tests would be beneficial.  Especially if one fails and the other doesn't.  Also I think most will see both jobs because treeherder defaults to displaying both tier 1 and 2 jobs.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Carsten, mind giving me feedback from a sheriffs standpoint? Which group/job naming would be better for you?

Comment 10

[Carsten Book [:Tomcat]]

(In reply to Henrik Skupin (:whimboo) from comment #9)
> Carsten, mind giving me feedback from a sheriffs standpoint? Which group/job
> naming would be better for you?

i agree with garndt here:

I do think that differentiating between local and remote tests would be beneficial.  Especially if one fails and the other doesn't. 

There is tier-2 mentioned in treeherder but i guess with the same name and different tiers would be for sheriffs and developers confusing :)

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8766928 [details]
Bug 1283596 - Update firefox-ui-tests task template files for local and remote tests.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61648/diff/2-3/

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

https://reviewboard.mozilla.org/r/61648/#review58488

I have made the appropriate changes in my updated commit.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The updated patch looks great and is what Carsten wanted to see:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=674379133ee2

I'm going to land this patch now. CC'ing Dustin given that he is doing all this work on the task kind stuff.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8766927 [details]
Bug 1283596 - Add --tag option to firefox-ui-tests mozharness scripts.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61646/diff/2-3/

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8766928 [details]
Bug 1283596 - Update firefox-ui-tests task template files for local and remote tests.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61648/diff/3-4/

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7b33d4c571ee
Add --tag option to firefox-ui-tests mozharness scripts. r=automatedtester
https://hg.mozilla.org/integration/autoland/rev/aaf789f8f366
Update firefox-ui-tests task template files for local and remote tests. r=garndt

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Both commits got backed out due to syntax error:
https://hg.mozilla.org/integration/autoland/rev/0ed118774084

Something went wrong with the rebase and I simply missed a comma. I will update the patch and for safety run a single try test again.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8766927 [details]
Bug 1283596 - Add --tag option to firefox-ui-tests mozharness scripts.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61646/diff/3-4/

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8766928 [details]
Bug 1283596 - Update firefox-ui-tests task template files for local and remote tests.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61648/diff/4-5/

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a6a81552d25b
Add --tag option to firefox-ui-tests mozharness scripts. r=automatedtester
https://hg.mozilla.org/integration/autoland/rev/31c0f3a2f81c
Update firefox-ui-tests task template files for local and remote tests. r=garndt

Comment 21

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/a6a81552d25b
https://hg.mozilla.org/mozilla-central/rev/31c0f3a2f81c