Bump tier level of Firefox UI tests with local test data on Treeherder to Tier-1

RESOLVED FIXED in Firefox 50

Status

Testing
Firefox UI Tests
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: whimboo, Assigned: whimboo)

Tracking

49 Branch
mozilla50
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox50 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(2 attachments)

(Assignee)

Description

2 years ago
As discussed on bug 1272228 we want to bump the Firefox UI test results to tier level 1 for the tests with local test data only. The tests with remote test data like lots of the security and safebrowsing tests will remain at tier-2 level.
(Assignee)

Comment 1

2 years ago
Created attachment 8766927 [details]
Bug 1283596 - Add --tag option to firefox-ui-tests mozharness scripts.

Review commit: https://reviewboard.mozilla.org/r/61646/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/61646/
(Assignee)

Comment 2

2 years ago
Created attachment 8766928 [details]
Bug 1283596 - Update firefox-ui-tests task template files for local and remote tests.

Review commit: https://reviewboard.mozilla.org/r/61648/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/61648/
(Assignee)

Comment 3

2 years ago
Comment on attachment 8766927 [details]
Bug 1283596 - Add --tag option to firefox-ui-tests mozharness scripts.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61646/diff/1-2/
Attachment #8766927 - Attachment description: Bug 1283596 - Add --tag option to firefox-ui-tests mozharness scripts → Bug 1283596 - Add --tag option to firefox-ui-tests mozharness scripts.
Attachment #8766927 - Flags: review?(dburns)
Attachment #8766928 - Flags: review?(garndt)
(Assignee)

Comment 4

2 years ago
Comment on attachment 8766928 [details]
Bug 1283596 - Update firefox-ui-tests task template files for local and remote tests.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61648/diff/1-2/
(Assignee)

Comment 5

2 years ago
https://reviewboard.mozilla.org/r/61648/#review58488

::: taskcluster/ci/legacy/tasks/tests/fx_linux64_fxui_functional_local.yml:16
(Diff revision 2)
> +  extra:
> +    suite:
> +      name: firefox-ui
> +      flavor: functional local
> +    treeherder:
> +      symbol: l-en-US

Given that both types will definitely stay in different tiers I wonder if it would make more sense to use a different group symbol here.

Right now we have:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e6aad7127de7&selectedJob=23185271

It might be confusing to see two entries for tc-Fxfn() with different tiers. Maybe the following might be better?

tc-Fxfn-l(en-US) and tc-Fxfn-r[Tier-2](en-US)
Comment on attachment 8766927 [details]
Bug 1283596 - Add --tag option to firefox-ui-tests mozharness scripts.

https://reviewboard.mozilla.org/r/61646/#review58490
Attachment #8766927 - Flags: review?(dburns) → review+

Updated

2 years ago
Attachment #8766928 - Flags: review?(garndt) → review+

Comment 7

2 years ago
Comment on attachment 8766928 [details]
Bug 1283596 - Update firefox-ui-tests task template files for local and remote tests.

https://reviewboard.mozilla.org/r/61648/#review58524

I don't see any obvious problems with this, so if the task graph renders how you are expecting this should be good

Comment 8

2 years ago
(In reply to Henrik Skupin (:whimboo) from comment #5)
> https://reviewboard.mozilla.org/r/61648/#review58488
> 
> ::: taskcluster/ci/legacy/tasks/tests/fx_linux64_fxui_functional_local.yml:16
> (Diff revision 2)
> > +  extra:
> > +    suite:
> > +      name: firefox-ui
> > +      flavor: functional local
> > +    treeherder:
> > +      symbol: l-en-US
> 
> Given that both types will definitely stay in different tiers I wonder if it
> would make more sense to use a different group symbol here.
> 
> Right now we have:
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=e6aad7127de7&selectedJob=23185271
> 
> It might be confusing to see two entries for tc-Fxfn() with different tiers.
> Maybe the following might be better?
> 
> tc-Fxfn-l(en-US) and tc-Fxfn-r[Tier-2](en-US)

I do think that differentiating between local and remote tests would be beneficial.  Especially if one fails and the other doesn't.  Also I think most will see both jobs because treeherder defaults to displaying both tier 1 and 2 jobs.
(Assignee)

Comment 9

2 years ago
Carsten, mind giving me feedback from a sheriffs standpoint? Which group/job naming would be better for you?
Flags: needinfo?(cbook)
(Assignee)

Updated

2 years ago
Blocks: 1283791
(In reply to Henrik Skupin (:whimboo) from comment #9)
> Carsten, mind giving me feedback from a sheriffs standpoint? Which group/job
> naming would be better for you?

i agree with garndt here:

I do think that differentiating between local and remote tests would be beneficial.  Especially if one fails and the other doesn't. 

There is tier-2 mentioned in treeherder but i guess with the same name and different tiers would be for sheriffs and developers confusing :)
Flags: needinfo?(cbook)
(Assignee)

Comment 11

2 years ago
Comment on attachment 8766928 [details]
Bug 1283596 - Update firefox-ui-tests task template files for local and remote tests.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61648/diff/2-3/
(Assignee)

Comment 12

2 years ago
https://reviewboard.mozilla.org/r/61648/#review58488

I have made the appropriate changes in my updated commit.
(Assignee)

Comment 13

2 years ago
The updated patch looks great and is what Carsten wanted to see:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=674379133ee2

I'm going to land this patch now. CC'ing Dustin given that he is doing all this work on the task kind stuff.
(Assignee)

Comment 14

2 years ago
Comment on attachment 8766927 [details]
Bug 1283596 - Add --tag option to firefox-ui-tests mozharness scripts.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61646/diff/2-3/
(Assignee)

Comment 15

2 years ago
Comment on attachment 8766928 [details]
Bug 1283596 - Update firefox-ui-tests task template files for local and remote tests.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61648/diff/3-4/

Comment 16

2 years ago
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7b33d4c571ee
Add --tag option to firefox-ui-tests mozharness scripts. r=automatedtester
https://hg.mozilla.org/integration/autoland/rev/aaf789f8f366
Update firefox-ui-tests task template files for local and remote tests. r=garndt
(Assignee)

Comment 17

2 years ago
Both commits got backed out due to syntax error:
https://hg.mozilla.org/integration/autoland/rev/0ed118774084

Something went wrong with the rebase and I simply missed a comma. I will update the patch and for safety run a single try test again.
(Assignee)

Comment 18

2 years ago
Comment on attachment 8766927 [details]
Bug 1283596 - Add --tag option to firefox-ui-tests mozharness scripts.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61646/diff/3-4/
(Assignee)

Comment 19

2 years ago
Comment on attachment 8766928 [details]
Bug 1283596 - Update firefox-ui-tests task template files for local and remote tests.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61648/diff/4-5/

Comment 20

2 years ago
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a6a81552d25b
Add --tag option to firefox-ui-tests mozharness scripts. r=automatedtester
https://hg.mozilla.org/integration/autoland/rev/31c0f3a2f81c
Update firefox-ui-tests task template files for local and remote tests. r=garndt

Comment 21

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/a6a81552d25b
https://hg.mozilla.org/mozilla-central/rev/31c0f3a2f81c
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
status-firefox50: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla50
(Assignee)

Updated

2 years ago
Blocks: 1284889
You need to log in before you can comment on or make changes to this bug.