replace tsvg with tsvgx and tscroll with tscrollx

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: jmaher, Assigned: jmaher)

Tracking

(Depends on: 2 bugs)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments)

(Assignee)

Description

4 years ago
modifications to these tests have resulted in us creating useful tests, these have been running side by side for a while, time to do the replacement!
(Assignee)

Comment 1

4 years ago
Created attachment 779790 [details] [diff] [review]
switch tsvg/tscroll with 'x' versions for trunk (1.0)

this will need to ride the trains.
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #779790 - Flags: review?(armenzg)

Comment 2

4 years ago
Comment on attachment 779790 [details] [diff] [review]
switch tsvg/tscroll with 'x' versions for trunk (1.0)

Review of attachment 779790 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM.
Attachment #779790 - Flags: review?(armenzg) → review+
(Assignee)

Comment 3

4 years ago
Created attachment 779798 [details] [diff] [review]
and for the mobile tests (1.0)
Attachment #779798 - Flags: review?(armenzg)

Updated

4 years ago
Attachment #779798 - Flags: review?(armenzg) → review+
(Assignee)

Comment 4

4 years ago
landed, will wait for the reconfig tomorrow:
https://hg.mozilla.org/build/buildbot-configs/rev/5c56fd7fe000
https://hg.mozilla.org/build/buildbot-configs/rev/970e83957e9f

Comment 5

4 years ago
STOP THE PRESS!

While I think this replacement is good, I also think we need a wider consensus before deprecating talos tests. I'll post to dev.platform and inform of this coming change, and let's act according to the feedback there.

Meanwhile we can unhide the x-tests on tbpl and make them fully active - if they're not already (e.g. email-report regressions to devs, visible by default on tbpl etc).
(Assignee)

Comment 6

4 years ago
backed out due to bug 888899 being backed out and we need this fixed on osx first:
https://hg.mozilla.org/build/buildbot-configs/rev/e916c85b249e
https://hg.mozilla.org/build/buildbot-configs/rev/7fae5a3a155f

Comment 7

4 years ago
Following some reservations about comment 5 from Callek on IRC:

- The main reason for the delay is that "ASAP iterations mode" (which is the main difference the X tests bring) is not fully stable yet (bug 888899 to make it work on OSX, bug 880036 for temporarily hanged Firefox when it can't keep up, bug 884955 to overcome this hang).

- As for my relation to this bug and my request to back it out: I'm the one who noticed that the old tests are not optimal due to their iterations mode, added the X tests (modified versions of the old tests), and suggested to joel on IRC to deprecate the old tests. Unfortunately, it got file and landed before I noticed that it was filed. So I requested to back it out as soon as I could, before this reaches production.

- I still think that exposing this to a wider audience is in place before deprecating talos tests which were our benchmark for a long while, as sub-optimal as they are.
Depends on: 888899, 884955, 880036

Comment 8

4 years ago
Shouldn't this be needed?

+++ b/mozilla/project_branches.py
@@ -267,16 +267,19 @@ PROJECT_BRANCHES = {
     },
     'cedar': {
         'mozharness_tag': 'default',
         'mozharness_talos': True,
         'lock_platforms': True,
         'enable_talos': True,
         'talos_suites': {
             'xperf': 1,
+            'otherx': 1,
+            'svgx': 1,
+            'rafx': 1,
         },

On another note, are we making the replacing *only* on mozilla-central?
Are these new suites only supposed to run on m-c?
(Assignee)

Comment 9

4 years ago
this new suites should run on all trunk based branches.  we will roll out to aurora on the next merge and so forth.

Comment 10

4 years ago
Comment on attachment 779790 [details] [diff] [review]
switch tsvg/tscroll with 'x' versions for trunk (1.0)

(In reply to Joel Maher (:jmaher) from comment #9)
> this new suites should run on all trunk based branches.  we will roll out to
> aurora on the next merge and so forth.

If so, the currently r+'ed patch would not do that. Changing to r-.

I've also noticed that we're not disabling them but changing the coalescing preference:
+BRANCHES['mozilla-central']['svgr_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)
+BRANCHES['mozilla-central']['other_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)

This does *not* disable the tests but sets coalescing to False for these jobs.
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config_common.py#46

Sorry. My review was wrong.
Attachment #779790 - Flags: review+ → review-

Updated

4 years ago
Depends on: 897415
In production
(In reply to Rail Aliiev [:rail] from comment #11)
> In production

But not really since it was backed out (/me goes to make maintenance page clear)
Before we remove the old tests, we should run the old and new ones in parallel, to be able to assess the effectiveness of the new ones.

While I can see the new tests on desktop platforms (tscrollx, tsvgx at T(T) when ?showall=1), On mobile platforms I can see the new tsvgx_nochrome (also T(T)) but not a variant of the new scroll test, and also not the old tscroll.

Can we turn on both the old and the new tscroll tests on mobile before we deprecate the old ones?
(Assignee)

Comment 14

4 years ago
tscroll is not supported on mobile.  So we have a few weeks of side by side data as it stands.
Created attachment 785205 [details] [diff] [review]
Adapt the ASAP mode prefs to the change in bug 888899 part 2

Bug 888899 part 2 - which is in inbound but haven't landed yet - changes the prefs which are used to set ASAP mode (which tsvgx and tscrollx use).

If the prefs which are used for tsvgx and tscrollx don't match, then they will give different results.

This is expected, and is the main reason for the delay in deprecating the old tests.
Attachment #785205 - Flags: review?(jmaher)
(Assignee)

Updated

4 years ago
Attachment #785205 - Flags: review?(jmaher) → review+
(Assignee)

Comment 16

4 years ago
landed on talos:
http://hg.mozilla.org/build/talos/rev/4063ef2a221e

This needs a talos deployment before it is resolved.
(Assignee)

Updated

4 years ago
Blocks: 900913

Updated

4 years ago
Depends on: 854746, 845943
Product: mozilla.org → Release Engineering
Update from an email thread with clint, taras, joel and myself about a week ago: we decided to land the new tests fully production on all trees, and keep the old tests for 3 weeks on m-c only and hidden.

Right now it's still the other way around (old tests fully production, new tests only on m-c and hidden).
Updated the Talos wiki sections of tsvg/x, tscroll/x and tsvg_opacity (tsvg_opacity included reference to ASAP even though it wasn't touched and doesn't use ASAP mode). Would appreciate some review/English-editing.

https://wiki.mozilla.org/index.php?title=Buildbot%2FTalos%2FTests&diff=702982&oldid=699505
Flags: needinfo?(mbrubeck)
Flags: needinfo?(jmaher)
(Assignee)

Comment 19

4 years ago
I have done a few minor edits as well as adding a section for tart:
https://wiki.mozilla.org/Buildbot/Talos/Tests
Flags: needinfo?(jmaher)
Looks good.  Thanks!
Flags: needinfo?(mbrubeck)
(Assignee)

Updated

4 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.