replace tsvg with tsvgx and tscroll with tscrollx



6 years ago
11 months ago


(Reporter: jmaher, Assigned: jmaher)


(Depends on: 1 bug)

Firefox Tracking Flags

(Not tracked)



(3 attachments)



6 years ago
modifications to these tests have resulted in us creating useful tests, these have been running side by side for a while, time to do the replacement!

Comment 1

6 years ago
this will need to ride the trains.
Assignee: nobody → jmaher
Attachment #779790 - Flags: review?(armenzg)

Comment 2

6 years ago
Comment on attachment 779790 [details] [diff] [review]
switch tsvg/tscroll with 'x' versions for trunk (1.0)

Review of attachment 779790 [details] [diff] [review]:

Attachment #779790 - Flags: review?(armenzg) → review+

Comment 3

6 years ago
Attachment #779798 - Flags: review?(armenzg)


6 years ago
Attachment #779798 - Flags: review?(armenzg) → review+

While I think this replacement is good, I also think we need a wider consensus before deprecating talos tests. I'll post to dev.platform and inform of this coming change, and let's act according to the feedback there.

Meanwhile we can unhide the x-tests on tbpl and make them fully active - if they're not already (e.g. email-report regressions to devs, visible by default on tbpl etc).
Following some reservations about comment 5 from Callek on IRC:

- The main reason for the delay is that "ASAP iterations mode" (which is the main difference the X tests bring) is not fully stable yet (bug 888899 to make it work on OSX, bug 880036 for temporarily hanged Firefox when it can't keep up, bug 884955 to overcome this hang).

- As for my relation to this bug and my request to back it out: I'm the one who noticed that the old tests are not optimal due to their iterations mode, added the X tests (modified versions of the old tests), and suggested to joel on IRC to deprecate the old tests. Unfortunately, it got file and landed before I noticed that it was filed. So I requested to back it out as soon as I could, before this reaches production.

- I still think that exposing this to a wider audience is in place before deprecating talos tests which were our benchmark for a long while, as sub-optimal as they are.
Depends on: 888899, 884955, 880036

Comment 8

6 years ago
Shouldn't this be needed?

+++ b/mozilla/
@@ -267,16 +267,19 @@ PROJECT_BRANCHES = {
     'cedar': {
         'mozharness_tag': 'default',
         'mozharness_talos': True,
         'lock_platforms': True,
         'enable_talos': True,
         'talos_suites': {
             'xperf': 1,
+            'otherx': 1,
+            'svgx': 1,
+            'rafx': 1,

On another note, are we making the replacing *only* on mozilla-central?
Are these new suites only supposed to run on m-c?

Comment 9

6 years ago
this new suites should run on all trunk based branches.  we will roll out to aurora on the next merge and so forth.

Comment 10

6 years ago
Comment on attachment 779790 [details] [diff] [review]
switch tsvg/tscroll with 'x' versions for trunk (1.0)

(In reply to Joel Maher (:jmaher) from comment #9)
> this new suites should run on all trunk based branches.  we will roll out to
> aurora on the next merge and so forth.

If so, the currently r+'ed patch would not do that. Changing to r-.

I've also noticed that we're not disabling them but changing the coalescing preference:
+BRANCHES['mozilla-central']['svgr_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)
+BRANCHES['mozilla-central']['other_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)

This does *not* disable the tests but sets coalescing to False for these jobs.

Sorry. My review was wrong.
Attachment #779790 - Flags: review+ → review-
Depends on: 897415

Comment 11

6 years ago
In production
(In reply to Rail Aliiev [:rail] from comment #11)
> In production

But not really since it was backed out (/me goes to make maintenance page clear)
Before we remove the old tests, we should run the old and new ones in parallel, to be able to assess the effectiveness of the new ones.

While I can see the new tests on desktop platforms (tscrollx, tsvgx at T(T) when ?showall=1), On mobile platforms I can see the new tsvgx_nochrome (also T(T)) but not a variant of the new scroll test, and also not the old tscroll.

Can we turn on both the old and the new tscroll tests on mobile before we deprecate the old ones?

Comment 14

6 years ago
tscroll is not supported on mobile.  So we have a few weeks of side by side data as it stands.
Bug 888899 part 2 - which is in inbound but haven't landed yet - changes the prefs which are used to set ASAP mode (which tsvgx and tscrollx use).

If the prefs which are used for tsvgx and tscrollx don't match, then they will give different results.

This is expected, and is the main reason for the delay in deprecating the old tests.
Attachment #785205 - Flags: review?(jmaher)


6 years ago
Attachment #785205 - Flags: review?(jmaher) → review+

Comment 16

6 years ago
landed on talos:

This needs a talos deployment before it is resolved.


6 years ago
Blocks: 900913
Depends on: 854746, 845943
Product: → Release Engineering
Update from an email thread with clint, taras, joel and myself about a week ago: we decided to land the new tests fully production on all trees, and keep the old tests for 3 weeks on m-c only and hidden.

Right now it's still the other way around (old tests fully production, new tests only on m-c and hidden).
Updated the Talos wiki sections of tsvg/x, tscroll/x and tsvg_opacity (tsvg_opacity included reference to ASAP even though it wasn't touched and doesn't use ASAP mode). Would appreciate some review/English-editing.
Flags: needinfo?(mbrubeck)
Flags: needinfo?(jmaher)

Comment 19

6 years ago
I have done a few minor edits as well as adding a section for tart:
Flags: needinfo?(jmaher)
Looks good.  Thanks!
Flags: needinfo?(mbrubeck)


5 years ago
Last Resolved: 5 years ago
Resolution: --- → FIXED
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.