897054 - replace tsvg with tsvgx and tscroll with tscrollx

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

modifications to these tests have resulted in us creating useful tests, these have been running side by side for a while, time to do the replacement!

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: switch tsvg/tscroll with 'x' versions for trunk (1.0)

this will need to ride the trains.

Comment 2

[Armen [:armenzg]]

Comment on attachment 779790 [details] [diff] [review]
switch tsvg/tscroll with 'x' versions for trunk (1.0)

Review of attachment 779790 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: and for the mobile tests (1.0)

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

landed, will wait for the reconfig tomorrow:
https://hg.mozilla.org/build/buildbot-configs/rev/5c56fd7fe000
https://hg.mozilla.org/build/buildbot-configs/rev/970e83957e9f

Comment 5

[Avi Halachmi (:avih)]

STOP THE PRESS!

While I think this replacement is good, I also think we need a wider consensus before deprecating talos tests. I'll post to dev.platform and inform of this coming change, and let's act according to the feedback there.

Meanwhile we can unhide the x-tests on tbpl and make them fully active - if they're not already (e.g. email-report regressions to devs, visible by default on tbpl etc).

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

backed out due to bug 888899 being backed out and we need this fixed on osx first:
https://hg.mozilla.org/build/buildbot-configs/rev/e916c85b249e
https://hg.mozilla.org/build/buildbot-configs/rev/7fae5a3a155f

Comment 7

[Avi Halachmi (:avih)]

Following some reservations about comment 5 from Callek on IRC:

- The main reason for the delay is that "ASAP iterations mode" (which is the main difference the X tests bring) is not fully stable yet (bug 888899 to make it work on OSX, bug 880036 for temporarily hanged Firefox when it can't keep up, bug 884955 to overcome this hang).

- As for my relation to this bug and my request to back it out: I'm the one who noticed that the old tests are not optimal due to their iterations mode, added the X tests (modified versions of the old tests), and suggested to joel on IRC to deprecate the old tests. Unfortunately, it got file and landed before I noticed that it was filed. So I requested to back it out as soon as I could, before this reaches production.

- I still think that exposing this to a wider audience is in place before deprecating talos tests which were our benchmark for a long while, as sub-optimal as they are.

Comment 8

[Armen [:armenzg]]

Shouldn't this be needed?

+++ b/mozilla/project_branches.py
@@ -267,16 +267,19 @@ PROJECT_BRANCHES = {
     },
     'cedar': {
         'mozharness_tag': 'default',
         'mozharness_talos': True,
         'lock_platforms': True,
         'enable_talos': True,
         'talos_suites': {
             'xperf': 1,
+            'otherx': 1,
+            'svgx': 1,
+            'rafx': 1,
         },

On another note, are we making the replacing *only* on mozilla-central?
Are these new suites only supposed to run on m-c?

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

this new suites should run on all trunk based branches.  we will roll out to aurora on the next merge and so forth.

Comment 10

[Armen [:armenzg]]

Comment on attachment 779790 [details] [diff] [review]
switch tsvg/tscroll with 'x' versions for trunk (1.0)

(In reply to Joel Maher (:jmaher) from comment #9)
> this new suites should run on all trunk based branches.  we will roll out to
> aurora on the next merge and so forth.

If so, the currently r+'ed patch would not do that. Changing to r-.

I've also noticed that we're not disabling them but changing the coalescing preference:
+BRANCHES['mozilla-central']['svgr_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)
+BRANCHES['mozilla-central']['other_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)

This does *not* disable the tests but sets coalescing to False for these jobs.
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config_common.py#46

Sorry. My review was wrong.

Comment 11

[Rail Aliiev [:rail]]

In production

Comment 12

[Justin Wood (:Callek)]

(In reply to Rail Aliiev [:rail] from comment #11)
> In production

But not really since it was backed out (/me goes to make maintenance page clear)

Comment 13

[Avi Halachmi (:avih)]

Before we remove the old tests, we should run the old and new ones in parallel, to be able to assess the effectiveness of the new ones.

While I can see the new tests on desktop platforms (tscrollx, tsvgx at T(T) when ?showall=1), On mobile platforms I can see the new tsvgx_nochrome (also T(T)) but not a variant of the new scroll test, and also not the old tscroll.

Can we turn on both the old and the new tscroll tests on mobile before we deprecate the old ones?

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

tscroll is not supported on mobile.  So we have a few weeks of side by side data as it stands.

Comment 15

[Avi Halachmi (:avih)]

Attachment: Adapt the ASAP mode prefs to the change in bug 888899 part 2

Bug 888899 part 2 - which is in inbound but haven't landed yet - changes the prefs which are used to set ASAP mode (which tsvgx and tscrollx use).

If the prefs which are used for tsvgx and tscrollx don't match, then they will give different results.

This is expected, and is the main reason for the delay in deprecating the old tests.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

landed on talos:
http://hg.mozilla.org/build/talos/rev/4063ef2a221e

This needs a talos deployment before it is resolved.

Comment 17

[Avi Halachmi (:avih)]

Update from an email thread with clint, taras, joel and myself about a week ago: we decided to land the new tests fully production on all trees, and keep the old tests for 3 weeks on m-c only and hidden.

Right now it's still the other way around (old tests fully production, new tests only on m-c and hidden).

Comment 18

[Avi Halachmi (:avih)]

Updated the Talos wiki sections of tsvg/x, tscroll/x and tsvg_opacity (tsvg_opacity included reference to ASAP even though it wasn't touched and doesn't use ASAP mode). Would appreciate some review/English-editing.

https://wiki.mozilla.org/index.php?title=Buildbot%2FTalos%2FTests&diff=702982&oldid=699505

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

I have done a few minor edits as well as adding a section for tart:
https://wiki.mozilla.org/Buildbot/Talos/Tests

Comment 20

[Matt Brubeck (:mbrubeck)]

Looks good.  Thanks!