689649 - update buildbot config.py to adjust side by side talos staging for mozafterpaint

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: adjust side by side staging from central to beta (1.0)

now that we have been running side by side for a week we have a data point to look back on.

we did this on mozilla-central, and with the central->aurora merge, we now can just run the mozafterpaint tests on central & aurora without the old tests.

In addition we should run side by side on beta for a week, then turn it off.  That will only leave mozilla-release and shadow-central.

Comment 1

[Armen [:armenzg]]

Comment on attachment 562806 [details] [diff] [review]
adjust side by side staging from central to beta (1.0)

Review of attachment 562806 [details] [diff] [review]:
-----------------------------------------------------------------

I thought you wanted to do the switchover for all branches.

We can go this way as well.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: adjust side by side staging from central to beta and turn on for all others (1.0)

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: adjust side by side staging from central to beta and turn on for all others (1.1)

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: adjust side by side staging from central to beta and turn on for all others (2.0)

ok, since this is on for all, we need to explicitly turn it off for 1.9.2 and mozilla-release

Comment 5

[Armen [:armenzg]]

Attachment: adjust side by side staging from central to beta and turn on for all others (2.1)

I have done a comparison builder by builder and made this patch.

* We only use "OLD_BRANCH_" for 1.9.2 so I removed its use for m-a and m-b.
* I have added a talos exception on a11y.
* I moved chrome_mac so the sorting of the builders is the same across branches

Did you get the compare-talos patch sorted out? [1]

I am also not sure if we determined in our call if there were missing suites when comparing two changes from tbpl [2].

When would we be ready to land this?

[1] https://bitbucket.org/mconnor/compare-talos/changeset/db884c7f8c25
[2] http://perf.snarkfest.net/compare-talos/index.html?oldRevs=704f37801611&newRev=265d39da5c3d&tests=%20a11y,tdhtml,tdhtml_nochrome,tp4,tp4_memset,tp4_pbytes,tp4_rss,tp4_shutdown,tp4_xres,tp5,tp5_memset,tp5_pbytes,tp5_rss,tp5_shutdown,tp5_xres,dromaeo_basics,dromaeo_css,dromaeo_dom,dromaeo_jslib,dromaeo_sunspider,dromaeo_v8,tsspider,tsspider_nochrome,tgfx,tgfx_nochrome,tscroll,tsvg,tsvg_opacity,ts,ts_cold,ts_cold_generated_max,ts_cold_generated_max_shutdown,ts_cold_generated_med,ts_cold_generated_med_shutdown,ts_cold_generated_min,ts_cold_generated_min_shutdown,ts_cold_shutdown,ts_places_generated_max,ts_places_generated_max_shutdown,ts_places_generated_med,ts_places_generated_med_shutdown,ts_places_generated_min,ts_places_generated_min_shutdown,ts_shutdown,twinopen&submit=true

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: compare-talos update for all paint tests (1.0)

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 564250 [details] [diff] [review]
adjust side by side staging from central to beta and turn on for all others (2.1)

Review of attachment 564250 [details] [diff] [review]:
-----------------------------------------------------------------

this looks like it captures everything!

Comment 8

[Armen [:armenzg]]

Comment on attachment 564314 [details] [diff] [review]
compare-talos update for all paint tests (1.0)

Mike does this patch look good to you? If it is can you pull it into perf.snarkfest.net? Thanks!

Comment 9

[Armen [:armenzg]]

Comment on attachment 564250 [details] [diff] [review]
adjust side by side staging from central to beta and turn on for all others (2.1)

http://hg.mozilla.org/build/buildbot-configs/rev/d24187f91b51

This should go in with the next reconfig.
Probably tomorrow.

Comment 10

[Armen [:armenzg]]

We're all done here, right?

This went live on 10/06.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

well, we need to turn off side by side on m-b and make this the default for all but 1.9.2.

Comment 12

[Armen [:armenzg]]

Attachment: disable old_ suites for mozilla-beta

The newer ones get added by default so there is no need to explicitly show that we are adding them.

I also did some clean up.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 567790 [details] [diff] [review]
disable old_ suites for mozilla-beta

Review of attachment 567790 [details] [diff] [review]:
-----------------------------------------------------------------

this looks good to me.

Comment 14

[Armen [:armenzg]]

Comment on attachment 567790 [details] [diff] [review]
disable old_ suites for mozilla-beta

http://hg.mozilla.org/build/buildbot-configs/rev/3f8c229ccb8b

Can we enable mozAfterPaint on mozilla-release? That is the last place were mozAfterPaint has not yet been enabled. The old suite is currently only running for mozilla-release and 1.9.2.

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

Yes, we can do mozilla-release now.  Then we only will have to special case 1.9.2!

Comment 16

[Armen [:armenzg]]

Attachment: remove old_ suites from mozilla-release and add the mozAfterPaint ones

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 568190 [details] [diff] [review]
remove old_ suites from mozilla-release and add the mozAfterPaint ones

Review of attachment 568190 [details] [diff] [review]:
-----------------------------------------------------------------

this looks good.  Do we want to side by side on here, or just want for the next merge.  The more we can get rid of special cases the better :)

Comment 18

[Chris Cooper [:coop] (he/him)]

Comment on attachment 567790 [details] [diff] [review]
disable old_ suites for mozilla-beta

A reconfig that included this happened today.

Comment 19

[Armen [:armenzg]]

Attachment: run side by side mozAfterPaint in mozilla-release

As you mention, instead of replacing one for the other. Let's run them side by side for a little and then disable the old_ ones when I come back.

I carried forward the review and landed it as:
http://hg.mozilla.org/build/buildbot-configs/rev/5f86bf5d0a91

Comment 20

[[e-mail] (:bhearsum)]

This made it to production today.

Comment 21

[Armen [:armenzg]]

Attachment: disable side by side for mozilla-release

When would you like to disable this?
This branch only has activity from 6 weeks to 6 weeks so there has not been any activity to do side by side.
I don't see any reason to not just disable the old stuff now.

Comment 22

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 570846 [details] [diff] [review]
disable side by side for mozilla-release

Review of attachment 570846 [details] [diff] [review]:
-----------------------------------------------------------------

nice!  thanks for remembering this!

Comment 23

[Armen [:armenzg]]

Comment on attachment 570846 [details] [diff] [review]
disable side by side for mozilla-release

http://hg.mozilla.org/build/buildbot-configs/rev/964c08ef08d3

Comment 24

[Armen [:armenzg]]

This landed on production http://hg.mozilla.org/build/buildbot-configs/rev/891e4bd17d10