1373313 - compare the difference between devedition and regular builds on mozilla beta to determine if there is value in running talos on devedition

Status: RESOLVED FIXED

(Testing :: Talos, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we have added devedition builds which run tests on mozilla-beta- I want to determine for each test if the devedition data matches the non-devedition.  I think we have -addons as well.

If there are no differences, I would recommend disabling most if not all talos on duplicate configurations.  Given the volume of traffic on mozilla-beta we would not get alerts frequently (maybe a week later, could be too late...).  Also most if not all changes on beta have been landed on trunk so we will see the perf impact there.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:igoldan, when we are caught up with the main alerts, can you start taking a look at this?

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Sylvestre, one thing that came up while Joel and I were discussing this on IRC was that it appears we aren't currently PGOing DevEd builds. I believe it was accidental but before I proceed with filing a bug for fixing it, I wondered if you think it's even worth fixing? Sort of fits in with the question this bug is asking about DevEd performance.

Comment 3

[Ionuț Goldan [:igoldan]]

Sure, I will look into this. I will also need to ask you some questions on IRC.

Comment 4

[Sylvestre Ledru [:Sylvestre]]

Yeah, sounds like an accident!

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I have compared win7 to win7-devedition and there are a few differences:
nonmain_startup_fileio (devedition ~10% more):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=%5Bmozilla-beta,35af701fe20500913f0664f4959e017e36c660a3,1,1%5D&series=%5Bmozilla-beta,15b471bf178f0ca6afd84fb1fcfdc2dcc6880376,1,1%5D&zoom=1497228369553.178,1498722644000,14404.495182108789,26464.42027574175

main_startup_fileio (devedition <1% more- 500K):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=%5Bmozilla-beta,43ee725bb849b5e9bae46dae4d9956b53be0f054,1,1%5D&series=%5Bmozilla-beta,ac555d733ad90cd22e1fcc0b280b9d8ef3979d0b,1,1%5D&zoom=1497239395610.4531,1498722644000,70016854.10403134,71870786.688301

tsvg_static (devedition ~3% more):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=%5Bmozilla-beta,90dd115bdc48b6c6a358e6178f59990ea725ada9,1,1%5D&series=%5Bmozilla-beta,18cf40355e5b20164ab9307f83dd6d6eb6184aa8,1,1%5D&zoom=1497230062377.0457,1498722644000,59.17303386788243,64.62621738848168

cart (devedition ~2% more):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=%5Bmozilla-beta,9ce90fd3d84313259fa7090356c94522e0b386bb,1,1%5D&series=%5Bmozilla-beta,527871f319c65cf49b56a94a32fe5529bc3f2090,1,1%5D&zoom=1497233284556.3047,1498722644000,24.040449506781073,26.085393327005793

damp (devedition ~10% more):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=%5Bmozilla-beta,a4923de6da4d68fca9e4ba833355566244bad320,1,1%5D&series=%5Bmozilla-beta,1f8aed46624975327d00a6b936cee2c09768ef5d,1,1%5D

I can believe the damp changes given that devedition includes devtools and that would also account for the startup_fileio changes.

cart and tsvg_static are confusing, but minor.

We still have OSX to compare.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

for osx, here are the differences:
dromaeo_css (devedition about 2.5% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=%5Bmozilla-beta,3d2817f9ef3348e4f080c8b381b8fd9c4597bb16,1,1%5D&series=%5Bmozilla-beta,f7f5af64ddd1d28c773c718db6280ec3957fb3af,1,1%5D

damp (devedition about 2.5% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bmozilla-beta,be31b0524e6866c9e2c1ec439455142a91c9b409,1,1%5D&series=%5Bmozilla-beta,9663a154caf9b713fb951c8db24518c7e0ceddce,1,1%5D

tps (devedition about 2% faster):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bmozilla-beta,dc74ee20b2691ca17db67b12285e9ab19521581f,1,1%5D&series=%5Bmozilla-beta,d96c356354e1d01835f87142841749ee6d353184,1,1%5D

tp5o_webext main_rss (devedition about 10% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bmozilla-beta,3cce4ecc24953092a1726a6872dd350f69c31586,1,1%5D&series=%5Bmozilla-beta,3bed54700b8567df8ddc5ec701c764f89098c513,1,1%5D

tp5o_webext (devedition about 4% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bmozilla-beta,1dde7885edeab0dfa1e149dd1792c881d19d8d23,1,1%5D&series=%5Bmozilla-beta,0b8742886c4c6d9e9a7557eba42b88606f02417f,1,1%5D

ts_paint_webext (devedition about 6% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bmozilla-beta,ebfbddf3f798db29458e520fa3f69ae10083289b,1,1%5D&series=%5Bmozilla-beta,43b86e9d30eb1c24ef91a2f7f3610cb4ac75c647,1,1%5D

a11y (devedition about 4% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bmozilla-beta,a5b2f9afcf3b75a777907eb979dbab377acd16c6,1,1%5D&series=%5Bmozilla-beta,ed1f2229fdc30cdf9e79fe1440e6fda9cd277d21,1,1%5D

sessionrestore (devedition about 4% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bmozilla-beta,e8c56d3e301de5f1b1f690d62f6e3ec483df2f8a,1,1%5D&series=%5Bmozilla-beta,c22155824871783bb5d8f514b3a01b4c885034c0,1,1%5D

sessionrestore_no_auto_restore (devedition about 4% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bmozilla-beta,aaffa375582762ffaede6416cf855dbe870bf2b0,1,1%5D&series=%5Bmozilla-beta,e9484a55bdf595b7b86aea582bb9016cdd1ba956,1,1%5D

tpaint (devedition about 1% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bmozilla-beta,da50f951b6bcf1bd735bdd67ca1d0e56a873a715,1,1%5D&series=%5Bmozilla-beta,8faafec2fe5cc222e3f6c8d5b053fa72b8c93c8f,1,1%5D&zoom=1497691603513.8855,1498964140527.896,273.3707899457953,320.56180118175035

ts_paint (devedition about 6% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bmozilla-beta,477429f634f1a613c3519fd5747b7202bda2293b,1,1%5D&series=%5Bmozilla-beta,fd481b27e806dff6494d6a3007b5b4973cd28869,1,1%5D

cart (devedition about 5% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Bmozilla-beta,193d1018e16c98e7e61feec4380aa0fbc29caa6b,1,1%5D&series=%5Bmozilla-beta,e33315932aad2b93bf8b547a8b3f1e2e36692eb6,1,1%5D

tsvg_static (devedition about 5% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Bmozilla-beta,fe2b6418cb235cc5e32dbf3776632e83aa10e6b5,1,1%5D&series=%5Bmozilla-beta,1905980b9e0099eaef680b3f7fb04036349325b8,1,1%5D

tsvgr_opacity (devedition about 5% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Bmozilla-beta,62249199cd3e8b0397e9b800cb4870a4d654e416,1,1%5D&series=%5Bmozilla-beta,e36d4e6025ecbe2c591d4347294065bc90c453cf,1,1%5D

tsvgx (devedition about 5% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Bmozilla-beta,6434e7794a85e79e1c99f1ae11e348a718a9dcbf,1,1%5D&series=%5Bmozilla-beta,64f21cd01cea398ebff64ff834b78d50de29388c,1,1%5D

tp5o (devedition about 3% higher):
https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bmozilla-beta,d91b803c27aa207a6ac6a0c9212d217b8bc00b49,1,1%5D&series=%5Bmozilla-beta,106a1dea664e18062fca92be89188d2edc7c339a,1,1%5D

tp5o main_rss (devedition about 10% higher):
https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bmozilla-beta,d0bfce1d25caa9d99bbe227b2c40492cd93cb6c3,1,1%5D&series=%5Bmozilla-beta,d9b2865ade63ca5702db9cbd1572d36051440fd5,1,1%5D

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

one thing I noticed in both win7 and osx is that we have matching patterns on all the tests, we don't see one version getting a fix or regression to change the pattern from the other edition.  Given the fact that we have almost no alerts on beta (<10 per release), and that we are tracking an identical pattern I recommend removing the devedition talos tests.

:sylvestre, do you have any concerns with removing the devedition talos tests on mozilla-beta and just keeping the regular talos tests only?

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: buildbot configs patch for disabling talos on devedition

note: this still needs a taskcluster config- for this this is removing the builders.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: differences.txt

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: taskcluster config to remove talos from devedition

Comment 11

[Robert Wood [:rwood]]

Comment on attachment 8887063 [details] [diff] [review]
buildbot configs patch for disabling talos on devedition

Do these lines need to be updated also? I'm not sure...

ALL_TALOS_PLATFORMS and the 3 following lines after it:

https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla-tests/config.py#l255

And:

https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla-tests/config.py#l3097

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

good find!

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

removing ni for :sylvestre as per comment 4.

I am not able to run list_builder_differences.sh anymore, so I am reluctant to remove the extra lines in the file as pointed out in the review.  :rwood, should I go ahead an land the patch as is, or should I make the changes and monitor after it is deployed?

Comment 14

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher) (UTC-8) from comment #13)
> removing ni for :sylvestre as per comment 4.
> 
> I am not able to run list_builder_differences.sh anymore, so I am reluctant
> to remove the extra lines in the file as pointed out in the review.  :rwood,
> should I go ahead an land the patch as is, or should I make the changes and
> monitor after it is deployed?

Yeah I think it's safer to land it as-is, it looks like get_talos_slave_platforms will return a default 'slave_platforms' if the platform isn't found (the first case above).

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

buildbot configs landed:
https://hg.mozilla.org/build/buildbot-configs/rev/29ab993f3088241af71de9720af8044fec05617d

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/bac5058d778c
taskcluster config to disable devedition talos. r=rwood

Comment 17

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/bac5058d778c

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/b766d449efd2