1508218 - 13.15 - 35.94% Images (linux64-nightly, macosx64-nightly, windows10-64-nightly) regression on push fd2d2838220419e3e2682889598cfcded38211fd (Tue Nov 13 2018)

Status: VERIFIED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected an awsy regression from push:

https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?changeset=fd2d2838220419e3e2682889598cfcded38211fd

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 36%  Images windows10-64-nightly opt stylo     7,928,577.12 -> 10,778,195.66
 14%  Images macosx64-nightly opt stylo         7,088,142.67 -> 8,061,267.61
 13%  Images linux64-nightly opt stylo          6,475,322.93 -> 7,326,552.38

Improvements:

  4%  JS windows10-64-nightly opt stylo     106,347,823.43 -> 102,529,028.58
  4%  JS macosx64-nightly opt stylo         106,265,793.61 -> 102,521,636.10
  3%  JS linux64-nightly opt stylo          104,406,001.11 -> 100,784,847.72
  3%  JS windows7-32-nightly opt stylo      77,771,777.72 -> 75,659,388.68


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=17615

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/AWSY/Tests

Comment 1

[Bobby Holley (:bholley)]

When bug 1506643 landed, it accidentally changed the default tab in AWSY from about:newtab to about:blank, reducing measured memory usage (bug 1506643 comment 9) because it effectively reduced test coverage. So bug 1507267 reverted the change. We basically want to verify that the numbers have returned to the baseline we had before bug 1506643 landed.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Correct. And that's what I mentioned ahead of time on bug 1507267 comment 9.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Perhaps I'm reading this wrong, but to me it appears that bug 1506643 caused improvements to Images of around 5%, whereas this is a _regression_ to Images up to 36%. Is the same push likely to make an improvement to mozilla-central whilst regressing mozilla-beta? Can we uplift the fix in bug 1507267 to beta, otherwise we're unlikely to see if this corrects as a result.

Comment 4

[Ionuț Goldan [:igoldan]]

(In reply to Henrik Skupin (:whimboo) from comment #2)
> Correct. And that's what I mentioned ahead of time on bug 1507267 comment 9.

I was expecting what you said in that comment. Still, bug 1507267 wasn't uplifted to beta. Without this uplift, I should have only seen improvements.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Maybe the regression on beta is different from central because the code has been diverged since the last merge. As such an uplift of bug 1507267 might indicate it. Note that I already requested uplift of that spoken patch earlier today.

Comment 6

[Ionuț Goldan [:igoldan]]

This got uplifted to beta according to [1]. And indeed, all AWSY changes from comment 0 have been reverted.
This is just a subset of the new AWSY baseline:

== Change summary for alert #17888 (as of Wed, 21 Nov 2018 10:05:53 GMT) ==

Regressions:

  5%  JS macosx64-nightly opt stylo         102,002,043.56 -> 107,041,565.93
  3%  JS windows10-64-nightly opt stylo     102,272,960.36 -> 105,357,089.62

Improvements:

 13%  Images macosx64-nightly opt stylo     8,042,134.80 -> 6,960,424.07

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=17888


[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1507267#c15

Comment 7

[Ryan VanderMeulen [:RyanVM]]

We still run AWSY on esr60, though I don't know if anybody is actually monitoring the results.

Comment 8

[Liz Henry (:lizzard) (relman/hg->git project)]

If this didn't affect 63, why would it affect esr60?  Ionut, does this actually affect esr?

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Because bug 1506643 was uplifted there.