1459129 - 17.56% Images (windows10-64-nightly) regression on push 064c3804de2e967753bcba437523892da918f67a (Fri Apr 27 2018)

Status: RESOLVED INCOMPLETE

(Testing :: AWSY, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected an awsy regression from push:

https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?changeset=064c3804de2e967753bcba437523892da918f67a

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 18%  Images windows10-64-nightly opt stylo     6,961,605.60 -> 8,183,950.50


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=13037

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/AWSY/Tests

Comment 1

[Ionuț Goldan [:igoldan]]

This is a big regression. It appeared on integration branches in 2 steps -- Mar 20 & Apr 5, but Perfherder didn't catch them, because of the noise in tests.
Looking closer on mozilla-beta, this regression seems to happen also on the other platforms.

I already retrigger/bacfilled data points around the 2 dates I mentioned.

Comment 2

[Ionuț Goldan [:igoldan]]

RyanVM: Can you take a look over this beta bug and maybe pull in the relevant regression authors?

Comment 3

[Ryan VanderMeulen [:RyanVM]]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #1)
> I already retrigger/bacfilled data points around the 2 dates I mentioned.

What were the results of this? That would be a lot easier to look over than a giant merge pushlog range.

Comment 4

[Ionuț Goldan [:igoldan]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #3)
> (In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #1)
> > I already retrigger/bacfilled data points around the 2 dates I mentioned.
> 
> What were the results of this? That would be a lot easier to look over than
> a giant merge pushlog range.

I'm afraid it's pretty hard to tell, even after the 2nd round of retriggers on mozilla-inbound.
Still, with the new results, it's starting to look like both regressions originated from autoland.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Can you at least provide a rough range for around the time those regressions happened? I'm guessing it's going to be something imagelib-related.

Comment 6

[Ionuț Goldan [:igoldan]]

The first regression that happened around March 20: https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=mozilla-beta,1667474,0,4&series=mozilla-inbound,1645873,1,4&series=autoland,1655738,1,4&highlightedRevisions=f76b3a9e3005&zoom=1520819102970.9944,1522254555761.0498,6126404.494382023,9514044.943820225

The second regression that happened around April 5: https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=mozilla-beta,1667474,0,4&series=mozilla-inbound,1645873,1,4&series=autoland,1655738,1,4&highlightedRevisions=9e8eeee62ffb&highlightedRevisions=f76b3a9e3005&zoom=1522008172073.2043,1523818556562.1548,6750000,9800561.797752809

The highlighted data points are just for easy spotting the approximate moment when the regression got in.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Those points on Beta don't tell us anything as they're from Fx60. The initial regression range in #c0 was for the first Fx61 to Beta merge. But I guess that means we didn't see any corresponding regressions on Nightly 61 that might explain this?

Comment 8

[Ionuț Goldan [:igoldan]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #7)
> Those points on Beta don't tell us anything as they're from Fx60. The
> initial regression range in #c0 was for the first Fx61 to Beta merge. But I
> guess that means we didn't see any corresponding regressions on Nightly 61
> that might explain this?

That's right, we didn't see those corresponding regressions on Nightly.

Comment 9

[Ionuț Goldan [:igoldan]]

A narrower range for April 5 would be: 1fb1569b61d2 - 65e0fefdab51
As for March 20, the range would be: 5ed5a088c782 - ea23c2dda34a

These are both from autoland: https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=mozilla-beta,1667474,1,4&series=autoland,1655738,1,4&highlightedRevisions=5ed5a088c782&highlightedRevisions=ea23c2dda34a

Comment 10

[Ryan VanderMeulen [:RyanVM]]

If you saw a jump on Autoland and Beta at basically the same time, wouldn't that suggest an external change?

Comment 11

[Ryan VanderMeulen [:RyanVM]]

OK, in that March range, there's bug 1417155, which might be plausible?

Comment 12

[Vicky Chin [:vchin]]

Any updates on this bug?

Comment 13

[Florin Strugariu [:Bebe]]

There is no data available anymore for this regression.
Closing this as we can't test/reproduce anymore