1459545 - 4.67 - 44.68% rasterflood_gradient / tsvg_static / tsvgr_opacity / tsvgx (windows7-32) regression on push 7279e34fd6b82db30cc7b064b30359bbee6d7546 (Fri May 4 2018)

Status: RESOLVED INVALID

(Testing :: Talos, defect)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=7279e34fd6b82db30cc7b064b30359bbee6d7546

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 45%  tsvgx windows7-32 opt e10s stylo     178.98 -> 258.96
 44%  rasterflood_gradient windows7-32 pgo e10s stylo1,362.58 -> 757.92
 44%  tsvgx windows7-32 pgo e10s stylo     159.52 -> 230.20
 44%  rasterflood_gradient windows7-32 opt e10s stylo1,334.00 -> 745.58
 20%  tsvg_static windows7-32 opt e10s stylo45.07 -> 53.96
 15%  tsvg_static windows7-32 pgo e10s stylo46.45 -> 53.57
 10%  tsvgr_opacity windows7-32 pgo e10s stylo98.94 -> 108.93
  5%  tsvgr_opacity windows7-32 opt e10s stylo120.80 -> 126.43

Improvements:

  7%  tps windows7-32 opt e10s stylo     15.77 -> 14.66
  7%  tps windows7-32 pgo e10s stylo     13.46 -> 12.57
  6%  rasterflood_svg windows7-32 opt e10s stylo11,240.61 -> 10,521.20
  6%  rasterflood_svg windows7-32 pgo e10s stylo10,774.97 -> 10,090.77
  3%  tp6_google windows7-32 opt e10s stylo457.71 -> 444.50


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=13075

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

:jmaher Bug 1458638 seems to have updated the Windows 7 baselines. Are any of the alerts above unexpected?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

good news is all the changes here are win7- that is all that was changed.

Now I get a bit confused, why would rasterflood_gradient see a large regression, when rasterflood_svg see an improvement?  I might be able to get convince that SVG is faster with skia, but then both the svgx, svg_static, and svg_opacity regressed.

We are now testing the correct graphics mode for 32 bit Firefox, not sure if there are other things to investigate or change.


:rhunt, I will leave the decision up to you- I assume we want to accept these regressions, but do they look accurate?  is there any followup work to do?

Comment 3

[Ryan Hunt [:rhunt] (on leave until early May)]

The numbers here are correct. They're confusing because a lot has happened so it's hard to find a baseline.

Before March 30th we had windows + skia + old hardware.
After March 30th until about now we had windows + d2d + new hardware.

In this time period some significant skia only changes like bug 1454978 we landed. There might be others, I'm asking around.

And now we have windows + skia + new hardware + new skia only changes.

So I have no idea what things are due to what. With bug 1454978 it's possible our skia code is better rasterflood_gradient vs D2D, but still worse on rasterflood_svg.

So in the end we definitely want to take these results because we're testing skia on windows again, but I have no idea what results we are getting here.

It would be nice if we could somehow compare mozilla-central before March 30th on the new hardware with skia, and compare it to mozilla-central today on the new hardware with skia to know what has changed.

Comment 4

[Ryan Hunt [:rhunt] (on leave until early May)]

(In reply to Ryan Hunt [:rhunt] from comment #3)
> The numbers here are correct. They're confusing because a lot has happened
> so it's hard to find a baseline.
> 
> Before March 30th we had windows + skia + old hardware.
> After March 30th until about now we had windows + d2d + new hardware.
> 
> In this time period some significant skia only changes like bug 1454978 we
> landed. There might be others, I'm asking around.
> 
> And now we have windows + skia + new hardware + new skia only changes.
> 
> So I have no idea what things are due to what. With bug 1454978 it's
> possible our skia code is better rasterflood_gradient vs D2D, but still
> worse on rasterflood_svg.
> 
> So in the end we definitely want to take these results because we're testing
> skia on windows again, but I have no idea what results we are getting here.
> 
> It would be nice if we could somehow compare mozilla-central before March
> 30th on the new hardware with skia, and compare it to mozilla-central today
> on the new hardware with skia to know what has changed.

Or at the very least with the pref from bug 1454978 flipped on and off.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

here is a compare view with skia turned on for April 1st (the first day we used the new hardware) compared to the most recent 2 days on mozilla-central:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=2f0a14434aefc4cb355ff37b8074d786f679f0a7&framework=1&showOnlyImportant=1&selectedTimeRange=172800

a very similar set of changes- I am not really sure how to interpret this, it is all a bit confusing.

Comment 6

[Ryan Hunt [:rhunt] (on leave until early May)]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #5)
> here is a compare view with skia turned on for April 1st (the first day we
> used the new hardware) compared to the most recent 2 days on mozilla-central:
> https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-
> central&newProject=try&newRevision=2f0a14434aefc4cb355ff37b8074d786f679f0a7&f
> ramework=1&showOnlyImportant=1&selectedTimeRange=172800
> 
> a very similar set of changes- I am not really sure how to interpret this,
> it is all a bit confusing.

Interesting. So if I understand that correctly, you need to reverse the results from that link to see how we've changed since April. Because the new revision is old code, and the base is new code.

i.e tps is actually 37.30% better, but motionmark_animometer is actually 8.21% worse, and so on.

So my hunch is that most of these performance differences are from bug 1454978. The rasterflood, tps, tpaint, tsvgr_opacity, tsvgx, tp6_XXX improvements are expected and what I saw in my mock talos runs before flipping parallel painting on.

The motionmark_animometer, motionmark_htmlsuite, damp, sessionrestore, tp5o responsiveness, tp6_youtube regressions are not expected. In fact I don't think I ever saw motionmark results, so I must have missed it or ran the tests before it was turned on.

I want to dig into what's going on here further and see if I can address those performance regressions. But I think that would be best done in a dedicated bug, and this one can be closed. We're testing the right thing now.

Comment 7

[Ryan Hunt [:rhunt] (on leave until early May)]

I've filed bug 1459767 to investigate this further.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Thanks for the investigation, Ryan. Doesn't sound like there's any more work left to do in this bug.