1459767 - Investigate windows with skia POMTP talos regressions

Status: RESOLVED FIXED

(Core :: Graphics, defect, P3)

Description

[Ryan Hunt [:rhunt]]

Because of bug 1458638, we haven't been testing skia on talos on windows since about April 1 when new hardware was added.

This has been fixed, and this bug is about giving credit to commits that caused improvements, and finding out if we introduced any regressions.

:jmaher was able to get a talos run with skia on the commit that introduced the new hardware. Here's the compare view that shows what has changed since then [1]

Note: You need to reverse the results from this link. i.e. red is good and green is bad

[1] https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=2f0a14434aefc4cb355ff37b8074d786f679f0a7&framework=1&showOnlyImportant=1&selectedTimeRange=172800

Here are the performance improvements:

a11yr
   297.00 ± 1.31% 	< 	310.56 ± 1.53% 	4.57%
rasterflood_gradient
   749.00 ± 0.72% 	> 	453.60 ± 0.55% 	-39.44%
rasterflood_svg
   10,508.35 ± 0.20% 	< 	11,461.02 ± 0.77% 	9.07% 
sessionrestore
   256.60 ± 0.98% 	< 	276.20 ± 0.54% 	7.64%
stylebench
   53.49 ± 0.54% 	> 	51.58 ± 0.76% 	-3.58%
tabpaint
   55.04 ± 4.85% 	< 	60.63 ± 0.68% 	10.15%
tp5o
   233.29 ± 1.82% 	< 	247.24 ± 1.31% 	5.98%
tp5o_webext
   281.58 ± 1.16% 	< 	296.91 ± 0.51% 	5.45%
tp6_amazon
   296.40 ± 2.61% 	< 	329.25 ± 0.75% 	11.08%
tp6_facebook
   174.75 ± 2.53% 	< 	197.50 ± 1.43% 	13.02%
tp6_google
   445.50 ± 0.22% 	< 	464.75 ± 0.38% 	4.32%
tpaint
   159.84 ± 1.38% 	< 	164.41 ± 1.53% 	2.86%
tps
   14.65 ± 1.27% 	< 	20.11 ± 1.85% 	37.30%
ts_paint_heavy
   373.10 ± 0.54% 	< 	383.20 ± 1.08% 	2.71%
tsvgr_opacity
   125.07 ± 2.34% 	< 	147.31 ± 0.66% 	17.78%
tsvgx
   258.63 ± 1.46% 	< 	299.77 ± 1.42% 	15.91%

Here are the performance regressions:

damp
   87.46 ± 1.13% 	> 	79.10 ± 0.78% 	-9.56%
kraken
   1,058.22 ± 0.76% 	> 	1,026.40 ± 0.58% 	-3.01%
motionmark_animometer
   23.62 ± 0.66% 	< 	25.56 ± 0.40% 	8.21%
motionmark_htmlsuite
   21.76 ± 0.53% 	< 	22.40 ± 0.41% 	2.93%
sessionrestore_many_windows
   1,012.70 ± 0.97% 	> 	970.60 ± 0.75% 	-4.16%
tp5o responsiveness
   0.53 ± 6.56% 	> 	0.45 ± 3.01% 	-16.48%
tp6_youtube
   316.00 ± 4.78% 	> 	191.50 ± 5.17% 	-39.40%

Comment 1

[Ryan Hunt [:rhunt]]

Most of these changes are probably due to bug 1454978, I will do a talos run to try to see if that is the case.

Comment 2

[Ryan Hunt [:rhunt]]

Here is a try run that shows what happens if we disable POMTP with skia on windows. Red is good here because that means that we have gotten better with this enabled.

If this talos run is accurate, it looks like not all the improvements or regressions are due to POMTP.

For POMTP it looks like there are two regressions, animometer of ~3% and tsvg_static of about ~8%. The rest are improvements.

[1] https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&originalRevision=dc2ea9f1bf9d&newProject=try&newRevision=1405315a469bcb1ddcada38a896f9f581eb67caf&framework=1&showOnlyImportant=1

Comment 3

[Ryan Hunt [:rhunt]]

So now that I think about this further, that initial try run that :jmaher did shows us every perf change since april on equal configurations (skia vs. skia). But that doesn't mean these changes are all due to skia, they could be from other code changes that were accepted.

For example, the regression in tp6_youtube was from bug 1455115 which is stylo. My guess would be most of these changes are from non-skia changes and were probably caught by talos.

So I think the most tractable thing I can do here is to just focus on the skia POMTP changes as those are for sure graphics related and would not have been caught by talos.

Comment 4

[Ryan Hunt [:rhunt]]

Here's an update. Some improvements have landed for POMTP, such as bug 1462411 that should improve our situation here. Talos results from that indicate about a 8% improvement to tsvg (bug 1462411 comment 7).

Seeing as we're now much better on most tests, and the only remaining regression from that try run should be 3% on animometer, I think it's best to just focus on general improvements and close this bug.