1251699 - 5.7% TART regression on OS X with APZ disabled

Status: RESOLVED WORKSFORME

(Firefox :: General, defect, P1)

Description

[Jim Mathies [:jimm]]

5-10%. I find this surprising considering tab animation appears much smoother than non-e10s. 

https://treeherder.mozilla.org/perf.html#/graphs?series=[mozilla-central,e4e66cb9d2890771e77d7fc73930cef5ea9f8cdd,0]&series=[mozilla-central,956cd0f8f2d1cd69d9edfb1b787ecadc6594ae1d,0]&series=[mozilla-central,b1d144c18b37edd1852da21b96b47fa7fc35691f,1]&series=[mozilla-central,26f363f15d58cd426c6edeb593cbfc2fa76d17fe,1]

Comment 1

[Mike Conley (:mconley) (:⚙️)]

Here's a try push where I disabled APZ: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5eb1f8c13c9c

Here's a comparison against the fx-team base rev: https://treeherder.mozilla.org/perf.html#/compare?originalProject=fx-team&originalRevision=ad4d966f2367&newProject=try&newRevision=5eb1f8c13c9c&framework=1&showOnlyImportant=0


I'm seeing big improvements on Linux 64 and Win 7. Still waiting on retriggers for OS X and Win 8, but it looks compelling.

I have _no idea_ why APZ would affect TART, but the numbers here are pretty clear. Assuming OS X and Win 8 show the same results when their numbers come in (and the initial numbers suggest this), I _think_ we can chalk this one up to APZ.

Comment 2

[Mike Conley (:mconley) (:⚙️)]

Numbers with APZ disabled are in:

https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&originalRevision=a4929411c0aa&newProject=try&newRevision=850e39c8f668&framework=1&filter=TART&showOnlyImportant=0

Conclusions:

1) APZ causes a ~9.5% - ~12% TART regression with e10s enabled. This takes care of a good chunk of the regression across the board.

2) Doing some napkin math, I can re-write the e10s-only regressions (without APZ enabled) as:

           non-e10s (*)     e10s           Diff (%)
-----------------------------------------------------
Linux64:    4.99           5.01             +0.4%
OS X 10:    9.12           9.64             +5.7% **
Win 7  :    5.33           5.34             +1.8%
Win 8  :    4.95           4.92             -0.6%

All but the (potentially dubious - see **) OS X regression are below the 5% threshold.

So the next steps for this bug:

1) File a new bug for the APZ releated TART regressions and make sure the APZ team sees it.
2) Clear up whether or not there's still a regression for OS X

** The numbers here look a little noisy. I've done more retriggers, and I'm needinfo'ing myself to update this once they come in.

Comment 3

[Mike Conley (:mconley) (:⚙️)]

Filed the APZ-attributed regressions as bug 1253412.

Comment 4

[Mike Conley (:mconley) (:⚙️)]

OS X retriggers are in, and we're holding steady at a 5.7% regression on OS X.

New table:

           non-e10s        e10s           Diff (%)
-----------------------------------------------------
Linux64:    4.99           5.01             +0.4%
OS X 10:    9.11           9.63             +5.7%
Win 7  :    5.33           5.34             +1.8%
Win 8  :    4.95           4.92             -0.6%

Comment 5

[Mike Conley (:mconley) (:⚙️)]

A recent try push with APZ disabled shows that this regression has disappeared.

OS X:
https://treeherder.mozilla.org/perf.html#/e10s?repo=try&revision=cd629d4df5a4

Other platforms:
https://treeherder.mozilla.org/perf.html#/e10s?repo=try&revision=95526c1a74da