1174776 - 8.5%-30% tp5o talos regression with e10s enabled on all three platforms

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Mike Conley (:mconley) (:⚙️)]

Initial measurements on mozilla-central changeset 0093691d3715 comparing e10s to non-e10s shows the following data for Windows after several retriggers:

Win 7:

:( tp5o                 227.9 +/-  0% (6#)  [  +8.5%]     247.3 +/-  1% (6#)

Comment 1

[Mike Conley (:mconley) (:⚙️)]

We're seeing similar regressions on Linux 32 / Linux 64:

Linux 32:

:( tp5o                 269.0 +/-  0% (6#)  [ +26.4%]     339.9 +/-  2% (6#)


Linux 64:

:( tp5o                 239.0 +/-  1% (6#)  [ +29.9%]     310.4 +/-  1% (6#)

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

from:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=aad95360a002&exclusion_profile=false&filter-searchStr=talos

here are the results:

Linux:
:( tp5o                 274.8 +/-  0% (7#)  [ +25.9%]     345.9 +/-  2% (7#)
Linux64:
:( tp5o                 247.0 +/-  1% (7#)  [ +28.6%]     317.6 +/-  1% (7#)
Win7:
:( tp5o                 229.7 +/-  0% (7#)  [  +9.1%]     250.7 +/-  0% (7#)
WinXP:
:( tp5o                 212.7 +/-  1% (7#)  [ +12.4%]     239.0 +/-  0% (6#)
Win8:
:( tp5o                 215.1 +/-  0% (7#)  [  +9.5%]     235.6 +/-  2% (7#)
OSX 10.10:
:( tp5o                 280.6 +/-  4% (7#)  [ +20.6%]     338.5 +/-  5% (7#)


notice that OSX 10.10 has a 20% hit, the rest are roughly the same.

Comment 3

[Jim Mathies [:jimm]]

Should investigating this track m8? We can't roll out with major tp5 opt regressions.

Comment 4

[Mike Conley (:mconley) (:⚙️)]

I was able to get talos to spew out some profiles for me for both e10s and non-e10s try builds.

Linux64 numbers have come in. Here is the subtest comparison:

https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=try&originalRevision=3a36fc31a1a1&newProject=try&newRevision=01ee797fc2c9&originalSignature=bd72d04511c657c5c5040f1633fe73642fcdcb3b&newSignature=bd72d04511c657c5c5040f1633fe73642fcdcb3b

I've chosen two bad cases - xinhuanet.com and youtube.com.


# xinhuanet.com

non-e10s: http://people.mozilla.org/~bgirard/cleopatra/?zippedProfile=http://mozilla-releng-blobs.s3.amazonaws.com/blobs/Try-Non-PGO/sha512/3f1cf539d64869a3159e7cb1934e3b47112b5cfea8cdb5b8a2208f2d44c46ec326601ef87c617547dccd865533b2bc9497031b8bf7a91e797c3d2ce43036430f&pathInZip=profile_tp5o/xinhuanet.com/cycle_0.sps#report=de3f6a10f8926d6a876c537d1dd092d8e548e35d&filter=%5B%7B%22type%22%3A%22RangeSampleFilter%22,%22start%22%3A824068,%22end%22%3A825106%7D%5D&selection=0,1

e10s: http://people.mozilla.org/~bgirard/cleopatra/?zippedProfile=http://mozilla-releng-blobs.s3.amazonaws.com/blobs/Try-Non-PGO/sha512/6a7aa316e29e3130cc3de4fa966904b6739babcaa243a5e84c5331c1ec7c435f8c94f5631d5f31174f86e006f44dd4464c0c22bc4ac295db8c61c48859328eea&pathInZip=profile_tp5o/xinhuanet.com/cycle_0.sps#report=815a41510062b36c0d4e62613b2f74f0eb4df706&filter=%5B%7B%22type%22%3A%22RangeSampleFilter%22,%22start%22%3A896438,%22end%22%3A901213%7D%5D&selection=0,1


# youtube.com

non-e10s: http://people.mozilla.org/~bgirard/cleopatra/?zippedProfile=http://mozilla-releng-blobs.s3.amazonaws.com/blobs/Try-Non-PGO/sha512/3f1cf539d64869a3159e7cb1934e3b47112b5cfea8cdb5b8a2208f2d44c46ec326601ef87c617547dccd865533b2bc9497031b8bf7a91e797c3d2ce43036430f&pathInZip=profile_tp5o/youtube.com/cycle_0.sps#report=0e810e3b58dac0fe5f297991a08ffc622f4b1c5c&filter=%5B%7B%22type%22%3A%22RangeSampleFilter%22,%22start%22%3A918418,%22end%22%3A918849%7D%5D&selection=0,1,593,188,188,678,679,681,682,683,680,681,682,683,684,685,686,687,688,689,729,681,682,683,684,685,686,687,682,683,684,685,686,687,738,739,740,741,682,683,680,681,738,739,740,741,682,743,744,682,683,684,685,686,687,758,759,760,687,730,698,699,700,701,702,703,704

e10s: http://people.mozilla.org/~bgirard/cleopatra/?zippedProfile=http://mozilla-releng-blobs.s3.amazonaws.com/blobs/Try-Non-PGO/sha512/6a7aa316e29e3130cc3de4fa966904b6739babcaa243a5e84c5331c1ec7c435f8c94f5631d5f31174f86e006f44dd4464c0c22bc4ac295db8c61c48859328eea&pathInZip=profile_tp5o/youtube.com/cycle_0.sps#report=6d3c45633c3354fe2439532f1c5e680e2ae82b5b&filter=%5B%7B%22type%22%3A%22RangeSampleFilter%22,%22start%22%3A1048604,%22end%22%3A1049055%7D%5D&selection=0,1,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33


So the questions here are:

1) Do these profiles show us anything useful?
2) If (1), what bugs can we file based on these profiles?

Comment 5

[Benoit Girard (:BenWa)]

These profiles don't sync up. Let's fix that before we start doing more analysis.

Comment 6

[Benoit Girard (:BenWa)]

There was a bug in cleopatra which I've now fixed. Here are the same URL again that will cause re-processing and will give better syncing:

# xinhuanet.com

non-e10s: http://people.mozilla.org/~bgirard/cleopatra/?zippedProfile=http://mozilla-releng-blobs.s3.amazonaws.com/blobs/Try-Non-PGO/sha512/3f1cf539d64869a3159e7cb1934e3b47112b5cfea8cdb5b8a2208f2d44c46ec326601ef87c617547dccd865533b2bc9497031b8bf7a91e797c3d2ce43036430f&pathInZip=profile_tp5o/xinhuanet.com/cycle_0.sps#report=de3f6a10f8926d6a876c537d1dd092d8e548e35d&filter=%5B%7B%22type%22%3A%22RangeSampleFilter%22,%22start%22%3A824068,%22end%22%3A825106%7D%5D&selection=0,1

e10s: http://people.mozilla.org/~bgirard/cleopatra/?zippedProfile=http://mozilla-releng-blobs.s3.amazonaws.com/blobs/Try-Non-PGO/sha512/6a7aa316e29e3130cc3de4fa966904b6739babcaa243a5e84c5331c1ec7c435f8c94f5631d5f31174f86e006f44dd4464c0c22bc4ac295db8c61c48859328eea&pathInZip=profile_tp5o/xinhuanet.com/cycle_0.sps


# youtube.com

non-e10s: http://people.mozilla.org/~bgirard/cleopatra/?zippedProfile=http://mozilla-releng-blobs.s3.amazonaws.com/blobs/Try-Non-PGO/sha512/3f1cf539d64869a3159e7cb1934e3b47112b5cfea8cdb5b8a2208f2d44c46ec326601ef87c617547dccd865533b2bc9497031b8bf7a91e797c3d2ce43036430f&pathInZip=profile_tp5o/youtube.com/cycle_0.sps#report=0e810e3b58dac0fe5f297991a08ffc622f4b1c5c&filter=%5B%7B%22type%22%3A%22RangeSampleFilter%22,%22start%22%3A918418,%22end%22%3A918849%7D%5D&selection=0,1,593,188,188,678,679,681,682,683,680,681,682,683,684,685,686,687,688,689,729,681,682,683,684,685,686,687,682,683,684,685,686,687,738,739,740,741,682,683,680,681,738,739,740,741,682,743,744,682,683,684,685,686,687,758,759,760,687,730,698,699,700,701,702,703,704

e10s: http://people.mozilla.org/~bgirard/cleopatra/?zippedProfile=http://mozilla-releng-blobs.s3.amazonaws.com/blobs/Try-Non-PGO/sha512/6a7aa316e29e3130cc3de4fa966904b6739babcaa243a5e84c5331c1ec7c435f8c94f5631d5f31174f86e006f44dd4464c0c22bc4ac295db8c61c48859328eea&pathInZip=profile_tp5o/youtube.com/cycle_0.sps

Comment 7

[Mike Conley (:mconley) (:⚙️)]

Thanks BenWa.

One thing that jumps out immediately is that we spend wayyyyyyy more time cycle collecting with e10s enabled. The cycle collection occurs in the content process, and seems to cause the parent to sit around waiting a lot more.

Samples spend in nsCycleCollector::Collect:

xinhuanet.com:
non-e10s: 0
e10s: 466

youtube.com:
non-e10s: 11
e10s: 659

Comment 8

[Mike Conley (:mconley) (:⚙️)]

Filed bug 1184277 about the aggressive cycle collection.

Comment 9

[Benoit Girard (:BenWa)]

Here's what the youtube profile looks like now with bug 1184277 fixed:

Non e10s:
http://people.mozilla.org/~bgirard/cleopatra/?zippedProfile=http://mozilla-releng-blobs.s3.amazonaws.com/blobs/Try-Non-PGO/sha512/3f1cf539d64869a3159e7cb1934e3b47112b5cfea8cdb5b8a2208f2d44c46ec326601ef87c617547dccd865533b2bc9497031b8bf7a91e797c3d2ce43036430f&pathInZip=profile_tp5o/youtube.com/cycle_0.sps

e10s with bug 1184277 fixed:
http://people.mozilla.org/~bgirard/cleopatra/?zippedProfile=http://mozilla-releng-blobs.s3.amazonaws.com/blobs/Try/sha512/668bbef25bc4815042eb84b15d3850eda7a6db200988b8243e528ac81374f9c8984a7c6e8e6de33051c9edc8cee96095c9f416d13266a2dce2213c752eade5c2&pathInZip=profile_tp5o/youtube.com/cycle_0.sps

Comment 10

[Benoit Girard (:BenWa)]

It looks like part of the problem is we're hitting a similar issue that tab switch is hitting. With e10s enabled we will have the chrome painting extra frames and sometimes the content process ends up blocked waiting on the chrome paints.

Looks like fixing bug 1181333 would also help buy us 5-10 ms on youtube.

Comment 11

[Benoit Girard (:BenWa)]

Moving profiles that I collected in bug 1184763 here:
More (all of the \o/!) try pushes to understand the issue better:

without e10s with xrender without throbber:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8e23115d4ab0

without e10s without xrender without throbber:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a1e65c16ef11

with e10s without xrender without throbber:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=87dedb31301c

with e10s with xrender without throbber:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=25378a68a246

Comment 12

[Benoit Girard (:BenWa)]

Here's my conclusion from this data. We've got 73ms regression with e10s on youtube load which breaks down into this:

15ms: bug 1181333 (waiting on chrome to paint early frame before we unblock the content thread)
10ms: bug 1186139 (sync message: ContentLinkHandler.onLinkEvent())
10ms: If we turn off XRender before we ship e10s this should close the non-e10s/e10s gap by 10ms
10ms: Throbber - We're compositing more throbber frame which seems to steal about 10ms by paper cutting the profile
?ms: Extra e10s sampling overhead, we could gather more profiles without the profiler
18ms: Unexplained, we should re-measure once we've made progress on the above issues. Bug 1180916 might help here too.

Comment 13

[Jim Mathies [:jimm]]

inbound w/bug 1217571:

https://treeherder.mozilla.org/perf.html#/graphs?series=[mozilla-inbound,ca94ff5540bd7a90cd41306de55b1b69962aca1e,0]&series=[mozilla-inbound,92bf5647ee18b4288bce380ddc91f1fbf70be212,0]&series=[mozilla-inbound,06a2fc2ad84cc4c129b44d9b9f29b800c519e559,0]&series=[mozilla-inbound,3ec2958352e2cb98df247808db574209f4ab5eb8,0]&series=[mozilla-inbound,bd72d04511c657c5c5040f1633fe73642fcdcb3b,1]&series=[mozilla-inbound,dc5dc84a309cfa3f89307a1d195307e333707a12,1]

fixed!

Comment 14

[Mike Conley (:mconley) (:⚙️)]

Sweet! jmaher - cool if we close this out?