1393422 - 4.72% tart (windows10-64) regression on push e81a6f33863a (Thu Aug 17 2017)

Status: RESOLVED WONTFIX

(Firefox :: New Tab Page, defect)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=9a2c3ae25925b21fe7c95e858b9d7a5ffe4728f0&tochange=e81a6f33863a952d1cad99d0a3cfd2f0aeaf8b35

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  5%  tart summary windows10-64 opt e10s     5.63 -> 5.90


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=8913

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

Ed, could you take a look over this tab-related regression? I had to retrigger multiple times the test, to identify it.
Because of the noise in the perf results, I also want to know to which extent were you expecting this. Thanks!

Comment 2

[Ed Lee :Mardak]

Bug 1389701 made it so that newly loaded content processes would correctly manage new content pages instead of missing some of them when there are many new tabs. My guess is that tart was hitting the bug and *not* loading content scripts resulting in work incorrectly not being done, i.e., new tab page was failing to load and render correctly.

This means the test was measuring broken behavior before, and now it should be more consistent in measuring the correct thing.

Mossop, does that sound right, and what should we do about this regression?

Comment 3

[Dave Townsend [:mossop]]

That sounds right but I think we should validate it. Can you push to try without that change and some logging to verify that that is what was happening?

Comment 4

[Ed Lee :Mardak]

igoldan, is there a way to view the console errors from talos runs? A quick search on some of the raw logs doesn't seem to show any, so would doing a `dump()` make them appear?

no console errors logged:
https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-win64/autoland_win10_64_test-svgr-e10s-bm126-tests1-windows-build236.txt.gz

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

talos prints out all console errors (at least it did), so I would be surprised if it was not doing that. all of the stdout/stderr we see from the process is fed through this class:
http://searchfox.org/mozilla-central/source/testing/talos/talos/talos_process.py#55

So I believe we should see it, but likely there is a bug- maybe try tweaking that log handler?

Comment 6

[Ed Lee :Mardak]

I ended up just using dumps:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ecbb25231643c412c62aed700f5996149626e662&filter-searchStr=svg

Almost half of those runs ended up printing:

ED: typeof addMessageListener: undefined

So it was indeed running into the bug that was fixed by bug 1389701.

Comment 7

[Ed Lee :Mardak]

Including the runs from https://treeherder.mozilla.org/#/jobs?repo=try&revision=e746d2042159eecda2dd54b8b647f3f5876f9a8c&filter-searchStr=svg

For the runs that did have undefined and loaded incorrectly, the average tart was 5.54. And those that loaded correctly averaged 5.71. So that's about a 3% difference.

Comment 8

[Dave Townsend [:mossop]]

From IRC we think that this is likely an actual regression from activity stream being turned on that we just didn't see because activity stream was frequently broken in the TART test.

Comment 9

[Jim Mathies [:jimm]]

Any plans to try and address this regression?

Comment 10

[Ed Lee :Mardak]

Yes. We're trying to get prerendering of the page landed (primarily for faster about:home rendering, but it should help for about:newtab opening as well as many new tabs are opened and not always preloaded). k88hudson is working on that https://github.com/mozilla/activity-stream/issues/2813

Comment 11

[Ed Lee :Mardak]

k88hudson, any updates on prerendering and any initial Talos try runs?

Comment 12

[Kate Hudson :k88hudson]

I have a working patch right now in https://github.com/mozilla/activity-stream/pull/3359, I'll give it a try today to see what happens

Comment 13

[Jim Mathies [:jimm]]

(In reply to Kate Hudson :k88hudson from comment #12)
> I have a working patch right now in
> https://github.com/mozilla/activity-stream/pull/3359, I'll give it a try
> today to see what happens

Looks like this landed, do we have an update here?

Comment 14

[Ed Lee :Mardak]

It was just turned on via bug 1398819 and just merged to central 

https://hg.mozilla.org/mozilla-central/rev/8475a8df9f16

Should be able to trigger the jobs from autoland treeherder https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=8475a8df9f16b4199cb3222e56902daa698dce3a

Comment 15

[Erin Lancaster [:elan]]

:Mardak can this ticket be resolved as fixed? Does it need to be verified? Thank you.

Comment 16

[Ed Lee :Mardak]

Not exactly fixed. We're looking into another activity stream about:newtab tart regression in bug 1401682. That one is better understood. Although this one here is likely WONTFIX as it seems to be just that activity stream about:newtab has different content than tiles about:newtab. There was a similar tart readjust baseline when content services made UI changes to about:newtab.

Comment 17

[Ed Lee :Mardak]

I was doing some investigation into bug 1401682 and did some more testing and research and realized that when activity stream was first turned on, there was a tart improvement reported in bug 1381804 comment 16. windows10-64 opt had the largest improvement and is the test regressing in this bug. Bug 1389701 that was fixed leading to this regression made it so that tart was testing the actual about:newtab instead of a broken about:newtab as noted in comment 6.

A broken page renders faster than the actual about:newtab, so the turning on Activity Stream tart improvement was actually overstated. The "regression" here is just that Activity Stream in content process did not make as big as an improvement (but still an improvement over Tiles about:newtab).

Comment 18

[Ed Lee :Mardak]

Attachment: displaying nothing on activity stream

Just saving this somewhere. Here's tart numbers from hiding all activity stream content:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&originalRevision=a0eb21bf55e1c1ae0ba311e6f2273da05c712799&newProject=try&newRevision=d5ad46700a818c671e51337a2e46bba75c11850d&framework=1&filter=tart%20opt&showOnlyImportant=0

Looks like at best we can improve tart by 1.5% in terms of content rendering time right now.  Probably can affect what gets computed at the time of opening a tab though. Also that webrender/quantum has ~no performance change when showing something vs nothing for activity stream.