1447979 - 1.11 - 8.52% tp5o_webext responsiveness (linux64, windows10-64, windows7-32) regression on push e2ddf4dbfa33 (Wed Mar 21 2018)

Status: RESOLVED WORKSFORME

(Core :: DOM: Core & HTML, defect)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=243246b2f4406f0b2f118758220db5c7fdd215df&tochange=e2ddf4dbfa33d0da768cd8a46f2627e6c2aadbc6

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  9%  tp5o_webext responsiveness windows7-32 opt e10s stylo     2.91 -> 3.16
  3%  tp5o_webext responsiveness linux64 opt e10s stylo         1.63 -> 1.68
  3%  tp5o_webext responsiveness windows10-64 opt e10s stylo    2.89 -> 2.97
  3%  tp5o_webext responsiveness linux64 pgo e10s stylo         1.44 -> 1.48
  1%  tp5o_webext responsiveness windows10-64 pgo e10s stylo    2.56 -> 2.59

Improvements:

  6%  tabpaint windows10-64 opt e10s stylo     73.04 -> 68.47


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=12305

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

:jandem It seems like bug 1447578 caused these responsiveness perf regressions on multiple desktop OSes. Is this something we can fix or should we accept/backout?

Note: I am not entirely sure about this finding and I'm still waiting for the tests to finish. In about an hour I should be able to properly confirm.

Comment 2

[Jan de Mooij [:jandem]]

Bug 1447578 should not affect performance like this so it's pretty mysterious. We could bisect the patches maybe...

Comment 3

[Ionuț Goldan [:igoldan]]

(In reply to Jan de Mooij [:jandem] from comment #2)
> Bug 1447578 should not affect performance like this so it's pretty
> mysterious. We could bisect the patches maybe...

I triggered a Try bisect. Will inform you once the results arrive.

Comment 4

[Ionuț Goldan [:igoldan]]

Some of the results started to show up. Turns out the regressions are happening on Linux only. Those from Windows are invalid and must be ignored.

Also, it's possible that the previous bug 888600 is more related to this. :peterv Am I right to assume this?

From the looks of it [1], the perf regressions happens in 2 consecutive steps. That is bug 888600, followed by bug 1447979.

[1] https://treeherder.mozilla.org/perf.html#/graphs?series=mozilla-inbound,1640992,1,1&series=autoland,1646628,0,1&zoom=1521641812653.5955,1521660643113.4587,1.2563524184777657,1.899217168427632&selected=mozilla-inbound,1640992,318657,435837268,1

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Andrew, is there anyone on your team who has cycles to take a look at this?

Comment 6

[Jan de Mooij [:jandem]]

Do you still think bug 1447578 is related, or is this from bug 888600? Canceling NI because comment 4 seems to suggest that's true and I don't see how bug 1447578 is related.

Comment 7

[Peter Van der Beken [:peterv]]

Well, it seems I can't run this locally, it always fails with timeouts. I'll try to get some profiles from try.

Comment 8

[Marion Daly [:mdaly]]

Peter: update?

Comment 9

[Marion Daly [:mdaly]]

Any update on this?

Comment 10

[Jan de Mooij [:jandem]]

Moving to DOM because it sounds like this is from bug 888600.

Comment 11

[Steven DeTar [:sdetar]]

Peter: since this looks like this is from bug 888600. Were you able to get some profiles from try as stated in Comment 7?

Comment 12

[Mike Taylor [:miketaylr]]

Ionut, can you provide some more context as to what's regressing here? (Forgive my ignorance)

Comment 13

[Ionuț Goldan [:igoldan]]

(In reply to Mike Taylor [:miketaylr] from comment #12)
> Ionut, can you provide some more context as to what's regressing here?
> (Forgive my ignorance)

I tried to provide some gecko profiles, but without luck: the jobs fail [1].
I think :kmag can provide more details as to what this test is measuring and maybe give clues on what regressed here.

I looked again over the graph. It seems that after March 23, the regression was canceled by bug 1444119, which :peterv pushed.
I vote for closing this as WORKSFORME, given the time that passed.

:peterv if you confirm that bug 1444119 fixed something from bug 1447578, I would actually close this as FIXED.

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=243246b2f4406f0b2f118758220db5c7fdd215df&selectedJob=178283862&group_state=expanded&filter-searchStr=Linux%20x64%20opt%20Talos%20performance%20tests%20with%20e10s%20test-linux64%2Fopt-talos-g5-e10s%20T-e10s(g5-p)

Comment 14

[Marion Daly [:mdaly]]

:overholt, ideas on this?

Comment 15

[Andrew Overholt [:overholt]]

Let's wait on Peter's answer to comment 13.

Comment 16

[Andrew Overholt [:overholt]]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #13)
> (In reply to Mike Taylor [:miketaylr] from comment #12)
> > Ionut, can you provide some more context as to what's regressing here?
> > (Forgive my ignorance)
> 
> I tried to provide some gecko profiles, but without luck: the jobs fail [1].

Without a profile, this is going to be very difficult to diagnose (Peter also can't reproduce locally).

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

this is a really old bug, geckoProfile has changed enough in the last 9 weeks that it is no surprise Ionut is not able to get profiles.  Why are we just now looking into this.  Is Peter trying to reproduce this on code from march 21st?

Comment 18

[Peter Van der Beken [:peterv]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #17)
> this is a really old bug, geckoProfile has changed enough in the last 9
> weeks that it is no surprise Ionut is not able to get profiles.  Why are we
> just now looking into this.

I tried almost 2 months ago and couldn't get profiles either with the same errors: https://treeherder.mozilla.org/#/jobs?repo=try&revision=97327b05d2fb3fb975bb5db0914c543e8f9278b0&selectedJob=172893387

Comment 19

[Peter Van der Beken [:peterv]]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #13)
> (In reply to Mike Taylor [:miketaylr] from comment #12)
> :peterv if you confirm that bug 1444119 fixed something from bug 1447578, I
> would actually close this as FIXED.

Those patches are completely unrelated.