1725706 - Spurious 32% Performance improvement in PayPal fission (only) Linux (only)

Status: RESOLVED WORKSFORME

(Testing :: Performance, defect, P3)

Description

[Randell Jesup [:jesup] (needinfo me)]

Unexplained large improvement in PayPal (32.7%), for fission only (?)
The list of changes points to bug 1719194 as the most likely case, but it's not clear why that would cause such a change, for fission only.

Note that warmloads currently involve navigating to about:blank inbetween each load of paypal, and in fission that will cause us to use a new process for paypal on every new warm load, so that might be involved. Perhaps paypal gc's during load if the process exists (e10s), and avoids the GC in fission with a brand-new process?

We're changing warmload tests to be site/page1 -> site/page2 -> site/page1 etc, which might affect this case.

Profiles of a warmload in e10s and fission may well show the issue.

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

Dave, can you get profiles of paypal in e10s and fission, or let nbp and me how to do so? Thanks

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

https://treeherder.mozilla.org/perfherder/alerts?id=30926

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

FYI as to why I filed a bug on this:
Since it was unexpected, I'd like to know why it had this impact. Who knows, it might point to a way to help other sites. If we don't understand it, we could accidentally undo it later -- or it may point to a problem in how we're testing/measuring (i.e. illusionary gain, or a non-real-world gain). This also gives us a huge advantage over e10s for that one site if it's real, and if we could extend that it would be great.

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

== Change summary for alert #30926 (as of Fri, 13 Aug 2021 18:25:43 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
33%	paypal SpeedIndex	linux1804-64-shippable-qr	fission warm webrender	749.00 -> 503.92

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=30926

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Randell Jesup [:jesup] (needinfo me) from comment #1)

Dave, can you get profiles of paypal in e10s and fission, or let nbp and me how to do so? Thanks

I have triggered these, and you should be able to find the profiles here. Here are the steps I used to generate the profiles:

1. From the alert you can click the graph icon to see the graph view.
2. Click the first datapoint showing the improvement and then click the job link to see the job in Treeherder.
3. You can then click the job link in the job details panel (first link on the left of the panel) to filter out other jobs. By removing references to Fission and 'vismet' from the filter, you can see both the parent Fission and non-Fission jobs (the profile jobs will also show up here).
4. For each job, click to show the job panel, switch to the performance tab if not already selected and click "Generate performance profile".

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

This shows the cold-load improvement: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&selected=3412671,1391074368&series=autoland,3381783,1,13&series=mozilla-central,3746452,1,13&series=mozilla-central,3747961,1,13&series=mozilla-central,3412671,1,13&timerange=7776000&zoom=1626996090147,1627550228445,570.9313725490197,963.4738562091504

Note that this change is linux-only, which may help track down what happened. Warm and cold load (32% and 26%)

This appears to be the changes that could be the cause: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=0c272222c17b3edd7190a24d7171c51eb2f009ba&tochange=5f5c284c5fe32909513debed7b4c3e416f3e7f31

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

Note: the videos do NOT back up there being an improvement here in fission; if anything they show a severe regression in fission. The profiles (from above) back this up; FNBP happens way earlier for e10s/webrender than for fission; the page loads quicker, etc.

Perhaps something is wrong with the visualmetrics calculation - incorrectly setting the start or end point?

This is concerning, since this throws into question all our SpeedIndex/etc measurements

Also, what from the list of checkins above from the first rev that showed this change could have caused this? I don't see any real candidates

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This is indeed concerning. I recall we have seen alerts when the size of the viewport changed (MR1), and when the URL bar colour changed. Perhaps this is a similar issue. :jesup I see you have been discussing this issue with Peter Hedenskog. Could you summarise any findings here?

Comment 9

[Bas Schouten (:bas.schouten)]

qf- since it doesn't look like this is a real thing.

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

This appears to be an issue related to timing and when the video ends; jmaher's checkin of an updated browsertime caused the spurious improvement to disappear; this appears to be solely because of the change in when we check for page done.

Comment 11

[Kimberly Sereduck :kimberlythegeek]

:jesup should I close this bug, or does this need to be investigated further?

Comment 12

[Randell Jesup [:jesup] (needinfo me)]

We believe this was related to the time period used for recording. It was not a real improvement.