Closed Bug 1725706 Opened 4 years ago Closed 4 years ago

Spurious 32% Performance improvement in PayPal fission (only) Linux (only)

Categories

(Testing :: Performance, defect, P3)

Unspecified
Linux
defect

Tracking

(Performance Impact:none)

RESOLVED WORKSFORME
Performance Impact none

People

(Reporter: jesup, Unassigned)

Details

(Whiteboard: [fission-perf])

Unexplained large improvement in PayPal (32.7%), for fission only (?)
The list of changes points to bug 1719194 as the most likely case, but it's not clear why that would cause such a change, for fission only.

Note that warmloads currently involve navigating to about:blank inbetween each load of paypal, and in fission that will cause us to use a new process for paypal on every new warm load, so that might be involved. Perhaps paypal gc's during load if the process exists (e10s), and avoids the GC in fission with a brand-new process?

We're changing warmload tests to be site/page1 -> site/page2 -> site/page1 etc, which might affect this case.

Profiles of a warmload in e10s and fission may well show the issue.

Dave, can you get profiles of paypal in e10s and fission, or let nbp and me how to do so? Thanks

Flags: needinfo?(nicolas.b.pierron)
Flags: needinfo?(dave.hunt)

FYI as to why I filed a bug on this:
Since it was unexpected, I'd like to know why it had this impact. Who knows, it might point to a way to help other sites. If we don't understand it, we could accidentally undo it later -- or it may point to a problem in how we're testing/measuring (i.e. illusionary gain, or a non-real-world gain). This also gives us a huge advantage over e10s for that one site if it's real, and if we could extend that it would be great.

== Change summary for alert #30926 (as of Fri, 13 Aug 2021 18:25:43 GMT) ==

Improvements:

Ratio Test Platform Options Absolute values (old vs new)
33% paypal SpeedIndex linux1804-64-shippable-qr fission warm webrender 749.00 -> 503.92

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=30926

(In reply to Randell Jesup [:jesup] (needinfo me) from comment #1)

Dave, can you get profiles of paypal in e10s and fission, or let nbp and me how to do so? Thanks

I have triggered these, and you should be able to find the profiles here. Here are the steps I used to generate the profiles:

  1. From the alert you can click the graph icon to see the graph view.
  2. Click the first datapoint showing the improvement and then click the job link to see the job in Treeherder.
  3. You can then click the job link in the job details panel (first link on the left of the panel) to filter out other jobs. By removing references to Fission and 'vismet' from the filter, you can see both the parent Fission and non-Fission jobs (the profile jobs will also show up here).
  4. For each job, click to show the job panel, switch to the performance tab if not already selected and click "Generate performance profile".
Flags: needinfo?(dave.hunt)
OS: Unspecified → Linux
Summary: 32% Performance improvement in PayPal fission (only) warmloads → 32% Performance improvement in PayPal fission (only) Linux (only)

Note: the videos do NOT back up there being an improvement here in fission; if anything they show a severe regression in fission. The profiles (from above) back this up; FNBP happens way earlier for e10s/webrender than for fission; the page loads quicker, etc.

Perhaps something is wrong with the visualmetrics calculation - incorrectly setting the start or end point?

This is concerning, since this throws into question all our SpeedIndex/etc measurements

Also, what from the list of checkins above from the first rev that showed this change could have caused this? I don't see any real candidates

Flags: needinfo?(nicolas.b.pierron) → needinfo?(dave.hunt)
Summary: 32% Performance improvement in PayPal fission (only) Linux (only) → Spurious 32% Performance improvement in PayPal fission (only) Linux (only)

This is indeed concerning. I recall we have seen alerts when the size of the viewport changed (MR1), and when the URL bar colour changed. Perhaps this is a similar issue. :jesup I see you have been discussing this issue with Peter Hedenskog. Could you summarise any findings here?

Flags: needinfo?(dave.hunt) → needinfo?(rjesup)

qf- since it doesn't look like this is a real thing.

Whiteboard: [qf][fission-perf] → [qf-][fission-perf]

This appears to be an issue related to timing and when the video ends; jmaher's checkin of an updated browsertime caused the spurious improvement to disappear; this appears to be solely because of the change in when we check for page done.

Flags: needinfo?(rjesup)
Component: JavaScript Engine → Performance
Product: Core → Testing
Priority: -- → P3

:jesup should I close this bug, or does this need to be investigated further?

Flags: needinfo?(rjesup)

We believe this was related to the time period used for recording. It was not a real improvement.

Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(rjesup)
Resolution: --- → WORKSFORME
Performance Impact: --- → -
Whiteboard: [qf-][fission-perf] → [fission-perf]
You need to log in before you can comment on or make changes to this bug.