1592918 - 2.39 - 8.9% raptor-tp6-reddit-firefox-cold fcp / raptor-tp6-yandex-firefox-cold / raptor-tp6-yandex-firefox-cold fcp (linux64-shippable-qr, windows10-64-shippable-qr) regression on push f526a659b51ac9e46cb9c8abc1f685e5f9eaf372 (Tue October 29 2019)

Status: RESOLVED FIXED

(Firefox :: Address Bar, defect, P3)

Description

[Florin Strugariu [:Bebe]]

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=f526a659b51ac9e46cb9c8abc1f685e5f9eaf372

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

9% raptor-tp6-yandex-firefox-cold fcp linux64-shippable-qr opt 310.75 -> 338.42
7% raptor-tp6-yandex-firefox-cold fcp windows10-64-shippable-qr opt 278.88 -> 298.83
7% raptor-tp6-yandex-firefox-cold fcp windows10-64-shippable-qr opt 277.92 -> 297.00
5% raptor-tp6-reddit-firefox-cold fcp windows10-64-shippable-qr opt 287.50 -> 303.08
2% raptor-tp6-yandex-firefox-cold linux64-shippable-qr opt 654.61 -> 670.26

Improvements:

12% raptor-tp6-slides-firefox-cold loadtime windows7-32-shippable opt 2,285.17 -> 2,007.75
3% raptor-tp6-slides-firefox-cold windows7-32-shippable opt 1,256.47 -> 1,215.59

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=23607

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/TestEngineering/Performance/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling

Comment 1

[Marco Bonardo [:mak]]

Hm, I doubt that changing the urlbar design can have a real impact on page loads. If it does, it looks like a bug in the Raptor measurement.

Since this was a nightly only enabled feature, and we're just moving back to the Release/Beta status quo, I doubt this would block any release. It would still be interesting to understand how/why a Firefox UI change affects these Raptor Tp6 tests.

What is fcp?

Comment 2

[Robert Wood [:rwood]]

(In reply to Marco Bonardo [:mak] from comment #1)

What is fcp?

Hi, fcp = 'first-contentful-paint', one of the page-load measurements taken by Raptor.

https://developer.mozilla.org/en-US/docs/Glossary/First_contentful_paint

https://wiki.mozilla.org/TestEngineering/Performance/Raptor#Page-Load_Tests

Comment 3

[Marco Bonardo [:mak]]

How does the load happen? How can the urlbar design influence it?

Comment 4

[Robert Wood [:rwood]]

(In reply to Marco Bonardo [:mak] from comment #3)

How does the load happen? How can the urlbar design influence it?

Raptor uses a web extension to load test pages (which are mitmproxy page recordings played back through a local proxy). I don't know anything about the urlbar design and don't know if it's possible that is effecting the first-contentful-paint value or not. Maybe someone on the performance team might have some input or ideas on how that can be checked? Perhaps via gecko profiling?

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

URLbar design could in theory affect pageload if the target of the load is injected as keystrokes. Is raptor loading pages via marionette? Even if it isn't, the URLbar is updated at the start of pageload with the new URL. Depending on if this delays some other aspects this could affect pageload. Even extra paints for the bar could affect it. Not sure if these things are happening, but they could

Comment 6

[Marco Bonardo [:mak]]

We should check what happens when we'll re-enable the megabar in Nightly, comparing the 2 moves may give us a better idea of what exactly is shifting.

Comment 7

[Andrew Creskey [:acreskey]]

I'm collecting profiles from try with screencaptures enabled to see if we can can spot a difference.

Comment 8

[Andrew Creskey [:acreskey]]

Sorry for the delay.
Looking into this but when I push a job to try from these older revisions I see no jobs (nor can I add them).
e.g.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=aeb305e65a1de9fac4874cbb6f6aed20fffa9e1e

But regardless, I don't think there's much we can do except as :mak said, see see what happens when the megabar is re-enabled.

Comment 9

[Marco Bonardo [:mak]]

The regressions reverted now that we re-enabled the new design in bug 1599784.

https://treeherder.mozilla.org/perf.html#/alerts?id=24254

Yandex and Reddit seem the ones most affected in both reports.
This doesn't sound like an urlbar bug anymore, the regression is actually an improvement since the new design is our target, even if it's likely there's a problem in how things are measured if the ui shouldn't influence measurements too much.

Note, you may compare profiles with browser.urlbar.update1 set to true and false, and compare those.
How do we want to proceed here?

Comment 10

[Andrew Creskey [:acreskey]]

Thank you Marco.
I personally don't see anything actionable here.

Comment 11

[Marco Bonardo [:mak]]

Fine, anyway the "regression" reverted.