1675502 - 5.38 - 5.62% booking ContentfulSpeedIndex (android-hw-g5-7-0-arm7-api-16-shippable) regression on push 4f69567e1cb1837b0260ac6f174aa9799b622417 (Sat October 31 2020)

Status: RESOLVED WONTFIX

(Core :: Panning and Zooming, defect)

Description

[Florin Strugariu [:Bebe]]

Perfherder has detected a browsertime performance regression from push 4f69567e1cb1837b0260ac6f174aa9799b622417. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
6%	booking	ContentfulSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	cold	1,090.75 -> 1,152.00
5%	booking	ContentfulSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	cold	1,088.75 -> 1,147.33

Improvements:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
9%	bing	ContentfulSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm	734.00 -> 670.50
4%	facebook	Similarity2D	android-hw-g5-7-0-arm7-api-16-shippable	cold	0.95 -> 0.99
2%	facebook	Similarity2D	android-hw-p2-8-0-android-aarch64-shippable	cold	0.88 -> 0.91

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Timothy Nikkel (:tnikkel)]

11% regression on booking cold, 9% improvement on bing warm. And similarity improvements (I think this means rendered better?). Pretty close to a wash. I'll do a few try pushes to try to narrow down what part of the pref flipped on code caused the regression, but I think I will probably just recommend we take this.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1671331

Comment 3

[Timothy Nikkel (:tnikkel)]

Okay, these tests are really hard to find. They don't even show up in ./mach try chooser --full. you have to push and then add new jobs search, and they are under vismet, which is not clear from the name.

Comment 4

[Timothy Nikkel (:tnikkel)]

And retriggers don't seem to work for those jobs.

Comment 5

[Timothy Nikkel (:tnikkel)]

And if you use add new jobs to get more runs the new jobs have identical numbers to the previous ones, which can't be true, so it must not be generating a new set of numbers.

So this is basically a giant pain to actually investigate. If we actually care about this test we need to do a lot of work to get it up to the level of other perf tests.

Further the few results I do have suggest that my patch may not have caused a regression.

Comment 6

[Timothy Nikkel (:tnikkel)]

Okay, after many many pushes and wasted PGO builds on try (no other way to get more results) it is this hunk

https://searchfox.org/mozilla-central/rev/50215d649d4854812837f1343e8f47bd998dacb5/layout/generic/nsGfxScrollFrame.cpp#545

that causes the regression.

That hunk decides if a site gets scrollbars or not and the decision is slightly changed. booking.com seems to get scrollbars both before and after my change, so perhaps during loading we decide not to give it scrollbars before finally giving it scrollbars and this causes the regression?

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Timothy Nikkel (:tnikkel) from comment #5)

And if you use add new jobs to get more runs the new jobs have identical numbers to the previous ones, which can't be true, so it must not be generating a new set of numbers.

So this is basically a giant pain to actually investigate. If we actually care about this test we need to do a lot of work to get it up to the level of other perf tests.

Further the few results I do have suggest that my patch may not have caused a regression.

Sorry for the frustration this has caused. We're aware of the challenges around running and retriggering these tests, and improving this workflow is a priority for the performance team. I believe bug 1677559 will help with this particular scenario.

:sparky can you confirm :tnikkel's theory on the scrollbars loading later?

Comment 8

[Greg Mierzwinski [:sparky]]

Attachment: cold-side-by-side.mp4

Yup it does load the scroll bar a bit later. It looks like the page load is slower overall as well.

Comment 9

[Greg Mierzwinski [:sparky]]

Resolving this performance alert as WONTFIX given that it was likely only a change in scroll bar that caused it. :tnikkel, feel free to change the resolution if needed.