1699737 - 9.82 - 13.86% booking / booking FirstVisualChange / booking fcp (android-hw-g5-7-0-arm7-api-16-shippable) regression on push 5b32fd7d93c1753382e0ddc257c425908cc41889 (Wed March 10 2021)

Status: RESOLVED WORKSFORME

(Remote Protocol :: Marionette, defect)

Description

[Florin Strugariu [:Bebe]]

Perfherder has detected a browsertime performance regression from push 5b32fd7d93c1753382e0ddc257c425908cc41889. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
14%	booking	fcp	android-hw-g5-7-0-arm7-api-16-shippable	nocondprof warm webrender	503.77 -> 573.58
10%	booking	FirstVisualChange	android-hw-g5-7-0-arm7-api-16-shippable	nocondprof warm webrender	572.38 -> 629.50
10%	booking		android-hw-g5-7-0-arm7-api-16-shippable	nocondprof warm webrender	779.28 -> 855.80

Improvements:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
51%	youtube	LastVisualChange	windows10-64-shippable	nocondprof warm	4,610.00 -> 2,260.00
50%	youtube	LastVisualChange	windows10-64-shippable	nocondprof warm	4,545.00 -> 2,256.67
19%	booking	PerceptualSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	nocondprof warm webrender	983.62 -> 794.17
8%	espn	LastVisualChange	android-hw-g5-7-0-arm7-api-16-shippable	nocondprof warm webrender	6,051.67 -> 5,573.08
5%	espn	LastVisualChange	android-hw-g5-7-0-arm7-api-16-shippable	nocondprof warm webrender	6,093.38 -> 5,766.50

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

If there is indeed such a bad regression it would have to do with the changes on bug 1663757. Bug 1664881 just makes use of the documentURI as exposed via the BrowsingContext class now.

Nika, do you have any ideas, why we got those huge changes for both regressions and improvements?

Comment 2

[Nika Layzell [:nika] (ni? for response)]

Your changes would've caused a request for the current URI of a BrowsingContext to return more immediately, as it's only fetching the data from the current local process rather than going async over IPC, so there's a chance that the way that these particular benchmarks use marionette was causing timing to change?

The fact this caused such a significant change to performance numbers makes me somewhat worried about how these values are being measured by our performance test suites.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Andrew and Greg, maybe one one of you could further help to understand how those numbers are measured?

Comment 4

[Julian Descottes [:jdescottes]]

FWIW If you look at individual graphs from the alert (eg graph 1, graph 2) the values seem quite flaky.

Comment 5

[Greg Mierzwinski [:sparky]]

Attachment: 24.mp4

:bebe can you run some more retriggers/backfill for the booking regression? It looks like the vismet changes might be from this patch, and it's hard to tell from the fcp graph: https://treeherder.mozilla.org/jobs?repo=autoland&selectedJob=332796908&group_state=expanded&searchStr=browsertime%2Cbooking%2Candroid&revision=537e54c210e7637da745d40d3060f5f685cc9cb2

I'm not sure which metric you're asking about, but the "LastVisualChange" is the last time something visually changed on the page. The youtube improvements were definitely caused by this patch and it's because the end of the youtube videos no longer has a video link that changes. I've added a video example of what was happening before, watch how the middle video thumbnail on the first row shows a broken link that then becomes a play sign at the end of the test (very briefly). With this patch applied, the video thumbnail stays broken throughout the whole test.

Comment 6

[Florin Strugariu [:Bebe]]

did some retrigers.
Also notice that the alert came back to it's original values.
https://treeherder.mozilla.org/perfherder/graphs?timerange=1209600&series=autoland,2891214,1,13

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #6)

did some retrigers.
Also notice that the alert came back to it's original values.
https://treeherder.mozilla.org/perfherder/graphs?timerange=1209600&series=autoland,2891214,1,13

So what were the results? Maybe you could help to interpret the results? Thanks.

Comment 8

[Florin Strugariu [:Bebe]]

All booking alerts regressed/improved fully on 5b32fd7d93c1753382e0ddc257c425908cc41889 with numbers going up in 2d1aaf9b970047d847d51a380c476c669731d715 also.
The booking alerts came back to the original values on: e1fa9dc7464ba49efdf422116ac2c92e2d40a200

ESPN alerts regressed/improved 5b32fd7d93c1753382e0ddc257c425908cc41889 and did not came back to the original values
Youtube alerts improved on 5b32fd7d93c1753382e0ddc257c425908cc41889 and the values did not changed since

in my opinion booking might be affected by 2d1aaf9b970047d847d51a380c476c669731d715 but i can't say that 100% the rest of the alerts are 100% from this patch

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #8)

All booking alerts regressed/improved fully on 5b32fd7d93c1753382e0ddc257c425908cc41889 with numbers going up in 2d1aaf9b970047d847d51a380c476c669731d715 also.
The booking alerts came back to the original values on: e1fa9dc7464ba49efdf422116ac2c92e2d40a200

Regressions were only mentioned for booking in comment 0. Given your last comment all alerts for that test came back to their original values. So it means that there is no more work to do on this bug?

Comment 10

[Florin Strugariu [:Bebe]]

Yes we can mark this as fixed

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks.