1666213 - 3.37 - 25.65% allrecipes/instagram/youtube-watch/youtube-watch ContentfulSpeedIndex (android-hw-g5-7-0-arm7-api-16-shippable, android-hw-p2-8-0-android-aarch64-shippable) regression on push 4951412ca28ee33b50f4f8af722d01703feb7099 (Tue September 15 2020)

Status: RESOLVED INVALID

(Remote Protocol :: Marionette, defect)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Perfherder has detected a browsertime performance regression from push 4951412ca28ee33b50f4f8af722d01703feb7099. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

26% instagram ContentfulSpeedIndex android-hw-p2-8-0-android-aarch64-shippable opt cold 815.46 -> 1,024.58
15% allrecipes ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,532.17 -> 2,910.08
12% allrecipes ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,586.54 -> 2,887.00
4% youtube-watch LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable opt warm 1,968.33 -> 2,051.83
3% youtube-watch loadtime android-hw-g5-7-0-arm7-api-16-shippable opt warm 1,595.79 -> 1,649.50

Improvements:

23% google-maps ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 1,454.25 -> 1,121.33
3% allrecipes SpeedIndex android-hw-p2-8-0-android-aarch64-shippable opt warm 1,145.58 -> 1,112.33
3% allrecipes loadtime android-hw-p2-8-0-android-aarch64-shippable opt warm 1,923.83 -> 1,869.17
2% allrecipes PerceptualSpeedIndex android-hw-p2-8-0-android-aarch64-shippable opt warm 989.50 -> 965.58

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Browsertime doesn't use the geckodriver / Marionette navigation commands but just a call to Execute Async Script. So I don't see how this could have been influenced browsertime. Andrew, do you have any ideas?

Comment 2

[Andrew Creskey [:acreskey]]

This is quite interesting.
I've just starting looking at the impact of this change via a modified Browsertime that actually uses webdriver navigation.
While it's running nicely locally, so far in CI this change is failing for me.
Let me see if I can find out why this is happening by profiling.

Comment 3

[Andrew Creskey [:acreskey]]

But ultimately Bug 1612831 is a change to the test harness to make it more closely follow the user experience.
This regression wouldn't impact users -- it's making our tests more realistic.

Comment 4

[Alexandru Ionescu (needinfo me) [:alexandrui]]

I did some retriggers maybe we caught an outliar.

Comment 5

[Alexandru Ionescu (needinfo me) [:alexandrui]]

I can't manage to retrigger vismet on a different instance of the test. Sparky, could you please do 5 retriggers on this range?

Comment 6

[Greg Mierzwinski [:sparky]]

:alexandrui, we have docs for this here: https://wiki.mozilla.org/TestEngineering/Performance/Raptor/Browsertime#Retriggering_Browsertime_Visual_Metrics_Tasks

I've retriggered the tasks.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Any updates here? Greg, can you please have a look for the retriggered tasks?

Comment 8

[Greg Mierzwinski [:sparky]]

It's a legitimate change, not an outlier. I think we should resolve this as WONTFIX/INVALID like the other alert we were looking at.

Videos from this patch: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OEtXXru9T9CoCb8ALEOHVg/runs/0/artifacts/public/test_info/browsertime-results.tgz

Videos from previous patch (without regression): https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LxPM3DwjRG2YpYdCryy1xQ/runs/0/artifacts/public/test_info/browsertime-results.tgz

Comment 9

[Andrew Creskey [:acreskey]]

I don't see any issues in the videos but I agree that this should be resolved at WONTFIX/INVALID.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thank you both for checking!