Closed Bug 1668241 Opened 4 years ago Closed 4 years ago

Google-maps recording is broken on mobile

Categories

(Testing :: Raptor, defect, P1)

defect

Tracking

(firefox86 fixed)

RESOLVED FIXED
86 Branch
Tracking Status
firefox86 --- fixed

People

(Reporter: sparky, Assigned: alexandrui)

References

Details

(Keywords: perf-alert)

Attachments

(3 files)

The google-maps recording is broken because it doesn't load up the actual map before finishing. It's either because of browsertime or the recording itself.

See this comment for some sample videos: https://bugzilla.mozilla.org/show_bug.cgi?id=1667938#c9

I don't know if this is the case here, but the --pageLoadStrategy", "none" that we configure browsertime means that, to my understanding, we're not waiting for the load event. (We are only waiting for the root url to download and then 5 seconds).
This may not be sufficient for sites that populate asynchronously.

https://searchfox.org/mozilla-central/rev/dfd9c0f72f9765bd4a187444e0c1e19e8834a506/testing/raptor/raptor/browsertime/base.py#214

Is this still an issue, :sparky?

Flags: needinfo?(gmierz2)

:sparky, I see no failing job in the screenshot attached.

Flags: needinfo?(gmierz2)

(In reply to Alexandru Ionescu (needinfo me) [:alexandrui] from comment #6)

:sparky, I see no failing job in the screenshot attached.

This bug isn't about failing jobs, it's about the incomplete page load. You can download the browsertime-results.tgz from the job linked in comment 4 to see the problem described in comment 0.

Flags: needinfo?(gmierz2)
Assignee: nobody → aionescu
Priority: P2 → P1
Status: NEW → ASSIGNED

According to this comparison there will be some regressions, but giving the times were low because the map wasn't loading, it's expected.

Pushed by aionescu@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/52f2c63efff8 Google-maps recording is broken on mobile r=perftest-reviewers,Bebe
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → 86 Branch

== Change summary for alert #28120 (as of Thu, 17 Dec 2020 08:26:46 GMT) ==

Regressions:

Ratio Suite Test Platform Options Absolute values (old vs new)
121% google-maps SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 849.75 -> 1,879.92
46% google-maps SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 1,753.08 -> 2,556.42
44% google-maps fcp android-hw-p2-8-0-android-aarch64-shippable cold nocondprof webrender 441.75 -> 635.29
30% google-maps FirstVisualChange android-hw-p2-8-0-android-aarch64-shippable cold nocondprof webrender 740.75 -> 961.17
29% google-maps loadtime android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 1,495.54 -> 1,922.50
28% google-maps ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 1,685.75 -> 2,162.75
28% google-maps loadtime android-hw-p2-8-0-android-aarch64-shippable cold nocondprof webrender 996.62 -> 1,271.25
23% google-maps PerceptualSpeedIndex android-hw-p2-8-0-android-aarch64-shippable cold nocondprof webrender 880.17 -> 1,083.92
16% google-maps PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 862.38 -> 1,002.08
16% google-maps SpeedIndex android-hw-p2-8-0-android-aarch64-shippable cold nocondprof webrender 1,444.71 -> 1,672.58
16% google-maps PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 864.75 -> 1,000.42
14% google-maps ContentfulSpeedIndex android-hw-p2-8-0-android-aarch64-shippable cold nocondprof webrender 1,188.29 -> 1,358.08
12% google-maps FirstVisualChange android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 426.96 -> 479.25
12% google-maps PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 1,746.25 -> 1,955.42
11% google-maps FirstVisualChange android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 806.17 -> 894.33
11% google-maps fcp android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 884.67 -> 981.33
9% google-maps FirstVisualChange android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 1,681.08 -> 1,836.33

Improvements:

Ratio Suite Test Platform Options Absolute values (old vs new)
60% google-maps LastVisualChange android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 4,102.50 -> 1,621.42
59% google-maps LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 6,290.50 -> 2,550.92
58% google-maps loadtime android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,086.67 -> 460.75
47% google-maps loadtime android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 569.83 -> 300.96
46% google-maps android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 839.34 -> 449.92
36% google-maps android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 450.30 -> 286.92
33% google-maps LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 6,313.92 -> 4,233.83
29% google-maps SpeedIndex android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 1,177.58 -> 834.08
29% google-maps ContentfulSpeedIndex android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 918.92 -> 650.92
20% google-maps android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 1,013.69 -> 807.90
7% google-maps fcp android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 763.46 -> 713.08

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=28120

Keywords: perf-alert
Flags: needinfo?(aionescu)

Giving the previous recording wasn't loading the map, I would say that the regressions are expected. I'm a bit concerned about the 121% of SpeedIndex.

Flags: needinfo?(aionescu)
Attached video cold-side-by-side.mp4

I wouldn't be concerned about the regressions even the 121% one. Here's a before/after video showing that everything's working now.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: