1732083 - [meta] Resolve intermittent and noisy speedIndex tests

Status: NEW

(Testing :: Raptor, defect, P2)

Description

[Denis Palmeiro [:denispal]]

Meta bug to track any intermittent problems surrounding speedindex. Things such as missing images in recordings, or intermittent popups distorting the results.

Comment 1

[Kash Shampur [:kshampur] ⌚EST]

Hey Denis are these still an issue?

Comment 2

[Denis Palmeiro [:denispal]]

yeah, this should still be an issue unless the recordings were changed.

Comment 3

[Greg Mierzwinski [:sparky]]

The recording for nytimes has been updated on Jan 31: https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tooltool-manifests/playback/mitm7-linux-firefox-nytimes.manifest

For the other test pages, all of them were updated in January except for Pinterest.

Comment 4

[Denis Palmeiro [:denispal]]

I have not checked every single site, but there are still a lot of problems. Some examples are gdocs, netflix, nytimes, office, and twitch.

Comment 5

[Denis Palmeiro [:denispal]]

Attachment: twitch-1-scaled.mp4

Comment 6

[Denis Palmeiro [:denispal]]

Attachment: netflix-1-scaled.mp4

Comment 7

[Denis Palmeiro [:denispal]]

Attachment: nytimes-1-scaled.mp4

Comment 8

[Greg Mierzwinski [:sparky]]

Based on some discussions in element, some of these are broken both locally, and in CI (twitch) whereas others are issues relating to playing back newer recordings on older firefox versions (netflix).

We also found that there are issues with imdb.