Closed Bug 1733434 Opened 3 years ago Closed 3 years ago

CNN live site test intermittently fails often

Categories

(Testing :: Raptor, defect, P1)

defect

Tracking

(firefox95 fixed)

RESOLVED FIXED
95 Branch
Tracking Status
firefox95 --- fixed

People

(Reporter: sparky, Assigned: sparky)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

The CNN live site test is currently failing very often (see bug 1680214). The purpose of this site was to test (and sheriff) a page with a high number of processes. This test has been failing for some time though and at one point, we tried to switch to another website (www.foxnews.com) but it also failed intermittently quite often: https://treeherder.mozilla.org/jobs?repo=try&revision=2467b904159b57b72feb7fce0ba8488ea9d9608e

From some discussions I had with jesup a while ago, it was suggested that this is happening because there are "are 3rd-party iframes/ads that are taking forever to load": https://matrix.to/#/!ECdrESRnYfSJXBzaMX:mozilla.org/$jZtuf9nAqnlIwhHjidirsD-wnGHWKjqrrMXwgG-dK6E?via=mozilla.org&via=matrix.org&via=nordgedanken.dev

We'll need to figure out how to solve this issue so we can start sheriffing those tests again.

Assignee: nobody → gmierz2
Priority: P2 → P1

Based on this last try run I've made, iframes are definitely part of the problem. I can't read/access all of them in a webpage because of cross-origin issues but for those that I can access I'm seeing that they never/intermittently get into a "complete" readyState.

Try run that shows this (note how all tests fail after a 2-minute timeout): https://treeherder.mozilla.org/jobs?repo=try&revision=f12f831516788b2a85ce748bdbde4630ec5df67e

I've made some changes to exclude the iframe checks. In this push, I have the default pageCompleteCheck method but it waits for 15s rather than 5s and I also added a window.stop() call in the hopes that it'll stop some of the extra loading going on at the end of a test: https://treeherder.mozilla.org/jobs?repo=try&revision=764a39105906b6e25932961339d7bc030db3bd1c

Another try run here: https://treeherder.mozilla.org/jobs?repo=try&revision=256b7df69f03515b052a26b7ac5a150551674f64

I've noticed that the video on this page autplays which is not a default behaviour for Firefox so I've attempted to disable it. The prefs weren't getting set when I changed them in the common user.js profile prefs which was because browsertime was overwriting them: https://github.com/gmierz/browsertime/commit/814fa863894e4c2aeb4cbccba928864b268c125f

Blocks: 1734414

I've managed to make some progress on this.

Here's a try run with no changes: https://treeherder.mozilla.org/jobs?repo=try&revision=8e59b58767d25f402fbae9fdc8aa3ab4a819ba13

In this try run, I've decreased the number of cycles to 10 and disabled the media autoplay: https://treeherder.mozilla.org/jobs?repo=try&revision=256b7df69f03515b052a26b7ac5a150551674f64

The patch shows a 50-75% decrease in the occurrence of this failure.

This patch reduces the number of browser-cycles that a live-site test (cnn in this case) does from 25 down to 5. Furthermore, we also disable media autoplay for all live site tests. This change required an update to browsertime to prevent default prefs from being set and overriding our prefs.

See Also: → 1734654
Pushed by gmierz2@outlook.com:
https://hg.mozilla.org/integration/autoland/rev/e6df1ed94379
Reduce browser-cycles and disable media autoplay in live site tests. r=perftest-reviewers,kimberlythegeek
Blocks: 1735390

Backed out changeset e6df1ed94379 (Bug 1733434) for causing btime and awsy failures.
Backout link: https://hg.mozilla.org/integration/autoland/rev/d1e8ca45b22f6aa52e537cb45c28d2eb79e59794
Push with failures, btime, awsy failures.

Flags: needinfo?(gmierz2)
Pushed by malexandru@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8c15d07ec653
Reduce browser-cycles and disable media autoplay in live site tests. r=perftest-reviewers,kimberlythegeek

Re-landed changeset e6df1ed94379 since the failures were caused by another push, sorry for the inconvenience!
Re-land link: https://hg.mozilla.org/integration/autoland/rev/8c15d07ec653ac53feb382e4771799589915ed58

Flags: needinfo?(gmierz2)
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 95 Branch
Blocks: 1734654
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: