1574432 - Investigate noise in raptor-tp6-yahoo-news-firefox loadtime linux64-shippable opt

Status: RESOLVED WONTFIX

(Testing :: Raptor, defect, P3)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Subtest traptor-tp6-yahoo-news-firefox loadtime linux64-shippable opt from the traptor-tp6 suite presents a lot of noise, on linux64-shippable platform.

The noise levels affect the sheriffing process & hide even obvious perf changes from Perfherder's alert detection system.

Graph

The subtest should either be improved or it should not generate any more alerts, as the overall raptor-tp6-yahoo-news-firefox is more stable.

Comment 1

[Florin Strugariu [:Bebe]]

Not sure what we can do with this test.
The test runs file locally and it's loading properly not sure what we can do to make the test more stable.

Rob If you have any suggestion I'm more than happy to try

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This might be a good candidate for comparing with browsertime.

Comment 3

[Robert Wood [:rwood]]

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #1)

Not sure what we can do with this test.
The test runs file locally and it's loading properly not sure what we can do to make the test more stable.

Rob If you have any suggestion I'm more than happy to try

No suggestions, if the data is too noisy I'd suggest disabling the test on linux (but leaving it available on try), and as Dave noted eventually compare it with browsertime.

Comment 4

[Intermittent Failures Robot]

1 failures in 3801 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64-ref-hw-2017: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1574432&startday=2019-10-07&endday=2019-10-13&tree=all

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Is this still an issue? We're not running the warm page load on integration any more. The cold page load looks reasonably stable to me.

https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=autoland,2134604,1,10&timerange=2592000

Comment 6

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Dave, I'm not able anymore to see warm tp6 tests in the graph, so I guess this bug is not relevant anymore.

Comment 7

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Dave, I'm not able anymore to see warm tp6 tests in the graph, so I guess this bug is not relevant anymore.

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Alexandru Ionescu :alexandrui (needinfo me) from comment #7)

Dave, I'm not able anymore to see warm tp6 tests in the graph, so I guess this bug is not relevant anymore.

Right, that's what I said in comment 5. I wasn't certain if this bug was limited to warm page load, but your comment suggests that it is. Thanks.