1554969 - Run raptor-youtube-playback-firefox jobs on integration branches with alerting enabled

Status: RESOLVED FIXED

(Testing :: Raptor, task, P2)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Currently the tests only run on mozilla-central, but ultimately we also want to see them on integration branches. Before we are able to enable that it has to be clear which alert_threshold has to be used for the platforms, all dependent on the reported noise.

If the noise it too high it might be worth to enable only stable platforms (those with lesser dropped frames) first, and enhance support for others step by step.

Ionuts will be able to help out in case of questions for perfherder results.

Btw. here are all the results:

https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=mozilla-central,2041824,1,10&series=mozilla-central,2042980,1,10&series=mozilla-central,2042402,1,10&series=mozilla-central,2051456,1,10&series=mozilla-central,2042691,1,10&series=mozilla-central,2051167,1,10&series=mozilla-central,2043154,1,10&series=mozilla-central,2043558,1,10

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I would suggest alerting on the dropped frames subtests using our standard 2% unless there's another suggestion? Ionut, could you take a look over the data from mozilla-central to see if this would generate an excessive amount of alerts?

Comment 2

[Ionuț Goldan [:igoldan]]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #1)

I would suggest alerting on the dropped frames subtests using our standard 2% unless there's another suggestion? Ionut, could you take a look over the data from mozilla-central to see if this would generate an excessive amount of alerts?

The graphs look pretty decent.
I never attempted to programmatically simulate alert creation based on mozilla-central graphs, so we have more precision.
Dave, if you or someone else used a tool/script for this before, please share it.

Anyway, we can enable alerting for them. If we're spammed with many alerts, it's easy to turn them off again.
It will cost us 2-3 days of extra Perf sheriff work.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriff from comment #2)

(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #1)

I would suggest alerting on the dropped frames subtests using our standard 2% unless there's another suggestion? Ionut, could you take a look over the data from mozilla-central to see if this would generate an excessive amount of alerts?

The graphs look pretty decent.
I never attempted to programmatically simulate alert creation based on mozilla-central graphs, so we have more precision.
Dave, if you or someone else used a tool/script for this before, please share it.

There's Joel's doalerts, however I'm not sure how useful this is when running against the existing data as we're not currently running against integration branches.

Anyway, we can enable alerting for them. If we're spammed with many alerts, it's easy to turn them off again.
It will cost us 2-3 days of extra Perf sheriff work.

Let's limit this bug to enabling integration for the tests running on desktop Firefox, and only alert on th 1X playback speeds for now. We can then use doalerts after a few weeks to see the impact of enabling alerts for the other subtests. We don't want to alert on the suite level value. Here's a list of the subtests we want to alert on, based on this ActiveData query:

• H264.1080p30@1X_dropped_frames
• H264.1080p60@1X_dropped_frames
• H264.1440p30@1X_dropped_frames
• H264.144p15@1X_dropped_frames
• H264.2160p30@1X_dropped_frames
• H264.240p30@1X_dropped_frames
• H264.360p30@1X_dropped_frames
• H264.480p30@1X_dropped_frames
• H264.720p30@1X_dropped_frames
• H264.720p60@1X_dropped_frames
• VP9.1080p30@1X_dropped_frames
• VP9.1080p60@1X_dropped_frames
• VP9.1440p30@1X_dropped_frames
• VP9.1440p60@1X_dropped_frames
• VP9.144p30@1X_dropped_frames
• VP9.2160p30@1X_dropped_frames
• VP9.2160p60@1X_dropped_frames
• VP9.240p30@1X_dropped_frames
• VP9.360p30@1X_dropped_frames
• VP9.480p30@1X_dropped_frames
• VP9.720p30@1X_dropped_frames
• VP9.720p60@1X_dropped_frames

Comment 4

[Marian Raiciof [:marauder]]

Push to try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=900fe76d92bfcd186bfbb7a72e73d3372883493e

Comment 5

[Marian Raiciof [:marauder]]

The following two patches will apply for this bug too, so i will close this ticket as fixed:
https://phabricator.services.mozilla.com/D40776
https://phabricator.services.mozilla.com/D43049