1677068 - [browsertime] Review visual metrics that should generate regression/improvement alerts

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P1)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

For the moment we're alerting on several visual metrics that are not bringing value to us, but the opposite, is bringing confusion, so there's the proposal to keep only the Speedindex ones:

• ContentfulSpeedIndex
• PerceptualSpeedIndex
• SpeedIndex

Jesup, desnis, what's your input on this?

Comment 1

[Greg Mierzwinski [:sparky]]

videoRecordingStart is the only one we shouldn't be alerting on. The others are useful:

• ContentfulSpeedIndex
• PerceptualSpeedIndex
• SpeedIndex
• FirstVisualChange
• LastVisualChange

Comment 2

[Alexandru Ionescu (needinfo me) [:alexandrui]]

I'm confused as Dave mentioned only the *SpeedIndex ones.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Alexandru Ionescu (needinfo me) [:alexandrui] from comment #2)

I'm confused as Dave mentioned only the *SpeedIndex ones.

sparky is the expert here, if he says there's value in alerting on First/LastVisualChange then I'm not going to disagree.

Comment 4

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Sparky, can I get some help here? Can't find any explicit reference of videoRecordingStart except METRICS_FIELDS which doesn't seem to be used anywhere. I am also having hard times running visual metrics locally so it's taking a long time to run on try on every change I make. Could you suggest where can I disable videoRecordingStart?

Comment 5

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Latest findings reveal that visual metrics have no explicit shouldAlert flag in python/mozperftest/mozperftest/metrics/visualmetrics.py. Have to add it.

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Alexandru Ionescu (needinfo me) [:alexandrui] from comment #5)

Latest findings reveal that visual metrics have no explicit shouldAlert flag in python/mozperftest/mozperftest/metrics/visualmetrics.py. Have to add it.

That is correct, by default Perfherder will alert on metrics without an explicit shouldAlert=false.

Comment 7

[Greg Mierzwinski [:sparky]]

(In reply to Alexandru Ionescu (needinfo me) [:alexandrui] from comment #4)

Sparky, can I get some help here? Can't find any explicit reference of videoRecordingStart except METRICS_FIELDS which doesn't seem to be used anywhere. I am also having hard times running visual metrics locally so it's taking a long time to run on try on every change I make. Could you suggest where can I disable videoRecordingStart?

You're looking in the wrong code - that's mozperftest not raptor. You'll want to modify the code here: https://searchfox.org/mozilla-central/source/taskcluster/docker/visual-metrics/run-visual-metrics.py

Comment 8

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Bug 1677068 Disable videoRecordingStart visual metric r?#perftest

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7643eaee62a3
Disable videoRecordingStart visual metric r=perftest-reviewers,Bebe

Comment 10

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/7643eaee62a3