1931740 - Backfills on "Base content JS opt fission" for Linux and Mac shows numbers that are most probably incorrect and noisy (Windows is very stable)

Status: NEW

(Testing :: AWSY, defect, P3)

Description

[Mayank Bansal]

Look at this overall graph : https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&replicates=0&series=autoland,3806445,1,4&timerange=2592000

1. This area has two datapoints. The one on top was the original datapoint. The one at the bottom was the result of backfill. Why would a backfill value show a 2kb-3kb lower value?
2. This shows a 2KB improvement from a rename change.
3. This whole area has effectively tri-modal behaviour.
4. This shows a 2KB improvement from a CSS change
5. This shows a 2kb improvement from an improvement to search-indexer

Look at the corresponding graph for Windows, which also had backfills. But this is very consistent with 0 noise.
I didnt check for MacOS.

Comment 1

[Mayank Bansal]

Look at the similar range for MacOs: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&replicates=0&series=autoland,3807462,1,4&timerange=7776000&zoom=1730434301459,1732125829683,1498072.3985405292,1531820.9796795167

This is also noisy and incorrect like Linux.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:kshampur, could you have a look please?

For more information, please visit BugBot documentation.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

@mayank are you seeing this just for backfills, or also retries? The links in comment 0 are no longer working - could you attach screenshots for the issues you're seeing? Do we see the same variance if we exclude backfills and retriggers?

Comment 4

[Mayank Bansal]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #3)

The links in comment 0 are no longer working - could you attach screenshots for the issues you're seeing?

I have updated the links that will work for few days. Is there a way to have "permanent links" that will open the graph in the same timerange and with same parameters?

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #3)

@mayank are you seeing this just for backfills, or also retries? Do we see the same variance if we exclude backfills and retriggers?

I observed them when I did backfills with default settings. These settings look like this :

depth: 19/<insert number>
inclusive: false
retrigger: true
times: 1

Comment 5

[Mayank Bansal]

Attachment: Overall graph.png

Comment 6

[Mayank Bansal]

Attachment: Slide1.PNG