Closed Bug 1675774 Opened 4 years ago Closed 3 years ago

[mozproy] Drop zero-sized patches from recording at replay time

Categories

(Testing :: Mozbase, defect, P2)

Product:
Testing
Component:
Mozbase
Version:
Firefox 84
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: alexandrui, Assigned: alexandrui)

References

Details

(Keywords: perf-alert)

Attachments

(1 obsolete file)

Currently the recordings are capturing the zero-sized patches that sometimes cause intermittent failures by sometimes waiting on patches that has no content. We need to drop those to avoid causing intermittents in the future.

Attachment #9186249 - Attachment description: Bug 1675774 Drop zero-sized patches from recording at replay time r?#perftest,sparky → Bug 1675774 - Drop zero-sized patches from recording at replay time. r?#perftest,sparky
Pushed by gmierz2@outlook.com:
https://hg.mozilla.org/integration/autoland/rev/ece60a1dbd9d
Drop zero-sized patches from recording at replay time. r=sparky,perftest-reviewers

https://hg.mozilla.org/mozilla-central/rev/ece60a1dbd9d

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
status-firefox84: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 84 Branch

Sparky, aren't the mb/mbu fails related to the patch? I did another try to make sure of that.

Flags: needinfo?(gmierz2)

None of those failures are related to this patch.

Flags: needinfo?(gmierz2)
Regressions: 1676630

Backed out for performance regressions (bug 1676630)

Backout: https://hg.mozilla.org/integration/autoland/rev/88f6a6c2a328a0abc4421e5428b9b1c7a527c8b3

Status: RESOLVED → REOPENED
status-firefox84: fixed → ---
Flags: needinfo?(aionescu)
Resolution: FIXED → ---
Target Milestone: 84 Branch → ---

I will investigate and come back with a fixed patch.

Flags: needinfo?(aionescu)

(In reply to Alexandru Ionescu (needinfo me) [:alexandrui] from comment #7)

I will investigate and come back with a fixed patch.

I would like to understand the issue and the impact it's having before we invest further time into this bug. Could you provide some more details?

Flags: needinfo?(aionescu)

I am having hard times (again) running browsertime locally. As soon as I fix them, I'll run the tests locally and investigate what's happening.

Flags: needinfo?(aionescu)

(In reply to Alexandru Ionescu (needinfo me) [:alexandrui] from comment #9)

I am having hard times (again) running browsertime locally. As soon as I fix them, I'll run the tests locally and investigate what's happening.

This doesn't answer my question. Once we understand the original issue and the impact we can prioritise this appropriately. I don't want to spend time investigating/fixing this patch if it doesn't provide enough value relative to the effort.

Flags: needinfo?(aionescu)

I'll try to understand why is it impacting visual metrics this way as I'm very curious, but with low priority.

Flags: needinfo?(aionescu)

(In reply to Andreea Pavel [:apavel] from comment #6)

Backed out for performance regressions (bug 1676630)

Backout: https://hg.mozilla.org/integration/autoland/rev/88f6a6c2a328a0abc4421e5428b9b1c7a527c8b3

== Change summary for alert #27627 (as of Fri, 13 Nov 2020 02:29:23 GMT) ==

Regressions:

Ratio Suite Test Platform Options Absolute values (old vs new)
26% raptor-tp6-twitter-firefox-cold loadtime windows7-32-shippable 1,427.17 -> 1,794.33
25% raptor-tp6-twitter-firefox-cold loadtime windows10-64-shippable 1,441.58 -> 1,797.42
24% raptor-tp6-twitter-firefox-cold loadtime windows10-64-shippable-qr webrender 1,441.83 -> 1,793.08
8% raptor-tp6-reddit-firefox-cold loadtime windows10-64-shippable-qr webrender 2,752.50 -> 2,979.92
8% raptor-tp6-reddit-firefox-cold loadtime windows10-64-shippable 2,737.50 -> 2,961.75
8% raptor-tp6-reddit-firefox-cold loadtime windows7-32-shippable 2,709.08 -> 2,927.67
6% raptor-tp6-twitter-firefox-cold windows10-64-shippable 454.24 -> 483.12
6% raptor-tp6-twitter-firefox-cold windows7-32-shippable 457.21 -> 483.95
5% raptor-tp6-twitter-firefox-cold windows10-64-shippable-qr webrender 450.16 -> 474.49
3% raptor-tp6-reddit-firefox-cold windows7-32-shippable 644.61 -> 661.35

Improvements:

Ratio Suite Test Platform Options Absolute values (old vs new)
115% raptor-tp6-paypal-firefox-cold replay-confidence windows10-64-shippable 45.17 -> 97.00
113% raptor-tp6-paypal-firefox-cold replay-confidence windows10-64-shippable-qr webrender 45.50 -> 97.00
111% raptor-tp6-paypal-firefox-cold replay-confidence windows7-32-shippable 45.92 -> 97.00
34% raptor-tp6-instagram-firefox-cold replay-confidence windows10-64-shippable-qr webrender 71.83 -> 96.00
33% raptor-tp6-instagram-firefox-cold replay-confidence windows10-64-shippable 72.00 -> 96.00
33% raptor-tp6-instagram-firefox-cold replay-confidence windows7-32-shippable 72.17 -> 96.00
30% raptor-tp6-twitch-firefox-cold replay-confidence windows7-32-shippable 72.00 -> 93.25
27% raptor-tp6-twitch-firefox-cold replay-confidence windows10-64-shippable-qr webrender 72.00 -> 91.58
27% raptor-tp6-twitch-firefox-cold replay-confidence windows10-64-shippable 73.83 -> 93.58
25% raptor-tp6-yahoo-mail-firefox-cold replay-confidence windows10-64-shippable 62.25 -> 78.08
25% raptor-tp6-yahoo-mail-firefox-cold replay-confidence windows7-32-shippable 62.08 -> 77.58
25% raptor-tp6-yahoo-mail-firefox-cold replay-confidence windows10-64-shippable-qr webrender 62.75 -> 78.33
12% raptor-tp6-netflix-firefox-cold replay-confidence windows10-64-shippable 76.58 -> 85.67
11% raptor-tp6-netflix-firefox-cold replay-confidence windows7-32-shippable 79.00 -> 88.00
11% raptor-tp6-netflix-firefox-cold replay-confidence windows10-64-shippable-qr webrender 76.50 -> 85.08
11% raptor-tp6-reddit-firefox-cold replay-confidence windows7-32-shippable 85.67 -> 94.92
10% raptor-tp6-reddit-firefox-cold replay-confidence windows10-64-shippable-qr webrender 85.83 -> 94.58
10% raptor-tp6-reddit-firefox-cold replay-confidence windows10-64-shippable 85.92 -> 94.33
9% raptor-tp6-netflix-firefox-cold fcp windows10-64-shippable-qr webrender 330.50 -> 301.67
8% raptor-tp6-youtube-firefox-cold fcp windows10-64-shippable 515.31 -> 476.08
7% raptor-tp6-amazon-firefox-cold replay-confidence windows10-64-shippable-qr webrender 80.20 -> 85.58
7% raptor-tp6-youtube-firefox-cold fcp windows10-64-shippable-qr webrender 493.58 -> 460.58
6% raptor-tp6-youtube-firefox-cold fcp windows7-32-shippable 515.35 -> 482.00
6% raptor-tp6-linkedin-firefox-cold replay-confidence windows10-64-shippable 85.33 -> 90.67
6% raptor-tp6-yahoo-news-firefox-cold replay-confidence windows10-64-shippable-qr webrender 86.00 -> 91.25
6% raptor-tp6-yahoo-news-firefox-cold replay-confidence windows7-32-shippable 86.00 -> 91.25
6% raptor-tp6-yahoo-news-firefox-cold replay-confidence windows10-64-shippable 86.00 -> 91.17
6% raptor-tp6-amazon-firefox-cold replay-confidence windows7-32-shippable 80.90 -> 85.67
6% raptor-tp6-linkedin-firefox-cold replay-confidence windows7-32-shippable 85.08 -> 90.00
6% raptor-tp6-linkedin-firefox-cold replay-confidence windows10-64-shippable-qr webrender 85.83 -> 90.58
5% raptor-tp6-youtube-firefox-cold windows10-64-shippable-qr webrender 968.49 -> 920.64
5% raptor-tp6-youtube-firefox-cold windows7-32-shippable 989.73 -> 942.20
4% raptor-tp6-youtube-firefox-cold replay-confidence windows7-32-shippable 92.42 -> 96.00
4% raptor-tp6-youtube-firefox-cold replay-confidence windows10-64-shippable-qr webrender 92.50 -> 96.00
4% raptor-tp6-youtube-firefox-cold replay-confidence windows10-64-shippable 92.69 -> 96.17
4% raptor-tp6-google-firefox-cold replay-confidence windows7-32-shippable 83.00 -> 86.00
4% raptor-tp6-microsoft-firefox-cold replay-confidence windows7-32-shippable 74.89 -> 77.58
3% raptor-tp6-google-firefox-cold replay-confidence windows10-64-shippable 83.00 -> 85.75
3% raptor-tp6-google-firefox-cold replay-confidence windows10-64-shippable-qr webrender 82.80 -> 85.50
3% raptor-tp6-imdb-firefox-cold replay-confidence windows10-64-shippable 91.00 -> 93.33
3% raptor-tp6-yandex-firefox-cold replay-confidence windows7-32-shippable 82.00 -> 84.08
3% raptor-tp6-pinterest-firefox-cold replay-confidence windows10-64-shippable 80.00 -> 82.00
3% raptor-tp6-pinterest-firefox-cold replay-confidence windows10-64-shippable-qr webrender 80.00 -> 82.00
3% raptor-tp6-pinterest-firefox-cold replay-confidence windows7-32-shippable 80.00 -> 82.00
2% raptor-tp6-yandex-firefox-cold replay-confidence windows10-64-shippable 82.00 -> 84.00
2% raptor-tp6-yandex-firefox-cold replay-confidence windows10-64-shippable-qr webrender 82.00 -> 84.00
2% raptor-tp6-imdb-firefox-cold replay-confidence windows10-64-shippable-qr webrender 90.84 -> 93.00
2% raptor-tp6-imdb-firefox-cold replay-confidence windows7-32-shippable 90.80 -> 92.92
2% raptor-tp6-facebook-redesign-firefox-cold replay-confidence windows10-64-shippable-qr webrender 87.00 -> 88.83
2% raptor-tp6-outlook-firefox-cold replay-confidence windows10-64-shippable 98.00 -> 100.00
2% raptor-tp6-outlook-firefox-cold replay-confidence windows10-64-shippable-qr webrender 98.00 -> 100.00
2% raptor-tp6-outlook-firefox-cold replay-confidence windows7-32-shippable 98.00 -> 100.00

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=27627

Assignee: aionescu → nobody
Priority: P1 → P2

[Tracking Requested - why for this release]:

status-firefox85: --- → affected
tracking-firefox85: --- → ?

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #14)

[Tracking Requested - why for this release]:

Please explain why you're requesting tracking. It's not clear to me the issue and impact here.

Flags: needinfo?(aionescu)

I believe this was a mistake but :bebe can correct me if I'm wrong. This bug caused a performance regression when it landed leading to it being backed out. When we file independant regression bugs we set the status flag for the affected version. We likely need to improve our workflow for improvements and backouts as setting the flags on this bug doesn't make a lot of sense.

status-firefox85: affected → ---
tracking-firefox85: ? → ---
Flags: needinfo?(aionescu) → needinfo?(fstrugariu)

You are right. In my view we should not change the status flag on bugs that are not linked directly to performance alerts.

As we decided to only add a comment in the causing bug for improvements and not log a new bug for it. and we are tracking these status flags we should find a improvement in our process for these cases

Flags: needinfo?(fstrugariu)

Closing this as incomplete as the recording that was failing because of zero-sized patches is not failing anymore thus there's nothing we can test on to make sure the patch is doing what's supposed to and not bringing regressions with it.
The solution would be to drop the exact zero-sized patches with the exact status code(s) that are causing the issue.
Code to start with is attached to this bug and to Bug 1628627.

Status: REOPENED → RESOLVED
Closed: 4 years ago → 3 years ago
Resolution: --- → INCOMPLETE
Assignee: nobody → aionescu
Attachment #9186249 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: