Closed Bug 1553038 Opened 2 years ago Closed 2 years ago

Possible timebombs inside Raptor's Facebook & Instagram page recordings

Categories

(Testing :: Raptor, defect, P2)

All
Android
defect

Tracking

(firefox69 wontfix, firefox70 wontfix)

RESOLVED WORKSFORME
Tracking Status
firefox69 --- wontfix
firefox70 --- wontfix

People

(Reporter: igoldan, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: perf, perf-alert, regression)

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=a22ba2fa077a0b0f9fc21bc0b1bcbb6737331f74

Regressions:

66% raptor-tp6m-facebook-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 pgo 740.42 -> 1,229.00
63% raptor-tp6m-facebook-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 opt 794.58 -> 1,298.75
54% raptor-tp6m-instagram-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 pgo 259.67 -> 398.83
53% raptor-tp6m-facebook-geckoview-cold android-hw-p2-8-0-android-aarch64 pgo 966.95 -> 1,476.21
51% raptor-tp6m-facebook-geckoview-cold android-hw-p2-8-0-android-aarch64 opt 1,038.72 -> 1,563.30
46% raptor-tp6m-instagram-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 opt 270.00 -> 393.00
40% raptor-tp6m-instagram-geckoview-cold android-hw-p2-8-0-android-aarch64 pgo 472.52 -> 662.19
36% raptor-tp6m-instagram-geckoview-cold android-hw-p2-8-0-android-aarch64 opt 497.61 -> 674.47
24% raptor-tp6m-facebook-geckoview-cold loadtime android-hw-p2-8-0-android-aarch64 opt 1,993.08 -> 2,469.25
21% raptor-tp6m-facebook-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 opt 1,554.00 -> 1,874.33
20% raptor-tp6m-facebook-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 pgo 1,505.85 -> 1,804.33
19% raptor-tp6m-instagram-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 pgo 575.25 -> 684.75
19% raptor-tp6m-instagram-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 opt 592.17 -> 702.75
18% raptor-tp6m-facebook-geckoview-cold android-hw-g5-7-0-arm7-api-16 opt 1,967.53 -> 2,313.97
14% raptor-tp6m-instagram-geckoview-cold android-hw-g5-7-0-arm7-api-16 pgo 941.45 -> 1,075.62
14% raptor-tp6m-instagram-geckoview-cold android-hw-g5-7-0-arm7-api-16 opt 979.29 -> 1,113.01
12% raptor-tp6m-facebook-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 opt 3,319.04 -> 3,712.17
12% raptor-tp6m-facebook-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 3,191.31 -> 3,563.58

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/Performance_sheriffing/Raptor

Initially, this regression resembled an infra change. However, we should have seen regressions for all Android tests, not for only 2 tests (Facebook & Instagram).

This made us think that maybe these 2 site recordings have some sort of timebombs hardcoded in their *.js code. Basically, there are some datetime literals which condition some business logic; once current time is bigger than those literals, other business logic branches execute, impacting the time metrics we record.

We should check our existing recordings to at least confirm this is the case.

Summary: Possible timebomb inside Raptor's Facebook & Instagram page recordings → Possible timebombs inside Raptor's Facebook & Instagram page recordings
See Also: → 1552669

Investigations from bug 1548090 revealed that at least one of our recordings (Google Maps) has timebombs in it. This further corroborates our theory from comment 1.

Oh, I forgot to add the link to the alert summary.

Component: General → Raptor

The timebomb strikes again, on the same tests:

== Change summary for alert #21075 (as of Fri, 24 May 2019 11:23:13 GMT) ==

Regressions:

95% raptor-tp6m-facebook-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 pgo 655.25 -> 1,277.17
83% raptor-tp6m-facebook-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 opt 667.07 -> 1,222.08
80% raptor-tp6m-instagram-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 opt 225.58 -> 406.17
75% raptor-tp6m-facebook-geckoview-cold android-hw-p2-8-0-android-aarch64 pgo 877.05 -> 1,535.91
67% raptor-tp6m-instagram-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 pgo 235.42 -> 394.00
66% raptor-tp6m-facebook-geckoview-cold android-hw-p2-8-0-android-aarch64 opt 892.94 -> 1,482.61
56% raptor-tp6m-instagram-geckoview-cold android-hw-p2-8-0-android-aarch64 opt 436.54 -> 679.96
44% raptor-tp6m-instagram-geckoview-cold android-hw-p2-8-0-android-aarch64 pgo 453.56 -> 652.24
37% raptor-tp6m-facebook-geckoview-cold loadtime android-hw-p2-8-0-android-aarch64 pgo 1,774.67 -> 2,439.42
36% raptor-tp6m-instagram-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 pgo 469.29 -> 639.54
33% raptor-tp6m-instagram-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 opt 483.92 -> 641.67
32% raptor-tp6m-facebook-geckoview-cold loadtime android-hw-p2-8-0-android-aarch64 opt 1,797.14 -> 2,374.42
29% raptor-tp6m-facebook-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 opt 1,392.58 -> 1,801.33
28% raptor-tp6m-facebook-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 pgo 1,332.42 -> 1,698.92
26% raptor-tp6m-instagram-geckoview-cold android-hw-g5-7-0-arm7-api-16 pgo 808.73 -> 1,022.19
25% raptor-tp6m-facebook-geckoview-cold android-hw-g5-7-0-arm7-api-16 opt 1,786.96 -> 2,232.56
23% raptor-tp6m-instagram-geckoview-cold android-hw-g5-7-0-arm7-api-16 opt 841.50 -> 1,033.76
23% raptor-tp6m-facebook-geckoview-cold android-hw-g5-7-0-arm7-api-16 pgo 1,708.20 -> 2,092.52
16% raptor-tp6m-facebook-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 opt 3,077.67 -> 3,584.50
13% raptor-tp6m-facebook-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 2,969.38 -> 3,343.58
10% raptor-tp6m-instagram-geckoview-cold loadtime android-hw-p2-8-0-android-aarch64 opt 2,142.92 -> 2,347.17

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=21075

Priority: -- → P2

Another time bomb:

== Change summary for alert #21244 (as of Mon, 03 Jun 2019 05:32:01 GMT) ==

Regressions:

12% raptor-tp6m-instagram-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 pgo 346.56 -> 389.00
12% raptor-tp6m-instagram-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 opt 350.83 -> 393.42
7% raptor-tp6m-instagram-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 pgo 593.82 -> 634.21
6% raptor-tp6m-instagram-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 opt 607.74 -> 641.71

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=21244

Duplicate of this bug: 1557262

== Change summary for alert #21340 (as of Thu, 06 Jun 2019 13:07:55 GMT) ==

Improvements:

22% raptor-tp6-facebook-firefox-cold linux64-shippable-qr opt 858.36 -> 667.01
20% raptor-tp6-facebook-firefox-cold fcp linux64-shippable-qr opt 766.76 -> 611.75

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=21340

== Change summary for alert #21495 (as of Sat, 15 Jun 2019 04:04:35 GMT) ==

Improvements:

26% raptor-tp6-facebook-firefox-cold windows10-64-shippable-qr opt 851.11 -> 632.92
24% raptor-tp6-facebook-firefox-cold fcp windows10-64-shippable-qr opt 753.54 -> 572.33
24% raptor-tp6-facebook-firefox-cold loadtime windows10-64-shippable-qr opt 1,511.67 -> 1,151.83

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=21495

== Change summary for alert #22114 (as of Wed, 24 Jul 2019 16:23:31 GMT) ==

Regressions:

15% raptor-tp6m-youtube-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 2,275.40 -> 2,608.58
11% raptor-tp6m-youtube-geckoview-cold android-hw-p2-8-0-android-aarch64 pgo 1,223.21 -> 1,355.17
10% raptor-tp6m-youtube-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 pgo 1,227.08 -> 1,355.33
9% raptor-tp6m-youtube-geckoview-cold android-hw-g5-7-0-arm7-api-16 pgo 1,686.92 -> 1,847.06
9% raptor-tp6m-youtube-geckoview-cold android-hw-p2-8-0-arm7-api-16 pgo 1,218.81 -> 1,333.40
8% raptor-tp6m-youtube-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 pgo 1,776.77 -> 1,914.00

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=22114

and

== Change summary for alert #22139 (as of Thu, 25 Jul 2019 16:40:24 GMT) ==

Improvements:

14% raptor-tp6m-youtube-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 2,603.12 -> 2,245.17
11% raptor-tp6m-youtube-geckoview-cold loadtime android-hw-p2-8-0-arm7-api-16 pgo 1,820.96 -> 1,622.25

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=22139

Are we still seeing this behavior on recent builds?
Maybe moving to no upstream conn and new recordings fixed it

Flags: needinfo?(marian.raiciof)
Flags: needinfo?(igoldan)
Flags: needinfo?(alexandru.ionescu)

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #10)

Are we still seeing this behavior on recent builds?
Maybe moving to no upstream conn and new recordings fixed it

I've skimmed some random signatures & didn't find weird spikes. Let's leave this open for 3 extra weeks & if it doesn't reproduce then we'll close it as fixed.
I'll schedule a reminder.

Flags: needinfo?(igoldan)
Flags: needinfo?(alexandru.ionescu)
Flags: needinfo?(marian.raiciof)

Can this be closed or do you still need it to stay open?

(In reply to Ionuț Goldan [:igoldan], Performance Sheriff from comment #11)

I've skimmed some random signatures & didn't find weird spikes. Let's leave this open for 3 extra weeks & if it doesn't reproduce then we'll close it as fixed.
I'll schedule a reminder.

Hi Ionut, can this issue be closed now? Thanks :)

Flags: needinfo?(rwood) → needinfo?(igoldan)

Ah, just saw :igoldan's note on slack, he confirmed yes we can close this issue now "I didn't see any more weird spikes".

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → WORKSFORME
Flags: needinfo?(igoldan)
You need to log in before you can comment on or make changes to this bug.