1572057 - Investigate spike in raptor-tp6m-youtube-geckoview tests between 24 Jul 2019 and 25 Jul 2019

Status: RESOLVED FIXED

(Testing :: Raptor, task, P2)

Description

[Florin Strugariu [:Bebe]]

== Change summary for alert #22114 (as of Wed, 24 Jul 2019 16:23:31 GMT) ==

Regressions:

15% raptor-tp6m-youtube-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 2,275.40 -> 2,608.58
11% raptor-tp6m-youtube-geckoview-cold android-hw-p2-8-0-android-aarch64 pgo 1,223.21 -> 1,355.17
10% raptor-tp6m-youtube-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 pgo 1,227.08 -> 1,355.33
9% raptor-tp6m-youtube-geckoview-cold android-hw-g5-7-0-arm7-api-16 pgo 1,686.92 -> 1,847.06
9% raptor-tp6m-youtube-geckoview-cold android-hw-p2-8-0-arm7-api-16 pgo 1,218.81 -> 1,333.40
8% raptor-tp6m-youtube-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 pgo 1,776.77 -> 1,914.00

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=22114

and

== Change summary for alert #22139 (as of Thu, 25 Jul 2019 16:40:24 GMT) ==

Improvements:

14% raptor-tp6m-youtube-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 2,603.12 -> 2,245.17
11% raptor-tp6m-youtube-geckoview-cold loadtime android-hw-p2-8-0-arm7-api-16 pgo 1,820.96 -> 1,622.25

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=22139

Comment 1

[Florin Strugariu [:Bebe]]

https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=autoland,2006990,1&series=mozilla-inbound,73a625944202775f16abca6996548e7938d3d0aa,0

Check for infra change on commit: 585b6ae0dd0d59e88b882fccfceeddcd6e89229b

Comment 2

[Ionuț Goldan [:igoldan]]

The regression noticed in this bug happened 5 times, on a weekly basis, starting June 21, then July 15, July 19, July 24, July 29.
After the re recording of raptor-tp6m-youtube-geckoview-cold using mitmproxy 4.0.4, the regular spikes stopped occuring.

Conclusion is that the old recording made network connections which affected the perf metrics.

This is the most plausible explanation, because the graphs resemble some environmental changes.
Also, initial suspicion of infra changes didn't held, because the retriggers didn't show new metrics with low values.

We will close this bug as FIXED if we won't see this spike pattern occurring between the July 31 - Aug 22 time range.

Comment 3

[Ionuț Goldan [:igoldan]]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriff from comment #2)

We will close this bug as FIXED if we won't see this spike pattern occurring between the July 31 - Aug 22 time range.

2 weeks have passed and no sign of spikes noticed.