1602657 - Set appropriate value for post-startup-delay in (and only in) CI, for conditioned-profile runs

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P1)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

As generally agreed on in today's Browsertime triage meeting, let's do the following, for post-startup-delay:

• if in CI, and running on a known conditioned-profile enabled platform/build variant, automatically set/apply post-startup-delay = to a known, agreed-upon value (likely not 0; we'll need to test) which we're happy with, for abating browser settle-time
• if we're running locally, just respect the value of post-startup-delay (if present); else, do nothing (advanced users can and will know to add that, and we can document better, too)

--
Currently, via bug 1596346, we're doing this automatically in all environments, but I'm removing that (https://searchfox.org/mozilla-central/rev/23d4bffcad365e68d2d45776017056b76ca9a968/testing/raptor/raptor/raptor.py#175-178) over in bug 1600838.

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Bug 1602657 - Set appropriate value for post_startup_delay in CI

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

Latest try push below, which still fails unittests; I'd love feedback/help, to get this in soon, and correctly :-)

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=a4594152de18a5cb3f8fab2bc9ec5e69b11da231

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

This should now be ready for review, again; the most-recent try push with added logging, here: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=4c3d64ab9d47e81942e772826890ed21df9fed3b

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

I also did a try push from vanilla mozilla-central, to suss out the intermittents on Browsertime mobile (tp6m-c-1) seen in the previous push; they are still there, so should be unrelated to this patch: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=d8368dd28bd0488364b6d92eda2a773381577c50

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/80f5c59d5df4
Set appropriate value for post-startup-delay in (and only in) CI, for conditioned-profile runs. r=perftest-reviewers,rwood

Comment 6

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/80f5c59d5df4

Comment 7

[Cristian Brindusan [:cbrindusan]]

Backed out changeset 80f5c59d5df4 (Bug 1602657) as requested by sdonner on slack.
https://hg.mozilla.org/integration/autoland/rev/325da10270e4bbc644df0fd29a6c3f6ef536d262

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backout merged: https://hg.mozilla.org/mozilla-central/rev/325da10270e4

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

Hey Alex! Did I do the compare correctly, here: https://treeherder.mozilla.org/perf.html#/compare?originalProject=autoland&originalRevision=80f5c59d5df430ecff942865c54c7bb3c8ee4257&newProject=try&newRevision=b702ce549a0efe21164dc4966931c08e3b9c5573&framework=10 between the initial perf-regressing commit[0] and my recent push to try[1], which includes fixes for bug 1607953 and bug 1565804, which are purported to help make some tests more performant/stable/accurate?

Or, did I not run enough (of the right?) tests? Thx!

[0] https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=80f5c59d5df430ecff942865c54c7bb3c8ee4257
[1] https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=d8c9d60cc8f6a94e8b594f9853c2d7553683555f

Comment 10

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Hey Stephen! Looks like the revision from [1] in the compare link doesn't have the same commit hash. In the compare link you used b702ce549a0efe21164dc4966931c08e3b9c5573 but in the jobs link from [1] the revision is d8c9d60cc8f6a94e8b594f9853c2d7553683555f
This should be the correct comparison, except you don't really have the same jobs triggered, but I will take care of that. By the time you come back to office you should have the data you need.
PS: I am going to write a document about how the compare view can be used. It is WIP for the moment

Comment 11

[(PTO June 19-July 7) Ryan VanderMeulen]

Backout landed on Beta also per bug 1607750 comment 6.
https://hg.mozilla.org/releases/mozilla-beta/rev/d2dfae9a06bf

Comment 12

[Tarek Ziadé (:tarek)]

performance comparisons

52d0976feda8b1f2061cad0ea4eef647bd481067 == original
352cb828808e63cead3206c3c60162dc241c61c5 == post_startup_delay = 1000
6bd2031e761083cacb6bc5b21b84ecfe6ff0cbc1 == post_startup_delay = 5000
bea7d8f0cde25a7b50cadb271ac428c065dffd69 == post_startup_delay = 15000
45f3987c34bb4152e0476276f992940dddf5753e == post_startup_delay == 30000 (if we see changes in this one, something's broken)

30s vs 1s :
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=52d0976feda8b1f2061cad0ea4eef647bd481067&newProject=try&newRevision=352cb828808e63cead3206c3c60162dc241c61c5&framework=10&pageTitle=post+startup+delay+-+30s+vs+1s

30s vs 5s :
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=52d0976feda8b1f2061cad0ea4eef647bd481067&newProject=try&newRevision=6bd2031e761083cacb6bc5b21b84ecfe6ff0cbc1&framework=10&pageTitle=post+startup+delay+-+30s+vs+5s

30s vs 15s :
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=52d0976feda8b1f2061cad0ea4eef647bd481067&newProject=try&newRevision=bea7d8f0cde25a7b50cadb271ac428c065dffd69&framework=10&pageTitle=post+startup+delay+-+30s+vs+15s

30s vs 30s
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=52d0976feda8b1f2061cad0ea4eef647bd481067&newProject=try&newRevision=45f3987c34bb4152e0476276f992940dddf5753e&framework=10&pageTitle=post+startup+delay+-+30s+vs+30s
Sorry I did not filter out chrome/chromium runs...

Comment 13

[Tarek Ziadé (:tarek)]

updated runs with stats in mitm_netlocs.json

c50229e0f6626843ea392a134babb7c003b2490b = 1s
a94fe85efb5a32d3b27b4202a3c3c98cf64d35eb = 1s and network.connectivity-service.enabled=False
e15143b9c64086541028a92a14fd8a38bf5a1635 = 10s
94a069ac1162f97d6f5b7a9b65779a70d4f397d4 = original

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=94a069ac1162f97d6f5b7a9b65779a70d4f397d4&newProject=try&newRevision=c50229e0f6626843ea392a134babb7c003b2490b&framework=10&pageTitle=post+startup+delay+-+original+vs+1s

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=94a069ac1162f97d6f5b7a9b65779a70d4f397d4&newProject=try&newRevision=a94fe85efb5a32d3b27b4202a3c3c98cf64d35eb&framework=10&pageTitle=post+startup+delay+-+original+vs+1s+network+off

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=94a069ac1162f97d6f5b7a9b65779a70d4f397d4&newProject=try&newRevision=e15143b9c64086541028a92a14fd8a38bf5a1635&framework=10&pageTitle=post+startup+delay+-+original+vs+10s

more runs:

c09a9411d04582191a767b488e38ea6601350834 = all tp6- with 1 s
1fb0ea9f3c9c4163c71e859e41779917ba734af3 = all tp6- baseline

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=1fb0ea9f3c9c4163c71e859e41779917ba734af3&newProject=try&newRevision=c09a9411d04582191a767b488e38ea6601350834&framework=10&pageTitle=post+startup+delay+-+original+vs+1s

Comment 14

[Tarek Ziadé (:tarek)]

I think we have enough data at this point with the current quality of our recording. I'll defer the final decision to Dave before landing

Comment 15

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The only concern I have is the improvement we see on macOS (7-14%). Bebe: could you rebase and push to try just for macOS with screenshots enabled? This will give us the confidence value to compare and show us if there are any major visual differences when changing the post startup delay.

Comment 16

[Florin Strugariu [:Bebe]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cd07d5a6f9890409c9cd745dd548c49fe1d49fb3

Comment 17

[Tarek Ziadé (:tarek)]

Bebe, your push has some code failures

Comment 18

[Florin Strugariu [:Bebe]]

My bad...
https://treeherder.mozilla.org/#/jobs?repo=try&revision=84489775b3cedf8095ea5a3469fb857cde19b085

Comment 19

[Florin Strugariu [:Bebe]]

Looked at the screenshots and all seems OK

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1a28609bb405
Set appropriate value for post_startup_delay in CI r=perftest-reviewers,Bebe

Comment 21

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/1a28609bb405