1525017 - Significant noise seen when testing tp6 on the ux machines

Status: NEW

(Testing :: Raptor, defect, P3)

Description

[Denis Palmeiro [:denispal]]

I have been seeing variations of up to 50% when running tp6 on the -ux machines. Here is an example of two try runs that are identical in code changes. Many of the ux results have 30-50% variations.

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=2d278281bdd60bac4046a26e4e195b348b49661f&newProject=try&newRevision=3ea8e596da68a9f445b0f6413924d7c3570a9d83&framework=10

Comment 1

[Andrew Creskey [:acreskey]]

After noticing how long startup tasks continue to run on android, I ran a quick test where I increased raptor's post_startup_delay from 30 seconds to 90.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=559dc2dac7ec6fa67dcc983096f3f436b518b49b&selectedJob=231574715

There are still a lot of timeouts on the ux hardware, but overall this seems to significantly improve both noise and performance:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=559dc2dac7ec6fa67dcc983096f3f436b518b49b&framework=10&selectedTimeRange=2592000

What are the thoughts on increasing the startup delay?
Obviously it reduces test throughput, but it looks like the browser might not be ready even after 30 seconds (!).

I think a broader test that covered more hardware would be useful too.

Comment 2

[Andrew Creskey [:acreskey]]

:sphilp asked how long this would increase run time.
I happen to be running test-windows10-64-ux/opt-raptor-tp6-1-firefox-e10s locally and it takes 19 minutes to run on the reference laptop.
The browser is loaded 4 times for this test, once for each site, so adding 60 seconds to each browser startup would bring total test time to 23 minutes, if I'm not mistaken.
Maybe an additional 30 seconds is sufficient to reduce the noise?

I was told that the openh264 codec is downloaded after a minute, this could be introducing some noise.

Comment 3

[Andrew Creskey [:acreskey]]

For Bug 1502138 I ran the same test (extend startup delay to 90s) and other platforms this significantly reduces noise as well.
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-inbound&newProject=try&newRevision=3bbe84b59015f54cfe174a0d2430ec5fb2571ace&framework=10&selectedTimeRange=2592000

e.g.
osx-10-10 ... -45.72%

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Thanks :acreskey, I've raised bug 1536090 to experiment with changing the settle time.