Significant noise seen when testing tp6 on the ux machines
Categories
(Testing :: Raptor, defect, P3)
Tracking
(Not tracked)
People
(Reporter: denispal, Unassigned)
References
(Depends on 1 open bug, Blocks 1 open bug)
Details
I have been seeing variations of up to 50% when running tp6 on the -ux machines. Here is an example of two try runs that are identical in code changes. Many of the ux results have 30-50% variations.
Comment 1•6 years ago
|
||
After noticing how long startup tasks continue to run on android, I ran a quick test where I increased raptor's post_startup_delay
from 30 seconds to 90.
There are still a lot of timeouts on the ux hardware, but overall this seems to significantly improve both noise and performance:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=559dc2dac7ec6fa67dcc983096f3f436b518b49b&framework=10&selectedTimeRange=2592000
What are the thoughts on increasing the startup delay?
Obviously it reduces test throughput, but it looks like the browser might not be ready even after 30 seconds (!).
I think a broader test that covered more hardware would be useful too.
Comment 2•6 years ago
|
||
:sphilp asked how long this would increase run time.
I happen to be running test-windows10-64-ux/opt-raptor-tp6-1-firefox-e10s
locally and it takes 19 minutes to run on the reference laptop.
The browser is loaded 4 times for this test, once for each site, so adding 60 seconds to each browser startup would bring total test time to 23 minutes, if I'm not mistaken.
Maybe an additional 30 seconds is sufficient to reduce the noise?
I was told that the openh264 codec is downloaded after a minute, this could be introducing some noise.
Comment 3•6 years ago
|
||
For Bug 1502138 I ran the same test (extend startup delay to 90s) and other platforms this significantly reduces noise as well.
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-inbound&newProject=try&newRevision=3bbe84b59015f54cfe174a0d2430ec5fb2571ace&framework=10&selectedTimeRange=2592000
e.g.
osx-10-10 ... -45.72%
Comment 4•6 years ago
|
||
Thanks :acreskey, I've raised bug 1536090 to experiment with changing the settle time.
Updated•6 years ago
|
Updated•6 years ago
|
Updated•5 years ago
|
Updated•2 years ago
|
Comment 5•3 days ago
|
||
Closing as invalid, we don't have any -ux machines anymore, additionally this was running web-extension which we are not doing anymore.
Description
•