Open Bug 1525017 Opened 1 year ago Updated 1 month ago

Significant noise seen when testing tp6 on the ux machines

Categories

(Testing :: Raptor, defect, P3)

Version 3
Desktop
Windows 10
defect

Tracking

(Not tracked)

People

(Reporter: denispal, Unassigned)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

I have been seeing variations of up to 50% when running tp6 on the -ux machines. Here is an example of two try runs that are identical in code changes. Many of the ux results have 30-50% variations.

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=2d278281bdd60bac4046a26e4e195b348b49661f&newProject=try&newRevision=3ea8e596da68a9f445b0f6413924d7c3570a9d83&framework=10

After noticing how long startup tasks continue to run on android, I ran a quick test where I increased raptor's post_startup_delay from 30 seconds to 90.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=559dc2dac7ec6fa67dcc983096f3f436b518b49b&selectedJob=231574715

There are still a lot of timeouts on the ux hardware, but overall this seems to significantly improve both noise and performance:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=559dc2dac7ec6fa67dcc983096f3f436b518b49b&framework=10&selectedTimeRange=2592000

What are the thoughts on increasing the startup delay?
Obviously it reduces test throughput, but it looks like the browser might not be ready even after 30 seconds (!).

I think a broader test that covered more hardware would be useful too.

Flags: needinfo?(dave.hunt)

:sphilp asked how long this would increase run time.
I happen to be running test-windows10-64-ux/opt-raptor-tp6-1-firefox-e10s locally and it takes 19 minutes to run on the reference laptop.
The browser is loaded 4 times for this test, once for each site, so adding 60 seconds to each browser startup would bring total test time to 23 minutes, if I'm not mistaken.
Maybe an additional 30 seconds is sufficient to reduce the noise?

I was told that the openh264 codec is downloaded after a minute, this could be introducing some noise.

Thanks :acreskey, I've raised bug 1536090 to experiment with changing the settle time.

Flags: needinfo?(dave.hunt)
Priority: -- → P1
Priority: P1 → P2
Depends on: 1547135
Priority: P2 → P3
You need to log in before you can comment on or make changes to this bug.