Open Bug 1525017 Opened 6 years ago Updated 2 years ago

Significant noise seen when testing tp6 on the ux machines


(Testing :: Raptor, defect, P3)

Version 3
Windows 10


(Not tracked)


(Reporter: denispal, Unassigned)


(Depends on 1 open bug, Blocks 1 open bug)


I have been seeing variations of up to 50% when running tp6 on the -ux machines. Here is an example of two try runs that are identical in code changes. Many of the ux results have 30-50% variations.

After noticing how long startup tasks continue to run on android, I ran a quick test where I increased raptor's post_startup_delay from 30 seconds to 90.

There are still a lot of timeouts on the ux hardware, but overall this seems to significantly improve both noise and performance:

What are the thoughts on increasing the startup delay?
Obviously it reduces test throughput, but it looks like the browser might not be ready even after 30 seconds (!).

I think a broader test that covered more hardware would be useful too.

Flags: needinfo?(dave.hunt)

:sphilp asked how long this would increase run time.
I happen to be running test-windows10-64-ux/opt-raptor-tp6-1-firefox-e10s locally and it takes 19 minutes to run on the reference laptop.
The browser is loaded 4 times for this test, once for each site, so adding 60 seconds to each browser startup would bring total test time to 23 minutes, if I'm not mistaken.
Maybe an additional 30 seconds is sufficient to reduce the noise?

I was told that the openh264 codec is downloaded after a minute, this could be introducing some noise.

For Bug 1502138 I ran the same test (extend startup delay to 90s) and other platforms this significantly reduces noise as well.

osx-10-10 ... -45.72%

Thanks :acreskey, I've raised bug 1536090 to experiment with changing the settle time.

Flags: needinfo?(dave.hunt)
Priority: -- → P1
Priority: P1 → P2
See Also: → 1561324
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.