1664552 - Decrease pageload complete check start time and remove await on about:blank navigation

Status: RESOLVED INVALID

(Testing :: Raptor, task, P3)

Description

[Greg Mierzwinski [:sparky]]

This bug is to investigate the impacts of decreasing the page complete check start time and the removal of the wait on the about:blank navigation. If the analysis shows everything is fine, then we'll make the changes.

All of these analyses will be using chimera mode since it will be used for all browsertime tests in the future.

Comment 1

[Greg Mierzwinski [:sparky]]

This is interesting: https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=Oo5SuvbzTPaGC0KovYkqCA.0&revision=95355e44b8b1457f355f977e3e27bbef4ee6a3ee

I decreased the --pageCompleteCheckStartWait value from 5 seconds to 1 second and also removed the await call for the about:blank navigation. This option dictates when selenium should first start to check for the pageload completeness. Interestingly, it also seems to have increased the number of intermittents that we are seeing. For comparison, here is the same run but with the current, original settings: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0be634d07e922922abffdf1a744a1d17e8a05d14

This makes me wonder if we should make a trade-off between the intermittent rates and speed of the tests. I think that it boils down to what is more expensive: time spent debugging intermittents, or time spent running the test? I believe that the former is more expensive given the complexity of intermittent failures but I need some data to go further here.

I'm running a test which only has the await removed to be sure that that wasn't the cause of this increase here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ea305355aa07f7557eb79f372b8b38bded53c9bc

I'm also running a test which has the await removed, and the pageload check start decreased to 3s rather than 1s here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=520452ad7a79a2afd184872d19f6e1f55f6168e6

Comment 2

[Greg Mierzwinski [:sparky]]

The issue was coming from the await removal based on the try run above.

Comment 3

[Greg Mierzwinski [:sparky]]

This doesn't seem relevant now since we've had a lot of changes to the testing scripts in this time.