Page timeouts are ignored during cold page-load test suites

RESOLVED FIXED in Firefox 68

Status

defect
P1
normal
RESOLVED FIXED
5 months ago
3 months ago

People

(Reporter: rwood, Assigned: Bebe)

Tracking

Version 3
mozilla68
Points:
---

Firefox Tracking Flags

(firefox68 fixed)

Details

Attachments

(1 attachment)

When running Raptor cold page-load (i.e. tp6m-c-1) if there is a timeout when loading the test page during any one of the browser cycles, that particular browser cycle finishes but then the other browser cycles continue. Then the test will finish and will just include the successful browser cycles in the results (i.e. 9 out of 10 if one timed out). The timeout for a page load is visible in the logs, but the next browser cycle continues - and as long as there is one successful page load then the results will be reported.

Instead, if any one of the page-loads times out, that should stop all remaining browser cycles, and the entire cold load test should fail out reporting the page timeout.

:bebe could you pick this up now that Rob is on PTO?

Assignee: rwood → fstrugariu
Flags: needinfo?(fstrugariu)

Investigated:

Looks like there are two issues here:

  1. we don't check for timeouts between pagecycles
  2. sometimes the timeout from wait_for_test_finish overtires all exceptions
Flags: needinfo?(fstrugariu)

I suspect this is the cause of strange errors in the Fenix tests, such as https://treeherder.mozilla.org/#/jobs?repo=fenix&revision=449fffe9330101428ffa1fb4234c83b13f2e2faf&selectedJob=244078368 where we fail with a KeyError when attempting to combine the results.

:rwood could you review Bebe's patch and land it if it looks good to you? This will likely still give us errors on the Fenix jobs but at least they will be timeouts and much easier to understand.

Flags: needinfo?(rwood)

Yes sorry it was the first one in my review queue today, I'm on it now :) thanks

Flags: needinfo?(rwood)

Is there a try push for this please :bebe?

Flags: needinfo?(fstrugariu)
Flags: needinfo?(fstrugariu)
Pushed by fstrugariu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/921e77da2c82
Page timeouts are ignored during cold page-load test suites r=rwood,davehunt
Status: ASSIGNED → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
You need to log in before you can comment on or make changes to this bug.