Open Bug 1449065 Opened 2 years ago Updated 2 years ago

[wpt-sync] Sync PR 10186 - wptrunner: Always restart on test error

Categories

(Testing :: web-platform-tests, enhancement, P4)

enhancement

Tracking

(Not tracked)

People

(Reporter: mozilla.org, Unassigned)

References

()

Details

(Whiteboard: [wptsync downstream])

Sync web-platform-tests PR 10186 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/w3c/web-platform-tests/pull/10186
Details from upstream follow.

Mike Pennisi <mike@mikepennisi.com> wrote:
>  wptrunner: Always restart on test error
>  
>  Because a harness error may describe an unrecoverable browser state, the
>  browser should always be restarted when such an error is encountered
>  (even in the presence of the `--no-restart-on-unexpected` flag).
>  
>  ---
>  
>  @jgraham @gsnedders If `wpt run --no-restart-on-unexpected` is used to execute
>  a test file like this:
>  
>  ```html
>  <script>
>  window.opener.close();
>  </script>
>  ```
>  
>  Then after that test fails with the status "ERROR", all subsequent tests fail
>  for the same reason. By interpreting the error as unrecoverable, we can avoid
>  invalidating test runs in these cases.
>  
>  That's a contrived example, though. As an alternative, we might be able to
>  address it by detecting a missing window and applying the status "CRASH". I
>  started researching this possibility, but it started to look like this logic
>  would be dependent on the browser, executor, and/or transport. I'm also
>  thinking that there are many more subtle ways the environment could become
>  corrupted, and we'd probably never be able to detect them all.
>  
>  In contrast, what I'm proposing here is much more heavy-handed. There are many
>  kinds of harness errors that don't require full restarts, so this patch will
>  make running WPT with `--no-restart-on-unexpected` slower. My inclination is to
>  prefer correctness and completeness over speed since test execution time
>  (unlike correctness) can be improved by consumers by using more machines. I
>  also get the sense that not many consumers have a use for this particular flag,
>  so I'm guessing that we wouldn't want to take on the complexity overhead of a
>  more nuanced solution.
>  
>  What do you think?
You need to log in before you can comment on or make changes to this bug.