Open Bug 1449065 Opened 3 years ago Updated 3 years ago
[wpt-sync] Sync PR 10186 - wptrunner: Always restart on test error
Sync web-platform-tests PR 10186 into mozilla-central (this bug is closed when the sync is complete). PR: https://github.com/w3c/web-platform-tests/pull/10186 Details from upstream follow. Mike Pennisi <email@example.com> wrote: > wptrunner: Always restart on test error > > Because a harness error may describe an unrecoverable browser state, the > browser should always be restarted when such an error is encountered > (even in the presence of the `--no-restart-on-unexpected` flag). > > --- > > @jgraham @gsnedders If `wpt run --no-restart-on-unexpected` is used to execute > a test file like this: > > ```html > <script> > window.opener.close(); > </script> > ``` > > Then after that test fails with the status "ERROR", all subsequent tests fail > for the same reason. By interpreting the error as unrecoverable, we can avoid > invalidating test runs in these cases. > > That's a contrived example, though. As an alternative, we might be able to > address it by detecting a missing window and applying the status "CRASH". I > started researching this possibility, but it started to look like this logic > would be dependent on the browser, executor, and/or transport. I'm also > thinking that there are many more subtle ways the environment could become > corrupted, and we'd probably never be able to detect them all. > > In contrast, what I'm proposing here is much more heavy-handed. There are many > kinds of harness errors that don't require full restarts, so this patch will > make running WPT with `--no-restart-on-unexpected` slower. My inclination is to > prefer correctness and completeness over speed since test execution time > (unlike correctness) can be improved by consumers by using more machines. I > also get the sense that not many consumers have a use for this particular flag, > so I'm guessing that we wouldn't want to take on the complexity overhead of a > more nuanced solution. > > What do you think?
You need to log in before you can comment on or make changes to this bug.