Closed Bug 1417230 Opened 2 years ago Closed 2 years ago

Test verification of web-platform test fails if test is skipped

Categories

(Testing :: General, defect)

defect
Not set

Tracking

(firefox59 fixed)

RESOLVED FIXED
mozilla59
Tracking Status
firefox59 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Attachments

(1 file)

./mach web-platform-tests /media-source/mediasource-duration.html --verify
 0:00.00 LOG: MainThread INFO :::
 0:00.00 LOG: MainThread INFO ::: Running test verification step "Running tests in a loop 10 times"...
 0:00.00 LOG: MainThread INFO :::
 0:02.84 LOG: MainThread INFO ## All results ##

 0:02.84 LOG: MainThread INFO ### /media-source/mediasource-duration.html ###
 0:02.84 LOG: MainThread INFO | Subtest |            Results            | Messages |
 0:02.84 LOG: MainThread INFO |---------|-------------------------------|----------|
 0:02.84 LOG: MainThread INFO |         | **SKIP: 1/10, MISSING: 9/10** |          |
 0:02.84 LOG: MainThread INFO 
 0:02.84 LOG: MainThread INFO ## Unstable results ##

 0:02.84 LOG: MainThread INFO |                    Test                   | Subtest |            Results            | Messages |
 0:02.84 LOG: MainThread INFO |-------------------------------------------|---------|-------------------------------|----------|
 0:02.84 LOG: MainThread INFO | `/media-source/mediasource-duration.html` |         | **SKIP: 1/10, MISSING: 9/10** |          |
 0:02.84 LOG: MainThread INFO 
 0:02.84 LOG: MainThread INFO ::: Running tests in a loop 10 times : FAIL
 0:02.84 LOG: MainThread INFO :::
 0:02.84 LOG: MainThread ERROR ::: Test verification FAIL
 0:02.84 LOG: MainThread INFO :::
Oh, interesting. Probably just need to filter out anything that had at least one skip result; I'm not very worried about things not being disabled consistently.
This just allows a skipped test to pass verification:

./mach web-platform-tests /media-source/mediasource-duration.html --verify
 0:00.00 LOG: MainThread INFO :::
 0:00.00 LOG: MainThread INFO ::: Running test verification step "Running tests in a loop 10 times"...
 0:00.00 LOG: MainThread INFO :::
 0:02.84 LOG: MainThread INFO ## All results ##

 0:02.84 LOG: MainThread INFO ### /media-source/mediasource-duration.html ###
 0:02.84 LOG: MainThread INFO | Subtest |          Results          | Messages |
 0:02.84 LOG: MainThread INFO |---------|---------------------------|----------|
 0:02.84 LOG: MainThread INFO |         | SKIP: 1/10, MISSING: 9/10 |          |
 0:02.84 LOG: MainThread INFO 
 0:02.84 LOG: MainThread INFO :::
 0:02.84 LOG: MainThread INFO ::: Running test verification step "Running tests in a loop with restarts 5 times"...
 0:02.84 LOG: MainThread INFO :::
 0:05.75 LOG: MainThread INFO ## All results ##

 0:05.75 LOG: MainThread INFO ### /media-source/mediasource-duration.html ###
 0:05.75 LOG: MainThread INFO | Subtest | Results | Messages |
 0:05.75 LOG: MainThread INFO |---------|---------|----------|
 0:05.75 LOG: MainThread INFO |         | SKIP    |          |
 0:05.75 LOG: MainThread INFO 
 0:05.75 LOG: MainThread INFO :::
 0:05.75 LOG: MainThread INFO ::: Running test verification step "Running tests in a loop 10 times with flags chaos_mode_flags=3"...
 0:05.75 LOG: MainThread INFO :::
 0:08.62 LOG: MainThread INFO ## All results ##

 0:08.62 LOG: MainThread INFO ### /media-source/mediasource-duration.html ###
 0:08.62 LOG: MainThread INFO | Subtest |          Results          | Messages |
 0:08.62 LOG: MainThread INFO |---------|---------------------------|----------|
 0:08.62 LOG: MainThread INFO |         | SKIP: 1/10, MISSING: 9/10 |          |
 0:08.62 LOG: MainThread INFO 
 0:08.62 LOG: MainThread INFO :::
 0:08.62 LOG: MainThread INFO ::: Running test verification step "Running tests in a loop with restarts 5 times with flags chaos_mode_flags=3"...
 0:08.62 LOG: MainThread INFO :::
 0:11.50 LOG: MainThread INFO ## All results ##

 0:11.50 LOG: MainThread INFO ### /media-source/mediasource-duration.html ###
 0:11.50 LOG: MainThread INFO | Subtest | Results | Messages |
 0:11.50 LOG: MainThread INFO |---------|---------|----------|
 0:11.50 LOG: MainThread INFO |         | SKIP    |          |
 0:11.50 LOG: MainThread INFO 
 0:11.50 LOG: MainThread INFO ::: Running tests in a loop 10 times : PASS
 0:11.50 LOG: MainThread INFO ::: Running tests in a loop with restarts 5 times : PASS
 0:11.50 LOG: MainThread INFO ::: Running tests in a loop 10 times with flags chaos_mode_flags=3 : PASS
 0:11.50 LOG: MainThread INFO ::: Running tests in a loop with restarts 5 times with flags chaos_mode_flags=3 : PASS
 0:11.50 LOG: MainThread INFO :::
 0:11.50 LOG: MainThread INFO ::: Test verification PASS
 0:11.50 LOG: MainThread INFO :::
Attachment #8928294 - Flags: review?(james)
Comment on attachment 8928294 [details] [diff] [review]
allow SKIP result

Review of attachment 8928294 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/web-platform/tests/tools/wptrunner/wptrunner/stability.py
@@ +86,5 @@
>  
>  def is_inconsistent(results_dict, iterations):
>      """Return whether or not a single test is inconsistent."""
> +    if 'SKIP' in results_dict:
> +        return len(results_dict) > 1

Just `return False` I guess; we don't need to check there were multiple results in this case since I suppose that's different for repeat-without-restart and repeat-with-restart.
Attachment #8928294 - Flags: review?(james) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ff6af466f196
Do not fail when a skipped web-platform test is verified; r=jgraham
https://hg.mozilla.org/mozilla-central/rev/ff6af466f196
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla59
https://hg.mozilla.org/projects/oak/rev/ff6af466f196b774e339f3814653d8eac07c9107
Bug 1417230 - Do not fail when a skipped web-platform test is verified; r=jgraham
You need to log in before you can comment on or make changes to this bug.