1417230 - Test verification of web-platform test fails if test is skipped

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

./mach web-platform-tests /media-source/mediasource-duration.html --verify
 0:00.00 LOG: MainThread INFO :::
 0:00.00 LOG: MainThread INFO ::: Running test verification step "Running tests in a loop 10 times"...
 0:00.00 LOG: MainThread INFO :::
 0:02.84 LOG: MainThread INFO ## All results ##

 0:02.84 LOG: MainThread INFO ### /media-source/mediasource-duration.html ###
 0:02.84 LOG: MainThread INFO | Subtest |            Results            | Messages |
 0:02.84 LOG: MainThread INFO |---------|-------------------------------|----------|
 0:02.84 LOG: MainThread INFO |         | **SKIP: 1/10, MISSING: 9/10** |          |
 0:02.84 LOG: MainThread INFO 
 0:02.84 LOG: MainThread INFO ## Unstable results ##

 0:02.84 LOG: MainThread INFO |                    Test                   | Subtest |            Results            | Messages |
 0:02.84 LOG: MainThread INFO |-------------------------------------------|---------|-------------------------------|----------|
 0:02.84 LOG: MainThread INFO | `/media-source/mediasource-duration.html` |         | **SKIP: 1/10, MISSING: 9/10** |          |
 0:02.84 LOG: MainThread INFO 
 0:02.84 LOG: MainThread INFO ::: Running tests in a loop 10 times : FAIL
 0:02.84 LOG: MainThread INFO :::
 0:02.84 LOG: MainThread ERROR ::: Test verification FAIL
 0:02.84 LOG: MainThread INFO :::

Comment 1

[James Graham [:jgraham]]

Oh, interesting. Probably just need to filter out anything that had at least one skip result; I'm not very worried about things not being disabled consistently.

Comment 2

[Geoff Brown [:gbrown]]

Attachment: allow SKIP result

This just allows a skipped test to pass verification:

./mach web-platform-tests /media-source/mediasource-duration.html --verify
 0:00.00 LOG: MainThread INFO :::
 0:00.00 LOG: MainThread INFO ::: Running test verification step "Running tests in a loop 10 times"...
 0:00.00 LOG: MainThread INFO :::
 0:02.84 LOG: MainThread INFO ## All results ##

 0:02.84 LOG: MainThread INFO ### /media-source/mediasource-duration.html ###
 0:02.84 LOG: MainThread INFO | Subtest |          Results          | Messages |
 0:02.84 LOG: MainThread INFO |---------|---------------------------|----------|
 0:02.84 LOG: MainThread INFO |         | SKIP: 1/10, MISSING: 9/10 |          |
 0:02.84 LOG: MainThread INFO 
 0:02.84 LOG: MainThread INFO :::
 0:02.84 LOG: MainThread INFO ::: Running test verification step "Running tests in a loop with restarts 5 times"...
 0:02.84 LOG: MainThread INFO :::
 0:05.75 LOG: MainThread INFO ## All results ##

 0:05.75 LOG: MainThread INFO ### /media-source/mediasource-duration.html ###
 0:05.75 LOG: MainThread INFO | Subtest | Results | Messages |
 0:05.75 LOG: MainThread INFO |---------|---------|----------|
 0:05.75 LOG: MainThread INFO |         | SKIP    |          |
 0:05.75 LOG: MainThread INFO 
 0:05.75 LOG: MainThread INFO :::
 0:05.75 LOG: MainThread INFO ::: Running test verification step "Running tests in a loop 10 times with flags chaos_mode_flags=3"...
 0:05.75 LOG: MainThread INFO :::
 0:08.62 LOG: MainThread INFO ## All results ##

 0:08.62 LOG: MainThread INFO ### /media-source/mediasource-duration.html ###
 0:08.62 LOG: MainThread INFO | Subtest |          Results          | Messages |
 0:08.62 LOG: MainThread INFO |---------|---------------------------|----------|
 0:08.62 LOG: MainThread INFO |         | SKIP: 1/10, MISSING: 9/10 |          |
 0:08.62 LOG: MainThread INFO 
 0:08.62 LOG: MainThread INFO :::
 0:08.62 LOG: MainThread INFO ::: Running test verification step "Running tests in a loop with restarts 5 times with flags chaos_mode_flags=3"...
 0:08.62 LOG: MainThread INFO :::
 0:11.50 LOG: MainThread INFO ## All results ##

 0:11.50 LOG: MainThread INFO ### /media-source/mediasource-duration.html ###
 0:11.50 LOG: MainThread INFO | Subtest | Results | Messages |
 0:11.50 LOG: MainThread INFO |---------|---------|----------|
 0:11.50 LOG: MainThread INFO |         | SKIP    |          |
 0:11.50 LOG: MainThread INFO 
 0:11.50 LOG: MainThread INFO ::: Running tests in a loop 10 times : PASS
 0:11.50 LOG: MainThread INFO ::: Running tests in a loop with restarts 5 times : PASS
 0:11.50 LOG: MainThread INFO ::: Running tests in a loop 10 times with flags chaos_mode_flags=3 : PASS
 0:11.50 LOG: MainThread INFO ::: Running tests in a loop with restarts 5 times with flags chaos_mode_flags=3 : PASS
 0:11.50 LOG: MainThread INFO :::
 0:11.50 LOG: MainThread INFO ::: Test verification PASS
 0:11.50 LOG: MainThread INFO :::

Comment 3

[James Graham [:jgraham]]

Comment on attachment 8928294 [details] [diff] [review]
allow SKIP result

Review of attachment 8928294 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/web-platform/tests/tools/wptrunner/wptrunner/stability.py
@@ +86,5 @@
>  
>  def is_inconsistent(results_dict, iterations):
>      """Return whether or not a single test is inconsistent."""
> +    if 'SKIP' in results_dict:
> +        return len(results_dict) > 1

Just `return False` I guess; we don't need to check there were multiple results in this case since I suppose that's different for repeat-without-restart and repeat-with-restart.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ff6af466f196
Do not fail when a skipped web-platform test is verified; r=jgraham

Comment 5

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/ff6af466f196

Comment 6

[Sylvestre Ledru [:Sylvestre]]

https://hg.mozilla.org/projects/oak/rev/ff6af466f196b774e339f3814653d8eac07c9107
Bug 1417230 - Do not fail when a skipped web-platform test is verified; r=jgraham