[webvtt] WebVTT reftests from web-platform-tests intermittently fail to load

RESOLVED WORKSFORME

Status

()

defect
RESOLVED WORKSFORME
5 years ago
4 years ago

People

(Reporter: jgraham, Unassigned)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

Particularly on Windows it seems that these tests are highly intermittent, either returning timeout or fail with probability of about 50%.

I am considering disabling all of these tests at least on Windows until this problem is fixed.
Blocks: 945222
For example from a recent Windows 8 Run on cedar:

TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/2_cues_overlapping_partially_move_down.html == /webvtt/rendering/cues-with-video/processing-model/2_cues_overlapping_partially_move_down-ref.html | expected FAIL | | took 15656ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/2_cues_overlapping_partially_move_up.html == /webvtt/rendering/cues-with-video/processing-model/2_cues_overlapping_partially_move_up-ref.html | expected FAIL | | took 15605ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/2_tracks.html == /webvtt/rendering/cues-with-video/processing-model/2_tracks-ref.html | expected FAIL | | took 15603ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/3_tracks.html == /webvtt/rendering/cues-with-video/processing-model/3_tracks-ref.html | expected FAIL | | took 15638ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_end.html == /webvtt/rendering/cues-with-video/processing-model/align_end-ref.html | expected FAIL | | took 15618ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_end_wrapped.html == /webvtt/rendering/cues-with-video/processing-model/align_end_wrapped-ref.html | expected FAIL | | took 15647ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_middle.html == /webvtt/rendering/cues-with-video/processing-model/align_middle-ref.html | expected FAIL | | took 15619ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_middle_position_50.html == /webvtt/rendering/cues-with-video/processing-model/align_middle_position_50-ref.html | expected FAIL | | took 15648ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_middle_position_gt_50.html == /webvtt/rendering/cues-with-video/processing-model/align_middle_position_gt_50-ref.html | expected FAIL | | took 15637ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_middle_position_lt_50.html == /webvtt/rendering/cues-with-video/processing-model/align_middle_position_lt_50-ref.html | expected FAIL | | took 15621ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_middle_position_lt_50_size_gt_maximum_size.html == /webvtt/rendering/cues-with-video/processing-model/align_middle_position_lt_50_size_gt_maximum_size-ref.html | expected FAIL | | took 15669ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_middle_wrapped.html == /webvtt/rendering/cues-with-video/processing-model/align_middle_wrapped-ref.html | expected FAIL | | took 15636ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_start.html == /webvtt/rendering/cues-with-video/processing-model/align_start-ref.html | expected FAIL | | took 15679ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_start_wrapped.html == /webvtt/rendering/cues-with-video/processing-model/align_start_wrapped-ref.html | expected FAIL | | took 15687ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/audio_has_no_subtitles.html == /webvtt/rendering/cues-with-video/processing-model/audio_has_no_subtitles-ref.html | expected PASS | | took 15639ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/basic.html == /webvtt/rendering/cues-with-video/processing-model/basic-ref.html | expected FAIL | | took 15668ms
Ryan: I understand that you have been working with the media devs on some other cases of unstable tests? Do you have any pointers to that work and/or know if this could be added to the list of issues to get fixed? It is one of a small number of intermittents blocking web-platform-tests.
Flags: needinfo?(ryanvm)
I'd start with Ralph or Rick.
Flags: needinfo?(ryanvm) → needinfo?(giles)
Sorry, I haven't gotten to this. Rick, please look if you have time. Otherwise I'll see if I reproduce next week.
Flags: needinfo?(rick.eyre)
I probably won't be able to take a look for the next week or so. That being said I might be able to this weekend, depending. I'll post back if I do.
Flags: needinfo?(rick.eyre)
I have now disabled the dom_override_* tests which seem to be the ones that are still unstable. I'm doing a cedar run to see if the tests go consistently green now. It would be great if we could reenable these tests (or determine that they are racy) because it seems likely that instability here corresponds to a user-visible issue.
Sorry, I haven't had time to look at this, and don't expect to soon.
Flags: needinfo?(giles)
Inactive; closing (see bug 1180138).
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.