1018936 - [webvtt] WebVTT reftests from web-platform-tests intermittently fail to load

Status: RESOLVED WORKSFORME

(Core :: Audio/Video, defect)

Description

[James Graham [:jgraham]]

Particularly on Windows it seems that these tests are highly intermittent, either returning timeout or fail with probability of about 50%.

I am considering disabling all of these tests at least on Windows until this problem is fixed.

Comment 1

[James Graham [:jgraham]]

For example from a recent Windows 8 Run on cedar:

TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/2_cues_overlapping_partially_move_down.html == /webvtt/rendering/cues-with-video/processing-model/2_cues_overlapping_partially_move_down-ref.html | expected FAIL | | took 15656ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/2_cues_overlapping_partially_move_up.html == /webvtt/rendering/cues-with-video/processing-model/2_cues_overlapping_partially_move_up-ref.html | expected FAIL | | took 15605ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/2_tracks.html == /webvtt/rendering/cues-with-video/processing-model/2_tracks-ref.html | expected FAIL | | took 15603ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/3_tracks.html == /webvtt/rendering/cues-with-video/processing-model/3_tracks-ref.html | expected FAIL | | took 15638ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_end.html == /webvtt/rendering/cues-with-video/processing-model/align_end-ref.html | expected FAIL | | took 15618ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_end_wrapped.html == /webvtt/rendering/cues-with-video/processing-model/align_end_wrapped-ref.html | expected FAIL | | took 15647ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_middle.html == /webvtt/rendering/cues-with-video/processing-model/align_middle-ref.html | expected FAIL | | took 15619ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_middle_position_50.html == /webvtt/rendering/cues-with-video/processing-model/align_middle_position_50-ref.html | expected FAIL | | took 15648ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_middle_position_gt_50.html == /webvtt/rendering/cues-with-video/processing-model/align_middle_position_gt_50-ref.html | expected FAIL | | took 15637ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_middle_position_lt_50.html == /webvtt/rendering/cues-with-video/processing-model/align_middle_position_lt_50-ref.html | expected FAIL | | took 15621ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_middle_position_lt_50_size_gt_maximum_size.html == /webvtt/rendering/cues-with-video/processing-model/align_middle_position_lt_50_size_gt_maximum_size-ref.html | expected FAIL | | took 15669ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_middle_wrapped.html == /webvtt/rendering/cues-with-video/processing-model/align_middle_wrapped-ref.html | expected FAIL | | took 15636ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_start.html == /webvtt/rendering/cues-with-video/processing-model/align_start-ref.html | expected FAIL | | took 15679ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/align_start_wrapped.html == /webvtt/rendering/cues-with-video/processing-model/align_start_wrapped-ref.html | expected FAIL | | took 15687ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/audio_has_no_subtitles.html == /webvtt/rendering/cues-with-video/processing-model/audio_has_no_subtitles-ref.html | expected PASS | | took 15639ms
TEST-END TEST-UNEXPECTED-TIMEOUT | /webvtt/rendering/cues-with-video/processing-model/basic.html == /webvtt/rendering/cues-with-video/processing-model/basic-ref.html | expected FAIL | | took 15668ms

Comment 2

[James Graham [:jgraham]]

Ryan: I understand that you have been working with the media devs on some other cases of unstable tests? Do you have any pointers to that work and/or know if this could be added to the list of issues to get fixed? It is one of a small number of intermittents blocking web-platform-tests.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

I'd start with Ralph or Rick.

Comment 4

[Ralph Giles (:rillian)]

Sorry, I haven't gotten to this. Rick, please look if you have time. Otherwise I'll see if I reproduce next week.

Comment 5

[Rick Eyre (:reyre)]

I probably won't be able to take a look for the next week or so. That being said I might be able to this weekend, depending. I'll post back if I do.

Comment 6

[James Graham [:jgraham]]

I have now disabled the dom_override_* tests which seem to be the ones that are still unstable. I'm doing a cedar run to see if the tests go consistently green now. It would be great if we could reenable these tests (or determine that they are racy) because it seems likely that instability here corresponds to a user-visible issue.

Comment 7

[Ralph Giles (:rillian)]

Sorry, I haven't had time to look at this, and don't expect to soon.

Comment 8

[Ed Morley [:emorley]]

Inactive; closing (see bug 1180138).