Closed Bug 1332564 Opened 7 years ago Closed 7 years ago

Intermittent TEST-UNEXPECTED-PASS | /webvtt/rendering/cues-with-video/processing-model/selectors/cue_function/class_object/class_white-space_nowrap.html | expected FAIL

Categories

(Testing :: web-platform-tests, defect)

Product:
Testing
Component:
web-platform-tests
Version:
Version 3
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox52 unaffected, firefox53 fixed, firefox54 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla54
Tracking Flags:
Tracking Status
firefox52 --- unaffected
firefox53 --- fixed
firefox54 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: jmaher)

References

(Depends on 2 open bugs)

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell disabled])

Attachments

(1 file)

disable web-platform-tests webvtt/rendering/*/selectors/*
634 bytes, patch
gbrown
: review+
Details | Diff | Splinter Review 
this is narrowed down to a small range:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=win%20x64%20wr-e10s&tochange=efeda43ab7b20d9216bf72b5f97a3dd715312e2a&fromchange=5abe47a8bfe0fd31d5cedc5bf01bdfdbae9ed7bf&selectedJob=72322746

I suspect these failures are related to the large update :jgraham did for wpt:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=cf7aaeb55da022f9b66b17e89e406e7ca03c7636

this reminds me of bug 1332440 which started then, although this test has been around for quite some time in the tree.

:jgraham, can you comment on if we should mark this test as something to skip or we can fix this?  I am unclear on who would own this test or what bugzilla component we should put this in.
Flags: needinfo?(james)
See Also: → 1332484
looks like a common pattern in:
/webvtt/rendering/cues-with-video/processing-model/selectors/cue_function/*/*-space_nowrap.html
See Also: → 1332642, 1332605
The update fixed some brokenness in the wpt harness that may have caused these tests to go from permafail to intermittent pass. Since we weren't running them correctly before it's possible that the tests have always been broken, or that we have uncovered some implementation issue.

I don't know who works on WebVTT  support but looking at the git log suggests maybe rillian would be a good starting point.
Flags: needinfo?(james) → needinfo?(giles)
Benjamin, can you please take a look at these web platform tests and figure out if they're correct and why we're intermittent on them?
Flags: needinfo?(giles) → needinfo?(bechen)
See Also: → 1332543
I would prefer to disable/skip the whole tests under /webvtt/rendering/cues-with-video/processing-model/selectors.
Because the "selector for pseudo element" we don't implement yet. So it is meaningless if a test become PASS from FAIL.
Flags: needinfo?(bechen)
For the record I'm pretty skeptical of disabling tests just because we don't implement a feature yet. When we do come to implement the feature we are likley ot forget to reenable the tests and so miss the issues that they would have caught.
I guess if there is a bug that is on file to implement the features, that would be a way to mark this as blocked by that bug and we could potentially remember this.

On another note, what bugzilla component would /webvtt/rendering/cues-with-video/processing-model/selectors tests be in?

in fact /webvtt/rendering/ has nothing but /webvtt/rendering/cues-with-video/processing-model/ which includes a list of tests and directories:
* support/ *.vtt
* evil/ {tests}
* bidi/ {tests}

would it be save to assume that all the tests under /webvtt/rendering would be the same bugzilla component?
example patch that will disable this.

:bechen, can you comment on if we really should disable this and what work is tracked to implement this if we do disable it?
Flags: needinfo?(bechen)
Waitaminute. This and all the other unexpected-PASS bugs that we unexpectedly pass so often that I changed the summary of one to "Permaorange" instead of "Intermittent", it is unexpected that we pass them because we DON'T IMPLEMENT THE FEATURE THEY TEST?

How about if instead of disabling them because we don't implement the feature, we instead disable them because they are obviously and self-evidently bogus and incorrectly-written?
Maybe it is worth to file another bug to find out why those tests were PASS for a long time.
For example: the selectors/cue/color_rgba.html change the font color to rgba(128,255,128,0.5), but our implementation is white. So the testcase should be FAIL but it PASS on the try server.

Please disable(In reply to Joel Maher ( :jmaher) from comment #10)
> Created attachment 8833303 [details] [diff] [review]
> disable web-platform-tests webvtt/rendering/*/selectors/*
> 
> example patch that will disable this.
> 
> :bechen, can you comment on if we really should disable this and what work
> is tracked to implement this if we do disable it?

Please disable them first, after finishing bug 865395 and other relative bugs(::cue pseudo element and selector), I will re-enable the tests again. We can leave this bug open or file another one.
Flags: needinfo?(bechen)
Comment on attachment 8833303 [details] [diff] [review]
disable web-platform-tests webvtt/rendering/*/selectors/*

as we will go forward with disabling these tests, lets do it and let :bechen implement the features and re-enable when it fits into a proper development schedule.
Attachment #8833303 - Flags: review?(gbrown)
Comment on attachment 8833303 [details] [diff] [review]
disable web-platform-tests webvtt/rendering/*/selectors/*

Review of attachment 8833303 [details] [diff] [review]:
-----------------------------------------------------------------

Sounds good. Thanks!
Attachment #8833303 - Flags: review?(gbrown) → review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/df0afcf0c3b6
disable webvtt/rendering/* as features are not implemented in gecko. r=gbrown
https://hg.mozilla.org/mozilla-central/rev/df0afcf0c3b6
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox54: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla54
Assignee: nobody → jmaher
status-firefox52: --- → unaffected
status-firefox53: --- → affected
Whiteboard: [stockwell disabled]
Blocks: 1353689
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: