Intermittent TEST-UNEXPECTED-PASS | /webvtt/rendering/cues-with-video/processing-model/selectors/cue_function/class_object/class_white-space_nowrap.html | expected FAIL

RESOLVED FIXED in Firefox 53

Status

Testing
web-platform-tests
RESOLVED FIXED
3 months ago
25 days ago

People

(Reporter: Treeherder Bug Filer, Assigned: jmaher)

Tracking

(Depends on: 2 bugs, {intermittent-failure})

Version 3
mozilla54
intermittent-failure
Points:
---
Dependency tree / graph
Bug Flags:
in-testsuite -

Firefox Tracking Flags

(firefox52 unaffected, firefox53 fixed, firefox54 fixed)

Details

(Whiteboard: [stockwell disabled])

Attachments

(1 attachment)

(Reporter)

Description

3 months ago
treeherder
Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=70568061&repo=mozilla-inbound

https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win64/1484888004/mozilla-inbound_win8_64_test-web-platform-tests-reftests-e10s-bm111-tests1-windows-build142.txt.gz

Comment 1

3 months ago
12 failures in 690 pushes (0.017 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 9
* autoland: 2
* mozilla-central: 1

Platform breakdown:
* windows8-64: 11
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1332564&startday=2017-01-16&endday=2017-01-22&tree=all
(Assignee)

Comment 2

3 months ago
this is narrowed down to a small range:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=win%20x64%20wr-e10s&tochange=efeda43ab7b20d9216bf72b5f97a3dd715312e2a&fromchange=5abe47a8bfe0fd31d5cedc5bf01bdfdbae9ed7bf&selectedJob=72322746

I suspect these failures are related to the large update :jgraham did for wpt:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=cf7aaeb55da022f9b66b17e89e406e7ca03c7636

this reminds me of bug 1332440 which started then, although this test has been around for quite some time in the tree.

:jgraham, can you comment on if we should mark this test as something to skip or we can fix this?  I am unclear on who would own this test or what bugzilla component we should put this in.
Flags: needinfo?(james)

Updated

3 months ago
See Also: → bug 1332484
(Assignee)

Comment 3

3 months ago
looks like a common pattern in:
/webvtt/rendering/cues-with-video/processing-model/selectors/cue_function/*/*-space_nowrap.html
See Also: → bug 1332642, bug 1332605
The update fixed some brokenness in the wpt harness that may have caused these tests to go from permafail to intermittent pass. Since we weren't running them correctly before it's possible that the tests have always been broken, or that we have uncovered some implementation issue.

I don't know who works on WebVTT  support but looking at the git log suggests maybe rillian would be a good starting point.
Flags: needinfo?(james) → needinfo?(giles)
Benjamin, can you please take a look at these web platform tests and figure out if they're correct and why we're intermittent on them?
Flags: needinfo?(giles) → needinfo?(bechen)

Comment 6

3 months ago
38 failures in 749 pushes (0.051 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 14
* autoland: 11
* mozilla-central: 7
* mozilla-aurora: 6

Platform breakdown:
* windows8-64: 38

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1332564&startday=2017-01-23&endday=2017-01-29&tree=all

Updated

3 months ago
See Also: → bug 1332543
I would prefer to disable/skip the whole tests under /webvtt/rendering/cues-with-video/processing-model/selectors.
Because the "selector for pseudo element" we don't implement yet. So it is meaningless if a test become PASS from FAIL.
Flags: needinfo?(bechen)
For the record I'm pretty skeptical of disabling tests just because we don't implement a feature yet. When we do come to implement the feature we are likley ot forget to reenable the tests and so miss the issues that they would have caught.
(Assignee)

Comment 9

3 months ago
I guess if there is a bug that is on file to implement the features, that would be a way to mark this as blocked by that bug and we could potentially remember this.

On another note, what bugzilla component would /webvtt/rendering/cues-with-video/processing-model/selectors tests be in?

in fact /webvtt/rendering/ has nothing but /webvtt/rendering/cues-with-video/processing-model/ which includes a list of tests and directories:
* support/ *.vtt
* evil/ {tests}
* bidi/ {tests}

would it be save to assume that all the tests under /webvtt/rendering would be the same bugzilla component?
(Assignee)

Comment 10

3 months ago
Created attachment 8833303 [details] [diff] [review]
disable web-platform-tests webvtt/rendering/*/selectors/*

example patch that will disable this.

:bechen, can you comment on if we really should disable this and what work is tracked to implement this if we do disable it?
Flags: needinfo?(bechen)
Waitaminute. This and all the other unexpected-PASS bugs that we unexpectedly pass so often that I changed the summary of one to "Permaorange" instead of "Intermittent", it is unexpected that we pass them because we DON'T IMPLEMENT THE FEATURE THEY TEST?

How about if instead of disabling them because we don't implement the feature, we instead disable them because they are obviously and self-evidently bogus and incorrectly-written?

Comment 12

3 months ago
60 failures in 733 pushes (0.082 failures/push) were associated with this bug in the last 7 days. 

This is the #17 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* autoland: 20
* mozilla-inbound: 19
* mozilla-central: 11
* mozilla-aurora: 10

Platform breakdown:
* windows8-64: 58
* windows7-32-vm: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1332564&startday=2017-01-30&endday=2017-02-05&tree=all

Updated

3 months ago
Depends on: 865395, 1321489
Maybe it is worth to file another bug to find out why those tests were PASS for a long time.
For example: the selectors/cue/color_rgba.html change the font color to rgba(128,255,128,0.5), but our implementation is white. So the testcase should be FAIL but it PASS on the try server.

Please disable(In reply to Joel Maher ( :jmaher) from comment #10)
> Created attachment 8833303 [details] [diff] [review]
> disable web-platform-tests webvtt/rendering/*/selectors/*
> 
> example patch that will disable this.
> 
> :bechen, can you comment on if we really should disable this and what work
> is tracked to implement this if we do disable it?

Please disable them first, after finishing bug 865395 and other relative bugs(::cue pseudo element and selector), I will re-enable the tests again. We can leave this bug open or file another one.
Flags: needinfo?(bechen)
(Assignee)

Comment 14

3 months ago
Comment on attachment 8833303 [details] [diff] [review]
disable web-platform-tests webvtt/rendering/*/selectors/*

as we will go forward with disabling these tests, lets do it and let :bechen implement the features and re-enable when it fits into a proper development schedule.
Attachment #8833303 - Flags: review?(gbrown)
Comment on attachment 8833303 [details] [diff] [review]
disable web-platform-tests webvtt/rendering/*/selectors/*

Review of attachment 8833303 [details] [diff] [review]:
-----------------------------------------------------------------

Sounds good. Thanks!
Attachment #8833303 - Flags: review?(gbrown) → review+
(Assignee)

Updated

3 months ago
Duplicate of this bug: 1332484
(Assignee)

Updated

3 months ago
Duplicate of this bug: 1332642
(Assignee)

Updated

3 months ago
Duplicate of this bug: 1332605

Comment 19

3 months ago
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/df0afcf0c3b6
disable webvtt/rendering/* as features are not implemented in gecko. r=gbrown

Comment 20

3 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/df0afcf0c3b6
Status: NEW → RESOLVED
Last Resolved: 3 months ago
status-firefox54: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla54
(Assignee)

Updated

3 months ago
Duplicate of this bug: 1332543
Assignee: nobody → jmaher
status-firefox52: --- → unaffected
status-firefox53: --- → affected

Comment 22

3 months ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-aurora/rev/4895917438eb
status-firefox53: affected → fixed
Flags: in-testsuite-

Comment 23

3 months ago
13 failures in 836 pushes (0.016 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* autoland: 5
* mozilla-central: 3
* mozilla-aurora: 3
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* windows8-64: 13

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1332564&startday=2017-02-06&endday=2017-02-12&tree=all
(Assignee)

Updated

2 months ago
Whiteboard: [stockwell disabled]
Blocks: 1353689
You need to log in before you can comment on or make changes to this bug.