Closed Bug 1267486 Opened 8 years ago Closed 6 years ago

Intermittent webkit-text-fill-color-property-003.html | /compat/webkit-text-fill-color-property-003.html 798b8ed19e9fc078bba7ee8baf13f181c59ee8a0

Categories

(Testing :: web-platform-tests, defect)

Product:
Testing
Component:
web-platform-tests
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox48 affected, firefox49 affected)

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
firefox48 --- affected
firefox49 --- affected

People

(Reporter: KWierso, Unassigned)

Details

(Keywords: intermittent-failure) 

Jeremy, this is somewhat high on the Orange Factor list. Can you please take a look?
Flags: needinfo?(jeremychen)
(In reply to Andrew Overholt [:overholt] from comment #12)
> Jeremy, this is somewhat high on the Orange Factor list. Can you please take
> a look?

Sure. I took a look and found that this is likely a focus problem from reftest framework. (See bug 1292460 comment 17)

I suspect this a scenario that the test file is finishing loading while the framework (xul parts) is not. Therefore, the focus is changed to xul instead of the test file. However, on linux, the color of selected texts could be different between w/ and w/o focus. So, we would see texts within an unfocused color in the failed tests. Taking try runs to prove my assumptions.
Flags: needinfo?(jeremychen)
Second thoughts, this may not be chunk related since some of the failures are not in the first of a chunk. Still figuring out why the focus is been changed before taking a screenshot.
Here's a brief about my investigation:

From the test failures, it is very likely caused through following steps:
1. load test page and execute the script to select texts
2. somehow something (maybe relates to reftest framework or testing build bot environment) causes a focus change so the color of selected texts become another color which presents for selected texts w/o focus. so, an unexpected reftest screenshot is taken.

As comment 14 said, this is not chunk related. Some failures happen even though they are not the first test right after the testing browser restarted.

So, I add debugging focus-logs to see if there's any clue. However, focus-logs looks exact the same from failed tests to success tests. (https://treeherder.mozilla.org/#/jobs?repo=try&revision=e7855f3b89ff)

To ensure that a reftest screenshot is taken on test page while focus on it, I do have a patch which extends the waiting time for test page to take a reftest screenshot. This seems eliminate the symptom from 1~2 out of 20 (https://treeherder.mozilla.org/#/jobs?repo=try&revision=0a43d502f77c&filter-searchStr=Linux%20opt%20W3C%20Web%20Platform%20Reftests%20(Wr))to 0 out of 20 ( https://treeherder.mozilla.org/#/jobs?repo=try&revision=dcae8b31c79f)

But, I doubt if this is something worth to do for three reasons: 1) extending waiting time may not be reliable enough; 2) adding additional codes confuses developers to read/maintain the test; 3) the cost (TC/buildbot resource usage) is definitely increased.

There're still some mysteries need to be dug, e.g., why this only happens on linux platform? However, to do that, I might need to take much more time (probably less time for a reftest framework expert!?) to study reftest framework, and to tell the difference of environments between running tests on linux platform machines and those on other platform machines.

From what I can tell, the feature functions finely, the tests seems fine as well. Maybe we should just keep tracking this instead of working around it for now?
Flags: needinfo?(overholt)
Thanks for the analysis, Jeremy! I'm not sure what's best in this case but Ryan probably has thoughts.
As to reftest issues, Ted can probably help.
Flags: needinfo?(ted)
Flags: needinfo?(ryanvm)
Flags: needinfo?(overholt)
Linux focus issues are usually the domain of Neil, but he's on PTO until next week. This is the web-platform-test harness, not our regular reftest harness, so you might want to ping James first. But I suspect he'll end up deferring to Neil, so I don't know.
Flags: needinfo?(ted)
Flags: needinfo?(ryanvm)
Flags: needinfo?(james)
Yeah, I think Ryan is right that I don't know off the top of my head. It is possible that the wpt harness needs to be more aggressive about ensuring the window is focused when a screenshot is taken. I can check what the other reftest harness does ofc.
Flags: needinfo?(james) → needinfo?(enndeakin)
Unfortunately I don't know much about the wpt harness. Hopefully Enn can help you!
It looks like this doesn't happen anymore.
Flags: needinfo?(enndeakin)
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.