WPT intermittent test failures where we're comparing a screenshot of the completely wrong file
Categories
(Testing :: web-platform-tests, defect)
Tracking
(Not tracked)
People
(Reporter: dholbert, Unassigned)
References
Details
[Filing this bug to track an issue I've noticed a few times in the past and don't want to lose track of the various bugs/instances as it continues to happen.]
Occasionally we get an intermittent WPT test-failure for a reftest-style WPT test, and it turns out the WPT has a screenshot of the entirely wrong file that it's using for comparison, for some reason (at least, as shown in reftest-analyzer).
Not sure what's going wrong -- i.e. if some edge case is causing WPT to just trivially use a stale/wrong screenshot somehow, vs. if the screenshot is silently failing in Firefox (and so we're left with a stale screenshot buffer), vs. if Firefox itself is returning a stale screenshot for some reason.
The most recent instance of this is bug 1905324, which is supposedly a test-failure in hyphens-vertical-001.html, but in fact the reftest-screenshot for the testcase appears to be a screenshot of hyphens-span-001.html (which is two tests earlier in the log). See my notes in bug 1905324 comment 1. It also might have been what happened in bug 1861485, though the logs there have expired so I'm not sure. (I'm just going off of the notes that I left in comments there.)
Let's mark instances of this issue (in whatever WPT test it happens to occur in) as depending on this bug, to keep track of them.
|
||
Comment 1•1 year ago
|
||
The severity field is not set for this bug.
:jgraham, could you have a look please?
For more information, please visit BugBot documentation.
|
Reporter | |
Comment 2•1 year ago
|
||
This might have been the same thing that we observed in bug 1947259 (similar symptoms, but for regular reftests instead of WPT reftests).
Given no new instances of this in 10 months, I think we can close as WFM.
Description
•