Open Bug 1465961 Opened 2 years ago Updated 2 years ago

TVw jobs don't seem to respect our per-platform test skipping

Categories

(Testing :: web-platform-tests, enhancement)

Version 3
enhancement
Not set
normal

Tracking

(Not tracked)

People

(Reporter: dholbert, Unassigned)

Details

It seems that whenever we add a new web-platform-test that uses the "Ahem" font, it permafails on OS X TVw, which causes bugs to be filed & causes confusion/concern.  But then, nothing is visibly wrong in subsequent OS X WPT test-runs (I think because we skip these ahem-using tests for "normal" OS X wpt test runs, though I'm not sure)

(There is a known issue with Ahem font not loading on Mac bug 1425698).

EXPECTED RESULTS:
If we skip "ahem"-using WPT tests on Mac for "normal" (non-TVw) test runs, then we should do so on TVw test runs as well.  Or if we don't skip the tests, then we should do whatever is needed to make the two sorts of test runs consistent.

Places where this has caused trouble/confusion in the past:
- bug 1451123 (and followup bug 1452184)
- bug 1465664
jgraham, do you know what we can do here to avoid these Mac-TVw-only perma-oranges that seem to go away for "real" Mac wpt runs?

(aside from fixing bug 1425698, which is presumably somewhat mysterious and which I think we're working around via some skipping logic now though I'm not sure)
Flags: needinfo?(james)
So the problem is that we pass --exclude=css in to wptrunner in mozharness on OSX/Windows. Presumably we are missing that flag on the TVw entrypoint. This isn't related to Ahem in particular, it just happens that Ahem-using tests fail on OSX but not on Linux so the problem is always visible in that case. So there are three possible solutions here in order of difficulty/correctness:

* Pass --exclude=css to the TVw entrypoint on Windows/OSX
* Stop using the flags and instead put disabled: if os == "win" or os == "osx" in testing/web-platform/meta/css/__dir__.ini
* Run all the CSS tests on Windows/OSX

I expect to work on the third option after all hands. For now I guess the second option is probably the best one to try.
Flags: needinfo?(james)
Thanks - that makes sense.
You need to log in before you can comment on or make changes to this bug.