Closed Bug 1498357 (wpt.fyi-firefox-fails) Opened 6 years ago Closed 2 years ago

[meta] Tracking bug for web platform tests that only fail in Firefox

Categories

(Core :: General, enhancement, P3)

Product:
Core
Component:
General
Type:
enhancement

Tracking

()

Status:
RESOLVED INCOMPLETE
Tracking Flags:
Tracking Status
firefox64 --- affected

People

(Reporter: bzbarsky, Unassigned)

References

(Depends on 16 open bugs,
URL
)

Details

(Keywords: meta) 

See https://groups.google.com/forum/#!topic/mozilla.dev.platform/LMnQq-yqbxg for a list.  We should file specific bugs blocking this one for specific failures.

One obvious issue is streams, of course.
Keywords: meta
Priority: -- → P3
Summary: Tracking bug for web platform tests that only fail in Firefox → [meta] Tracking bug for web platform tests that only fail in Firefox
Depends on: 332175
Depends on: 1328138
Depends on: 1425874
Depends on: 1398354
Depends on: 1499547
Depends on: 1499556
Alias: wpt.fyi-firefox-fails
Depends on: 1500492
Filed https://github.com/web-platform-tests/wpt/issues/13626 for a failure due to a test with buggy assumptions ( flex-flexitem-percentage-prescation.html )
Depends on: 1500495
Depends on: 1500590
Depends on: 1360715
Depends on: 1469045
No longer depends on: 1469045
Depends on: 1500623
Depends on: 1120171
Depends on: 1500631
Depends on: 1501023
Depends on: 1501057
Depends on: 1502168
Given the recent news [1] it won't be as relevant to consider the status of EdgeHTML for prioritization in other engines. Given that and the unreliable results, I've updated my script to consider only Chrome, Firefox and Safari. I also the reports auto-updating on a daily basis:

https://foolip.github.io/ad-hoc-wpt-results-analysis/chrome-lone-failures.html
https://foolip.github.io/ad-hoc-wpt-results-analysis/firefox-lone-failures.html
https://foolip.github.io/ad-hoc-wpt-results-analysis/safari-lone-failures.html

[1] https://github.com/MicrosoftEdge/MSEdge/blob/master/README.md
Depends on: 1514227
Depends on: 1514230
Depends on: 1514234
Depends on: 1514237
Depends on: 1514243
Depends on: 1514264
Depends on: 1514265
Depends on: 1514266
Depends on: 1514267
Depends on: 1514269
Depends on: 1514271
Depends on: 1514276
Depends on: 1514301
Depends on: 1514362
Depends on: 1514367
Depends on: 1514371
Depends on: 1514375
Depends on: 1514376
Depends on: 1514396
Depends on: 1514397
Depends on: 1514399
Depends on: 1514401
Depends on: 1514486
Depends on: 144795

wpt.fyi now supports a structured-query syntax that supports searching by specific statuses, per-browser. Using this feature, a search for

!firefox:pass !firefox:ok (chrome:pass|chrome:ok) (edge:pass|edge:ok) (safari:pass|safari:ok)

will show "Firefox anomalies" / tests that only fail in firefox.

https://wpt.fyi/results/?label=master&product=chrome%5Bexperimental%5D&product=edge&product=firefox%5Bexperimental%5D&product=safari%5Bexperimental%5D&aligned&q=%21firefox%3Apass+%21firefox%3Aok+%28chrome%3Apass%7Cchrome%3Aok%29+%28edge%3Apass%7Cedge%3Aok%29+%28safari%3Apass%7Csafari%3Aok%29

Depends on: 1489308
Depends on: 1522940
Depends on: 1522963
Depends on: 1450027
Depends on: 1527874
Depends on: 1527875
Depends on: 1528689
Depends on: 1528692
Depends on: 1528706
Depends on: 1529019
Depends on: 1529051
Depends on: 1529054

https://jgraham.github.io/wptdash/ can be used to get a list of tests that fail in Firefox but pass in Chrome & Safari, based on wpt.fyi data (url subject ot change, suggestions for improvement welcome).

Depends on: 1529087
Depends on: 1529093
Depends on: 1529096
No longer depends on: 1529096
No longer depends on: 1514486
No longer depends on: 1527874
No longer depends on: 1527875
No longer depends on: 1529087
Depends on: 1529299
No longer depends on: 1514397
Depends on: 1529543

lukebjerring, is the wpt.fyi URL in comment #3 correct? In many of the columns I see "N / N" for Firefox, which seems weird.

Are the exclamation marks just in the wrong place? The insights tab on staging.wpt.fyi suggests "(firefox:!pass&firefox:!ok) (chrome:pass|chrome:ok) (edge:pass|edge:ok) (safari:pass|safari:ok)" as the query, but I ran into this bug:
https://github.com/web-platform-tests/wpt.fyi/issues/1167

Depends on: 25291
Depends on: 1533013
Depends on: 1543267
Depends on: 1092007
Depends on: 1545051
Depends on: 1546387
Depends on: 1546389
No longer depends on: 1546387
No longer depends on: 1529051
No longer depends on: 1514376
No longer depends on: 1514265
No longer depends on: 1529093
No longer depends on: 1514264
No longer depends on: 1500492
No longer depends on: 1514276
No longer depends on: 1514301
No longer depends on: 1514266
No longer depends on: 1514271
No longer depends on: 1514237
No longer depends on: 1500495
No longer depends on: 1514267
No longer depends on: 1514243
No longer depends on: 1514269
No longer depends on: 1514375
No longer depends on: 1529054
No longer depends on: 1514362
No longer depends on: 1501508
No longer depends on: 1543267
No longer depends on: 1539469
No longer depends on: 1514367
Depends on: 1549560
Depends on: 1568219
Depends on: 1531397
Depends on: 1574856
Depends on: 1578448
No longer depends on: 1502168
Depends on: 1598674
Depends on: 1607363
Depends on: 1607615
Depends on: 520969
Depends on: 1609491
Depends on: 1610819
Depends on: 1610821
Depends on: 1614329
Depends on: 1467437

This meta bug isn't used anymore. Shall we close it?

Flags: needinfo?(james)

Sure? Happy for anyone to reopen it if there's some use, but tracking per component might be easier.

Status: NEW → RESOLVED
Closed: 2 years ago
Flags: needinfo?(james)
Resolution: --- → INCOMPLETE
Depends on: 1903405
No longer depends on: 1903405
You need to log in before you can comment on or make changes to this bug.