Open Bug 913153 (css3test) Opened 11 years ago Updated 2 years ago

[meta] Improve score on CSS3Test.com

Categories

(Core :: Layout, task)

task

Tracking

()

People

(Reporter: fb+mozdev, Unassigned)

References

(Depends on 17 open bugs, Blocks 1 open bug, )

Details

(Keywords: meta)

Nightly is now at 67% (BTW: Chrome 29 stable is at 66% but including false positives). 

Fixing / unifying CSS image value rendering should give us a great improvement, though there's still a lot to do. 100% seems unrealistic e. g. due to the Speech test (unless that's something Mozilla is interested in supporting in Firefox). However, a solid >90% doesn't seem too far off, does it?
I did not check yet whether the tests make sense, are up-to-date (to the specs) or if there are false negatives, but I doubt there's any error. 

– image-resolution 
– icon & content: icon
– nav-*
– ::value, ::choices, ::repeat-*
– perspective: 0
– line-break
– overflow-wrap
– text-align: start end;
– word-spacing & letter-spacing (extended)
– hanging-punctuation
– text-decoration (extended) & text-emphasis*
– font-variant: none & (extended) & font-variant-ligatures: none
– attr(data-*) (that one's weird)
– 

… do not have a bug yet. Will file new ones as soon as I have time (anyone is welcome to do this, though). 

Bug 276079 needs a followup. `text-indent: hanging` should probably be handled by Bug 784648. Need to check if Bug 702508 covers all failing flex box tests. 

Left to check: Writing Modes part.
I think this shouldn't depend on metabugs like bug 631373; it should depend only on specific bugs that cause failures of these tests.
Yeah, sorry, that one should not be there.
There IS a speech bug. Yay! 


Also checked the Writing Modes part, bugs need to be filed for: 

– text-orientation
– caption-side
– text-combine-horizontal

Now I got everything in the test at least mentioned in this bug. Time to file bugs! Later, though.
Depends on: 47159, writing-mode
Bug 775618 may fix failing break-* tests.
Depends on: 775618
(In reply to Florian Bender from comment #1)
> – font-variant: none & (extended) & font-variant-ligatures: none

Ah, adventures in Webkit targeting...  These are the only two flavors of the "font-variant" subproperties that Webkit supports.
(In reply to John Daggett (:jtd) from comment #6)
> (In reply to Florian Bender from comment #1)
> > – font-variant: none & (extended) & font-variant-ligatures: none
> 
> Ah, adventures in Webkit targeting...  These are the only two flavors of the
> "font-variant" subproperties that Webkit supports.

Ok, oops.  The problem appears to be that these *values* were added after I wrote the initial patches for feature support.  Nothing to do with Webkit...  Will file a bug on these additions.
Depends on: 913264
Thanks for filing! 

There is also a failing test for `font-variant: sub lining-nums contextual ruby;`. Is there a bug for that, too? Is that behind a flag? All other tests (except 'none' & unicode-range) succeed.
Sorry, looks like the stuff in Bug 651693 (resp. depending bugs) needs to be switched on by default, unless Bug 872301 can be fixed in time. 

Should I add Bug 651693 back again to the dependancy list, then? Could not find a bug for flipping the pref …
There has been a big update to css3test.com with the following changes (according to https://github.com/LeaVerou/css3test/commits/gh-pages): 
– update Writing Modes test
– add dpcm test to image-resolution
– add CSS Regions test
– update CSS Text Decoration test
– update CSS Text test
– add Grid Layout test
– fix transform/perspective test

The following tests were changed (08th Sep), I'm also mentioning what the changes mean to this bug (whereby "[no change]" means no change to the previously posted lists, mostly because "sibling" tests were already failing, or new tests pass): 
– [no change] PASSing new direction and updated unicode-bidi properties
– [no change] still FAILing caption-side & text-combine-horizontal
– [no change] FAILing new image-resolution: *dpcm test
– [NEW] FAILing CSS Regions (added bug)
– [no change] still FAILing text-underline-position, text-emphasis*
– [no change] PASSing updated text-shadow test
– [no change] PASSing new word-wrap test
– [no change] still FAILing parts of text-align but
– [NEW] text-align: [char] test has been removed (removed dependent bug, sadly)
– [no change] still FAILing text-justify
– [NEW] FAILing CSS Grid (except `display: [inline-]grid;` but that's triggering XUL Grid; added bug)
– [NEW] PASSing perspective-origin & transform-origin (waiting on Bug 712484 to resolve)
Depends on: 674802, css-grid
No longer depends on: character-alignment
The test for `perspective: 0` is invalid ("Lengths must be positive.", http://www.w3.org/TR/css3-transforms/#perspective-property) and will likely be removed shortly (at least there's a pull request ready, see https://github.com/LeaVerou/css3test/pull/54), so there's one less bug to file. 

BTW, stock Nightly (2013-09-09) is now at 62% (including false positive for `display: (inline-)grid`) vs. Firefox 24b9 at 58% (mostly due to font magic) vs. Chrome 29 stable at 60% (including false positive for `background-repeat: space|round;`) vs. Safari 6.0.5 at 51% (but v7 will improve its score significantly, also with false positive).
Bug 914360 does not improve the score (the opposite), but it corrects it. Thus adding as dependency.
Depends on: 914360
Depends on: 921731
(In reply to Florian Bender from comment #1)
> – attr(data-*) (that one's weird)
This is an actual bug in Gecko: Bug 435426.
Depends on: 435426
Tom's Hardware Guide's "Web Browser Grand Prix" (tracking bug 933949) includes css3test.
Blocks: WBGP
Status: UNCONFIRMED → NEW
Ever confirmed: true
Depends on: 939901
Depends on: 943918
No longer depends on: 737785
No longer blocks: WBGP
Blocks: WBGP
Latest Nightly - 630 tests out of 1039
Firefox Beta -  557 tests out of 1039
Chrome 31 - 588 tests out of 1039
IE 11 - 587 tests out of 1039
Chrome 33 - 597 tests out of 1039
Opera 19 -  613 tests out of 1039
Depends on: 955857
Component: General → Tracking
Keywords: meta
QA Contact: chofmann
Latest Nightly -     667 tests out of 1338
Latest Aurora (29) - 667 tests out of 1338
Firefox Beta (28) -  589 tests out of 1338
Chrome 33 -          658 tests out of 1338
Safari 7 -           644 tests out of 1338 (minus some points for background-repeat)
Latest Nightly vs. Chrome 33 is not a fair comparison, you have to compare latest Nightly with Chrome Canary and latest stable of Chrome with latest stable of Firefox… ;)

Latest Firefox: 575 tests out of 1338
Latest Chrome: 658 tests out of 1338

Latest Nightly: 667 tests out of 1338
Latest Chrome Canary: 675 tests out of 1338

But good work @firefox devs, it's a huge jump from 589 to 667!
Sure. I must confess I was a bit too lazy to download Canary atm. Thanks for the numbers Sören.
Opera Next 20 got 813.
Note that pre-release versions are likely to have a higher score than the resulting release, e.g. most font tests will fail in Fx 29 Release because they are only enabled in Nightly and Aurora (by default).
Depends on: 1011369
Depends on: 1006326
No longer depends on: 613659
For now:

33 Nightly: 686 out of 1341, 59%
38  Canary: 709 out of 1341, 60%
Depends on: 686281
Depends on: 773296
Depends on: 1038663
Depends on: 1039488
Depends on: 1040668
Depends on: 1040695
Depends on: 660196
Depends on: css-shapes-1
Depends on: 960316
Depends on: 869828
Depends on: 1057180
Depends on: 1032922
Depends on: 1058519
Depends on: 1091885
Depends on: will-change
Depends on: enable-css-ruby
I am convinced you should be careful at not reaching any hard, solid conclusion regarding Lea Verou meta-test page. For many reasons.

- The page and tests need to be constantly updated: eg 'cursor: grab' and 'cursor: grabbing' are not being tested ( http://www.w3.org/TR/2015/CR-css-ui-3-20150707/#cursor ) and I have spotted other properties while editing this comment 
- The page tests properties from Working Drafts and not just from Candidate Recommendations only: eg Regions module
- The page claim Firefox supports properties that we know are not implemented at all: widows and orphans ( bug 137367 )
- The page or tests claim Firefox does not support 'text-combine-upright' but that is not true:
http://test.csswg.org/suites/css-writing-modes-3_dev/nightly-unstable/html/text-combine-upright-parsing-valid-001.htm
and bug 1097499 : no working implementation but there is support with Firefox 42 if/when layout.css.text-combine-upright.enabled is set to true
- There are many false positives and false negatives in those tests and that page: Chrome supports a lot more than what the page claims due to vendor-prefixes

The most important reason why this page should not be taken seriously (otherwise, at least, with a lot of salt) is that implementation is not being tested and Lea Verou is fully aware of this: "Caution: This test checks which CSS3 features the browser recognizes, not whether they are implemented correctly."
(In reply to Gérard Talbot from comment #23)
> - The page and tests need to be constantly updated: eg 'cursor: grab' and
> 'cursor: grabbing' are not being tested (
> http://www.w3.org/TR/2015/CR-css-ui-3-20150707/#cursor )

Filed an issue to implement those values.[1]

> and I have spotted
> other properties while editing this comment 

Would be good if you filed issues against those on the css3test GitHub repo.

> - The page tests properties from Working Drafts and not just from Candidate
> Recommendations only: eg Regions module

As far as I know the implementation criteria on css3test.com does not follow the status of the specifications but requires at least one working implementation of a spec.

> - The page claim Firefox supports properties that we know are not
> implemented at all: widows and orphans ( bug 137367 )

As the page states at the top, it only tests whether a browser recognizes a feature, not whether its implemented correctly.
While 'widows' and 'orphans' are not implemented yet, Firefox already exposes them through the DOM.
Non-the-less this should be mentioned on the page, so I created an issue for that.[2]

> - The page or tests claim Firefox does not support 'text-combine-upright'
> but that is not true:
> http://test.csswg.org/suites/css-writing-modes-3_dev/nightly-unstable/html/
> text-combine-upright-parsing-valid-001.htm

'text-combine-upright' is not tested yet.[3] Instead it still tests for 'text-combine-horizontal'.
Also, Firefox *does not support* 'text-combine-upright' yet.[4]

> - There are many false positives and false negatives in those tests and that7
> page: Chrome supports a lot more than what the page claims due to
> vendor-prefixes

Not sure what's the rule for the page when features are behind prefixes, though you should ask Lea about that.

> The most important reason why this page should not be taken seriously
> (otherwise, at least, with a lot of salt) is that implementation is not
> being tested and Lea Verou is fully aware of this: "Caution: This test
> checks which CSS3 features the browser recognizes, not whether they are
> implemented correctly."

Same counts in big parts for html5test.com (which is tracked in bug 802882, btw.). Though web authors refer to those pages for comparing how good browsers implement the (upcoming) web standards. Bad scores may result in bad reputation. So it should be somewhat important to Mozilla to get a higher rank on those pages.
And for implementers they provide at least a rough overview where feature implementation stands and may help to decide what to implement next.

If you believe css3test.com does not offer good enough measurement in that relation, you should talk to Lea and/or file bugs against it.

Sebastian

[1] https://github.com/LeaVerou/css3test/issues/103
[2] https://github.com/LeaVerou/css3test/issues/104
[3] https://github.com/LeaVerou/css3test/issues/105
[4] bug 1097499
Depends on: 968761
Depends on: 1285811
Component: Tracking → Layout
Depends on: 933562
Depends on: 546052
Depends on: nth-child-of
Depends on: 1373832
Depends on: 1373833
Depends on: 1373835
See Also: → html5test
We're still at 67% using Nightly.
Depends on: 1578503
Type: defect → task
Depends on: cross-fade
No longer depends on: 546052
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.