913153 - (css3test) [meta] Improve score on CSS3Test.com

Status: NEW

(Core :: Layout, task)

Description

[Florian Bender]

Nightly is now at 67% (BTW: Chrome 29 stable is at 66% but including false positives). 

Fixing / unifying CSS image value rendering should give us a great improvement, though there's still a lot to do. 100% seems unrealistic e. g. due to the Speech test (unless that's something Mozilla is interested in supporting in Firefox). However, a solid >90% doesn't seem too far off, does it?

Comment 1

[Florian Bender]

I did not check yet whether the tests make sense, are up-to-date (to the specs) or if there are false negatives, but I doubt there's any error. 

– image-resolution 
– icon & content: icon
– nav-*
– ::value, ::choices, ::repeat-*
– perspective: 0
– line-break
– overflow-wrap
– text-align: start end;
– word-spacing & letter-spacing (extended)
– hanging-punctuation
– text-decoration (extended) & text-emphasis*
– font-variant: none & (extended) & font-variant-ligatures: none
– attr(data-*) (that one's weird)
– 

… do not have a bug yet. Will file new ones as soon as I have time (anyone is welcome to do this, though). 

Bug 276079 needs a followup. `text-indent: hanging` should probably be handled by Bug 784648. Need to check if Bug 702508 covers all failing flex box tests. 

Left to check: Writing Modes part.

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I think this shouldn't depend on metabugs like bug 631373; it should depend only on specific bugs that cause failures of these tests.

Comment 3

[Florian Bender]

Yeah, sorry, that one should not be there.

Comment 4

[Florian Bender]

There IS a speech bug. Yay! 


Also checked the Writing Modes part, bugs need to be filed for: 

– text-orientation
– caption-side
– text-combine-horizontal

Now I got everything in the test at least mentioned in this bug. Time to file bugs! Later, though.

Comment 5

[Florian Bender]

Bug 775618 may fix failing break-* tests.

Comment 6

[John Daggett (:jtd)]

(In reply to Florian Bender from comment #1)
> – font-variant: none & (extended) & font-variant-ligatures: none

Ah, adventures in Webkit targeting...  These are the only two flavors of the "font-variant" subproperties that Webkit supports.

Comment 7

[John Daggett (:jtd)]

(In reply to John Daggett (:jtd) from comment #6)
> (In reply to Florian Bender from comment #1)
> > – font-variant: none & (extended) & font-variant-ligatures: none
> 
> Ah, adventures in Webkit targeting...  These are the only two flavors of the
> "font-variant" subproperties that Webkit supports.

Ok, oops.  The problem appears to be that these *values* were added after I wrote the initial patches for feature support.  Nothing to do with Webkit...  Will file a bug on these additions.

Comment 8

[Florian Bender]

Thanks for filing! 

There is also a failing test for `font-variant: sub lining-nums contextual ruby;`. Is there a bug for that, too? Is that behind a flag? All other tests (except 'none' & unicode-range) succeed.

Comment 9

[Florian Bender]

Sorry, looks like the stuff in Bug 651693 (resp. depending bugs) needs to be switched on by default, unless Bug 872301 can be fixed in time. 

Should I add Bug 651693 back again to the dependancy list, then? Could not find a bug for flipping the pref …

Comment 10

[Florian Bender]

There has been a big update to css3test.com with the following changes (according to https://github.com/LeaVerou/css3test/commits/gh-pages): 
– update Writing Modes test
– add dpcm test to image-resolution
– add CSS Regions test
– update CSS Text Decoration test
– update CSS Text test
– add Grid Layout test
– fix transform/perspective test

The following tests were changed (08th Sep), I'm also mentioning what the changes mean to this bug (whereby "[no change]" means no change to the previously posted lists, mostly because "sibling" tests were already failing, or new tests pass): 
– [no change] PASSing new direction and updated unicode-bidi properties
– [no change] still FAILing caption-side & text-combine-horizontal
– [no change] FAILing new image-resolution: *dpcm test
– [NEW] FAILing CSS Regions (added bug)
– [no change] still FAILing text-underline-position, text-emphasis*
– [no change] PASSing updated text-shadow test
– [no change] PASSing new word-wrap test
– [no change] still FAILing parts of text-align but
– [NEW] text-align: [char] test has been removed (removed dependent bug, sadly)
– [no change] still FAILing text-justify
– [NEW] FAILing CSS Grid (except `display: [inline-]grid;` but that's triggering XUL Grid; added bug)
– [NEW] PASSing perspective-origin & transform-origin (waiting on Bug 712484 to resolve)

Comment 11

[Florian Bender]

The test for `perspective: 0` is invalid ("Lengths must be positive.", http://www.w3.org/TR/css3-transforms/#perspective-property) and will likely be removed shortly (at least there's a pull request ready, see https://github.com/LeaVerou/css3test/pull/54), so there's one less bug to file. 

BTW, stock Nightly (2013-09-09) is now at 62% (including false positive for `display: (inline-)grid`) vs. Firefox 24b9 at 58% (mostly due to font magic) vs. Chrome 29 stable at 60% (including false positive for `background-repeat: space|round;`) vs. Safari 6.0.5 at 51% (but v7 will improve its score significantly, also with false positive).

Comment 12

[Florian Bender]

Bug 914360 does not improve the score (the opposite), but it corrects it. Thus adding as dependency.

Comment 13

[Florian Bender]

(In reply to Florian Bender from comment #1)
> – attr(data-*) (that one's weird)
This is an actual bug in Gecko: Bug 435426.

Comment 14

[Chris Peterson [:cpeterson]]

Tom's Hardware Guide's "Web Browser Grand Prix" (tracking bug 933949) includes css3test.

Comment 15

[Guilherme Lima]

Latest Nightly - 630 tests out of 1039
Firefox Beta -  557 tests out of 1039
Chrome 31 - 588 tests out of 1039
IE 11 - 587 tests out of 1039

Comment 16

[sjw]

Chrome 33 - 597 tests out of 1039
Opera 19 -  613 tests out of 1039

Comment 17

[Albert Scheiner [:alberts]]

Latest Nightly -     667 tests out of 1338
Latest Aurora (29) - 667 tests out of 1338
Firefox Beta (28) -  589 tests out of 1338
Chrome 33 -          658 tests out of 1338
Safari 7 -           644 tests out of 1338 (minus some points for background-repeat)

Comment 18

[Sören Hentzschel]

Latest Nightly vs. Chrome 33 is not a fair comparison, you have to compare latest Nightly with Chrome Canary and latest stable of Chrome with latest stable of Firefox… ;)

Latest Firefox: 575 tests out of 1338
Latest Chrome: 658 tests out of 1338

Latest Nightly: 667 tests out of 1338
Latest Chrome Canary: 675 tests out of 1338

But good work @firefox devs, it's a huge jump from 589 to 667!

Comment 19

[Albert Scheiner [:alberts]]

Sure. I must confess I was a bit too lazy to download Canary atm. Thanks for the numbers Sören.

Comment 20

[sjw]

Opera Next 20 got 813.

Comment 21

[Florian Bender]

Note that pre-release versions are likely to have a higher score than the resulting release, e.g. most font tests will fail in Fx 29 Release because they are only enabled in Nightly and Aurora (by default).

Comment 22

[entership12345]

For now:

33 Nightly: 686 out of 1341, 59%
38  Canary: 709 out of 1341, 60%

Comment 23

[Gérard Talbot]

I am convinced you should be careful at not reaching any hard, solid conclusion regarding Lea Verou meta-test page. For many reasons.

- The page and tests need to be constantly updated: eg 'cursor: grab' and 'cursor: grabbing' are not being tested ( http://www.w3.org/TR/2015/CR-css-ui-3-20150707/#cursor ) and I have spotted other properties while editing this comment 
- The page tests properties from Working Drafts and not just from Candidate Recommendations only: eg Regions module
- The page claim Firefox supports properties that we know are not implemented at all: widows and orphans ( bug 137367 )
- The page or tests claim Firefox does not support 'text-combine-upright' but that is not true:
http://test.csswg.org/suites/css-writing-modes-3_dev/nightly-unstable/html/text-combine-upright-parsing-valid-001.htm
and bug 1097499 : no working implementation but there is support with Firefox 42 if/when layout.css.text-combine-upright.enabled is set to true
- There are many false positives and false negatives in those tests and that page: Chrome supports a lot more than what the page claims due to vendor-prefixes

The most important reason why this page should not be taken seriously (otherwise, at least, with a lot of salt) is that implementation is not being tested and Lea Verou is fully aware of this: "Caution: This test checks which CSS3 features the browser recognizes, not whether they are implemented correctly."

Comment 24

[Sebastian Zartner [:sebo]]

(In reply to Gérard Talbot from comment #23)
> - The page and tests need to be constantly updated: eg 'cursor: grab' and
> 'cursor: grabbing' are not being tested (
> http://www.w3.org/TR/2015/CR-css-ui-3-20150707/#cursor )

Filed an issue to implement those values.[1]

> and I have spotted
> other properties while editing this comment 

Would be good if you filed issues against those on the css3test GitHub repo.

> - The page tests properties from Working Drafts and not just from Candidate
> Recommendations only: eg Regions module

As far as I know the implementation criteria on css3test.com does not follow the status of the specifications but requires at least one working implementation of a spec.

> - The page claim Firefox supports properties that we know are not
> implemented at all: widows and orphans ( bug 137367 )

As the page states at the top, it only tests whether a browser recognizes a feature, not whether its implemented correctly.
While 'widows' and 'orphans' are not implemented yet, Firefox already exposes them through the DOM.
Non-the-less this should be mentioned on the page, so I created an issue for that.[2]

> - The page or tests claim Firefox does not support 'text-combine-upright'
> but that is not true:
> http://test.csswg.org/suites/css-writing-modes-3_dev/nightly-unstable/html/
> text-combine-upright-parsing-valid-001.htm

'text-combine-upright' is not tested yet.[3] Instead it still tests for 'text-combine-horizontal'.
Also, Firefox *does not support* 'text-combine-upright' yet.[4]

> - There are many false positives and false negatives in those tests and that7
> page: Chrome supports a lot more than what the page claims due to
> vendor-prefixes

Not sure what's the rule for the page when features are behind prefixes, though you should ask Lea about that.

> The most important reason why this page should not be taken seriously
> (otherwise, at least, with a lot of salt) is that implementation is not
> being tested and Lea Verou is fully aware of this: "Caution: This test
> checks which CSS3 features the browser recognizes, not whether they are
> implemented correctly."

Same counts in big parts for html5test.com (which is tracked in bug 802882, btw.). Though web authors refer to those pages for comparing how good browsers implement the (upcoming) web standards. Bad scores may result in bad reputation. So it should be somewhat important to Mozilla to get a higher rank on those pages.
And for implementers they provide at least a rough overview where feature implementation stands and may help to decide what to implement next.

If you believe css3test.com does not offer good enough measurement in that relation, you should talk to Lea and/or file bugs against it.

Sebastian

[1] https://github.com/LeaVerou/css3test/issues/103
[2] https://github.com/LeaVerou/css3test/issues/104
[3] https://github.com/LeaVerou/css3test/issues/105
[4] bug 1097499

Comment 25

[Jens Hausdorf :jens1o [ni? me]]

We're still at 67% using Nightly.