Last Comment Bug 913153 - (css3test) [meta] Improve score on CSS3Test.com
(css3test)
: [meta] Improve score on CSS3Test.com
Status: NEW
: meta
Product: Core
Classification: Components
Component: Layout (show other bugs)
: unspecified
: All All
-- normal with 28 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
: chris hofmann
: Jet Villegas (:jet)
Mentors:
http://css3test.com/
Depends on: 47159 writing-mode 276079 363250 435426 549114 616436 css-grid 674802 700926 703217 770780 775618 784648 812990 914360 960316 1011369 css-shapes-1 css-text-emphasis basic-shape-ship 1285811 unicode-range 548372 624647 645642 660196 686281 702508 712484 720531 773296 842329 869828 913264 921731 939901 943918 955857 will-change 968761 1006326 1032922 1038663 enable-css-ruby 1039488 1040668 1040695 1057180 1058519 1091885
Blocks: WBGP
  Show dependency treegraph
 
Reported: 2013-09-05 11:47 PDT by Florian Bender
Modified: 2016-08-09 19:31 PDT (History)
34 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments

Description User image Florian Bender 2013-09-05 11:47:09 PDT
Nightly is now at 67% (BTW: Chrome 29 stable is at 66% but including false positives). 

Fixing / unifying CSS image value rendering should give us a great improvement, though there's still a lot to do. 100% seems unrealistic e. g. due to the Speech test (unless that's something Mozilla is interested in supporting in Firefox). However, a solid >90% doesn't seem too far off, does it?
Comment 1 User image Florian Bender 2013-09-05 12:39:55 PDT
I did not check yet whether the tests make sense, are up-to-date (to the specs) or if there are false negatives, but I doubt there's any error. 

– image-resolution 
– icon & content: icon
– nav-*
– ::value, ::choices, ::repeat-*
– perspective: 0
– line-break
– overflow-wrap
– text-align: start end;
– word-spacing & letter-spacing (extended)
– hanging-punctuation
– text-decoration (extended) & text-emphasis*
– font-variant: none & (extended) & font-variant-ligatures: none
– attr(data-*) (that one's weird)
– 

… do not have a bug yet. Will file new ones as soon as I have time (anyone is welcome to do this, though). 

Bug 276079 needs a followup. `text-indent: hanging` should probably be handled by Bug 784648. Need to check if Bug 702508 covers all failing flex box tests. 

Left to check: Writing Modes part.
Comment 2 User image David Baron :dbaron: ⌚️UTC-8 2013-09-05 12:48:59 PDT
I think this shouldn't depend on metabugs like bug 631373; it should depend only on specific bugs that cause failures of these tests.
Comment 3 User image Florian Bender 2013-09-05 13:33:22 PDT
Yeah, sorry, that one should not be there.
Comment 4 User image Florian Bender 2013-09-05 13:48:17 PDT
There IS a speech bug. Yay! 


Also checked the Writing Modes part, bugs need to be filed for: 

– text-orientation
– caption-side
– text-combine-horizontal

Now I got everything in the test at least mentioned in this bug. Time to file bugs! Later, though.
Comment 5 User image Florian Bender 2013-09-05 14:07:37 PDT
Bug 775618 may fix failing break-* tests.
Comment 6 User image John Daggett (:jtd) 2013-09-05 16:43:42 PDT
(In reply to Florian Bender from comment #1)
> – font-variant: none & (extended) & font-variant-ligatures: none

Ah, adventures in Webkit targeting...  These are the only two flavors of the "font-variant" subproperties that Webkit supports.
Comment 7 User image John Daggett (:jtd) 2013-09-05 16:48:21 PDT
(In reply to John Daggett (:jtd) from comment #6)
> (In reply to Florian Bender from comment #1)
> > – font-variant: none & (extended) & font-variant-ligatures: none
> 
> Ah, adventures in Webkit targeting...  These are the only two flavors of the
> "font-variant" subproperties that Webkit supports.

Ok, oops.  The problem appears to be that these *values* were added after I wrote the initial patches for feature support.  Nothing to do with Webkit...  Will file a bug on these additions.
Comment 8 User image Florian Bender 2013-09-06 02:07:07 PDT
Thanks for filing! 

There is also a failing test for `font-variant: sub lining-nums contextual ruby;`. Is there a bug for that, too? Is that behind a flag? All other tests (except 'none' & unicode-range) succeed.
Comment 9 User image Florian Bender 2013-09-06 03:11:14 PDT
Sorry, looks like the stuff in Bug 651693 (resp. depending bugs) needs to be switched on by default, unless Bug 872301 can be fixed in time. 

Should I add Bug 651693 back again to the dependancy list, then? Could not find a bug for flipping the pref …
Comment 10 User image Florian Bender 2013-09-09 13:49:21 PDT
There has been a big update to css3test.com with the following changes (according to https://github.com/LeaVerou/css3test/commits/gh-pages): 
– update Writing Modes test
– add dpcm test to image-resolution
– add CSS Regions test
– update CSS Text Decoration test
– update CSS Text test
– add Grid Layout test
– fix transform/perspective test

The following tests were changed (08th Sep), I'm also mentioning what the changes mean to this bug (whereby "[no change]" means no change to the previously posted lists, mostly because "sibling" tests were already failing, or new tests pass): 
– [no change] PASSing new direction and updated unicode-bidi properties
– [no change] still FAILing caption-side & text-combine-horizontal
– [no change] FAILing new image-resolution: *dpcm test
– [NEW] FAILing CSS Regions (added bug)
– [no change] still FAILing text-underline-position, text-emphasis*
– [no change] PASSing updated text-shadow test
– [no change] PASSing new word-wrap test
– [no change] still FAILing parts of text-align but
– [NEW] text-align: [char] test has been removed (removed dependent bug, sadly)
– [no change] still FAILing text-justify
– [NEW] FAILing CSS Grid (except `display: [inline-]grid;` but that's triggering XUL Grid; added bug)
– [NEW] PASSing perspective-origin & transform-origin (waiting on Bug 712484 to resolve)
Comment 11 User image Florian Bender 2013-09-09 14:35:12 PDT
The test for `perspective: 0` is invalid ("Lengths must be positive.", http://www.w3.org/TR/css3-transforms/#perspective-property) and will likely be removed shortly (at least there's a pull request ready, see https://github.com/LeaVerou/css3test/pull/54), so there's one less bug to file. 

BTW, stock Nightly (2013-09-09) is now at 62% (including false positive for `display: (inline-)grid`) vs. Firefox 24b9 at 58% (mostly due to font magic) vs. Chrome 29 stable at 60% (including false positive for `background-repeat: space|round;`) vs. Safari 6.0.5 at 51% (but v7 will improve its score significantly, also with false positive).
Comment 12 User image Florian Bender 2013-09-09 14:48:58 PDT
Bug 914360 does not improve the score (the opposite), but it corrects it. Thus adding as dependency.
Comment 13 User image Florian Bender 2013-10-18 14:48:44 PDT
(In reply to Florian Bender from comment #1)
> – attr(data-*) (that one's weird)
This is an actual bug in Gecko: Bug 435426.
Comment 14 User image Chris Peterson [:cpeterson] 2013-11-05 18:01:56 PST
Tom's Hardware Guide's "Web Browser Grand Prix" (tracking bug 933949) includes css3test.
Comment 15 User image Guilherme Lima 2013-12-08 11:56:52 PST
Latest Nightly - 630 tests out of 1039
Firefox Beta -  557 tests out of 1039
Chrome 31 - 588 tests out of 1039
IE 11 - 587 tests out of 1039
Comment 16 User image sjw 2013-12-09 02:11:07 PST
Chrome 33 - 597 tests out of 1039
Opera 19 -  613 tests out of 1039
Comment 17 User image Albert Scheiner [:alberts] 2014-02-27 06:48:46 PST
Latest Nightly -     667 tests out of 1338
Latest Aurora (29) - 667 tests out of 1338
Firefox Beta (28) -  589 tests out of 1338
Chrome 33 -          658 tests out of 1338
Safari 7 -           644 tests out of 1338 (minus some points for background-repeat)
Comment 18 User image Sören Hentzschel 2014-02-27 07:13:43 PST
Latest Nightly vs. Chrome 33 is not a fair comparison, you have to compare latest Nightly with Chrome Canary and latest stable of Chrome with latest stable of Firefox… ;)

Latest Firefox: 575 tests out of 1338
Latest Chrome: 658 tests out of 1338

Latest Nightly: 667 tests out of 1338
Latest Chrome Canary: 675 tests out of 1338

But good work @firefox devs, it's a huge jump from 589 to 667!
Comment 19 User image Albert Scheiner [:alberts] 2014-02-27 07:32:31 PST
Sure. I must confess I was a bit too lazy to download Canary atm. Thanks for the numbers Sören.
Comment 20 User image sjw 2014-02-27 07:44:57 PST
Opera Next 20 got 813.
Comment 21 User image Florian Bender 2014-02-27 07:54:53 PST
Note that pre-release versions are likely to have a higher score than the resulting release, e.g. most font tests will fail in Fx 29 Release because they are only enabled in Nightly and Aurora (by default).
Comment 22 User image entership12345 2014-07-14 22:50:34 PDT
For now:

33 Nightly: 686 out of 1341, 59%
38  Canary: 709 out of 1341, 60%
Comment 23 User image Gérard Talbot 2015-07-22 11:57:06 PDT
I am convinced you should be careful at not reaching any hard, solid conclusion regarding Lea Verou meta-test page. For many reasons.

- The page and tests need to be constantly updated: eg 'cursor: grab' and 'cursor: grabbing' are not being tested ( http://www.w3.org/TR/2015/CR-css-ui-3-20150707/#cursor ) and I have spotted other properties while editing this comment 
- The page tests properties from Working Drafts and not just from Candidate Recommendations only: eg Regions module
- The page claim Firefox supports properties that we know are not implemented at all: widows and orphans ( bug 137367 )
- The page or tests claim Firefox does not support 'text-combine-upright' but that is not true:
http://test.csswg.org/suites/css-writing-modes-3_dev/nightly-unstable/html/text-combine-upright-parsing-valid-001.htm
and bug 1097499 : no working implementation but there is support with Firefox 42 if/when layout.css.text-combine-upright.enabled is set to true
- There are many false positives and false negatives in those tests and that page: Chrome supports a lot more than what the page claims due to vendor-prefixes

The most important reason why this page should not be taken seriously (otherwise, at least, with a lot of salt) is that implementation is not being tested and Lea Verou is fully aware of this: "Caution: This test checks which CSS3 features the browser recognizes, not whether they are implemented correctly."
Comment 24 User image Sebastian Zartner [:sebo] 2015-07-22 14:31:00 PDT
(In reply to Gérard Talbot from comment #23)
> - The page and tests need to be constantly updated: eg 'cursor: grab' and
> 'cursor: grabbing' are not being tested (
> http://www.w3.org/TR/2015/CR-css-ui-3-20150707/#cursor )

Filed an issue to implement those values.[1]

> and I have spotted
> other properties while editing this comment 

Would be good if you filed issues against those on the css3test GitHub repo.

> - The page tests properties from Working Drafts and not just from Candidate
> Recommendations only: eg Regions module

As far as I know the implementation criteria on css3test.com does not follow the status of the specifications but requires at least one working implementation of a spec.

> - The page claim Firefox supports properties that we know are not
> implemented at all: widows and orphans ( bug 137367 )

As the page states at the top, it only tests whether a browser recognizes a feature, not whether its implemented correctly.
While 'widows' and 'orphans' are not implemented yet, Firefox already exposes them through the DOM.
Non-the-less this should be mentioned on the page, so I created an issue for that.[2]

> - The page or tests claim Firefox does not support 'text-combine-upright'
> but that is not true:
> http://test.csswg.org/suites/css-writing-modes-3_dev/nightly-unstable/html/
> text-combine-upright-parsing-valid-001.htm

'text-combine-upright' is not tested yet.[3] Instead it still tests for 'text-combine-horizontal'.
Also, Firefox *does not support* 'text-combine-upright' yet.[4]

> - There are many false positives and false negatives in those tests and that7
> page: Chrome supports a lot more than what the page claims due to
> vendor-prefixes

Not sure what's the rule for the page when features are behind prefixes, though you should ask Lea about that.

> The most important reason why this page should not be taken seriously
> (otherwise, at least, with a lot of salt) is that implementation is not
> being tested and Lea Verou is fully aware of this: "Caution: This test
> checks which CSS3 features the browser recognizes, not whether they are
> implemented correctly."

Same counts in big parts for html5test.com (which is tracked in bug 802882, btw.). Though web authors refer to those pages for comparing how good browsers implement the (upcoming) web standards. Bad scores may result in bad reputation. So it should be somewhat important to Mozilla to get a higher rank on those pages.
And for implementers they provide at least a rough overview where feature implementation stands and may help to decide what to implement next.

If you believe css3test.com does not offer good enough measurement in that relation, you should talk to Lea and/or file bugs against it.

Sebastian

[1] https://github.com/LeaVerou/css3test/issues/103
[2] https://github.com/LeaVerou/css3test/issues/104
[3] https://github.com/LeaVerou/css3test/issues/105
[4] bug 1097499

Note You need to log in before you can comment on or make changes to this bug.