Closed Bug 554013 (ietestcenter) Opened 14 years ago Closed 8 years ago

Tracking bug for Internet Explorer Testing Center failures

Categories

(Core Graveyard :: Tracking, defect)

Product:
Core Graveyard
Component:
Tracking
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: Gavin, Unassigned)

References

(Depends on 6 open bugs,
URL
)

Details

(Keywords: meta) 

      No description provided.
Depends on: 451134
Depends on: 554016
Note that the IE test center is only testing the new tests that MSFT worked on with W3C. We should try to pass them all, but like link:visited, some we might break intentionally! :)

ref: http://blogs.msdn.com/ie/archive/2010/03/19/the-internet-explorer-testcenter-welcome-page-clarification-corrections.aspx
Depends on: 427537
Keywords: meta
Depends on: 553805
Bug 576036 should probably depend on this bug.
Depends on: 589552
Depends on: 589555
Depends on: 589558
Depends on: 589561
Depends on: 589562
Depends on: 589563
Depends on: 589564
Depends on: 589565
Depends on: 589566
Depends on: 589567
Also depends on Bug 449157.  Why can't I edit that?
Depends on: 449157
Depends on: 498253
Also depends on Bug 495040.
Depends on: 589615
Depends on: 589617
Depends on: 589619
Depends on: 589621
Depends on: 589622
Depends on: 589623
Depends on: 589624
Depends on: 589625
Bugs created and number currently passing (listed number passing on MS's site
in parentheses) [IE9's score in square brackets]:

HTML5
Canvas:         12/15 (10) [12]
Media Elements: 16/23 (16) [20]
XHTML:           6/9   (4)  [9]
Selection:      12/17 (12) [17]
Depends on: 589638
Depends on: 589640
HTML5
Foreign Content: 21/24 (3) [24]
Bug 589563 was closed Invalid, so this should be:

Media Elements: 16/22 (16) [20]
HTML5 
getElementsByClassName 11/11 (11) [11]
Depends on: 589645
Depends on: 589646
Depends on: 589647
Depends on: 589648
Depends on: 589649
Depends on: 589650
Depends on: 589651
SVG: 49/56 (42) [56]
Percentages per web standard (reported by MS in parens) [IE9 in square brackets]:

HTML5: 80% (57%)  [94%]
SVG:   88% (75%) [100%]
Depends on: 589653
Depends on: 589654
Depends on: 589655
Depends on: 589657
Depends on: 589658
Depends on: 589660
Depends on: 589661
Depends on: 589662
CSS3
Fonts: 6/13 (5) [12]
Depends on: 589672
CSS3
Namespaces: 16/17 (16) [17]
Depends on: 589674
Depends on: 589676
Depends on: 589677
Depends on: 589678
Depends on: 589679
Depends on: 589680
Depends on: 589683
Depends on: 589684
Depends on: 589685
Depends on: 589687
Depends on: 589688
Depends on: 589689
Depends on: 589690
Depends on: 589693
Depends on: 589694
Depends on: 589695
Depends on: 589697
Depends on: 589698
Depends on: 589699
Depends on: 589700
Depends on: 589701
Depends on: 589702
Depends on: 589703
Depends on: 589704
Depends on: 589705
Depends on: 589706
Depends on: 589707
Depends on: 589708
Depends on: 589709
Depends on: 589710
Depends on: 589711
Darxus: please stop filing bugs on each individual failure. More diagnosis is needed - many of the failures stem from the same root cause.
No longer depends on: 589711
Depends on: 589712
Depends on: 548372
Gavin:  I suspected, but I don't see this getting done any other way.  Are you going to go through and create the individual bugs for each test that corresponds to exactly one flaw in mozilla?  It hasn't happened in the 5 months this tracking bug has existed.

It's driving me nuts, but I think we're better off creating a bug for each of them, and closing them as they're fixed, even if they're all one problem.


CSS3
Backgrounds & Borders: 18/52 (15) [50]
Also depends on Bug 451134.
Depends on: 483446
Bugs for the CSS3 Backgrounds & Borders tests are *not* opened, and need to be opened after bug 451134 is closed.
Also depends on:  bug 3512, bug 475891.
Depends on: 280391
Depends on: 501421
Depends on: 508561
Also depends on bug 376027.
Depends on: 590039
Filing one bug per failing testcase is not useful.  We should have one bug per underlying problem.  Maintaining data on each failing testcase would be better done using a google docs spreadsheet or a wiki.
I'd also point out that a few of the dependent bugs are invalid or probably invalid: bug 589654, bug 589617, bug 589619.
It looks to me like there's no way mozilla is ever going to pass the ietestcenter, and MS is going to remain the holder of the most standards compliant web browser.

Because the bugs I spent half my weekend opening keep getting closed in groups marked duplicates of a bug, with no remaining trace of the test cases.  And then those bugs get closed as invalid.

I've noticed a bunch of MS's tests are apparently invalid.  At least one other person is of the opinion that these bugs need to remain open until we get MS to fix those tests.  I have gotten a response from a real person at Microsoft.

And now I'm told I should be maintaining my own spreadsheet and figuring out which test cases to group how instead of filing a bug for each test case.

While nobody else has been willing to touch this in the five months since this bug was created.

It's not going to happen.

And I'm probably less than half way through the test cases.


I recommend leaving all the bugs open, marked as depending on the bugs they're currently marked as duplicates of, until the test cases are passed, or the tests are fixed.

If that sounds good, or you can come up with a better idea than has already been suggested, let me know.

It's been great seeing the patches coming through, but right now I feel like my time was mostly wasted.
"passing IE's testing center" isn't really a goal in and of itself, since it isn't a great measure of our feature set's utility to web authors/users. This is especially true considering the fact that some of its tests are bogus.

We have existing bugs on file that track the fixes required to pass entire swaths of tests, so tracking the state of individual tests in bugs isn't useful. Lack of activity in this bug isn't the bottleneck for getting the bugs fixed - we're not suffering from lack of "tracking" at this point. The bottleneck is generally code-level implementation work. Some of the implementation work required has just not been a priority, given our judgment of its relative importance compared to other things being worked on.
(In reply to comment #22)
> It looks to me like there's no way mozilla is ever going to pass the
> ietestcenter, and MS is going to remain the holder of the most standards
> compliant web browser.

That's a pretty ridiculous claim.

Let's be clear what's going on here -- Microsoft is publishing the tests that Microsoft passes, so they get 100%, which only goes down when people point out mistakes in tests that they've already published.

One can sample the features of Web standards in lots of different ways, and looking at tests that measure one particular sample says rather little about the whole, especially when the sample is as small as this one.
Depends on: 599630
Depends on: 607854
No longer depends on: 589685
Depends on: 405300
Depends on: 672043
Depends on: 574496
http://samples.msdn.microsoft.com/ietestcenter update:

----------------------+---------+--------------------+----------------+--------------+-----------------------+----------+
Web Standards 	      | Tests 	| IE 10 | FF 16.0.1  |	Opera 12.02   |	Safari 5.1.7 |	Chrome 22.0.1229.94  |	  IE 9  |
----------------------+---------+-------+------------+----------------+--------------+-----------------------+----------+
HTML5 	              | 326 	| 100% 	|   75%      |	   72% 	      |   74% 	     |       90% 	     |    36%   |
----------------------+---------+-------+------------+----------------+--------------+-----------------------+----------+
SVG 1.1 2nd edition   |	74 	| 100% 	|   93%      |	   93% 	      |   88% 	     |       97% 	     |    95%   |
----------------------+---------+--------------------+----------------+--------------+-----------------------+----------+
CSS                   |	236 	| 100% 	|   69%      |	   59% 	      |   63% 	     |       68% 	     |    62%   |
----------------------+---------+--------------------+----------------+--------------+-----------------------+----------+
DOM 	              | 127 	| 100% 	|   89%      |     85% 	      |   90% 	     |       94% 	     |   100%   |
----------------------+---------+--------------------+----------------+--------------+-----------------------+----------+
Web Performance       |	37 	| 100% 	|   66%      |      0% 	      |    0% 	     |       68% 	     |    30%   |
----------------------+---------+--------------------+----------------+--------------+-----------------------+----------+
Web Applications      |	471 	| 100% 	|   83%      |	   48% 	      |   33% 	     |       60% 	     |    21%   |
----------------------+---------+--------------------+----------------+--------------+-----------------------+----------+
JavaScript 	      | 6,318 	| 100% 	|   97%      |    100% 	      |   91% 	     |      100% 	     |    91%   |
----------------------+---------+--------------------+----------------+--------------+-----------------------+----------+
Marking all tracking bugs which haven't been updated since 2014 as INCOMPLETE.
If this bug is still relevant, please reopen it and move it into a bugzilla component related to the work
being tracked. The Core: Tracking component will no longer be used.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → INCOMPLETE
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.