226823 - (badtests) Remove standards-compliance tests from mozilla.org/quality

Status: RESOLVED FIXED

(www.mozilla.org :: General, defect)

Description

[Fabian Guisset]

As suggested by Hixie, let's discuss whether the so-called standards-compliance
tests (mainly the DOM and js ones) are worth keeping on mozilla.org
Arguments to keep them:
1) They might be modified later to become good
Arguments to remove them:
1) Most of them are invalid
2) They waste bug reporter's time: going through all the tests takes a lot of time
3) They waste triager's time
4) They waste developer's time, including lots of bugmail
5) There is the NIST DOM Test suite that could be used as a replacement. This
would also boost the interest of the mozilla community in this test suite.

Other arguments?

Comment 1

[Hixie (not reading bugmail)]

The HTML, XHTML, and CSS tests are just as bad, IMHO.

The main argument for removing them for me is:
6) Companies (both those working on Mozilla and those working on other products) 
are employing people with limited knowledge of standards, and using these tests 
to determine the compliance of products they care about. This wastes their time, 
their money, and the time of developers and QA when they subsequently report 
invalid bugs because of these tests.

Comment 2

[Johnny Stenback (:jst)]

I can't remember the last time those tests were used to report an actual
problem/regression in Mozilla, but I see people filing invalid bugs based on
those... So I vote for removing them, unless we can find soneone to maintain
them, and that hasn't happened yet, so...

Comment 3

[Boris Zbarsky [:bzbarsky]]

My vote is also for removal (though we should keep a tarball of them somewhere
(eg cvs) in case a maintainer comes along).

Comment 4

[Hixie (not reading bugmail)]

cc'ing brendan to keep him in the loop

Since we all seem agreed here, does someone who has a tree want to tarball the
files and cvs remove them?

Comment 5

[Boris Zbarsky [:bzbarsky]]

ccing some devmo people.

So we want to remove everything under www.mozilla.org/quality/standards/ from
the website, right?  Should we make a public announcement to that effect at this
point?  As a bonus, that may (not likely, though) find us a maintainer...

There are some legitimate bugs filed on some of those pages (some having nothing
to do with what the test is purporting to test), and for those we probably want
to attach a copy of the page to the bug, right?

Comment 6

[fantasai]

I've got a tgz of /quality/browser/standards now. What do you want me to do with it?

Comment 7

[Boris Zbarsky [:bzbarsky]]

If it's small enough to attach to this bug, I'd say just do that....

Comment 8

[fantasai]

2.8MB. It's up at http://fantasai.inkedblade.net/temp/tests.tgz
Note: it includes the CVS files

Comment 9

[Nigel McFarlane]

If this is an orphaned project, then that should be noted on an orphaned
project page somewhere?

Comment 10

[Boris Zbarsky [:bzbarsky]]

I don't believe we have such a page to note it on....

If we do find someone to maintain these, then it may be acceptable to just take
them down from the website temporarily (and not CVS remove them) until they have
been brought into some sort of shape.  My estimate is that this would take
hundreds, possibly thousands of man-hours of work to do.

Comment 11

[Hixie (not reading bugmail)]

> My estimate is that this would take hundreds, possibly thousands of man-hours 
> of work to do.

And the person doing that work would need to be proficient in the relevant 
specs. Someone that capable should not be wasting their time trying to fix those 
tests, the time would be much better spent simply writing new, quality tests.

I think we should cut the life support and just nuke these tests. We have about 
two dozen bugs that are open on them, I suggest we just attach the relevant 
files to those bugs then cvs remove the lot.

Comment 12

[Nigel McFarlane]

Never throw spare parts out, not ever. You'll just regret it
one day.

An alternate thought: create a brief doc giving an overview
of the status of the material, including its strengths and
weaknesses, and people's net experience with it, and then spray it
all with plastic sealant?

- N.

Comment 13

[Christian :Biesinger (don't email me, ping me on IRC)]

(In reply to comment #12)
> Never throw spare parts out, not ever. You'll just regret it
> one day.

that's what CVS is for.

Comment 14

[Hixie (not reading bugmail)]

> Never throw spare parts out, not ever. You'll just regret it one day.

As biesi pointed out, this would obviously still all be in CVS.


> An alternate thought: create a brief doc giving an overview of the status of 
> the material, including its strengths and weaknesses, and people's net 
> experience with it, and then spray it all with plastic sealant?

The tests are mostly lacking pass criteria, those that don't lack pass criteria
are mostly invalid, and those that aren't invalid are all tests that are so
simple that simply visiting the bug database itself is enough to check that
those tests would pass. Net experience with those tests is that they waste a lot
of people's time.

Comment 15

[Nigel McFarlane]

Removing docs entirely from the web (cvs remove) lowers Web access,
which is bad for the public. Thus I prefer a web-archiving approach.
But these files aren't annoying me personally, so do as you want.

If you do cvs remove them without a legacy web link, no-one will ever
find them again, even with Attics and so on in place.

- N.

Comment 16

[Boris Zbarsky [:bzbarsky]]

> no-one will ever find them again

That's rather the goal, actually, unless they get fixed and maintained.  I cced
people in case they felt there was a pressing need to maintain these tests, in
which case we need to find someone to do it right now or so.

Ian, could you do the honors and start attaching files to relevant bugs?

Comment 17

[Hixie (not reading bugmail)]

*** Bug 116720 has been marked as a duplicate of this bug. ***

Comment 18

[Hixie (not reading bugmail)]

> Ian, could you do the honors and start attaching files to relevant bugs?

Done. There are no open bugs that I know of that still refer to these tests.

Comment 19

[Boris Zbarsky [:bzbarsky]]

OK.  So what exact directories are we CVS removing?  quality/browser/standards?
 What about quality/ngdriver?

Comment 20

[Hixie (not reading bugmail)]

I checked for bugs with URL fields for both, so nuking both is thus fine by me. 
A number of the bugs blocked by this one are expecting the ngdriver stuff to be 
dealt with as well.

Comment 21

[Boris Zbarsky [:bzbarsky]]

OK.  So now we remove links to these files, and then nuke them.

Comment 22

[Boris Zbarsky [:bzbarsky]]

Anyone got a webtree and time to actually do this?

Comment 23

[Owen Marshall (Not reading bugmail)]

Yah, I have a gila tree and some time on my hands. I can take care of this. -->me

Do we want to pull ngdriver? I will spin a tar up for it as needed.

Comment 24

[Boris Zbarsky [:bzbarsky]]

Yes, I think we want to pull ngdriver.

Comment 25

[Owen Marshall (Not reading bugmail)]

docs/dom/grids/core_tab.html
docs/dom/grids/html_tab.html
These look to be fully based upon the testcases. I can fully pull them, or just
the links. This comes up a lot, honestly... if the page is built completely
around the tests, should I go ahead and pull those as well, as they won't make
much sense?

projects/ui/accessibility/unix/testcase/MAI/Image.html
CC'ing Aaron to evaluate the impact on these pages. From what I see the image
map links directly back to standards, so this probably could be pulled.

projects/minimo/testing/basic-test-plan.html
Same here, based fully on these tests, ccing chofman to evaluate.

Making progress... ;)

Comment 26

[OstGote!]

A lot of the pulled test cases are still linked, please look at Bug 259288, Bug
260893, and Bug 263851.

Comment 27

[Simon Paquet [:sipaq]]

*** Bug 269409 has been marked as a duplicate of this bug. ***

Comment 28

[Simon Paquet [:sipaq]]

*** Bug 259288 has been marked as a duplicate of this bug. ***

Comment 29

[Anne (:annevk)]

Why is this bug part of Core? Should it not be inside mozilla.org?

Anyway, what needs to be done? I have access to mozilla.org and I am willing to
fix it and fix related issues that might arise after this one is fixed.

From who does this need review? Has there been made a backup of all files and
stored somewhere on mozilla.org?

Comment 30

[OstGote!]

*** Bug 281898 has been marked as a duplicate of this bug. ***

Comment 31

[Anne (:annevk)]

*** Bug 280062 has been marked as a duplicate of this bug. ***

Comment 32

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I'll take this.

Comment 33

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

quality/ngdriver appears to already be removed.  I didn't touch anything there.

I made a tarball of quality/browser/standards/ from before timeless removed a
few of the tests in javascript and redirected every file in that directory to a
page that links to the tarball.

Is that what was supposed to happen?  What else needs doing?  There are probably
some bad links from various places, but they are redirected, so it's not a top
priority...

Comment 34

[Gérard Talbot]

*** Bug 193551 has been marked as a duplicate of this bug. ***

Comment 35

[timeless]

unfortunately these tests migrated into other people's repositories and are now being used to file bugs anyway.

and because the originals are so hard to find it took me a while to figure out where the were. and with the originals gone there's no easy/convenient way for me to find out that a specific test is invalid (i think we just got about 2 dozen bugs based on this test suite).

Comment 36

[Boris Zbarsky [:bzbarsky]]

The "other people" just need pointing out that their tests are bogus and to please stop filing bugs on them.

Comment 37

[timeless]

http://lists.kde.org/?l=kde-commits&m=111990521306752&w=2

altha: that directory mentions webcore, can you proxy the message to them?

Comment 38

[Joost de Valk]

I filed http://bugzilla.opendarwin.org/show_bug.cgi?id=9712 to let this be taken care of in WebKit.

Comment 39

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Fixed, per comment 33.

Comment 40

[vineet]

hgkjmhgfmvcnbgnhgfng