Closed Bug 226823 (badtests) Opened 21 years ago Closed 16 years ago

Remove standards-compliance tests from mozilla.org/quality

Categories

(www.mozilla.org :: General, defect)

Product:
www.mozilla.org
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: fabian, Assigned: dbaron)

References

(
URL
)

Details

As suggested by Hixie, let's discuss whether the so-called standards-compliance
tests (mainly the DOM and js ones) are worth keeping on mozilla.org
Arguments to keep them:
1) They might be modified later to become good
Arguments to remove them:
1) Most of them are invalid
2) They waste bug reporter's time: going through all the tests takes a lot of time
3) They waste triager's time
4) They waste developer's time, including lots of bugmail
5) There is the NIST DOM Test suite that could be used as a replacement. This
would also boost the interest of the mozilla community in this test suite.

Other arguments?
The HTML, XHTML, and CSS tests are just as bad, IMHO.

The main argument for removing them for me is:
6) Companies (both those working on Mozilla and those working on other products) 
are employing people with limited knowledge of standards, and using these tests 
to determine the compliance of products they care about. This wastes their time, 
their money, and the time of developers and QA when they subsequently report 
invalid bugs because of these tests.
I can't remember the last time those tests were used to report an actual
problem/regression in Mozilla, but I see people filing invalid bugs based on
those... So I vote for removing them, unless we can find soneone to maintain
them, and that hasn't happened yet, so...
My vote is also for removal (though we should keep a tarball of them somewhere
(eg cvs) in case a maintainer comes along).
Blocks: 53570
Blocks: 204799
Blocks: 226774
Blocks: 237488
cc'ing brendan to keep him in the loop

Since we all seem agreed here, does someone who has a tree want to tarball the
files and cvs remove them?
Alias: badtests
Blocks: 193548
Blocks: 193551
ccing some devmo people.

So we want to remove everything under www.mozilla.org/quality/standards/ from
the website, right?  Should we make a public announcement to that effect at this
point?  As a bonus, that may (not likely, though) find us a maintainer...

There are some legitimate bugs filed on some of those pages (some having nothing
to do with what the test is purporting to test), and for those we probably want
to attach a copy of the page to the bug, right?
Blocks: 49986, 116720, 163973, 195673, 230442
I've got a tgz of /quality/browser/standards now. What do you want me to do with it?
If it's small enough to attach to this bug, I'd say just do that....
2.8MB. It's up at http://fantasai.inkedblade.net/temp/tests.tgz
Note: it includes the CVS files
If this is an orphaned project, then that should be noted on an orphaned
project page somewhere?
I don't believe we have such a page to note it on....

If we do find someone to maintain these, then it may be acceptable to just take
them down from the website temporarily (and not CVS remove them) until they have
been brought into some sort of shape.  My estimate is that this would take
hundreds, possibly thousands of man-hours of work to do.
> My estimate is that this would take hundreds, possibly thousands of man-hours 
> of work to do.

And the person doing that work would need to be proficient in the relevant 
specs. Someone that capable should not be wasting their time trying to fix those 
tests, the time would be much better spent simply writing new, quality tests.

I think we should cut the life support and just nuke these tests. We have about 
two dozen bugs that are open on them, I suggest we just attach the relevant 
files to those bugs then cvs remove the lot.
Never throw spare parts out, not ever. You'll just regret it
one day.

An alternate thought: create a brief doc giving an overview
of the status of the material, including its strengths and
weaknesses, and people's net experience with it, and then spray it
all with plastic sealant?

- N.
(In reply to comment #12)
> Never throw spare parts out, not ever. You'll just regret it
> one day.

that's what CVS is for.
> Never throw spare parts out, not ever. You'll just regret it one day.

As biesi pointed out, this would obviously still all be in CVS.


> An alternate thought: create a brief doc giving an overview of the status of 
> the material, including its strengths and weaknesses, and people's net 
> experience with it, and then spray it all with plastic sealant?

The tests are mostly lacking pass criteria, those that don't lack pass criteria
are mostly invalid, and those that aren't invalid are all tests that are so
simple that simply visiting the bug database itself is enough to check that
those tests would pass. Net experience with those tests is that they waste a lot
of people's time.
Removing docs entirely from the web (cvs remove) lowers Web access,
which is bad for the public. Thus I prefer a web-archiving approach.
But these files aren't annoying me personally, so do as you want.

If you do cvs remove them without a legacy web link, no-one will ever
find them again, even with Attics and so on in place.

- N.
> no-one will ever find them again

That's rather the goal, actually, unless they get fixed and maintained.  I cced
people in case they felt there was a pressing need to maintain these tests, in
which case we need to find someone to do it right now or so.

Ian, could you do the honors and start attaching files to relevant bugs?
No longer depends on: 163971
No longer depends on: 31889
No longer depends on: 226361
Blocks: 230442
No longer depends on: 230442
Blocks: 116720
No longer depends on: 116720
*** Bug 116720 has been marked as a duplicate of this bug. ***
Blocks: 163973
No longer depends on: 163973
Blocks: 226772
No longer depends on: 226772
No longer depends on: character-alignment
No longer depends on: 14088
No longer depends on: 49986
No longer depends on: 168725
No longer depends on: 122311
No longer depends on: 224213
No longer depends on: 196745
No longer depends on: 195673
> Ian, could you do the honors and start attaching files to relevant bugs?

Done. There are no open bugs that I know of that still refer to these tests.
OK.  So what exact directories are we CVS removing?  quality/browser/standards?
 What about quality/ngdriver?
I checked for bugs with URL fields for both, so nuking both is thus fine by me. 
A number of the bugs blocked by this one are expecting the ngdriver stuff to be 
dealt with as well.
OK.  So now we remove links to these files, and then nuke them.
Blocks: 256558
Anyone got a webtree and time to actually do this?
Blocks: 256810
Yah, I have a gila tree and some time on my hands. I can take care of this. -->me

Do we want to pull ngdriver? I will spin a tar up for it as needed.
Assignee: general → owen-bugzilla
Yes, I think we want to pull ngdriver.
docs/dom/grids/core_tab.html
docs/dom/grids/html_tab.html
These look to be fully based upon the testcases. I can fully pull them, or just
the links. This comes up a lot, honestly... if the page is built completely
around the tests, should I go ahead and pull those as well, as they won't make
much sense?

projects/ui/accessibility/unix/testcase/MAI/Image.html
CC'ing Aaron to evaluate the impact on these pages. From what I see the image
map links directly back to standards, so this probably could be pulled.

projects/minimo/testing/basic-test-plan.html
Same here, based fully on these tests, ccing chofman to evaluate.

Making progress... ;)
Status: NEW → ASSIGNED
Blocks: 265107
No longer blocks: 265107
A lot of the pulled test cases are still linked, please look at Bug 259288, Bug
260893, and Bug 263851.
Blocks: 259288, 263851
Blocks: 269409
*** Bug 269409 has been marked as a duplicate of this bug. ***
*** Bug 259288 has been marked as a duplicate of this bug. ***
Blocks: 278489
Why is this bug part of Core? Should it not be inside mozilla.org?

Anyway, what needs to be done? I have access to mozilla.org and I am willing to
fix it and fix related issues that might arise after this one is fixed.

From who does this need review? Has there been made a backup of all files and
stored somewhere on mozilla.org?
Blocks: 278980
Blocks: 119020
Blocks: 280736
*** Bug 281898 has been marked as a duplicate of this bug. ***
No longer blocks: 278980
*** Bug 280062 has been marked as a duplicate of this bug. ***
No longer blocks: 116720, 259288, 263851, 269409, 280736
I'll take this.
Assignee: irixman+bugzilla → dbaron
Status: ASSIGNED → NEW
Component: DOM → webmaster@mozilla.org
Product: Core → mozilla.org
Version: Trunk → other
quality/ngdriver appears to already be removed.  I didn't touch anything there.

I made a tarball of quality/browser/standards/ from before timeless removed a
few of the tests in javascript and redirected every file in that directory to a
page that links to the tarball.

Is that what was supposed to happen?  What else needs doing?  There are probably
some bad links from various places, but they are redirected, so it's not a top
priority...
No longer blocks: 193551
*** Bug 193551 has been marked as a duplicate of this bug. ***
unfortunately these tests migrated into other people's repositories and are now being used to file bugs anyway.

and because the originals are so hard to find it took me a while to figure out where the were. and with the originals gone there's no easy/convenient way for me to find out that a specific test is invalid (i think we just got about 2 dozen bugs based on this test suite).
The "other people" just need pointing out that their tests are bogus and to please stop filing bugs on them.
http://lists.kde.org/?l=kde-commits&m=111990521306752&w=2

altha: that directory mentions webcore, can you proxy the message to them?
I filed http://bugzilla.opendarwin.org/show_bug.cgi?id=9712 to let this be taken care of in WebKit.
Product: mozilla.org → Websites
Assignee: dbaron → nobody
Status: NEW → RESOLVED
Closed: 16 years ago
QA Contact: ian → www-mozilla-org
Resolution: --- → WORKSFORME
Fixed, per comment 33.
Assignee: nobody → dbaron
Resolution: WORKSFORME → FIXED
hgkjmhgfmvcnbgnhgfng
Component: www.mozilla.org → General
Product: Websites → www.mozilla.org
You need to log in before you can comment on or make changes to this bug.