391756 - import the sqlite test suite into our test suite.

Status: RESOLVED WONTFIX

(Toolkit :: Storage, defect)

Description

[(not reading, please use [e-mail] instead)]

import the sqlite test suite.  

note, the test suite is tcl based.

currently, we've ported part of like.test into test_like.js

Comment 1

[park,kiwon]

Attachment: SQLite Test Suite Mocha

Hi, my name is Park, KiWon. 

I'd like take this part, so I checked the SQLite's test suite, and Firefox's test suite. I think converting TCL to Javascript is not difficult, but it is not flexible. 

Thus, I think that making it as functions and recall the functions such as the attached snap shop. If it is okay, I am working on this part.

Comment 2

[Shawn Wilsher :sdwilsh]

How hard is this to update when SQLite changes things?

Is there a way to automate this?

Comment 3

[Peter Weilbacher]

I love these bugs that are just opened without reasoning and with a half baked comment and no reference. ;-) As it currently stands, I don't understand the point of this one. The SQLite guys already run the testsuite before doing a release and verifies new features this way (and keep regressions form occurring). As Mozilla only takes official releases, what do you hope to gain by essentially doubling the same tests? (Other than just slowing down the operation of the test machines)?

Manually converting the SQLite Tcl testsuite to something else will be a _huge_ effort. We started looking at that when we didn't have Tcl available for OS/2, and gave up after only converting two of the tcl files. (We didn't try JS, don't remember what, may have been for a C framework, but the efforts will be comparable.) During test conversion you will also add some errors in the tests, so even after porting, until you have it converted to a stable form, it will again take quite a while.

Also, are you aware that the SQLite testsuite consists of now over a million tests and that it may take more than an hour to execute? There is a subset listed in the quick.test file (activated through veryquick.test) that only runs something like 25000 individual tests, maybe that is what you should be looking at.

Comment 4

[park,kiwon]

I think Firefox also needs its own SQLite test suite even though the SQLite has own test suite. Because the SQLite test suite (TCL base) is just for the SQLite(?)  I can use the test suite, but it is not easy to read TCL code (Not only to me).

If Firefox has its own SQLite Test Suite (JS base), and people can share their own test functions, and  then it can fix or find SQLite bugs which are not the SQLite’s bugs, it is the bugs that related bugs between Firefox and SQLite.
Thus, I think the Test Suite may have or is focused on these functions: 
   - SQLite feature tests on Firefox
   - performance test on Firefox
   - SQLite new version import tests 
          – Shawn, I think it needs configuration parts that setting 
            or changing the SQLite versions
   -share the test plan to others: download JS files into Test folder

Need more? Please add your idea..

Comment 5

[Peter Weilbacher]

I thought the tests for mozstorage would do all that. But in essence you are arguing for not porting the SQLite testsuite but for developing more independent tests?

Comment 6

[Shawn Wilsher :sdwilsh]

Comment on attachment 342979 [details]
SQLite Test Suite Mocha

So, I can see a value coming from having the test suite in our tree, and runnable (but not part of make check).  This will allow for folks to ensure that sqlite runs as expected on their system with our configuration.

However, we also do not want to make upgrading SQLite more difficult.  When it was difficult to upgrade, we simply didn't upgrade, and we do not want to get into that state again.

With that said, we should set this up such that it's fairly easy to update the tests, with a configure flag that will enable the tests to be ran with some target (maybe just a new make target - not sure there)

Comment 7

[David Humphrey (:humph)]

Ted, can you give some ideas here on how to go about doing Shawn's plan?  I'd like to get KiWon going on this before the term runs out.

Comment 8

[(not currently active) Ted Mielczarek]

Ok, sounds like the tests are very much tied to the version of SQLite we're using. I think as a first pass, maybe we can just add a tests/ dir under db/sqlite3/src, and then in that makefile do:
ifdef RUN_SQLITE_TESTS
DIRS = tests
endif

Then in db/sqlite3/src/tests, you can put a Makefile.in, and in a check target, run the tests. So something like:
check::
    run-tests arg1 arg2 # of course, change this to whatever command you need

Then, to test this, you can do make -C OBJDIR/db/sqlite3/src/ check RUN_SQLITE_TESTS=1, and it should go into your tests directory and run that command.

I'd say get it running some tests first, and then we'll sort out the specifics later.

Comment 9

[Shawn Wilsher :sdwilsh]

I'm not sure we really want to go down this path honestly.  SQLite already tests this stuff on our tier one platforms since we are a SQLite consortium member.

As a result, I'm going to say WONTFIX.

Comment 10

[(not currently active) Ted Mielczarek]

Are they testing with our exact build options, etc? Seems like there's still room for failure here.

Comment 11

[Shawn Wilsher :sdwilsh]

(In reply to comment #10)
> Are they testing with our exact build options, etc? Seems like there's still
> room for failure here.
They are.  I have to report any changes we ever make to them so they can update.