282733 - Database-Upgrade Regression Tests for checksetup on landfill

Status: RESOLVED FIXED

(Bugzilla :: bugzilla.org, enhancement)

Description

[Max Kanat-Alexander]

It's very difficult to make a general-purpose regression test script for
checksetup, because it's difficult to generate old databases on the fly.
However, landfill.bugzilla.org already has all the databases and tools in place
to run such a test.

Basically, the script would just clone a bunch of databases, upgrade them, and
then watch for warnings or errors.

It would have to only run every few hours, because otherwise it would hose landfill.

Comment 1

[Max Kanat-Alexander]

We need to:

1. Check if we can create an empty database
2. Check if we can run cleanly against the tip database
3. Check if we can upgrade from 2.8
4. Check if we can upgrade from intermediate versions

We could do 1, 2, and 3 on every checkin, and only do 4 once a day, or once
every six hours, perhaps.

Comment 2

[Max Kanat-Alexander]

Attachment: v1

Here it is. I've got it working on landfill, in an installation called
checksetup.

Right now it only works there, but it's easy to make it work on other
installations.

Soon I will integrate it into tinderbox.

I'm just posting it here for reference. :-)

Comment 3

[Max Kanat-Alexander]

Next steps are to:

(1) Integrate with Tinderbox
(2) Make it do some actual tests on the created DBs, like making a bunch of Bug
objects and User objects, and calling their methods.

Comment 4

[Max Kanat-Alexander]

Attachment: v2

OK, here's version 2. I basically cleaned up version 1.

Now, it needs a tinderclient wrapper. I know that I need to send email to some
email address at regular intervals... what's the format of the email? Who knows
this? justdave? Are there some docs about this somewhere in the tinderbox code?


Basically, the script just needs to continually check for changes in
checksetup, and then run the basic test if there have been changes. It should
write out two logs: one that is stdout+stderr, and one that is just stderr.

If test-checksetup.pl returned non-0, we're burning (red). If the stderr log
contains anything, we're orange.

Comment 5

[Max Kanat-Alexander]

Attachment: Tinderbox script, not working

I have NO IDEA why this script isn't working. It dies with:

Test complete. Failed 0 time(s).
Can't use string ("--------------------------------") as a SCALAR ref while
"strict refs" in use at ./tinderbox.pl line 74.

So that's in these lines of tinderbox.pl:

Log($log,`perl -W ./test-checksetup.pl 2> $error_log 2>&1`);
checkerrors($log);
Log($log,"test-checksetup.pl complete\n");
Log($log,"----------------------------------------------------\n\n");
									       
			     
##########################
# Check For Any Warnings #
##########################
Log($log,"---------------------------------------------\n");


Here's the CATCH: if I comment-out the line that runs test-checksetup, it works
fine. Of course, that defeats the whole purpose of the script!

Comment 6

[Max Kanat-Alexander]

Attachment: Working tinderbox script

OK, this one works and is now running on landfill.

Yes, I will soon modify it so that it runs the test even if other files than
checksetup have changed.

Right now it:

(1) Creates an empty database
(2) Runs against the tip database
(3) Upgrades 2.10

It doesn't do 2.8 because I know that burns, at the moment.

Comment 7

[Max Kanat-Alexander]

Attachment: Tinderbox Client, v3

OK, this tinderbox client is pretty much the good one. It runs checksetup if
there have been any changes in the installation.

By the way, this script is a bit cleaner than Devel::Tinderclient in a few
places, most noticeably the cool thing I did with IO::Tee. However, it's not as
configurable or generally useful (yet) as Devel::Tinderclient.

Comment 8

[Max Kanat-Alexander]

OK, we also have the full test, now. I'll post the two new scripts for basic and
full here tomorrow, and then this is resolved.

Comment 9

[Max Kanat-Alexander]

Attachment: Tinderbox Client, v4

OK, *this* is the final, currently-working Tinderbox client.

It has so many cool bells and whistles that I can't even begin to explain.
Hahaha.

Anyhow, this code is WAY cleaner than Devel::Tinderclient, but not as
configurable. 

However, if somebody was going to re-write Devel::Tinderclient, this would be a
good place to start. I also changed Tinderconfig.pm to have "use strict" and
"use constant" for the configuration variables.

Right now it's sending me the tinderbox messages, just to make sure that
everything works before I flood the landfill tinderbox. :-)

Comment 10

[Max Kanat-Alexander]

Attachment: Final Test-Checksetup Script (v3)

OK, here's what I have now for the test-checksetup script as it's running on
landfill.

It also does testing on each created database, now. Any further enhancements to
it will go in a future bug. :-)

Comment 11

[Max Kanat-Alexander]

Attachment: Final Tinderbox Client (v5)

OK, this client is running properly and working on landfill, now.

Comment 12

[Max Kanat-Alexander]

Attachment: Tinderconfig.pm for the Client

Note that I removed a lot of variables from this, but I also left some in that
don't need to be there. No big deal.

Comment 13

[Vlad Dascalu]

Comment on attachment 175279 [details]
Final Test-Checksetup Script (v3)

Just a small strange-worded text:

    # This is 2.08 is that it sorts properly.

Comment 14

[Max Kanat-Alexander]

Oh, that should have been: "This is 2.08 so that it sorts properly." :-) I fixed
it on the running copy on the server.