248175 - (bz-dbinstall) Installation needs to support DB-independence

Status: RESOLVED FIXED

(Bugzilla :: Installation & Upgrading, enhancement, P1)

Description

[Joel Peshkin]

Today, checksetup makes mysql assumptions all over the place.  For 2.18 and
prior, that is a safe assumption.  BEFORE anyone lands a DB-compatibility
solution, we need to fix this.

My suggestion is as follows....

First, move the code that upgrades sites from checksetup to legacychecksetup,
deleting the module checks and the administrator setup in the process.

The new checksetup should perform operations in the following order...
1) Check requirements
2) Determine current Table configuration (schema only)
  [If current predates new checksetup, call the old checksetup code]
3) Add new fields/tables to schema (or ask a DBA to do so)
4) Determine migrations to be done [cannot rely on information from step 2, we
may be run again after a DBA finishes the schema adds]
5) Do the migrations
6) Determine the schema deletions or renames to be done

I am suggesting we make a point of preserving the old upgrade code prior to the
first version that does this.  The whole approach still does raise some
interesting questions. Such as...

a) If extra fields or tables are in the DB and not in checksetup, do we
automatically remove them or do we keep a list of fields that checksetup has
ever wanted to remove.
b) Can we always determine what conversion code needs to run solely by looking
at the schema?

For example, we add a new field that never existed...
-->We should be able to just add it to the db-independent table description.

add a new field that needs program initialization
-->We add it to the db-independent table description.
-->init code checks for the field existing AND being uninitialized

change a field to a reference to another table
-->We add the new stuff to the db-independent table descriptions.
-->init code checks for the new field existing AND old field existing AND being
uninitialized
-->cleanup code checks for the new field existing AND old field existing AND
initilization already done --> deletes old field

etc...

Comment 1

[Dave Miller [:justdave]]

There's nothing wrong with automatically performing schema updates and data
conversions at the same time...  however, it all needs to happen from a separate
script that can be run by the DBA.

In the typical enterprise world, the application user ('bugs') is not going to
be granted enough permissions to create, alter, and drop tables.  The primary
checksetup.pl script, run by the application admin, should be able to tell if
changes are necessary, but not actually make them (of course, it can offer to
run the upgrade script, so it can all be done at once if needed).  The script
which actually makes the changes should prompt for a DBA username and password,
rather than using one from a data file.

The checking can always be done from the same script, and called in such a way
that it knows to log in with the normal bugs user and only check instead of
update, so we avoid duplication of code.

Ideally, to prevent stranding people using a specific database, there should NOT
be a separate upgrade/setup script for each database, but instead try to do it
in a database agnostic way as much as possible, and use some kind of common
schema definition that defines any extra attributes needed for tables/columns to
deal with the places where being agnostic isn't possible. (which is what bug
146679 is about I think)

Comment 2

[Ed Sabol]

(In reply to comment #1)
> Ideally, to prevent stranding people using a specific database, there should NOT
> be a separate upgrade/setup script for each database, but instead try to do it
> in a database agnostic way as much as possible, and use some kind of common
> schema definition that defines any extra attributes needed for tables/columns to
> deal with the places where being agnostic isn't possible. (which is what bug
> 146679 is about I think)

Certainly. I've always hoped that a future version of checksetup.pl would utilize the Schema modules 
that I and others have been working on in bug 146679, but I'm hoping someone else will take care of 
adapting checksetup.pl to utilize the Schema modules. Right now, the Schema object class only has a 
single public method (besides the constructor) that returns all of the SQL used to create the Bugzilla 
schema. Can anyone think of any other methods that checksetup.pl might need from the Schema class?

Comment 3

[Joel Peshkin]

Here's the point...

The existing (legacy) checksetup will have to get mysql legacy (the only kind
that exists) up to some key point.  Starting with that point, whatever does this
must carry all the schema forward in an agnostic way.

I think we are all saying the same thing. However, it is easy to forget that
this needs more than just a way forward alone.  The checksetup replacement must
know when it has to call the legacy checksetup for upgrading sites.

Comment 4

[Dave Miller [:justdave]]

Bugzilla 2.20 feature set is now frozen as of 15 Sept 2004.  Anything flagged
enhancement that hasn't already landed is being pushed out.  If this bug is
otherwise ready to land, we'll handle it on a case-by-case basis, please set the
blocking2.20 flag to '?' if you think it qualifies.

Comment 5

[Max Kanat-Alexander]

I'll take this, because it's one of two major blockers for bug 98304, which is
important to me.

I think what's really important is that checksetup needs to support
DB-independence. It doesn't have to be *replaced* in order to do this, it just
has to do it. We have years and years of stable code in checksetup, there's no
need to throw it all away just because it's large and complicated.

If we'd like to re-write checksetup for some other reason, I think that could be
another bug.

If anybody disagrees, and would actually like to write this code, they can feel
free to take this bug back from me. :-)

Comment 6

[Max Kanat-Alexander]

I'm going to start working on this on what will essentially be a large fork
(with the current patches in bug 146679 and bug 277617) and then merge some of
my changes back in.

Comment 7

[Max Kanat-Alexander]

This and all its deps is currently the major 2.20 blocker.

Comment 8

[Myk Melez [:myk] [@mykmelez]]

"If it's not a regression from 2.18 and it's not a critical problem with
something that's already landed, let's push it off." - Dave

Comment 9

[Max Kanat-Alexander]

Without this bug landing, no PostgreSQL installation will be able to upgrade to
2.20. So I'd say it blocks 2.20.

Comment 10

[Max Kanat-Alexander]

OK, this bug is now considered complete now that all blockers have been
resolved. From now on, the ability to install on a new DB should depend mostly
on just writing a new driver and the associated Schema module.