Open
Bug 300434
Opened 20 years ago
Updated 16 years ago
A Tool to seamlessly move a Product to another Bugzilla installation
Categories
(Bugzilla :: Bugzilla-General, enhancement)
Tracking
()
NEW
People
(Reporter: mkanat, Unassigned)
References
(Depends on 1 open bug)
Details
What we want is to be able to basically take a product entirely out of a
Bugzilla installation, and move it to some other Bugzilla installation. (Our
motivation for this is to be able to move the "Bugzilla" product to its own
server, bugzilla.bugzilla.org.)
The requirements would be:
1) Must move all bugs and attachments from the product.
2) Must create a new series of bug numbers (starting from "1" or whatever the
lowest number is in the new DB) without holes.
3) All dependencies must remain consistent.
4) Needs to also fix bug numbers/links that are in comments.
Also, ideally it would be able to make the original installation auto-redirect
to the new installation for the moved bugs.
Reporter | ||
Comment 1•20 years ago
|
||
Ohhh... another problem is Users. All DB users have to be flawlessly moved,
along with any groups that they belong to, and any associations that that group
has with this Product.
The script doesn't initially have to deal with the idea that there could already
be bugs in the target installation. It could just create a new DB to be used by
the new system.
Comment 2•20 years ago
|
||
This is basically what prompted me to re-hack importxml.pl in the first place.
Here is the system I used to move ximian products (we do it on a product by
product basis although it theoretically could be component etc.) to novell bugzilla:
1. Query for all the bugs in a particular product to be moved.
2. Capture the buglist in the resulting page
3. I have a little java program that a coworker slapped together to take a bug
and grab it from showbug.cgi?ctype=xml. I use a little perl script to loop
through the list of bugs, grabbing the xml of each one and spitting them
individually to a file.
4. Run an XSLT transform to clean up the xml and transform it to the current
version's DTD (correct the product, version, milestone data etc. for the new
installation)
5. Run a script which calls my modified importxml.pl and passes it the new bug
xml data for each bug in the list
This creates new bug numbers for each bug as it is just basically creating a new
bug. I use the original bug number as the alias but prepend some text string to
make it unique.
The modified importxml.pl also verifies group permissions and adds them
appropriately as well as deals with some of the issues in the dependent bugs to
bug 134294.
In our case we have done a wholesale import of the profiles data assuming that
the other systems were being shutdown and all were moving to the new bugzilla.
In our case we have made this all run from the command line as moving products
is usually considered a one time thing. I am not sure how necessary it is to put
this functionality into the bugzilla interface for this reason but what are your
thoughts?
I would be happy to condense this into a single tool that can be used for this
purpose. We also have some scripts that we ran on the data in MySQL to make the
bug #'s (and attachment #'s) in comments match their new ones etc.
I have also modified ctype=xml to produce attachment data on command (encoded as
base64) so it can be used with a system such as this.
Reporter | ||
Comment 3•20 years ago
|
||
Yeah, a command-line tool would be perfect. I wouldn't imagine that this would
be a frequent operation, either.
Comment 4•20 years ago
|
||
I think we can make this easier by limiting what we are trying to achieve.
For example, instead of saying "we need to seamlessly migrate users", let's say
"the users need to exist on the target system". Then, we can just clone the
b.m.o. users table. Everyone even keeps the same password.
Similarly, I think it's reasonable to require that the products, components,
milestones, flags etc. all exist in the target Bugzilla before the migration.
That's not hard to achieve as a prerequisite - why write the code to do it
automatically? It's not an efficient use of time.
So what we actually have to do here is XML export the bugs (with all their data)
as a single file, and then XML import them, doing the following:
- Fix up the bug numbers of any dependencies/dupes which are part of the same
migration
- Drop any which aren't, adding a comment which says "dropped dependencies" and
giving full URLs to the original Bugzilla
- Fix up any "bug XXXX" comments for bugs which are in the migration
- Change the rest to full URLs to the original Bugzilla
How does that sound? Easier? :-)
Gerv
Reporter | ||
Comment 5•20 years ago
|
||
(In reply to comment #4)
> How does that sound? Easier? :-)
Sounds more difficult for the admin, actually. And possibly more difficult to
code, because you're working with an existing database instead of creating one
fresh. Seems to me like creating a fresh database would be easier, and just
copying things between them if possible.
Of course, that's also a bit tricky with our current Bugzilla::DB structure,
but not entirely impossible.
Comment 6•20 years ago
|
||
Oh, I see - you are limiting this feature to moving the product into a clean
Bugzilla? I didn't realise that. Doesn't that mean you could only ever move a
single product to a target Bugzilla, because then it wouldn't be clean any more?
How would adding bugs to an existing database be harder than adding them to a
fresh, newly-checksetuped one?
I agree my proposal leads to more work for the admin, but we need to think about
how many times this would actually be used. If the answer is a small handful,
coding time would probably be greater than cumulative admin time. So it seems
reasonable to put some of the work on the admin who wants the move, rather than
the coders.
Gerv
Comment 7•20 years ago
|
||
(In reply to comment #4)
> "the users need to exist on the target system". Then, we can just clone the
> b.m.o. users table. Everyone even keeps the same password.
>
Agreed, that was the approach we took.
However, we also have a tool that periodically goes through and removes stale
accounts (those belonging to people that have never contributed to any bug as
well as terminated employees which get disabled)
> Similarly, I think it's reasonable to require that the products, components,
> milestones, flags etc. all exist in the target Bugzilla before the migration.
> That's not hard to achieve as a prerequisite - why write the code to do it
> automatically? It's not an efficient use of time.
Part of the move was to rename some or all of the components, milestones, etc.
We also used the transforms to merge several components or products into one.
If you are happy with the current names and structure than I totally agree.
Additionally however, we were moving accross grossly different version of
bugzilla in which some fields were absent entirely from the source data and
needed to be filled in for the destination.
>
> So what we actually have to do here is XML export the bugs (with all their data)
> as a single file, and then XML import them, doing the following:
We specifically broke it into one file per bug because we found a major
performace flaw in the libxml2 libraries upon which XML::Parser is dependent.
It basically creates a Big(O) of N^2 issue in which the larger the file (and any
attachment > 1MB runs into this) the longer the processing time but it grows
exponentially. We would run out of memory pretty quickly when exporting even a
relativly small number of bugs with their attachment data as a single file.
>
> - Fix up the bug numbers of any dependencies/dupes which are part of the same
> migration
> - Drop any which aren't, adding a comment which says "dropped dependencies" and
> giving full URLs to the original Bugzilla
This would certainly be handy. importxml.pl currently does not address
dependencies at all. Though it makes an attempt at dupes.
> - Fix up any "bug XXXX" comments for bugs which are in the migration
This is as easy as prepending the text delimiter to the bug number to make it an
alias. Since show_bug.cgi will take an ID or an alias it is a simple matter of
munging it in the comment.
The other approach which we took was one in which we had an offset that we added
to all the bug #'s and then went through and incremented all references to the
old bug by it's offset.
This only worked since we were migrating all bugs from all products in an
installation.
> - Change the rest to full URLs to the original Bugzilla
>
> How does that sound? Easier? :-)
>
> Gerv
>
(In reply to comment #6)
> Oh, I see - you are limiting this feature to moving the product into a clean
> Bugzilla? I didn't realise that. Doesn't that mean you could only ever move a
> single product to a target Bugzilla, because then it wouldn't be clean any more?
>
I agree that it is impractical to always assume you are moving into a clean
bugzilla.
> How would adding bugs to an existing database be harder than adding them to a
> fresh, newly-checksetuped one?
>
> I agree my proposal leads to more work for the admin, but we need to think about
> how many times this would actually be used. If the answer is a small handful,
> coding time would probably be greater than cumulative admin time. So it seems
> reasonable to put some of the work on the admin who wants the move, rather than
> the coders.
>
> Gerv
Precisly my point as well. Every migration will be unique to the installation
and making the tool generic enough should be a worthy goal. Regardless, the
admin is going to have to take some responsibility and effort to get everything
prepped. Moving bugs should be a one time affair, especially if it is multiple
products. I don't think we can be expected to provide tools for every situation.
In our case we have now migrated a whole database (accross different versions of
bugzilla), product by product, and bug records from a commercial system. By
providing a generic set of tools and a best practices outline I think we can
make it possible for most admins to move what needs to be moved.
If we are only talking about moving the bugzilla product to a new b.b.o database
than that is a specific instance and let's focus on it. If we are talking about
a generic process for moving products that we can make available to everyone
then I think the outiline I have provided is a good place to start.
Comment 8•20 years ago
|
||
There are a couple of issues:
1. Can we assume the tool has simultaneous access to both databases?
If so, we don't have to muck with XML formats, and conditionally migrating
related data is easier. We can just migrate the bugs one by one, migrating
related data like components and user accounts as needed.
2. Can we assume we're migrating to a new installation?
If so, we can migrate the user, product, etc. tables wholesale and then delete
data afterwards as necessary, which is easier than having to select the data
from the old database and insert it into the new one.
The most flexible tool would be one that doesn't make either assumption. But
both assumptions are true when migrating b.m.o's Bugzilla product to a new
installation (say b.b.o), so it may not make sense to premise the Bugzilla
product move (BPM) on the development of a flexible generic tool.
For example, to do the BPM, we could do the following (where bmo and bbo are the
b.m.o and b.b.o databases, respectively):
1. set up a new Bugzilla installation with database;
2. copy b.m.o's user, product, component, etc. tables to it via filesystem copies;
3. create an "old bug ID" column in the bugs table to store the original bug ID
while we migrate data over to it: ALTER TABLE bbo.bugs ADD COLUMN old_bug_id INT;
4. copy bugs from bmo to bbo: INSERT INTO bbo.bugs (old_bug_id, <all cols except
bug_id>) SELECT bug_id, <all cols except bug_id> FROM bmo.bugs WHERE product_id
= <Bugzilla product ID>;
5. copy comments from old to new, updating bug ID references in the meantime to
refer to the new bug IDs: INSERT INTO bbo.longdescs (<all cols>) SELECT
bbo.bugs.bug_id, bmo.longdescs.<all cols except bug_id> FROM bbo.bugs JOIN
bmo.longdescs ON bbo.bugs.old_bug_id = bmo.longdescs.bug_id;
6. copy other bug-related data using queries like the one in the previous step
for other bug-related tables (attachments, flags, etc.);
7. run a script that puts prefixed old bug numbers in the alias field (this
could also be done as part of the query in step #4);
8. run a script that updates bug references in comments.
Of these steps, only the last two require any code written, and those might well
be fairly simple Perl scripts. There are other issues, of course, like how to
turn interproduct bug references in comments into interinstallation references.
I think time would be better spent writing that code and resolving those issues
than working on a generic tool. Not only would the BPM get done much faster
(which, after all, is the goal that prompted the filing of this bug), but we'd
have real experience on which to base the development of a generic tool.
I should mention that I've done several bug migrations before, including some
between Bugzilla-like software and a Bugzilla installation. In my experience,
each one was unique enough that there wasn't a great deal of code reuse.
I'm not saying there's no advantage to a generic tool, and obviously Bugzilla to
Bugzilla xfers will probably be easier than the ones I did, but a generic tool
will still not be the holy grail, seamlessly migrating data from different
versions and dealing with every installation's unique requirements.
The real focus should be on our goal, the BPM, and how to make that happen.
Comment 9•20 years ago
|
||
Myk's right. Let's do this one with documented, manual steps and minimal code.
Then we can look at the larger problem.
Gerv
Comment 10•20 years ago
|
||
So what is the first step?
I would be happy to help out in this.
Comment 11•19 years ago
|
||
A ***2.16*** & MySQL specific script I created for moving Evolution and some others products to bugzilla.gnome.org:
http://cvs.gnome.org/viewcvs/bugmasters/tools/mvbugs/
Doesn't do everything, but most, e.g.
* Uses next available bug numbers & changes comments to reflect new bug ids
* Can also use increment for bug numbers
* Uses available attachment numbers & changes comment to reflect new attachment numbers
* Deals with the changed bug numbers whereever they are used (bugs_activity, blockers, etc)
* Creates all required user accounts + gives some persons editbugs/canconfirm
* Created whatever was used/required (products, components, keywords, etc)
* If a keyword/product/component/account already exists it will be used
* Bug numbers not moved will be replaced with URLs
Requires direct connection to two databases (old one and the new one). New database must contain the tables (can be empty, but they must exist). Is very fast due to the direct connection (~70.000 bugs took 30 min or so). Most of the time was spent moving the longdescs + attachment data over.
For the direct db connection a partial ximian backup (only the required products) was used (ximain used 2.10, so backup&restore&checksetup was needed). Backup was restored in a temp db at gnome.
Giving only some persons canconfirm+editbugs was necessary as everyone at ximian had those. We gave everyone who closed a bug those, but fortunately such things will not be necessary for b.m.o. Other groups should not be tranferred IMO. Restricted bugs where not moved.
For Evolution we added 200.000 to the existing bug numbers (50 -> 200.050, etc). This was preferred as a lot of documentation referred to these bug numbers. Was also easier on the developers. On bugzilla.ximian.com a hack was added to redirect any access to those bugs to bugzilla.gnome.org. Aliases might be a good / better solution.
Only after moving everything we cleaned up the keywords (cleanups should not be part of a script IMO).
Most of the script acts like a SELECT + INSERT statement, but slightly more advanced (allows a function per column to change the value). Still simple enough to avoid bugs. Lots of tables are moved using one generic function.
I can probably port it to 2.20 or whatever is required to move the Bugzilla bugs (+ ensure it ports the new tables etc).
However, if a general solution is needed I suggest to enhance importxml.pl.
You need to log in
before you can comment on or make changes to this bug.
Description
•