Open Bug 300434 Opened 20 years ago Updated 16 years ago

A Tool to seamlessly move a Product to another Bugzilla installation

Categories

(Bugzilla :: Bugzilla-General, enhancement)

Product:
Bugzilla
Component:
Bugzilla-General
Version:
2.20
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

()

People

(Reporter: mkanat, Unassigned)

References

(Depends on 1 open bug)

Details

What we want is to be able to basically take a product entirely out of a Bugzilla installation, and move it to some other Bugzilla installation. (Our motivation for this is to be able to move the "Bugzilla" product to its own server, bugzilla.bugzilla.org.) The requirements would be: 1) Must move all bugs and attachments from the product. 2) Must create a new series of bug numbers (starting from "1" or whatever the lowest number is in the new DB) without holes. 3) All dependencies must remain consistent. 4) Needs to also fix bug numbers/links that are in comments. Also, ideally it would be able to make the original installation auto-redirect to the new installation for the moved bugs.
Ohhh... another problem is Users. All DB users have to be flawlessly moved, along with any groups that they belong to, and any associations that that group has with this Product. The script doesn't initially have to deal with the idea that there could already be bugs in the target installation. It could just create a new DB to be used by the new system.
This is basically what prompted me to re-hack importxml.pl in the first place. Here is the system I used to move ximian products (we do it on a product by product basis although it theoretically could be component etc.) to novell bugzilla: 1. Query for all the bugs in a particular product to be moved. 2. Capture the buglist in the resulting page 3. I have a little java program that a coworker slapped together to take a bug and grab it from showbug.cgi?ctype=xml. I use a little perl script to loop through the list of bugs, grabbing the xml of each one and spitting them individually to a file. 4. Run an XSLT transform to clean up the xml and transform it to the current version's DTD (correct the product, version, milestone data etc. for the new installation) 5. Run a script which calls my modified importxml.pl and passes it the new bug xml data for each bug in the list This creates new bug numbers for each bug as it is just basically creating a new bug. I use the original bug number as the alias but prepend some text string to make it unique. The modified importxml.pl also verifies group permissions and adds them appropriately as well as deals with some of the issues in the dependent bugs to bug 134294. In our case we have done a wholesale import of the profiles data assuming that the other systems were being shutdown and all were moving to the new bugzilla. In our case we have made this all run from the command line as moving products is usually considered a one time thing. I am not sure how necessary it is to put this functionality into the bugzilla interface for this reason but what are your thoughts? I would be happy to condense this into a single tool that can be used for this purpose. We also have some scripts that we ran on the data in MySQL to make the bug #'s (and attachment #'s) in comments match their new ones etc. I have also modified ctype=xml to produce attachment data on command (encoded as base64) so it can be used with a system such as this.
Yeah, a command-line tool would be perfect. I wouldn't imagine that this would be a frequent operation, either.
I think we can make this easier by limiting what we are trying to achieve. For example, instead of saying "we need to seamlessly migrate users", let's say "the users need to exist on the target system". Then, we can just clone the b.m.o. users table. Everyone even keeps the same password. Similarly, I think it's reasonable to require that the products, components, milestones, flags etc. all exist in the target Bugzilla before the migration. That's not hard to achieve as a prerequisite - why write the code to do it automatically? It's not an efficient use of time. So what we actually have to do here is XML export the bugs (with all their data) as a single file, and then XML import them, doing the following: - Fix up the bug numbers of any dependencies/dupes which are part of the same migration - Drop any which aren't, adding a comment which says "dropped dependencies" and giving full URLs to the original Bugzilla - Fix up any "bug XXXX" comments for bugs which are in the migration - Change the rest to full URLs to the original Bugzilla How does that sound? Easier? :-) Gerv
(In reply to comment #4) > How does that sound? Easier? :-) Sounds more difficult for the admin, actually. And possibly more difficult to code, because you're working with an existing database instead of creating one fresh. Seems to me like creating a fresh database would be easier, and just copying things between them if possible. Of course, that's also a bit tricky with our current Bugzilla::DB structure, but not entirely impossible.
Oh, I see - you are limiting this feature to moving the product into a clean Bugzilla? I didn't realise that. Doesn't that mean you could only ever move a single product to a target Bugzilla, because then it wouldn't be clean any more? How would adding bugs to an existing database be harder than adding them to a fresh, newly-checksetuped one? I agree my proposal leads to more work for the admin, but we need to think about how many times this would actually be used. If the answer is a small handful, coding time would probably be greater than cumulative admin time. So it seems reasonable to put some of the work on the admin who wants the move, rather than the coders. Gerv
(In reply to comment #4) > "the users need to exist on the target system". Then, we can just clone the > b.m.o. users table. Everyone even keeps the same password. > Agreed, that was the approach we took. However, we also have a tool that periodically goes through and removes stale accounts (those belonging to people that have never contributed to any bug as well as terminated employees which get disabled) > Similarly, I think it's reasonable to require that the products, components, > milestones, flags etc. all exist in the target Bugzilla before the migration. > That's not hard to achieve as a prerequisite - why write the code to do it > automatically? It's not an efficient use of time. Part of the move was to rename some or all of the components, milestones, etc. We also used the transforms to merge several components or products into one. If you are happy with the current names and structure than I totally agree. Additionally however, we were moving accross grossly different version of bugzilla in which some fields were absent entirely from the source data and needed to be filled in for the destination. > > So what we actually have to do here is XML export the bugs (with all their data) > as a single file, and then XML import them, doing the following: We specifically broke it into one file per bug because we found a major performace flaw in the libxml2 libraries upon which XML::Parser is dependent. It basically creates a Big(O) of N^2 issue in which the larger the file (and any attachment > 1MB runs into this) the longer the processing time but it grows exponentially. We would run out of memory pretty quickly when exporting even a relativly small number of bugs with their attachment data as a single file. > > - Fix up the bug numbers of any dependencies/dupes which are part of the same > migration > - Drop any which aren't, adding a comment which says "dropped dependencies" and > giving full URLs to the original Bugzilla This would certainly be handy. importxml.pl currently does not address dependencies at all. Though it makes an attempt at dupes. > - Fix up any "bug XXXX" comments for bugs which are in the migration This is as easy as prepending the text delimiter to the bug number to make it an alias. Since show_bug.cgi will take an ID or an alias it is a simple matter of munging it in the comment. The other approach which we took was one in which we had an offset that we added to all the bug #'s and then went through and incremented all references to the old bug by it's offset. This only worked since we were migrating all bugs from all products in an installation. > - Change the rest to full URLs to the original Bugzilla > > How does that sound? Easier? :-) > > Gerv > (In reply to comment #6) > Oh, I see - you are limiting this feature to moving the product into a clean > Bugzilla? I didn't realise that. Doesn't that mean you could only ever move a > single product to a target Bugzilla, because then it wouldn't be clean any more? > I agree that it is impractical to always assume you are moving into a clean bugzilla. > How would adding bugs to an existing database be harder than adding them to a > fresh, newly-checksetuped one? > > I agree my proposal leads to more work for the admin, but we need to think about > how many times this would actually be used. If the answer is a small handful, > coding time would probably be greater than cumulative admin time. So it seems > reasonable to put some of the work on the admin who wants the move, rather than > the coders. > > Gerv Precisly my point as well. Every migration will be unique to the installation and making the tool generic enough should be a worthy goal. Regardless, the admin is going to have to take some responsibility and effort to get everything prepped. Moving bugs should be a one time affair, especially if it is multiple products. I don't think we can be expected to provide tools for every situation. In our case we have now migrated a whole database (accross different versions of bugzilla), product by product, and bug records from a commercial system. By providing a generic set of tools and a best practices outline I think we can make it possible for most admins to move what needs to be moved. If we are only talking about moving the bugzilla product to a new b.b.o database than that is a specific instance and let's focus on it. If we are talking about a generic process for moving products that we can make available to everyone then I think the outiline I have provided is a good place to start.
There are a couple of issues: 1. Can we assume the tool has simultaneous access to both databases? If so, we don't have to muck with XML formats, and conditionally migrating related data is easier. We can just migrate the bugs one by one, migrating related data like components and user accounts as needed. 2. Can we assume we're migrating to a new installation? If so, we can migrate the user, product, etc. tables wholesale and then delete data afterwards as necessary, which is easier than having to select the data from the old database and insert it into the new one. The most flexible tool would be one that doesn't make either assumption. But both assumptions are true when migrating b.m.o's Bugzilla product to a new installation (say b.b.o), so it may not make sense to premise the Bugzilla product move (BPM) on the development of a flexible generic tool. For example, to do the BPM, we could do the following (where bmo and bbo are the b.m.o and b.b.o databases, respectively): 1. set up a new Bugzilla installation with database; 2. copy b.m.o's user, product, component, etc. tables to it via filesystem copies; 3. create an "old bug ID" column in the bugs table to store the original bug ID while we migrate data over to it: ALTER TABLE bbo.bugs ADD COLUMN old_bug_id INT; 4. copy bugs from bmo to bbo: INSERT INTO bbo.bugs (old_bug_id, <all cols except bug_id>) SELECT bug_id, <all cols except bug_id> FROM bmo.bugs WHERE product_id = <Bugzilla product ID>; 5. copy comments from old to new, updating bug ID references in the meantime to refer to the new bug IDs: INSERT INTO bbo.longdescs (<all cols>) SELECT bbo.bugs.bug_id, bmo.longdescs.<all cols except bug_id> FROM bbo.bugs JOIN bmo.longdescs ON bbo.bugs.old_bug_id = bmo.longdescs.bug_id; 6. copy other bug-related data using queries like the one in the previous step for other bug-related tables (attachments, flags, etc.); 7. run a script that puts prefixed old bug numbers in the alias field (this could also be done as part of the query in step #4); 8. run a script that updates bug references in comments. Of these steps, only the last two require any code written, and those might well be fairly simple Perl scripts. There are other issues, of course, like how to turn interproduct bug references in comments into interinstallation references. I think time would be better spent writing that code and resolving those issues than working on a generic tool. Not only would the BPM get done much faster (which, after all, is the goal that prompted the filing of this bug), but we'd have real experience on which to base the development of a generic tool. I should mention that I've done several bug migrations before, including some between Bugzilla-like software and a Bugzilla installation. In my experience, each one was unique enough that there wasn't a great deal of code reuse. I'm not saying there's no advantage to a generic tool, and obviously Bugzilla to Bugzilla xfers will probably be easier than the ones I did, but a generic tool will still not be the holy grail, seamlessly migrating data from different versions and dealing with every installation's unique requirements. The real focus should be on our goal, the BPM, and how to make that happen.
Myk's right. Let's do this one with documented, manual steps and minimal code. Then we can look at the larger problem. Gerv
So what is the first step? I would be happy to help out in this.
A ***2.16*** & MySQL specific script I created for moving Evolution and some others products to bugzilla.gnome.org: http://cvs.gnome.org/viewcvs/bugmasters/tools/mvbugs/ Doesn't do everything, but most, e.g. * Uses next available bug numbers & changes comments to reflect new bug ids * Can also use increment for bug numbers * Uses available attachment numbers & changes comment to reflect new attachment numbers * Deals with the changed bug numbers whereever they are used (bugs_activity, blockers, etc) * Creates all required user accounts + gives some persons editbugs/canconfirm * Created whatever was used/required (products, components, keywords, etc) * If a keyword/product/component/account already exists it will be used * Bug numbers not moved will be replaced with URLs Requires direct connection to two databases (old one and the new one). New database must contain the tables (can be empty, but they must exist). Is very fast due to the direct connection (~70.000 bugs took 30 min or so). Most of the time was spent moving the longdescs + attachment data over. For the direct db connection a partial ximian backup (only the required products) was used (ximain used 2.10, so backup&restore&checksetup was needed). Backup was restored in a temp db at gnome. Giving only some persons canconfirm+editbugs was necessary as everyone at ximian had those. We gave everyone who closed a bug those, but fortunately such things will not be necessary for b.m.o. Other groups should not be tranferred IMO. Restricted bugs where not moved. For Evolution we added 200.000 to the existing bug numbers (50 -> 200.050, etc). This was preferred as a lot of documentation referred to these bug numbers. Was also easier on the developers. On bugzilla.ximian.com a hack was added to redirect any access to those bugs to bugzilla.gnome.org. Aliases might be a good / better solution. Only after moving everything we cleaned up the keywords (cleanups should not be part of a script IMO). Most of the script acts like a SELECT + INSERT statement, but slightly more advanced (allows a function per column to change the value). Still simple enough to avoid bugs. Lots of tables are moved using one generic function. I can probably port it to 2.20 or whatever is required to move the Bugzilla bugs (+ ensure it ports the new tables etc). However, if a general solution is needed I suggest to enhance importxml.pl.
You need to log in before you can comment on or make changes to this bug.