Last Comment Bug 519584 - (bz-migrate) Implement a framework for migrating from other systems
(bz-migrate)
: Implement a framework for migrating from other systems
Status: RESOLVED FIXED
[es-lambda]
:
Product: Bugzilla
Classification: Server Software
Component: Installation & Upgrading (show other bugs)
: 3.5
: All All
: -- enhancement (vote)
: Bugzilla 3.6
Assigned To: Max Kanat-Alexander
: default-qa
:
Mentors:
Depends on: 522396 522404 522428 522620
Blocks:
  Show dependency treegraph
 
Reported: 2009-09-29 17:45 PDT by Max Kanat-Alexander
Modified: 2010-02-22 15:33 PST (History)
4 users (show)
mkanat: approval+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Work In Progress (125.85 KB, patch)
2009-10-13 17:00 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
WIP 2 (129.26 KB, patch)
2009-10-13 18:15 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
WIP 3 (131.66 KB, patch)
2009-10-13 23:39 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
WIP 4 (142.69 KB, patch)
2009-10-14 18:13 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
WIP 5 (150.80 KB, patch)
2009-10-14 23:51 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
v1 (157.41 KB, patch)
2009-10-15 16:20 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
v2 (161.56 KB, patch)
2009-10-15 19:39 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
v3 (163.43 KB, patch)
2009-10-15 21:38 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
v4 (167.85 KB, patch)
2009-10-15 22:19 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
v5 (167.88 KB, patch)
2009-10-15 22:27 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
get_set_serial_sql for oracle v1 (766 bytes, patch)
2009-10-15 23:17 PDT, Xiaoou
mkanat: review-
Details | Diff | Splinter Review
v6 (170.98 KB, patch)
2009-10-16 23:04 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
v7 (171.53 KB, patch)
2009-10-17 01:21 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
v8 (176.55 KB, patch)
2009-10-18 23:46 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
v9 (177.60 KB, patch)
2009-10-19 00:45 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
v9.1 (177.12 KB, patch)
2009-10-19 00:47 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
v10 (177.98 KB, patch)
2009-10-19 23:31 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
get_set_serial_sql for oracle v2 (685 bytes, patch)
2009-10-21 02:04 PDT, Xiaoou
mkanat: review+
Details | Diff | Splinter Review

Description Max Kanat-Alexander 2009-09-29 17:45:20 PDT
It often happens that people want to switch from their current bug-tracker to Bugzilla. It would be nice to have a contrib/ script that understood various different other systems and could do an initial copy of another system into Bugzilla, for migration purposes.
Comment 1 Max Kanat-Alexander 2009-10-13 17:00:30 PDT
Created attachment 406140 [details] [diff] [review]
Work In Progress

So far, this only touches installation code, so after completion and testing, I plan to check this in as a module owner (without review). It's way too huge for anybody to have time to review, anyhow, and I can't imagine how you would test a GNATS migration (whereas I can do it pretty well, and I have a client who wants it and can verify whether or not it was done correctly).
Comment 2 Max Kanat-Alexander 2009-10-13 18:15:08 PDT
Created attachment 406152 [details] [diff] [review]
WIP 2

Further work--this now parses the history of a GNATS bug.
Comment 3 Max Kanat-Alexander 2009-10-13 23:39:03 PDT
Created attachment 406183 [details] [diff] [review]
WIP 3

The architecture's starting to fall into place a bit more. This reads users and bugs, now.
Comment 4 Max Kanat-Alexander 2009-10-14 18:13:05 PDT
Created attachment 406377 [details] [diff] [review]
WIP 4

This now fully reads bugs, users, and products (though not attachments yet).

It also fully creates users and products (though there are some problems with bugs as of yet).
Comment 5 Max Kanat-Alexander 2009-10-14 23:51:20 PDT
Created attachment 406404 [details] [diff] [review]
WIP 5

This version currently is importing all bugs from my small test (18 bugs in two products), though it isn't yet handling attachments.
Comment 6 Max Kanat-Alexander 2009-10-15 16:20:29 PDT
Created attachment 406566 [details] [diff] [review]
v1

This version is essentially complete, importing everything from the other bug-tracker and implementing all the options that I plan to. However, it needs more testing.

Also, I've discovered that running it under --dry-run increments the AUTO_INCREMENT counter on tables, because even if we rollback a transaction, we still get AUTO_INCREMENT updated. :-( So I'm going to have to write some DB methods to fix that.
Comment 7 Max Kanat-Alexander 2009-10-15 19:39:57 PDT
Created attachment 406616 [details] [diff] [review]
v2

A bunch of bug-fixes over v1.
Comment 8 Vitaly Fedrushkov 2009-10-15 20:21:11 PDT
(In reply to comment #6)
> Also, I've discovered that running it under --dry-run increments the
> AUTO_INCREMENT counter on tables, because even if we rollback a transaction, we
> still get AUTO_INCREMENT updated. :-( So I'm going to have to write some DB
> methods to fix that.

This can't be done without dangerous cheating.  In Oracle, SEQUENCE values never get back if you rollback a transaction.

Surely you're not trying to hide dry-run attempts?
Comment 9 Max Kanat-Alexander 2009-10-15 21:21:09 PDT
It can be done relatively non-dangerously, after the transaction is rolled back. At least, in MySQL and PostgreSQL it can. I assume there's a similar method in Oracle. 

The idea is that dry-run attempts shouldn't modify the database AT ALL.
Comment 10 Max Kanat-Alexander 2009-10-15 21:38:01 PDT
Created attachment 406631 [details] [diff] [review]
v3

This version fully imports the GNATS database that I am testing on.
Comment 11 Max Kanat-Alexander 2009-10-15 22:19:00 PDT
Created attachment 406637 [details] [diff] [review]
v4

And this version also does the rollback of the auto_increment values.

xiaoou--I need an implementation of get_set_serial_sql (which you can see used and defined for other DBs in this patch) for Oracle.
Comment 12 Max Kanat-Alexander 2009-10-15 22:19:35 PDT
CC'ing mockodin so he can know about the new Bugzilla::DB method I added.
Comment 13 Max Kanat-Alexander 2009-10-15 22:27:46 PDT
Created attachment 406638 [details] [diff] [review]
v5

And this version passes runtests, and can be checked in once the blockers are resolved.
Comment 14 Max Kanat-Alexander 2009-10-15 22:28:02 PDT
Holding approval until blockers are resolved.
Comment 15 Vitaly Fedrushkov 2009-10-15 22:49:17 PDT
(In reply to comment #9)
> It can be done relatively non-dangerously, after the transaction is rolled
> back. At least, in MySQL and PostgreSQL it can. I assume there's a similar
> method in Oracle. 

Are you assuming (a) database being in exclusive mode during migrate.pl, or (b) dry-run can be done using read transactions only, without UPDATEs rolled back?
Comment 16 Xiaoou 2009-10-15 23:17:12 PDT
Created attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1
Comment 17 Max Kanat-Alexander 2009-10-16 14:25:53 PDT
(In reply to comment #15)
> Are you assuming (a) database being in exclusive mode during migrate.pl, or (b)
> dry-run can be done using read transactions only, without UPDATEs rolled back?

  The way that dry-run works is that it rolls back the transaction and *then* resets the serial counters to MAX(column) + 1. I assume that the database isn't under heavy production load when running under dry-run. And I do mention that --dry-run shouldn't be run on a live database.
Comment 18 Max Kanat-Alexander 2009-10-16 14:35:50 PDT
Comment on attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1

xiaoou: This looks like it will increment the sequence by $value, instead of setting the next value to $value. Am I missing something?
Comment 19 Max Kanat-Alexander 2009-10-16 14:37:24 PDT
Comment on attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1

The Oracle docs say: "To restart the sequence at a different number, you must drop and re-create it." Is that what we should be doing here instead?
Comment 20 Max Kanat-Alexander 2009-10-16 23:04:58 PDT
Created attachment 406849 [details] [diff] [review]
v6

I made some improvements to the GNATS migrator in terms of version number and username parsing.
Comment 21 Max Kanat-Alexander 2009-10-17 01:21:01 PDT
Created attachment 406855 [details] [diff] [review]
v7

This version uses less memory during import and has some improvements in attachment parsing for GNATS.
Comment 22 Max Kanat-Alexander 2009-10-18 23:46:24 PDT
Created attachment 406977 [details] [diff] [review]
v8

This version introduces the ability to set resolutions for particular translated statuses, and moves more code up into the generic framework from the GNATS migrator.
Comment 23 Max Kanat-Alexander 2009-10-19 00:45:04 PDT
Created attachment 406980 [details] [diff] [review]
v9

Fix a few bugs in the GNATS migrator.
Comment 24 Max Kanat-Alexander 2009-10-19 00:47:17 PDT
Created attachment 406981 [details] [diff] [review]
v9.1

The original v9 I attached had extra local changes in it.
Comment 25 Max Kanat-Alexander 2009-10-19 23:31:35 PDT
Created attachment 407231 [details] [diff] [review]
v10

Improve the version parsing of the GNATS migrator, and make value-mapping case-insensitive (because Bugzilla is case-insensitive about field values anyway).
Comment 26 Max Kanat-Alexander 2009-10-19 23:32:20 PDT
Comment on attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1

This doesn't look like it's setting the sequence's value, it looks like it's incrementing the sequence by the value passed in, which is not what I wanted.
Comment 27 Xiaoou 2009-10-21 02:04:44 PDT
Created attachment 407489 [details] [diff] [review]
get_set_serial_sql for oracle v2

Yeah, v1 was assuming that the current sequence is 0.
This v2 will set the sequence value.
Comment 28 Max Kanat-Alexander 2009-10-23 22:30:56 PDT
Checking in Bugzilla.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla.pm,v  <--  Bugzilla.pm
new revision: 1.80; previous revision: 1.79
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/migrate.pl,v
done
Checking in migrate.pl;
/cvsroot/mozilla/webtools/bugzilla/migrate.pl,v  <--  migrate.pl
initial revision: 1.1
done
Checking in Bugzilla/DB.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB.pm,v  <--  DB.pm
new revision: 1.129; previous revision: 1.128
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate.pm,v
done
Checking in Bugzilla/Migrate.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate.pm,v  <--  Migrate.pm
initial revision: 1.1
done
Checking in Bugzilla/DB/Schema/Mysql.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB/Schema/Mysql.pm,v  <--  Mysql.pm
new revision: 1.23; previous revision: 1.22
done
Checking in Bugzilla/DB/Schema/Pg.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB/Schema/Pg.pm,v  <--  Pg.pm
new revision: 1.16; previous revision: 1.15
done
Checking in Bugzilla/Install/Filesystem.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Filesystem.pm,v  <--  Filesystem.pm
new revision: 1.40; previous revision: 1.39
done
Checking in Bugzilla/Install/Requirements.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Requirements.pm,v  <--  Requirements.pm
new revision: 1.72; previous revision: 1.71
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate/Gnats.pm,v
done
Checking in Bugzilla/Migrate/Gnats.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate/Gnats.pm,v  <--  Gnats.pm
initial revision: 1.1
done
Checking in contrib/bzdbcopy.pl;
/cvsroot/mozilla/webtools/bugzilla/contrib/bzdbcopy.pl,v  <--  bzdbcopy.pl
new revision: 1.9; previous revision: 1.8
done
Removing contrib/gnats2bz.pl;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnats2bz.pl,v  <--  gnats2bz.pl
new revision: delete; previous revision: 1.10
done
Removing contrib/gnatsparse/README;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/README,v  <--  README
new revision: delete; previous revision: 1.2
done
Removing contrib/gnatsparse/gnatsparse.py;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/gnatsparse.py,v  <--  gnatsparse.py
new revision: delete; previous revision: 1.6
done
Removing contrib/gnatsparse/magic.py;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/magic.py,v  <--  magic.py
new revision: delete; previous revision: 1.1
done
Removing contrib/gnatsparse/specialuu.py;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/specialuu.py,v  <--  specialuu.py
new revision: delete; previous revision: 1.1
done
Checking in template/en/default/global/messages.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/global/messages.html.tmpl,v  <--  messages.html.tmpl
new revision: 1.93; previous revision: 1.92
done
Checking in template/en/default/global/user-error.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/global/user-error.html.tmpl,v  <--  user-error.html.tmpl
new revision: 1.287; previous revision: 1.286
done
Checking in template/en/default/setup/strings.txt.pl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/setup/strings.txt.pl,v  <--  strings.txt.pl
new revision: 1.17; previous revision: 1.16
done
Comment 29 Max Kanat-Alexander 2009-10-23 22:31:51 PDT
And Xiaoou's patch:

Checking in Bugzilla/DB/Schema/Oracle.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB/Schema/Oracle.pm,v  <--  Oracle.pm
new revision: 1.12; previous revision: 1.11
done
Comment 30 Max Kanat-Alexander 2010-02-22 15:33:31 PST
Added to the release notes in bug 547466.

Note You need to log in before you can comment on or make changes to this bug.