Last Comment Bug 519584 - (bz-migrate) Implement a framework for migrating from other systems
(bz-migrate)
: Implement a framework for migrating from other systems
Status: RESOLVED FIXED
[es-lambda]
:
Product: Bugzilla
Classification: Server Software
Component: Installation & Upgrading (show other bugs)
: 3.5
: All All
-- enhancement (vote)
: Bugzilla 3.6
Assigned To: Max Kanat-Alexander
: default-qa
:
Mentors:
Depends on: 522396 522404 522428 522620
Blocks:
  Show dependency treegraph
 
Reported: 2009-09-29 17:45 PDT by Max Kanat-Alexander
Modified: 2010-02-22 15:33 PST (History)
4 users (show)
mkanat: approval+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Work In Progress (125.85 KB, patch)
2009-10-13 17:00 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
WIP 2 (129.26 KB, patch)
2009-10-13 18:15 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
WIP 3 (131.66 KB, patch)
2009-10-13 23:39 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
WIP 4 (142.69 KB, patch)
2009-10-14 18:13 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
WIP 5 (150.80 KB, patch)
2009-10-14 23:51 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
v1 (157.41 KB, patch)
2009-10-15 16:20 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
v2 (161.56 KB, patch)
2009-10-15 19:39 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
v3 (163.43 KB, patch)
2009-10-15 21:38 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
v4 (167.85 KB, patch)
2009-10-15 22:19 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
v5 (167.88 KB, patch)
2009-10-15 22:27 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
get_set_serial_sql for oracle v1 (766 bytes, patch)
2009-10-15 23:17 PDT, Xiaoou
mkanat: review-
Details | Diff | Splinter Review
v6 (170.98 KB, patch)
2009-10-16 23:04 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
v7 (171.53 KB, patch)
2009-10-17 01:21 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
v8 (176.55 KB, patch)
2009-10-18 23:46 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
v9 (177.60 KB, patch)
2009-10-19 00:45 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
v9.1 (177.12 KB, patch)
2009-10-19 00:47 PDT, Max Kanat-Alexander
no flags Details | Diff | Splinter Review
v10 (177.98 KB, patch)
2009-10-19 23:31 PDT, Max Kanat-Alexander
mkanat: review+
Details | Diff | Splinter Review
get_set_serial_sql for oracle v2 (685 bytes, patch)
2009-10-21 02:04 PDT, Xiaoou
mkanat: review+
Details | Diff | Splinter Review

Description User image Max Kanat-Alexander 2009-09-29 17:45:20 PDT
It often happens that people want to switch from their current bug-tracker to Bugzilla. It would be nice to have a contrib/ script that understood various different other systems and could do an initial copy of another system into Bugzilla, for migration purposes.
Comment 1 User image Max Kanat-Alexander 2009-10-13 17:00:30 PDT
Created attachment 406140 [details] [diff] [review]
Work In Progress

So far, this only touches installation code, so after completion and testing, I plan to check this in as a module owner (without review). It's way too huge for anybody to have time to review, anyhow, and I can't imagine how you would test a GNATS migration (whereas I can do it pretty well, and I have a client who wants it and can verify whether or not it was done correctly).
Comment 2 User image Max Kanat-Alexander 2009-10-13 18:15:08 PDT
Created attachment 406152 [details] [diff] [review]
WIP 2

Further work--this now parses the history of a GNATS bug.
Comment 3 User image Max Kanat-Alexander 2009-10-13 23:39:03 PDT
Created attachment 406183 [details] [diff] [review]
WIP 3

The architecture's starting to fall into place a bit more. This reads users and bugs, now.
Comment 4 User image Max Kanat-Alexander 2009-10-14 18:13:05 PDT
Created attachment 406377 [details] [diff] [review]
WIP 4

This now fully reads bugs, users, and products (though not attachments yet).

It also fully creates users and products (though there are some problems with bugs as of yet).
Comment 5 User image Max Kanat-Alexander 2009-10-14 23:51:20 PDT
Created attachment 406404 [details] [diff] [review]
WIP 5

This version currently is importing all bugs from my small test (18 bugs in two products), though it isn't yet handling attachments.
Comment 6 User image Max Kanat-Alexander 2009-10-15 16:20:29 PDT
Created attachment 406566 [details] [diff] [review]
v1

This version is essentially complete, importing everything from the other bug-tracker and implementing all the options that I plan to. However, it needs more testing.

Also, I've discovered that running it under --dry-run increments the AUTO_INCREMENT counter on tables, because even if we rollback a transaction, we still get AUTO_INCREMENT updated. :-( So I'm going to have to write some DB methods to fix that.
Comment 7 User image Max Kanat-Alexander 2009-10-15 19:39:57 PDT
Created attachment 406616 [details] [diff] [review]
v2

A bunch of bug-fixes over v1.
Comment 8 User image Vitaly Fedrushkov 2009-10-15 20:21:11 PDT
(In reply to comment #6)
> Also, I've discovered that running it under --dry-run increments the
> AUTO_INCREMENT counter on tables, because even if we rollback a transaction, we
> still get AUTO_INCREMENT updated. :-( So I'm going to have to write some DB
> methods to fix that.

This can't be done without dangerous cheating.  In Oracle, SEQUENCE values never get back if you rollback a transaction.

Surely you're not trying to hide dry-run attempts?
Comment 9 User image Max Kanat-Alexander 2009-10-15 21:21:09 PDT
It can be done relatively non-dangerously, after the transaction is rolled back. At least, in MySQL and PostgreSQL it can. I assume there's a similar method in Oracle. 

The idea is that dry-run attempts shouldn't modify the database AT ALL.
Comment 10 User image Max Kanat-Alexander 2009-10-15 21:38:01 PDT
Created attachment 406631 [details] [diff] [review]
v3

This version fully imports the GNATS database that I am testing on.
Comment 11 User image Max Kanat-Alexander 2009-10-15 22:19:00 PDT
Created attachment 406637 [details] [diff] [review]
v4

And this version also does the rollback of the auto_increment values.

xiaoou--I need an implementation of get_set_serial_sql (which you can see used and defined for other DBs in this patch) for Oracle.
Comment 12 User image Max Kanat-Alexander 2009-10-15 22:19:35 PDT
CC'ing mockodin so he can know about the new Bugzilla::DB method I added.
Comment 13 User image Max Kanat-Alexander 2009-10-15 22:27:46 PDT
Created attachment 406638 [details] [diff] [review]
v5

And this version passes runtests, and can be checked in once the blockers are resolved.
Comment 14 User image Max Kanat-Alexander 2009-10-15 22:28:02 PDT
Holding approval until blockers are resolved.
Comment 15 User image Vitaly Fedrushkov 2009-10-15 22:49:17 PDT
(In reply to comment #9)
> It can be done relatively non-dangerously, after the transaction is rolled
> back. At least, in MySQL and PostgreSQL it can. I assume there's a similar
> method in Oracle. 

Are you assuming (a) database being in exclusive mode during migrate.pl, or (b) dry-run can be done using read transactions only, without UPDATEs rolled back?
Comment 16 User image Xiaoou 2009-10-15 23:17:12 PDT
Created attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1
Comment 17 User image Max Kanat-Alexander 2009-10-16 14:25:53 PDT
(In reply to comment #15)
> Are you assuming (a) database being in exclusive mode during migrate.pl, or (b)
> dry-run can be done using read transactions only, without UPDATEs rolled back?

  The way that dry-run works is that it rolls back the transaction and *then* resets the serial counters to MAX(column) + 1. I assume that the database isn't under heavy production load when running under dry-run. And I do mention that --dry-run shouldn't be run on a live database.
Comment 18 User image Max Kanat-Alexander 2009-10-16 14:35:50 PDT
Comment on attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1

xiaoou: This looks like it will increment the sequence by $value, instead of setting the next value to $value. Am I missing something?
Comment 19 User image Max Kanat-Alexander 2009-10-16 14:37:24 PDT
Comment on attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1

The Oracle docs say: "To restart the sequence at a different number, you must drop and re-create it." Is that what we should be doing here instead?
Comment 20 User image Max Kanat-Alexander 2009-10-16 23:04:58 PDT
Created attachment 406849 [details] [diff] [review]
v6

I made some improvements to the GNATS migrator in terms of version number and username parsing.
Comment 21 User image Max Kanat-Alexander 2009-10-17 01:21:01 PDT
Created attachment 406855 [details] [diff] [review]
v7

This version uses less memory during import and has some improvements in attachment parsing for GNATS.
Comment 22 User image Max Kanat-Alexander 2009-10-18 23:46:24 PDT
Created attachment 406977 [details] [diff] [review]
v8

This version introduces the ability to set resolutions for particular translated statuses, and moves more code up into the generic framework from the GNATS migrator.
Comment 23 User image Max Kanat-Alexander 2009-10-19 00:45:04 PDT
Created attachment 406980 [details] [diff] [review]
v9

Fix a few bugs in the GNATS migrator.
Comment 24 User image Max Kanat-Alexander 2009-10-19 00:47:17 PDT
Created attachment 406981 [details] [diff] [review]
v9.1

The original v9 I attached had extra local changes in it.
Comment 25 User image Max Kanat-Alexander 2009-10-19 23:31:35 PDT
Created attachment 407231 [details] [diff] [review]
v10

Improve the version parsing of the GNATS migrator, and make value-mapping case-insensitive (because Bugzilla is case-insensitive about field values anyway).
Comment 26 User image Max Kanat-Alexander 2009-10-19 23:32:20 PDT
Comment on attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1

This doesn't look like it's setting the sequence's value, it looks like it's incrementing the sequence by the value passed in, which is not what I wanted.
Comment 27 User image Xiaoou 2009-10-21 02:04:44 PDT
Created attachment 407489 [details] [diff] [review]
get_set_serial_sql for oracle v2

Yeah, v1 was assuming that the current sequence is 0.
This v2 will set the sequence value.
Comment 28 User image Max Kanat-Alexander 2009-10-23 22:30:56 PDT
Checking in Bugzilla.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla.pm,v  <--  Bugzilla.pm
new revision: 1.80; previous revision: 1.79
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/migrate.pl,v
done
Checking in migrate.pl;
/cvsroot/mozilla/webtools/bugzilla/migrate.pl,v  <--  migrate.pl
initial revision: 1.1
done
Checking in Bugzilla/DB.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB.pm,v  <--  DB.pm
new revision: 1.129; previous revision: 1.128
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate.pm,v
done
Checking in Bugzilla/Migrate.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate.pm,v  <--  Migrate.pm
initial revision: 1.1
done
Checking in Bugzilla/DB/Schema/Mysql.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB/Schema/Mysql.pm,v  <--  Mysql.pm
new revision: 1.23; previous revision: 1.22
done
Checking in Bugzilla/DB/Schema/Pg.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB/Schema/Pg.pm,v  <--  Pg.pm
new revision: 1.16; previous revision: 1.15
done
Checking in Bugzilla/Install/Filesystem.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Filesystem.pm,v  <--  Filesystem.pm
new revision: 1.40; previous revision: 1.39
done
Checking in Bugzilla/Install/Requirements.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Requirements.pm,v  <--  Requirements.pm
new revision: 1.72; previous revision: 1.71
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate/Gnats.pm,v
done
Checking in Bugzilla/Migrate/Gnats.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate/Gnats.pm,v  <--  Gnats.pm
initial revision: 1.1
done
Checking in contrib/bzdbcopy.pl;
/cvsroot/mozilla/webtools/bugzilla/contrib/bzdbcopy.pl,v  <--  bzdbcopy.pl
new revision: 1.9; previous revision: 1.8
done
Removing contrib/gnats2bz.pl;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnats2bz.pl,v  <--  gnats2bz.pl
new revision: delete; previous revision: 1.10
done
Removing contrib/gnatsparse/README;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/README,v  <--  README
new revision: delete; previous revision: 1.2
done
Removing contrib/gnatsparse/gnatsparse.py;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/gnatsparse.py,v  <--  gnatsparse.py
new revision: delete; previous revision: 1.6
done
Removing contrib/gnatsparse/magic.py;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/magic.py,v  <--  magic.py
new revision: delete; previous revision: 1.1
done
Removing contrib/gnatsparse/specialuu.py;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/specialuu.py,v  <--  specialuu.py
new revision: delete; previous revision: 1.1
done
Checking in template/en/default/global/messages.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/global/messages.html.tmpl,v  <--  messages.html.tmpl
new revision: 1.93; previous revision: 1.92
done
Checking in template/en/default/global/user-error.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/global/user-error.html.tmpl,v  <--  user-error.html.tmpl
new revision: 1.287; previous revision: 1.286
done
Checking in template/en/default/setup/strings.txt.pl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/setup/strings.txt.pl,v  <--  strings.txt.pl
new revision: 1.17; previous revision: 1.16
done
Comment 29 User image Max Kanat-Alexander 2009-10-23 22:31:51 PDT
And Xiaoou's patch:

Checking in Bugzilla/DB/Schema/Oracle.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB/Schema/Oracle.pm,v  <--  Oracle.pm
new revision: 1.12; previous revision: 1.11
done
Comment 30 User image Max Kanat-Alexander 2010-02-22 15:33:31 PST
Added to the release notes in bug 547466.

Note You need to log in before you can comment on or make changes to this bug.