519584 - (bz-migrate) Implement a framework for migrating from other systems

Status: RESOLVED FIXED

(Bugzilla :: Installation & Upgrading, enhancement)

Description

[Max Kanat-Alexander]

It often happens that people want to switch from their current bug-tracker to Bugzilla. It would be nice to have a contrib/ script that understood various different other systems and could do an initial copy of another system into Bugzilla, for migration purposes.

Comment 1

[Max Kanat-Alexander]

Attachment: Work In Progress

So far, this only touches installation code, so after completion and testing, I plan to check this in as a module owner (without review). It's way too huge for anybody to have time to review, anyhow, and I can't imagine how you would test a GNATS migration (whereas I can do it pretty well, and I have a client who wants it and can verify whether or not it was done correctly).

Comment 2

[Max Kanat-Alexander]

Attachment: WIP 2

Further work--this now parses the history of a GNATS bug.

Comment 3

[Max Kanat-Alexander]

Attachment: WIP 3

The architecture's starting to fall into place a bit more. This reads users and bugs, now.

Comment 4

[Max Kanat-Alexander]

Attachment: WIP 4

This now fully reads bugs, users, and products (though not attachments yet).

It also fully creates users and products (though there are some problems with bugs as of yet).

Comment 5

[Max Kanat-Alexander]

Attachment: WIP 5

This version currently is importing all bugs from my small test (18 bugs in two products), though it isn't yet handling attachments.

Comment 6

[Max Kanat-Alexander]

Attachment: v1

This version is essentially complete, importing everything from the other bug-tracker and implementing all the options that I plan to. However, it needs more testing.

Also, I've discovered that running it under --dry-run increments the AUTO_INCREMENT counter on tables, because even if we rollback a transaction, we still get AUTO_INCREMENT updated. :-( So I'm going to have to write some DB methods to fix that.

Comment 7

[Max Kanat-Alexander]

Attachment: v2

A bunch of bug-fixes over v1.

Comment 8

[Vitaly Fedrushkov]

(In reply to comment #6)
> Also, I've discovered that running it under --dry-run increments the
> AUTO_INCREMENT counter on tables, because even if we rollback a transaction, we
> still get AUTO_INCREMENT updated. :-( So I'm going to have to write some DB
> methods to fix that.

This can't be done without dangerous cheating.  In Oracle, SEQUENCE values never get back if you rollback a transaction.

Surely you're not trying to hide dry-run attempts?

Comment 9

[Max Kanat-Alexander]

It can be done relatively non-dangerously, after the transaction is rolled back. At least, in MySQL and PostgreSQL it can. I assume there's a similar method in Oracle. 

The idea is that dry-run attempts shouldn't modify the database AT ALL.

Comment 10

[Max Kanat-Alexander]

Attachment: v3

This version fully imports the GNATS database that I am testing on.

Comment 11

[Max Kanat-Alexander]

Attachment: v4

And this version also does the rollback of the auto_increment values.

xiaoou--I need an implementation of get_set_serial_sql (which you can see used and defined for other DBs in this patch) for Oracle.

Comment 12

[Max Kanat-Alexander]

CC'ing mockodin so he can know about the new Bugzilla::DB method I added.

Comment 13

[Max Kanat-Alexander]

Attachment: v5

And this version passes runtests, and can be checked in once the blockers are resolved.

Comment 14

[Max Kanat-Alexander]

Holding approval until blockers are resolved.

Comment 15

[Vitaly Fedrushkov]

(In reply to comment #9)
> It can be done relatively non-dangerously, after the transaction is rolled
> back. At least, in MySQL and PostgreSQL it can. I assume there's a similar
> method in Oracle. 

Are you assuming (a) database being in exclusive mode during migrate.pl, or (b) dry-run can be done using read transactions only, without UPDATEs rolled back?

Comment 16

[Xiaoou]

Attachment: get_set_serial_sql for oracle v1

Comment 17

[Max Kanat-Alexander]

(In reply to comment #15)
> Are you assuming (a) database being in exclusive mode during migrate.pl, or (b)
> dry-run can be done using read transactions only, without UPDATEs rolled back?

  The way that dry-run works is that it rolls back the transaction and *then* resets the serial counters to MAX(column) + 1. I assume that the database isn't under heavy production load when running under dry-run. And I do mention that --dry-run shouldn't be run on a live database.

Comment 18

[Max Kanat-Alexander]

Comment on attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1

xiaoou: This looks like it will increment the sequence by $value, instead of setting the next value to $value. Am I missing something?

Comment 19

[Max Kanat-Alexander]

Comment on attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1

The Oracle docs say: "To restart the sequence at a different number, you must drop and re-create it." Is that what we should be doing here instead?

Comment 20

[Max Kanat-Alexander]

Attachment: v6

I made some improvements to the GNATS migrator in terms of version number and username parsing.

Comment 21

[Max Kanat-Alexander]

Attachment: v7

This version uses less memory during import and has some improvements in attachment parsing for GNATS.

Comment 22

[Max Kanat-Alexander]

Attachment: v8

This version introduces the ability to set resolutions for particular translated statuses, and moves more code up into the generic framework from the GNATS migrator.

Comment 23

[Max Kanat-Alexander]

Attachment: v9

Fix a few bugs in the GNATS migrator.

Comment 24

[Max Kanat-Alexander]

Attachment: v9.1

The original v9 I attached had extra local changes in it.

Comment 25

[Max Kanat-Alexander]

Attachment: v10

Improve the version parsing of the GNATS migrator, and make value-mapping case-insensitive (because Bugzilla is case-insensitive about field values anyway).

Comment 26

[Max Kanat-Alexander]

Comment on attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1

This doesn't look like it's setting the sequence's value, it looks like it's incrementing the sequence by the value passed in, which is not what I wanted.

Comment 27

[Xiaoou]

Attachment: get_set_serial_sql for oracle v2

Yeah, v1 was assuming that the current sequence is 0.
This v2 will set the sequence value.

Comment 28

[Max Kanat-Alexander]

Checking in Bugzilla.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla.pm,v  <--  Bugzilla.pm
new revision: 1.80; previous revision: 1.79
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/migrate.pl,v
done
Checking in migrate.pl;
/cvsroot/mozilla/webtools/bugzilla/migrate.pl,v  <--  migrate.pl
initial revision: 1.1
done
Checking in Bugzilla/DB.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB.pm,v  <--  DB.pm
new revision: 1.129; previous revision: 1.128
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate.pm,v
done
Checking in Bugzilla/Migrate.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate.pm,v  <--  Migrate.pm
initial revision: 1.1
done
Checking in Bugzilla/DB/Schema/Mysql.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB/Schema/Mysql.pm,v  <--  Mysql.pm
new revision: 1.23; previous revision: 1.22
done
Checking in Bugzilla/DB/Schema/Pg.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB/Schema/Pg.pm,v  <--  Pg.pm
new revision: 1.16; previous revision: 1.15
done
Checking in Bugzilla/Install/Filesystem.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Filesystem.pm,v  <--  Filesystem.pm
new revision: 1.40; previous revision: 1.39
done
Checking in Bugzilla/Install/Requirements.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Requirements.pm,v  <--  Requirements.pm
new revision: 1.72; previous revision: 1.71
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate/Gnats.pm,v
done
Checking in Bugzilla/Migrate/Gnats.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate/Gnats.pm,v  <--  Gnats.pm
initial revision: 1.1
done
Checking in contrib/bzdbcopy.pl;
/cvsroot/mozilla/webtools/bugzilla/contrib/bzdbcopy.pl,v  <--  bzdbcopy.pl
new revision: 1.9; previous revision: 1.8
done
Removing contrib/gnats2bz.pl;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnats2bz.pl,v  <--  gnats2bz.pl
new revision: delete; previous revision: 1.10
done
Removing contrib/gnatsparse/README;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/README,v  <--  README
new revision: delete; previous revision: 1.2
done
Removing contrib/gnatsparse/gnatsparse.py;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/gnatsparse.py,v  <--  gnatsparse.py
new revision: delete; previous revision: 1.6
done
Removing contrib/gnatsparse/magic.py;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/magic.py,v  <--  magic.py
new revision: delete; previous revision: 1.1
done
Removing contrib/gnatsparse/specialuu.py;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/specialuu.py,v  <--  specialuu.py
new revision: delete; previous revision: 1.1
done
Checking in template/en/default/global/messages.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/global/messages.html.tmpl,v  <--  messages.html.tmpl
new revision: 1.93; previous revision: 1.92
done
Checking in template/en/default/global/user-error.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/global/user-error.html.tmpl,v  <--  user-error.html.tmpl
new revision: 1.287; previous revision: 1.286
done
Checking in template/en/default/setup/strings.txt.pl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/setup/strings.txt.pl,v  <--  strings.txt.pl
new revision: 1.17; previous revision: 1.16
done

Comment 29

[Max Kanat-Alexander]

And Xiaoou's patch:

Checking in Bugzilla/DB/Schema/Oracle.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB/Schema/Oracle.pm,v  <--  Oracle.pm
new revision: 1.12; previous revision: 1.11
done

Comment 30

[Max Kanat-Alexander]

Added to the release notes in bug 547466.