The default bug view has changed. See this FAQ.
Bug 519584 (bz-migrate)

Implement a framework for migrating from other systems

RESOLVED FIXED in Bugzilla 3.6

Status

()

Bugzilla
Installation & Upgrading
--
enhancement
RESOLVED FIXED
8 years ago
7 years ago

People

(Reporter: Max Kanat-Alexander, Assigned: Max Kanat-Alexander)

Tracking

Bugzilla 3.6
Dependency tree / graph
Bug Flags:
approval +

Details

(Whiteboard: [es-lambda])

Attachments

(2 attachments, 16 obsolete attachments)

v10
177.98 KB, patch
Max Kanat-Alexander
: review+
Details | Diff | Splinter Review
685 bytes, patch
Max Kanat-Alexander
: review+
Details | Diff | Splinter Review
(Assignee)

Description

8 years ago
It often happens that people want to switch from their current bug-tracker to Bugzilla. It would be nice to have a contrib/ script that understood various different other systems and could do an initial copy of another system into Bugzilla, for migration purposes.
(Assignee)

Comment 1

8 years ago
Created attachment 406140 [details] [diff] [review]
Work In Progress

So far, this only touches installation code, so after completion and testing, I plan to check this in as a module owner (without review). It's way too huge for anybody to have time to review, anyhow, and I can't imagine how you would test a GNATS migration (whereas I can do it pretty well, and I have a client who wants it and can verify whether or not it was done correctly).
Assignee: installation → mkanat
Status: NEW → ASSIGNED
(Assignee)

Comment 2

8 years ago
Created attachment 406152 [details] [diff] [review]
WIP 2

Further work--this now parses the history of a GNATS bug.
Attachment #406140 - Attachment is obsolete: true
(Assignee)

Updated

8 years ago
Target Milestone: --- → Bugzilla 3.6
(Assignee)

Comment 3

8 years ago
Created attachment 406183 [details] [diff] [review]
WIP 3

The architecture's starting to fall into place a bit more. This reads users and bugs, now.
Attachment #406152 - Attachment is obsolete: true
(Assignee)

Updated

8 years ago
Alias: bz-migrate
(Assignee)

Updated

8 years ago
Depends on: 522396
(Assignee)

Updated

8 years ago
Depends on: 522404
(Assignee)

Comment 4

8 years ago
Created attachment 406377 [details] [diff] [review]
WIP 4

This now fully reads bugs, users, and products (though not attachments yet).

It also fully creates users and products (though there are some problems with bugs as of yet).
Attachment #406183 - Attachment is obsolete: true
(Assignee)

Updated

8 years ago
Depends on: 522428
(Assignee)

Comment 5

8 years ago
Created attachment 406404 [details] [diff] [review]
WIP 5

This version currently is importing all bugs from my small test (18 bugs in two products), though it isn't yet handling attachments.
Attachment #406377 - Attachment is obsolete: true
(Assignee)

Comment 6

8 years ago
Created attachment 406566 [details] [diff] [review]
v1

This version is essentially complete, importing everything from the other bug-tracker and implementing all the options that I plan to. However, it needs more testing.

Also, I've discovered that running it under --dry-run increments the AUTO_INCREMENT counter on tables, because even if we rollback a transaction, we still get AUTO_INCREMENT updated. :-( So I'm going to have to write some DB methods to fix that.
Attachment #406404 - Attachment is obsolete: true
(Assignee)

Updated

8 years ago
Depends on: 522620
(Assignee)

Comment 7

8 years ago
Created attachment 406616 [details] [diff] [review]
v2

A bunch of bug-fixes over v1.
Attachment #406566 - Attachment is obsolete: true

Comment 8

8 years ago
(In reply to comment #6)
> Also, I've discovered that running it under --dry-run increments the
> AUTO_INCREMENT counter on tables, because even if we rollback a transaction, we
> still get AUTO_INCREMENT updated. :-( So I'm going to have to write some DB
> methods to fix that.

This can't be done without dangerous cheating.  In Oracle, SEQUENCE values never get back if you rollback a transaction.

Surely you're not trying to hide dry-run attempts?
(Assignee)

Comment 9

8 years ago
It can be done relatively non-dangerously, after the transaction is rolled back. At least, in MySQL and PostgreSQL it can. I assume there's a similar method in Oracle. 

The idea is that dry-run attempts shouldn't modify the database AT ALL.
(Assignee)

Comment 10

8 years ago
Created attachment 406631 [details] [diff] [review]
v3

This version fully imports the GNATS database that I am testing on.
Attachment #406616 - Attachment is obsolete: true
Attachment #406631 - Flags: review+
(Assignee)

Comment 11

8 years ago
Created attachment 406637 [details] [diff] [review]
v4

And this version also does the rollback of the auto_increment values.

xiaoou--I need an implementation of get_set_serial_sql (which you can see used and defined for other DBs in this patch) for Oracle.
Attachment #406631 - Attachment is obsolete: true
Attachment #406637 - Flags: review+
(Assignee)

Comment 12

8 years ago
CC'ing mockodin so he can know about the new Bugzilla::DB method I added.
(Assignee)

Comment 13

8 years ago
Created attachment 406638 [details] [diff] [review]
v5

And this version passes runtests, and can be checked in once the blockers are resolved.
Attachment #406637 - Attachment is obsolete: true
Attachment #406638 - Flags: review+
(Assignee)

Comment 14

8 years ago
Holding approval until blockers are resolved.
Flags: approval?

Comment 15

8 years ago
(In reply to comment #9)
> It can be done relatively non-dangerously, after the transaction is rolled
> back. At least, in MySQL and PostgreSQL it can. I assume there's a similar
> method in Oracle. 

Are you assuming (a) database being in exclusive mode during migrate.pl, or (b) dry-run can be done using read transactions only, without UPDATEs rolled back?

Comment 16

8 years ago
Created attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1
Attachment #406643 - Flags: review?(mkanat)
(Assignee)

Comment 17

8 years ago
(In reply to comment #15)
> Are you assuming (a) database being in exclusive mode during migrate.pl, or (b)
> dry-run can be done using read transactions only, without UPDATEs rolled back?

  The way that dry-run works is that it rolls back the transaction and *then* resets the serial counters to MAX(column) + 1. I assume that the database isn't under heavy production load when running under dry-run. And I do mention that --dry-run shouldn't be run on a live database.
(Assignee)

Comment 18

8 years ago
Comment on attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1

xiaoou: This looks like it will increment the sequence by $value, instead of setting the next value to $value. Am I missing something?
(Assignee)

Comment 19

8 years ago
Comment on attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1

The Oracle docs say: "To restart the sequence at a different number, you must drop and re-create it." Is that what we should be doing here instead?
(Assignee)

Comment 20

8 years ago
Created attachment 406849 [details] [diff] [review]
v6

I made some improvements to the GNATS migrator in terms of version number and username parsing.
Attachment #406638 - Attachment is obsolete: true
Attachment #406849 - Flags: review+
(Assignee)

Comment 21

8 years ago
Created attachment 406855 [details] [diff] [review]
v7

This version uses less memory during import and has some improvements in attachment parsing for GNATS.
Attachment #406849 - Attachment is obsolete: true
Attachment #406855 - Flags: review+
(Assignee)

Comment 22

8 years ago
Created attachment 406977 [details] [diff] [review]
v8

This version introduces the ability to set resolutions for particular translated statuses, and moves more code up into the generic framework from the GNATS migrator.
Attachment #406855 - Attachment is obsolete: true
Attachment #406977 - Flags: review+
(Assignee)

Comment 23

8 years ago
Created attachment 406980 [details] [diff] [review]
v9

Fix a few bugs in the GNATS migrator.
Attachment #406977 - Attachment is obsolete: true
(Assignee)

Updated

8 years ago
Attachment #406977 - Attachment description: v9 → v8
(Assignee)

Comment 24

8 years ago
Created attachment 406981 [details] [diff] [review]
v9.1

The original v9 I attached had extra local changes in it.
Attachment #406980 - Attachment is obsolete: true
(Assignee)

Comment 25

8 years ago
Created attachment 407231 [details] [diff] [review]
v10

Improve the version parsing of the GNATS migrator, and make value-mapping case-insensitive (because Bugzilla is case-insensitive about field values anyway).
Attachment #406981 - Attachment is obsolete: true
Attachment #407231 - Flags: review+
(Assignee)

Comment 26

8 years ago
Comment on attachment 406643 [details] [diff] [review]
get_set_serial_sql for oracle v1

This doesn't look like it's setting the sequence's value, it looks like it's incrementing the sequence by the value passed in, which is not what I wanted.
Attachment #406643 - Flags: review?(mkanat) → review-

Comment 27

8 years ago
Created attachment 407489 [details] [diff] [review]
get_set_serial_sql for oracle v2

Yeah, v1 was assuming that the current sequence is 0.
This v2 will set the sequence value.
Attachment #406643 - Attachment is obsolete: true
Attachment #407489 - Flags: review?(mkanat)
(Assignee)

Updated

8 years ago
Attachment #407489 - Flags: review?(mkanat) → review+
(Assignee)

Updated

8 years ago
Whiteboard: [es-lambda]
(Assignee)

Comment 28

8 years ago
Checking in Bugzilla.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla.pm,v  <--  Bugzilla.pm
new revision: 1.80; previous revision: 1.79
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/migrate.pl,v
done
Checking in migrate.pl;
/cvsroot/mozilla/webtools/bugzilla/migrate.pl,v  <--  migrate.pl
initial revision: 1.1
done
Checking in Bugzilla/DB.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB.pm,v  <--  DB.pm
new revision: 1.129; previous revision: 1.128
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate.pm,v
done
Checking in Bugzilla/Migrate.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate.pm,v  <--  Migrate.pm
initial revision: 1.1
done
Checking in Bugzilla/DB/Schema/Mysql.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB/Schema/Mysql.pm,v  <--  Mysql.pm
new revision: 1.23; previous revision: 1.22
done
Checking in Bugzilla/DB/Schema/Pg.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB/Schema/Pg.pm,v  <--  Pg.pm
new revision: 1.16; previous revision: 1.15
done
Checking in Bugzilla/Install/Filesystem.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Filesystem.pm,v  <--  Filesystem.pm
new revision: 1.40; previous revision: 1.39
done
Checking in Bugzilla/Install/Requirements.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Requirements.pm,v  <--  Requirements.pm
new revision: 1.72; previous revision: 1.71
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate/Gnats.pm,v
done
Checking in Bugzilla/Migrate/Gnats.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Migrate/Gnats.pm,v  <--  Gnats.pm
initial revision: 1.1
done
Checking in contrib/bzdbcopy.pl;
/cvsroot/mozilla/webtools/bugzilla/contrib/bzdbcopy.pl,v  <--  bzdbcopy.pl
new revision: 1.9; previous revision: 1.8
done
Removing contrib/gnats2bz.pl;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnats2bz.pl,v  <--  gnats2bz.pl
new revision: delete; previous revision: 1.10
done
Removing contrib/gnatsparse/README;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/README,v  <--  README
new revision: delete; previous revision: 1.2
done
Removing contrib/gnatsparse/gnatsparse.py;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/gnatsparse.py,v  <--  gnatsparse.py
new revision: delete; previous revision: 1.6
done
Removing contrib/gnatsparse/magic.py;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/magic.py,v  <--  magic.py
new revision: delete; previous revision: 1.1
done
Removing contrib/gnatsparse/specialuu.py;
/cvsroot/mozilla/webtools/bugzilla/contrib/gnatsparse/specialuu.py,v  <--  specialuu.py
new revision: delete; previous revision: 1.1
done
Checking in template/en/default/global/messages.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/global/messages.html.tmpl,v  <--  messages.html.tmpl
new revision: 1.93; previous revision: 1.92
done
Checking in template/en/default/global/user-error.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/global/user-error.html.tmpl,v  <--  user-error.html.tmpl
new revision: 1.287; previous revision: 1.286
done
Checking in template/en/default/setup/strings.txt.pl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/setup/strings.txt.pl,v  <--  strings.txt.pl
new revision: 1.17; previous revision: 1.16
done
Status: ASSIGNED → RESOLVED
Last Resolved: 8 years ago
Flags: approval? → approval+
Resolution: --- → FIXED
(Assignee)

Comment 29

8 years ago
And Xiaoou's patch:

Checking in Bugzilla/DB/Schema/Oracle.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/DB/Schema/Oracle.pm,v  <--  Oracle.pm
new revision: 1.12; previous revision: 1.11
done
(Assignee)

Updated

7 years ago
Keywords: relnote
(Assignee)

Comment 30

7 years ago
Added to the release notes in bug 547466.
Keywords: relnote
You need to log in before you can comment on or make changes to this bug.