Closed Bug 749586 Opened 12 years ago Closed 11 years ago

Import bugzilla.instantbird.org into bugzilla.mozilla.org

Categories

(bugzilla.mozilla.org :: Administration, task)

Product:
bugzilla.mozilla.org
Component:
Administration
Version:
Production
Type:
task
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: clokep, Unassigned)

References

(
URL
)

Details

I'm filing this bug per the email conversation (between myself, Florian, Gerv, Mark Banner and David Bienvenu) about how to manage the chat/ code that is now in comm-central and filing bugs against it.

The chat/ code that is now in Thunderbird was originally developed for Instantbird and its bugs are currently tracked in bugzilla.instantbird.org (BIO). As the chat/ code is actively developed and not treated as an upstream repository (e.g. like sqlite) and is user facing, it is expected that testers and users will file bugs against it. In order to avoid a large burden on not only Instantbird and Thunderbird developers, but also on users and testers, it would seem prudent to merge the Bugzilla instances.

There are a variety of reasons for this:
 - It isn't really appropriate to lands things in c-c with commit messages pointing to the Instantbird Bugzilla. This has caused the instantbird and comm-central repositories to diverge.
 - It is difficult for developers to work across two Bugzilla instances (especially when bugs might depend on each other), in fact there have already been several duplicate bugs filed in bugzilla.mozilla.org (BMO) that were already filed in bugzilla.instantbird.org. Although we can link to these other bugs, it is not as efficient as being able to mark it as a duplicate as the conversation about fixing the bug is split across two locations.
 - Testers and users should not be expected to use a different Bugzilla to file bugs for different components of a product (e.g. if all chat/ bugs were filed in BMO but other Instantbird bugs were filed on BIO; or if all chat/ bugs stayed in BIO).
 - By having components split across two Bugzillas, it becomes impossible to move bugs between components. E.g. many of our bugs that are caused by a misbehaving protocol get filed under "Conversation" or "Contact list", as that's where the user sees the issue occur; these bugs would need to be in a shared "chat" component.

One concern about doing this is we would like the comments with bug and attachment references in the Instantbird Bugzilla to automatically be re-linkified to the new correct bug number.

Florian suggested the following could occur to map BIO components into BMO:
 - Instantbird/* -> Client Software/Instantbird/*
 - Core/* -> Components/Chat Core/* (some of the bugs currently filed in Thunderbird/Instant Messaging would be moved/duped here)
 - Websites/* -> Other/Instantbird Servers/*
 - Addons/* -> Client Software/Instantbird/Demo add-ons

Additionally, would it be possible to have an "Instantbird" product listed between Calendar and Camino on this page: https://bugzilla.mozilla.org/enter_bug.cgi?

What needs to be done technically and administration wise to make this happen? I'd like for this to happen before the next merge date; as end-users will be able to turn on IM in Thunderbird at that point via a hidden pref (although it won't be on by default) so I'm expecting a deluge of bugs to be filed. Thanks!
dkl and glob: I asked for this bug to be filed so that you could chip in with your expertise on whether and how it would be possible to merge the InstantBird Bugzilla into ours.

My thoughts were that we could perhaps use the migration framework that Max wrote to write a Bugzilla -> Bugzilla migrator. It shouldn't be too complex and I'm sure other people would find it very useful. (We often get asked how to merge 2 Bugzillas.) We would get massive bonus points for updating bug and attachment references to reference the new numbers in b.m.o.

Gerv
i disagree that "it shouldn't be too complex"; this is a reasonably complicated task due to the different ways bugzilla can be configured and extended via bugzilla addons.

i've started a brain dump on the items we'll need to pay attention to at https://bmo.etherpad.mozilla.org/bugzilla-instantbird-merge
URL: https://bmo.etherpad.mozilla.org/bugz...
Patrick, yeah maintaining a separate Bugzilla instance sounds like a pain. However, as glob pointed out, this would be nontrivial. The bmo team is currently very busy with a couple high priority tasks, the most important being some large changes to the release-flag system, which in its current implementation is causing increasing performance degradation. It is imperative that we fix this before the end of the quarter, along with a couple other important tasks like upgrading Bugzilla to 4.2. We should be able to get to your migration after these are out of the way, but I think it is unlikely to occur before Q3.

However in the meantime we could create the necessary products and components in bmo. Is it feasible for you to then turn off write access to the Instantbird Bugzilla, or at least direct people to bmo? I realize you wouldn't have any back history in bmo, but at least new users could start filing bugs there.
(In reply to Mark Côté ( :mcote ) from comment #3)
> Patrick, yeah maintaining a separate Bugzilla instance sounds like a pain.
> However, as glob pointed out, this would be nontrivial.
I'm beginning to see that on the etherpad!

> The bmo team is
> currently very busy with a couple high priority tasks, the most important
> being some large changes to the release-flag system, which in its current
> implementation is causing increasing performance degradation. It is
> imperative that we fix this before the end of the quarter, along with a
> couple other important tasks like upgrading Bugzilla to 4.2.
Understood, I was just trying to get across our "best case" scenario -- I definitely understand there are priorities.

> We should be
> able to get to your migration after these are out of the way, but I think it
> is unlikely to occur before Q3.
Well it seems that IM-in-TB will be in Thunderbird 15 at the earliest: released on 2012-08-28 and in beta on 2012-07-17. That falls into Q3...it'd be great to get it done sometime before this time frame, if possible! :)

> However in the meantime we could create the necessary products and
> components in bmo. Is it feasible for you to then turn off write access to
> the Instantbird Bugzilla, or at least direct people to bmo? I realize you
> wouldn't have any back history in bmo, but at least new users could start
> filing bugs there.
I'll need to talk to Florian about this. Personally I'd prefer migrating it all at once, I think. But it might be best to at least add a chat core component.

Thanks for the help and support! :)
Patrick/Florian: can you re-review https://bmo.etherpad.mozilla.org/bugzilla-instantbird-merge ? There are a few questions there for you.

Gerv
(In reply to Gervase Markham [:gerv] from comment #5)
> Patrick/Florian: can you re-review
> https://bmo.etherpad.mozilla.org/bugzilla-instantbird-merge ? There are a
> few questions there for you.

Done, thanks for the heads up.
Patrick, thanks for your understanding. :) We will schedule this for beginning of Q3 or very end of Q2 if we get our other priorities out of the way. Let us know if we can create any components for you in the mean time (not sure which, if any, of the the ones listed in the etherpad would be useful in the near future--we can create any or all of them).
(In reply to Mark Côté ( :mcote ) from comment #7)
> Patrick, thanks for your understanding. :) We will schedule this for
> beginning of Q3 or very end of Q2 if we get our other priorities out of the
> way.
Of course!

> Let us know if we can create any components for you in the mean time
> (not sure which, if any, of the the ones listed in the etherpad would be
> useful in the near future--we can create any or all of them).
Will do. :)

Something Florian and I discussed briefly, which I'm unsure if it would even be wanted...there's a variety of bugs in BMO with links to BIO bugs and vice versa. It *could* be desirable to change those to directly point to the new bug links (i.e. http://bugzilla.instantbird.org/show_bug.cgi?id=123 --> http://bugzilla.mozilla.org/show_bug.cgi?id=xxx or even better: bug xxx; where xxx is the new bug number).

This would involved editing "old" comments though which is probably not desirable, but just thought I would bring this up.
(In reply to Mark Côté ( :mcote ) from comment #7)
> Patrick, thanks for your understanding. :) We will schedule this for
> beginning of Q3 or very end of Q2 if we get our other priorities out of the
> way.

Mark, I was just curious if there is any update on this?  (I figured it was a good time to check as it is now mid-Q3. If there are still other priorities, I understand...but of course we'd like to get this as soon as possible to avoid more bugs like bug 753807, bug 776901 and bug 779117.) Thanks!
Ah sorry, we had more carry-over work from Q2 than expected. I'll try to schedule something definite.
Yeah, so, deploying 4.2 is still our main priority and hit some snags (see blog posts from glob), so that work is probably going to carry us to the end of Q3. However we don't have too many high-priority items for Q4, so we'll try to get on that as soon as we can in October. Many apologies for the delays.
(In reply to Mark Côté ( :mcote ) from comment #11)
> However we don't have too many high-priority items for Q4, so we'll
> try to get on that as soon as we can in October. Many apologies for the
> delays.

My quarterly ping about this bug! :) How's Q4 looking?

For reference, Thunderbird 15 with chat support has now been released and...it is getting a bit confusing trying to mark duplicates across both Bugzillas, so we'd really love this soon!
Hey sorry for the delay. I believe you already talked to Byron, but we are *hoping* he can get to it in November or December, after some B2G work and the Bugzilla 4.2 upgrade. Will let you know as we get closer.
I saw that it looks like the Bugzilla 4.2 upgrade will be happening soon (March 5th or something?); has there been further thought given to this bug?

I briefly talked to glob about this in #bmo earlier this week...he had mentioned it was discussed but could not remember the results and that mcote might know. Thanks!
Many apologies that this whole process has dragged out so long.  I think the various delays in deploying 4.2 are indicative of the neglect that BMO suffered for so long before Byron and Dave took over, and there remains a lot to be done, much of it planned a year or two ago.  Given the amount of work required and that it would be largely a single-use project, I'm forced to admit that we don't really have the resources to do a full migration as you requested, and I don't want to continue to punt on you indefinitely.

If you have some development time, however, there is still a way for you to switch over to BMO in a relatively painless way.  After we create products, components, and such, you could write a script that uses the bzAPI REST interface to create bugs on BMO and thus copy the Instantbird bugs over.  You wouldn't be able to keep the timestamps on individual comments, but you could include the timestamp and user in the text of the new comment (which would also remain searchable).  Such an approach was actually used recently to migrate B2G github issues to BMO bugs.  It shouldn't be too difficult and you wouldn't be blocked on us.  We'd be happy to offer support of course.

Again sorry for taking so long to reach this conclusion.
if it helps, i have code which performed a github to bmo import at https://gist.github.com/globau/3774581
(In reply to Mark Côté ( :mcote ) from comment #15)
> Given the amount of
> work required and that it would be largely a single-use project, I'm forced
> to admit that we don't really have the resources to do a full migration as
> you requested, and I don't want to continue to punt on you indefinitely.
I understand the decision, although I am disappointed in it. Thanks for being forthcoming and honest.

> If you have some development time, however, there is still a way for you to
> switch over to BMO in a relatively painless way.  After we create products,
> components, and such, you could write a script that uses the bzAPI REST
> interface to create bugs on BMO and thus copy the Instantbird bugs over. 
> You wouldn't be able to keep the timestamps on individual comments, but you
> could include the timestamp and user in the text of the new comment (which
> would also remain searchable).
The timestamps don't bother me, but the user is kind of annoying (you can't search easily for "bugs I filed", etc.) I assume there's no alternative to this? (You can't create a bug / comment as a different user in the BzAPI, most likely.)

Would we still be able to do cross-referencing of bugs and attachments using this? I.e. we'd like "bug xyz" to correctly link to "bug abc" after the import. (I can guess that if you're careful about the order you recreate things in you can maintain an internal map of all bugs / attachments and do it that way.)
(In reply to Patrick Cloke [:clokep] from comment #17)
> The timestamps don't bother me, but the user is kind of annoying (you can't
> search easily for "bugs I filed", etc.) I assume there's no alternative to
> this? (You can't create a bug / comment as a different user in the BzAPI,
> most likely.)

correct - none of the bugzilla APIs (not the native xmlrpc/json/jsonp, nor the external bzapi proxy) allow you to perform actions as another user.

> Would we still be able to do cross-referencing of bugs and attachments using
> this? I.e. we'd like "bug xyz" to correctly link to "bug abc" after the
> import. (I can guess that if you're careful about the order you recreate
> things in you can maintain an internal map of all bugs / attachments and do
> it that way.)

while i can't speak for how the bzapi works, when you file a bug using the xmlrpc api, the bug id is returned, so you could use that to build a mapping table and update comments.

one option is to not turn off bugzilla.instantbird.org, rather switch it into a read-only mode.  migrate just open bugs, with a single comment which refers to instantbird's bug entry.
I was also dreaming of writing some code here, but I think mcote's honesty should be mine: I'm not going to get around to it either.

(In reply to Patrick Cloke [:clokep] from comment #17)
> The timestamps don't bother me, but the user is kind of annoying (you can't
> search easily for "bugs I filed", etc.) I assume there's no alternative to
> this? (You can't create a bug / comment as a different user in the BzAPI,
> most likely.)

If there was a small group of people who really wanted this info to be correct, and one person they all trusted, you could get them all to change their Bugzilla passwords to something temporary, give the temporary password to the migration person, and then change it back later.

As others have suggested, a little bit of careful analysis should allow you to update most bug references (if two bugs reference each other, you'll have a problem).

It also seems to me that migrating only open bugs may be the right way forward. You could knock up a simple BzAPI-using web page which allowed you to search across both Bugzillas at once. (I'm happy to provide an instance of BzAPI pointed at bugzilla.instantbird.org on the api-dev machine; just ask.)

Gerv
(In reply to Byron Jones ‹:glob› from comment #18)
> correct - none of the bugzilla APIs (not the native xmlrpc/json/jsonp, nor
> the external bzapi proxy) allow you to perform actions as another user.

Wrong! You can export/import bugs from one Bugzilla installation to another one using importxml.pl. Timestamps are not altered, same for the original users (reporter, assignee, commenters, etc...).
(In reply to Frédéric Buclin from comment #20)
> Wrong! You can export/import bugs from one Bugzilla installation to another
> one using importxml.pl. Timestamps are not altered, same for the original
> users (reporter, assignee, commenters, etc...).

importxml.pl isn't an api.
(In reply to Byron Jones ‹:glob› from comment #21)
> importxml.pl isn't an api.

But this bug is not about the existence of some API to do what Patrick asks. This bug is about it's possible to import BIO bugs into BMO without loosing users as much as possible, and the answer is yes: use importxml.pl.
(In reply to Frédéric Buclin from comment #22)
> (In reply to Byron Jones ‹:glob› from comment #21)
> > importxml.pl isn't an api.
> 
> But this bug is not about the existence of some API to do what Patrick asks.
> This bug is about it's possible to import BIO bugs into BMO without loosing
> users as much as possible, and the answer is yes: use importxml.pl.

Byron/Mark: Would this actually help / is this an option?
(In reply to Patrick Cloke [:clokep] from comment #23)
> Byron/Mark: Would this actually help / is this an option?

not really, because the bulk of the work in in data mapping .. basically generating the xml which could then be imported.
glob: is the data mapping something the InstantBird team could do? So all the BMO team had to do would be to import the data?

Gerv
(In reply to Gervase Markham [:gerv] from comment #25)
> glob: is the data mapping something the InstantBird team could do? So all
> the BMO team had to do would be to import the data?

i don't think that avenue is the best approach as we unfortunately don't have the time to provide guidance for a direct database import, nor spend time doing qa and trial runs of the import.  no one on the bmo team have used import.xml, so there will be lead time in getting us to a point where we're happy with taking that approach. for a single one-off import of a small dataset, it's proven hard to find the time to allocate to it.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WONTFIX
So it looks like the best way forward for the IB team is:

- Get me to make a BzAPI instance for your Bugzilla
- Write a script which queries for a subset of bugs in bugzilla.instantbird.org that you still care about and re-files them in bmo, in the best order possible for updating inter-bug references
- You can test your script by setting up your own bmo instance using the publicly-available bmo data dump
- For bonus points, get some key contributors to use temporary passwords for 24 hours so when you do the migration you can preserve some of the "reporter" and "commenter" information correctly

I'm sorry we can't be more help :-|

Gerv
(In reply to Gervase Markham [:gerv] from comment #27)
> So it looks like the best way forward for the IB team is:
> 
> - Get me to make a BzAPI instance for your Bugzilla
I'm unsure if I'll be using BzAPI vs. the built in JSONP interface (after looking a little bit, I don't see what makes BzAPI any better, but if someone wants to explain this to me, feel free to ping me in #instantbird). I'll let you know if I'd like this set up.

> - Write a script which queries for a subset of bugs in
> bugzilla.instantbird.org that you still care about and re-files them in bmo,
> in the best order possible for updating inter-bug references
> - You can test your script by setting up your own bmo instance using the
> publicly-available bmo data dump
> - For bonus points, get some key contributors to use temporary passwords for
> 24 hours so when you do the migration you can preserve some of the
> "reporter" and "commenter" information correctly
> 
> I'm sorry we can't be more help :-|
Thanks for trying / looking at this.

Would I need to file a bug / use this bug when I'm ready to actually perform this merge or just go ahead and do it? (Would I be rate-limited / killed in any way as a "spam" bot?)
Resolution: WONTFIX → FIXED
On the production bzAPI, at least, you shouldn't have any problems, but a friendly ping in #bteam would be appreciated. :)
Depends on: 951944
You need to log in before you can comment on or make changes to this bug.