10837 - [FEATURE] Import tool for 4.x address books

Status: VERIFIED FIXED

(SeaMonkey :: MailNews: Address Book & Contacts, defect, P3)

Description

[daver]

David,  where do we stand on this?

Comment 1

[davidmc]

I was supposed to start in one milestone, then finish in a later milestone. I

have not started this yet at all, though.  I am still working on sorting; I have

been syncing my source tree since last Wednesday, for a total of about four

calendar days slip, just for syncing.  I have had a long sequence of build system

failures on my Mac, due mainly to corrupt project data which seems somehow

related to xpidl changes.



After I do sorting, I can start on the ipmort tool.  Starting is easy, because it

only involves stripping things out of teh 4.5 code so it can build in a

standalone fashion.  So that is just abunch of violent hacking and stubbing for

two of three days.  Finishing the import tool is both hard and undefined; I tried

to avoid being given the task, since it is undefined.  But Phil made it clear in

no uncertain terms that he wanted me to do it.



So far, the best theory I have heard suggested is that Tony Robinson has some

kind of plugin system for import, and this might be used as the frontend.  All I

will have is a backend for import, and providing a frontend is undefined.  I will

not be able to provide an frontend by myself, and that's why I should not own the

finishing phase for the import tool.

Comment 2

[Phil Peterson]

Bulk move mail/news M10 bugs to M11

Comment 3

[Phil Peterson]

Not on the B1 list. Triage to M15 since it's probably > 1 milestone's worth of
work.

Comment 4

[davidmc]

I've started working on this task; yesterday I set up a Mac Codewarrior project
and started compiling, and triaging dependency problems on the fly.

There is very bad news, which invalidates all time estimates I have given for
this task.  I no longer know whether the task can be done, or if it can be done,
I have no idea how long it could take to remove all the problems.

The problem is the underlying third party db used by address books, because it
seems the headers (and presumably the sources) have been hacked to depend on
header files (especially basic integer types, such as those in NSPR), and I am
very strongly forbidden to look at the sources for that db anymore, on pain of
execution by electric chair, or something with a similar dramatic flavor.

Technically, it's unclear whether I am allowed to see the header files at all
either, since they contain code and I'm not supposed to see the code, even when
it might appear in a source level debugger.  My comfort zone is only seeing the
APIs used in our own code, but not seeing the definitions of APIs in the header
files, because this seems to best support the spirit of the constraint.

Someone who understands this third party db will have to go into the source which
I am not allowed to see, and remove all dependencies on header files which will
not be present in the standalone import tool project configuration.  If I did
this, I'd remove dependencies on other header files entirely, but someone else
might take another approach.

Such changes to the underlying db might have subtle effects in either linkage or
runtime behavior that I will have great trouble resolving without being able to
see the causes of the effects myself.  So I would expect some long and drawn out
handshaking in development with another engineer trying to fiddle with the db
while I am trying to use it, but without being able to see the db.

I will follow up more on this topic in another separate entry to this bug report.

Comment 5

[davidmc]

I constructed the Codewarrior project by taking a snapshot of libaddr from David
Bienvenu's 4.7 tree on my Linux box, and then also taking a snapshot of header
files only from the third party db, but without looking in these files because
I'm not allowed to see the sources.  Then I started compiling.  The libaddr files
were coming along nicely, but the files which apply the third party db had
hangups from missing header files and undefined types.  I tried whacking the
lines that include external header files from the db headers, without otherwise
looking at the db code, but this merely caused undefined types such as int32,
etc.  Now I can go back to working on the libaddr files which don't use the db,
but I don't see how I'll make much progress until the db dependencies are fixed.

Comment 6

[davidmc]

My original theory was that the db would build standalone, and that it was not
entwined with code from 4.5, but that assumption turns out to be wrong.  The
degree of dependency is probably very, very small, such as the addition of new
methods overloaded to deal with native types specific to NSPR data formats; but
any degree of dependency is a showstopper when I have an absolute injunction not
to see the sources.

Comment 7

[David :Bienvenu]

It turns out that the address book code doesn't use those methods (read and
writeint32). I can remove them from my copy of those files and you can re-ftp, I
guess.

Comment 8

[davidmc]

My first impulse was to declare this task impossible, and write off the
possibility of importing 4.x family address books to mozilla.  However, that's
too pessimistic an outlook.  It is still possible to do, it's just much harder.
But I don't know how hard; I get very uncomfortable when there is no way to say
whether a final success state is even reachable.  There is some potential for the
task to be a time swamp that eats time and never reaches a conclusion.

My reponse to such risk is to write off the task as undoable to control risk.
But perhaps someone else would like to buy the risk and play craps themselves, as
long as it does not require that I must also take responsibility for the outcome.

Comment 9

[David :Bienvenu]

OK, I've removed the int32 stuff from my tree
~client45/ns/lib/libneo/nstream.h,cpp

Comment 10

[davidmc]

On the Mac there were more missing header files than just the NSPR integer types.
It also seemed that some header file related to AppleScript constants was being
included, and whacking this include caused some contants to become undefined when
some other constants resemling AppleScript selectors could not be found.

I worry that there might be more such things, and finding them involves looking
at the header files, which feels questionable to me.

Comment 11

[davidmc]

I will post summaries of error messages generated by the Mac compiler.

Comment 12

[davidmc]

I've decided my approach was going down a blind alley, and now
I'm going to cut my losses and change the general strategy.

Now the goal is to generate an ldif file, and this ldif file
can be imported by Mozilla 5.0 code or associated tools; the
code to write this ldif in the 4.5 style is already present
in the old 4.t5 codebase (which is where I started from)
and requires much less massaging to make work for the current
problem.  We can still have auto-import be very auto-magic
for users, by hiding that this path uses a two step process.

So I'm going to scrap my recent work, which was going in the
direction of generalized import to any destination MDB database.
A lot of earlier work I did on this task is still valid which
does not involve the MDB stuff in particular; but making this
change involves eating the cost of MDB import done so far.

Eating this cost and switching to an ldif text-based data flow
path looks cheaper though, since I still had a bunch of work
to do with the direct MDB import, especially with debugging
and integration.  Basically the ldif path is much close to the
floor in terms of complexity, and has low integration risks,
although it is generally no efficient for large import content.

Another major risk in the MDB import approach I was using, was
that handing over this code to someone else had greater chance
for confusion over what was meant by the import architecture.
In contrast, making a 4.5 style ldif file has obvious results.

I talked with Candace and she says mozilla has at least first
draft versions of code for importing ldif to address books, and
that this might be used by Tony's tools for importing.   The
new approach I plan to use for creating ldif should fit right
into other data migration paths he must already handle.

Comment 13

[davidmc]

assigning to bienvenu, so he can pass it on to whoever owns the stuff I did.

Comment 14

[David :Bienvenu]

accepting

Comment 15

[(not reading, please use [e-mail] instead)]

adding me to the cc list.

Comment 16

[(not reading, please use [e-mail] instead)]

*** Bug 30190 has been marked as a duplicate of this bug. ***

Comment 17

[(not reading, please use [e-mail] instead)]

I just marked a migration bug a duplicate of this one.

I think this bug covers the automatic migration of addressbooks issue.

david b, since I'm the migration lackey, would you like to pawn this bug off on
me?

I wouldn't hold it against you.  At least, not for long.

Comment 18

[David :Bienvenu]

I still have to get DavidMc's stuff to build and run on mozilla for windows, 
linux, and Mac, checked into CVS, with project and make files - this will be a 
week's work, probably, so it'll be a while before it's in any state to get 
pawned off :-(

Comment 19

[David :Bienvenu]

giving to Seth to administer the coup de grace to this bug. CC'ing JF who kindly 
volunteered to help with the mac project.

Comment 20

[(not reading, please use [e-mail] instead)]

accepting.  I'll work on this tomorrow.

Comment 21

[(not reading, please use [e-mail] instead)]

I've got this working.

there are some polish issues that need to be finished.

I'll show david b what I've got tomorrow or thursday, and start checking it in.

Comment 22

[(not reading, please use [e-mail] instead)]

automatic migration of 4.x addressbooks works.
import of 4.x addressbooks works.

it only works in the commercial build.

ducarroz is going to turn it on for the mac tonight.

marking fixed.

Comment 23

[Jean-Francois Ducarroz]

Fixed on Mac as well.

Comment 24

[ppandit]

Using 6/27 release build on nbaca's NT system

Able to import address book entries from a 4.7 profile. 

Par