119360 - addressbook export to ldif -> import is lossy

Status: VERIFIED FIXED

(SeaMonkey :: MailNews: Address Book & Contacts, defect, P2)

Description

[(not reading, please use [e-mail] instead)]

addressbook export to ldif  -> import is lossy

(or it will be once I check in export to ldif)

more details coming soon.

Comment 1

[(not reading, please use [e-mail] instead)]

note export to .csv,.tab,.txt -> import is also lossy, but that is going to be 
a wontfix.  I'll log a new bug on that soon.

Comment 2

[(not reading, please use [e-mail] instead)]

here's what we are losing when we go ab -> ldif -> ab

home address 2
home city
home state
home country
home zip

work address 2

additional email

Comment 3

[(not reading, please use [e-mail] instead)]

we can (somewhat) avoid the loss of home address 2 and work address 2.

when writing out home address 1, we could write out:
home address 1 + home address 2

when writing out work address 1, we could write out:
work address 1 + work address 2

That would mean on AB -> LDIF -> AB, we'd not come out with the exact same, but 
it would be less lossy.

we'd still need figure out these attributes:

home city
home state
home country
home zip

additional email

Comment 4

[(not reading, please use [e-mail] instead)]

see related bug http://bugzilla.mozilla.org/show_bug.cgi?id=119420

Comment 5

[Hakon]

work phone (also)

In my case, I've noticed one single contact to have only the first row in
/other/notes persisting.

Comment 6

[Hakon]

Another thing that happens now and then is that the first contact of the
imported address book is a blank one. Because of that, there's then one contact
missing from anywhere of the original address book. Isn't this a "major"?
(2002012103)

Comment 7

[Ninoschka Baca]

Trunk build 2002-02-06-03: WinMe
To summarize the fields with missing data:

- additional email address
- work Phone
- home address2
- home city
- home state/province
- home zip
- home country
- work address2

Comment 8

[Ralf Angeli]

Regarding the work phone, this seems to be a simple typo in the export function.
In the generated ldif file the respective field is called "telehoneNumber" and
not "telephoneNumber". As a workaround, doing a search and replace with these
strings lets you import the entries for work phone correctly.

Comment 9

[Ninoschka Baca]

*** Bug 129532 has been marked as a duplicate of this bug. ***

Comment 10

[Hakon]

Notes is also lossy!
To reproduce:
1. Create a new address book
2. Create a new card
3. Enter in notes:
line1 [shift+enter]
line2 [shift+enter]
line3 [shift+enter]
line4 [shift+enter]

4. Close; Export the address book
5. Import

Result: Only "Line1" remains.

Comment 11

[Ninoschka Baca]

*** Bug 136358 has been marked as a duplicate of this bug. ***

Comment 12

[Ninoschka Baca]

Marking nsbeta1 because it would be nice to have one format that
exported/imported correctly. This will atleast place it on the radar to be fixed
at some point.

Comment 13

[Rich Ibbotson]

Attachment: proposed fix

The attached patch should fix the lost fields issue and the truncation of the
notes one.  Since the lost fields aren't described in RFC2798 and weren't in
4.x, I had to create new attribute names along the lines of the existing ones:

home city: homeLocality
home state: homeSt
home zip code: homePostalCode
home country: homeC
home address 2: homePostalAddress2
work address 2: postalAddress2
additional e-mail: secondaryMail

This doesn't address the "blank first contact" issue mentioned in comment 6
above.	Has that been reproduced?  I haven't seen it happen in 0.9.9.

Comment 14

[Hakon]

Is this bug going to be fixed till 1.0? After all, it's quite severe.

Comment 15

[Dan Meek]

I hope that Seth is communicating with [e-mail] on this, as cavin has 
been working on what appears to be the same problem.

This bug is crippling to the potential success of Mozilla.  One of the first 
things a user will try to do is import her address book from Netscape 4.x.  Upon 
finding a large fraction of the lists gone, and the remaining lists consisting 
often of either too few addresses or entirely wrong addresses, the natural 
reaction will be to consider Mozilla "not ready for prime time," to say the 
least.  I have wanted to switch to Mozilla for a year but cannot, due to this 
problem and the image it creates in my mind of a program that is potentially 
damaging to my data.

Comment 16

[Ninoschka Baca]

*** Bug 140987 has been marked as a duplicate of this bug. ***

Comment 17

[Scott MacGregor]

Cavin, does this bug look like a dup of one you are already working on (per the
comment #15? If so can we mark this a dup of that? 

Comment 18

[(not reading, please use [e-mail] instead)]

I'll start with the patch, from a quick glance, the patch looks like a good 
start. (nice work, [e-mail])

I'm working on a related dataloss bug, also.

Comment 19

[(not reading, please use [e-mail] instead)]

I think 1.0 is too soon, they won't take this.  for starters, let's say 1.1

Comment 20

[(not reading, please use [e-mail] instead)]

see related-to-mutliple-lines-in-notes-bug #115906

working on a fix for that, too.

Comment 21

[Cavin Song]

No, this bug is different from bug 62084 (comment #15) in that 62084 deals with
import of ldif files only, whereas this bug makes sure that when we export
addresses to ldif files and then import them back into addrbook we don't lose
info like home address, additional e-mail etc. This is how I understand it.

Comment 22

[(not reading, please use [e-mail] instead)]

Attachment: patch, for the exporting of notes issue. (strings with \r or \n are not LDIF safe "as is"). based on contributors patch.

Comment 23

[(not reading, please use [e-mail] instead)]

the next step is to take care of the second issue in the contributors patch.

we need to fix export / import to handle the additional fields, while trying 
not to break backwards compatability with 4.x and existing LDIF standards.

Comment 24

[David :Bienvenu]

Comment on attachment 82309 [details] [diff] [review]
patch, for the exporting of notes issue.  (strings with \r or \n are not LDIF safe "as is").  based on contributors patch.

sr=bienvenu

Comment 25

[Dan Mosedale (:dmosedale, :dmose)]

Comment on attachment 82309 [details] [diff] [review]
patch, for the exporting of notes issue.  (strings with \r or \n are not LDIF safe "as is").  based on contributors patch.

r=dmose

Comment 26

[(not reading, please use [e-mail] instead)]

http://bugzilla.mozilla.org/attachment.cgi?id=82309&action=view has landed on
the trunk.

next step, the other part of the proposed fix.

Comment 27

[Dan Mosedale (:dmosedale, :dmose)]

Attachment: example LDIF export from 4.x

Just for reference, I created an addressbook card in 4.x and with every field
filled in (though the netscape conference dialog was modal to itself, so a few
fields there may be missing, but I don't think that stuff even exists any
more).
Here's the LDIF export from 4.x

Comment 28

[Dan Mosedale (:dmosedale, :dmose)]

FWIW, our long-term LDAP schema thinking has been going on in bug 116692 and bug
118454.

For now, we clearly need some sort of ad-hoc extension here that will work
reasonably for the rest of the Mozilla data without wrecking backword and
forward compatibility, and giving us the flexibility to define a real schema
once we're not so rushed.  

One thing I'd propose doing is adding an objectclass to the current export data
to make it possible for importers to understand the version of this file. 
Perhaps "objectclass=mozillaTransitionalPerson".  Then someday we obsolete this
in favor of "objectclass=mozillaPerson".

Whether or not we should qualify all attributes that are new since 4.x as
xmozilla*, I'm not sure.  mcs, do you have any thoughts on this?

Comment 29

[Mark Smith [:mcs]]

I think it is a good idea to use a prefix when naming proprietary attributes
that do not appear (and are unlikely to appear) in standards documents such as
RFCs. I am not sure xmozilla would be my choice, but it is somewhat arbitrary
what you use. Maybe mozilla-ab- (for mozilla address book)?

Comment 30

[(not reading, please use [e-mail] instead)]

double checking what dmose saw:

when we export from 4.x to LDIF, and then import that LDIF into mozilla,
everything comes in.  (expect the conference stuff, that we no longer support)

One issue is that the url from 4.x (homeurl, in ldif) comes in as the home URL,
but the rest of the fields are treated like work fields.  I've logged a bug for
this, see http://bugzilla.mozilla.org/show_bug.cgi?id=142634

back to the real issue, mozilla: ab -> ldif -> ab, I'll come up with a patch for
this, following the original patch, and the suggestions by dmose and mcs.

Comment 31

[(not reading, please use [e-mail] instead)]

Attachment: patch, fix the lossy ness, when going from mozilla -> ldif -> mozilla, add new objectclass as well. again, based on contributor's patch.

Comment 32

[(not reading, please use [e-mail] instead)]

Attachment: output from mozilla

Comment 33

[(not reading, please use [e-mail] instead)]

Take a look at http://bugzilla.mozilla.org/attachment.cgi?id=82599&action=view 
to see a full card, once exported to ldif.

Here's what new:

objectclass: mozillaAbPersonUnsupported
mozilla-ab-mail2: email2
mozilla-ab-homePostalAddress2: a2
mozilla-ab-homeL: a3
mozilla-ab-homeSt: a4
mozilla-ab-homePostalCode: a5
mozilla-ab-homeC: a6
mozilla-ab-postalAddress2: b2

the object class, the prefix ("mozilla-ab-") and even the attribute (in case 
someone has an objection to homeL or homeC) are easily changed, so let me know 
if you have objections.

Comment 34

[(not reading, please use [e-mail] instead)]

got email from mcs, he's ok with this approach.

Comment 35

[Dan Mosedale (:dmosedale, :dmose)]

Comment on attachment 82598 [details] [diff] [review]
patch, fix the lossy ness, when going from mozilla -> ldif -> mozilla, add new objectclass as well. again, based on contributor's patch.

r=dmose; with two caveats:

I suggest we use "Obsolete" rather than "Unsupported" for the object class.

mcs: in the standards stuff I've seen, the convention seems to be interCaps for
attribute name.  Is there some reason to prefer hyphens (ie "mozilla-ab-") over
that (ie "mozillaAb")?

Comment 36

[David :Bienvenu]

Comment on attachment 82598 [details] [diff] [review]
patch, fix the lossy ness, when going from mozilla -> ldif -> mozilla, add new objectclass as well. again, based on contributor's patch.

sr=bienvenu

Comment 37

[Mark Smith [:mcs]]

Intercaps vs. hyphens is an arbitrary choice. I personally have no strong
preference, but you are right that most of the standard attribute types and
object classes use intercaps.

Comment 38

[Roland Felnhofer]

Actually I don't see any reason and benefit in 're-inventing the wheel' by
redefining the missing attributes for export and import when there is anyhow
already a proposed schema?

objectclass: mozillaAbPersonUnsupported
mozilla-ab-mail2: email2		-> xMozillaSecondEmail
mozilla-ab-homePostalAddress2: a2	-> (not defined so far) xMozillahomePostalAddress2
mozilla-ab-homeL: a3			-> xMozillahomeLocalityName
mozilla-ab-homeSt: a4			-> xMozillahomeStreet
mozilla-ab-homePostalCode: a5		-> xMozillahomePostalCode
mozilla-ab-homeC: a6			-> xMozillahomeCountryName
mozilla-ab-postalAddress2: b2		-> (not defined so far) xMozillapostalAddress2

I have no strong opinions about writing an upper or lower case 'home/Home'.
Also I would go with Seth's proposal to, when writing out home address 1 and 
work address 1 to combine them.

home address 1 + home address 2 = homePostalAddress
work address 1 + work address 2  = streetAddress

That would help us to introduce (or better reduce) proprietary attributes.

With my proposal we would not conflict with what is already proposed in
Bug#:118454 (obviously not ;-)). Further more we would already work in that
direction.

Downside of my proposal?

Comment 39

[Kristof Petr]

Im on Roland's side. He did great work on last 1/2 year
and his work is ignored. No reason to temporary lasttime workarounds.

We should use attributes from #116692, what will not change (I hope).

I guess people from #119360 accidentaly missed #116692, so they did propose
their own solution.

Comment 40

[(not reading, please use [e-mail] instead)]

>Actually I don't see any reason and benefit in 're-inventing the wheel' by
>redefining the missing attributes for export and import when there is anyhow
>already a proposed schema?

I'm ok with your suggestions.  Let me check with dmose.

Note, your proposal has some interCaps inconsistencies, and we've switched from 
Unsupported to Obsolete.

Paying close attention to intercaps, is this what you are suggesting:

objectclass: mozillaAbPersonObsolete
xMozillaSecondEmail
xMozillaHomePostalAddress2
xMozillaHomeLocalityName
xMozillaHomeStreet
xMozillaHomePostalCode
xMozillaHomeCountryName
xMozillaPostalAddress2

> Also I would go with Seth's proposal to, when writing out home address 1 and 
> work address 1 to combine them.
> home address 1 + home address 2 = homePostalAddress
> work address 1 + work address 2  = streetAddress
> That would help us to introduce (or better reduce) proprietary attributes.

There's a problem with this idea.  The problem is that if you export to LDIF 
and then re-import, you won't get the same thing.  The mozilla addressbook UI
makes a distinction between address1 and address2.  If we combine them when 
going to ldif, and then re-import, it's problematic.  But, exporting to a single
filed (homePostalAddress or streeAddress) would allow other applications to 
import our LDIF easier.  But, those other applications could easily handle our 
LDIF, and our mozillaAbPersonObsolete object class.  Let's take this issue to 
another bug.  For now, I want to leave the address fields as two attributes.

I'll prepare a new patch, for review.

Comment 41

[(not reading, please use [e-mail] instead)]

mcs writes:  "I am not sure xmozilla would be my choice, but it is somewhat 
arbitrary what you use."

Let me check with mark as well.

Comment 42

[Mark Smith [:mcs]]

This document will soon be an RFC:

http://www.ietf.org/internet-drafts/draft-ietf-ldapbis-iana-05.txt

It specifies registration procedures for various LDAP extensions, including
schema. It says the names that begin with x- are reserved for experiments and
will never be registered... which implies that x-mozilla would be a good prefix
to use. Historically, people how have defined new LDAP schema have not used x-
though; they have just picked a prefix "out of the air." For example, for
Netscape server products, we use a prefix of "ns" (kind of short, I know). I
think there are two reasonable choices for a prefix:

a) x-mozilla

b) mozilla

If you want to be as standards compliant as possible, choose a). If you are
happy to do something similar to what everyone else has historically done,
choose b) because it is simpler.

Comment 43

[(not reading, please use [e-mail] instead)]

from that RFC:

  Descriptors beginning with "x-" are for Private Use and SHALL NOT be
  registered.

  Descriptors beginning with "e-" are reserved for experiments.  IANA
  SHALL register any descriptor beginning with "e-" on a First Come
  First Served basis.

Note, I'm not sure what it means to register a descriptor.  Our attribute names 
aren't experimental, but they are private.

before submitting a final patch, I'd like to hear what dmose has to say.

Comment 44

[Mark Smith [:mcs]]

A descriptor can be, among other things, the name of an LDAP schema element
(attribute type or object class). From that same RFC-to-be:

> LDAP allows short descriptive names (or descriptors) to be used
> instead of a numeric Object Identifier to identify protocol extensions
>[RFC2251], schema elements [RFC2252], LDAP URL [RFC2255] extensions,
> and other objects.  Descriptors SHALL be restricted to UTF-8
> case-insensitive strings ...

They can be registered with the Internet Assigned Numbers Authority (IANA) but
registration involves a review process within the IETF... and you do not want to
deal with that for these attributes and OCs.

Comment 45

[Roland Felnhofer]

Seth, yes that's actually what I want to propose! Sorry I did a little bit of a
dirty copy-and-past job so I messed up the interCaps. Thank you for preparing a
new patch.

In respect of  xMozilla(Home)PostalAddress2 I see the reason and agree with you
in respect of  Mozilla import - export as well as the benefit we would gain for
other applications by getting rid of  xMozilla(Home)PostalAddress2.

In respect of  'x-mozilla' versus 'mozilla' I would prefer 'mozilla'. Simply
because it looks better.

Comment 46

[Dan Mosedale (:dmosedale, :dmose)]

I prefer mozilla*, I think.  Not only does it look neater and fit with existing
convention, if we someday wanted to register these attributes, we could. 
Whereas using x-mozilla would specifically prevent that.

Comment 47

[(not reading, please use [e-mail] instead)]

ok, mozillaFooBar it is.  new patch coming soon...

Comment 48

[(not reading, please use [e-mail] instead)]

Attachment: fix

new patch

Comment 49

[(not reading, please use [e-mail] instead)]

here's the new output:

objectclass: mozillaAbPersonObsolete
mozillaSecondEmail: email2
mozillaHomePostalAddress2: a2
mozillaHomeLocalityName: a3
mozillaHomeStreet: a4
mozillaHomePostalCode: a5
mozillaHomeCountryName: a6
mozillaPostalAddress2: b2

Comment 50

[(not reading, please use [e-mail] instead)]

Comment on attachment 82994 [details] [diff] [review]
fix

carrying forward the r/sr

Comment 51

[(not reading, please use [e-mail] instead)]

fixed on the trunk.

I think we want this for the branch.

Comment 52

[Asa Dotzler [:asa]]

Comment on attachment 82994 [details] [diff] [review]
fix

a=asa (on behalf of drivers) for checkin to the 1.0 branch

Comment 53

[scottputterman]

adding adt1.0.0+ for checkin into the 1.0 branch.

Comment 54

[(not reading, please use [e-mail] instead)]

fixed on both branch and trunk.  now on to #142940...

Comment 55

[Roland Felnhofer]

I just encountered an error in the ldif export !!!

mozillaHomeStreet: maps to State/Province 

Comment 56

[(not reading, please use [e-mail] instead)]

zoinks! 

I must have gotten confused from comment #38 and never turned on the brain.

the good news, is that I made the same mistake in import and export.

so if you export from mozilla to LDIF and then import (how we tell people to
move addressbooks), it will "work".

I'll get a patch ready.

Comment 57

[(not reading, please use [e-mail] instead)]

Attachment: supplimental fix.

wee, I suck.  "St" became "Street", not "State". 

note, export -> import will work, as I made the same mistake twice.

Comment 58

[Dan Mosedale (:dmosedale, :dmose)]

Comment on attachment 83069 [details] [diff] [review]
supplimental fix.

r=dmose

Comment 59

[(not reading, please use [e-mail] instead)]

I just landed the last supplimental patch on the trunk.  It should go on the 1.0
branch as well.

Comment 60

[Ninoschka Baca]

Note to Qa: Check the Mozilla 1.0 branch and then the trunk build.
Create a card where are the fields are populated, also include some lists,
export then import and all entries/fields should appear.

Comment 61

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 83069 [details] [diff] [review]
supplimental fix.

a=[e-mail] for branch checkin; please make sure it's in trunk as well
shaver sr'd it I believe

Comment 62

[(not reading, please use [e-mail] instead)]

completely fixed now on the trunk and the mozilla 1.0 branch.

Comment 63

[Ninoschka Baca]

Trunk build 2002-05-20: WinMe, Linux RH 7.1, Mac 9.1, Mac 10.1.3: Fixed
Branch build 2002-05-20: WinMe, Linux RH 7.1, Mac 9.1, Mac 10.1.3: Fixed
Marking Verified/Fixed on trunk, and verified1.0.0 for the branch.

Comment 64

[paul w brown]

This bug can still use some work ... the import is still not as good as itshould
be ... you can't move you address entries from one address book to another (like
to  eliminate duplicate entries) ... and most importantly, there is no support
what so ever for .nab files and the standards based vcard entries.  At minimum,
the address book should be able to import and export vard formats, in my opinion.