444093 - Spec out nsIAbItem::uuid more clearly

Status: RESOLVED FIXED

(MailNews Core :: Address Book, defect)

Description

[Joshua Cranmer [:jcranmer]]

As the summary says, this needs to be more clearly defined, which first means we need to do a lot more work on directories.

Comment 1

[Dan Mosedale (:dmosedale, :dmose)]

As much as I don't want this to block, I think life is going to suck a lot for sync extensions if we ship without this.  I have the beginnings of a UUID mixin interface/class from when I was working on Weave that might be useful to use here.

Comment 2

[Howard Chu]

Fwiw, LDAP directories already support the notion of a UUID. In OpenLDAP (and a few others) it's the entryUUID operational attribute. On MSAD it's objectGUID. Their text representations may differ, but they are semantically the same thing (and identical when converted to binary).

It's not a good idea to rely on LDAP DNs as unique identifiers because entry DNs can always be modified/renamed. UUID/GUID attributes are generated by a directory server at entry creation time and never changed thereafter.

In OpenLDAP we use the entryUUID when synchronizing entries in LDAP replication. It might make sense to take the same approach here for PalmSync or any other synchronization tasks. The notion of locally-unique IDs that can lose their meaning or uniqueness when moving a Card from place to place sounds pretty problematic, and should probably be avoided.

Comment 3

[Dan Mosedale (:dmosedale, :dmose)]

(In reply to comment #2)

> Fwiw, LDAP directories already support the notion of a UUID. In OpenLDAP (and a
> few others) it's the entryUUID operational attribute. On MSAD it's objectGUID.
> Their text representations may differ, but they are semantically the same thing
> (and identical when converted to binary).

Great.  Shouldn't be hard to use the nsIAbLDAPAttributeMap functionality to use one of those (if extant).  In the case where there isn't one of those, would it be better to generate our own or to use the DN?

> In OpenLDAP we use the entryUUID when synchronizing entries in LDAP
> replication. It might make sense to take the same approach here for PalmSync or
> any other synchronization tasks. The notion of locally-unique IDs that can lose
> their meaning or uniqueness when moving a Card from place to place sounds
> pretty problematic, and should probably be avoided.

There are various edge cases where there end up being multiple addressbooks containing the same object.  One such is two different views on an LDAP tree (eg public vs. authenticated, possibly even exposing different data), and I expect there will be similar cases around CardDAV in the future.  vCard v3 (as well as the latest draft specifically describe the "UID" property as being "used to uniquely identify the object that the vCard represents, perhaps for this reason.

My suspicion is that we want to do something a lot like calendar does: have an "id" attribute that is only guaranteed to be unique across a single collection, and a "hashId" attribute which is computed from the tuple of (id, calendar).

Comment 4

[Dan Mosedale (:dmosedale, :dmose)]

I've done a bunch of thinking about how we might best move this forward; but I'm still trying to generate a coherent set of steps from this.  I'll try and update this bug more over the weekend, if not sooner.

Comment 5

[Joshua Cranmer [:jcranmer]]

Let me add where I was coming from in the original megapatch before it was pulled.

[Assume "to me" or similar phrases in the next few paragraphs so I'm not so repetitive.]
I saw the UUID as a generic, implementation-agnostic way to refer to a card. Before, this was serviced primarily by nsIAbMDBCard::DbRowID and nsIAbLDAPCard::DN [1]; it would be an identifier guaranteed to be unique by virtue of the limitations of protocol whereas something akin to popping out of uuidgen would only be "most likely unique" (compare to message ids).

Most of my design decisions impressed upon the UUID documentation were in conjunction with the need to merge the unique-within-directory nature of the card's implementation with the actually unique (using URIs) for the directory. I don't think mailing lists entered my mind then (they just did right now...).
</history>

In terms of the "right" way to do this, I'm not quite sure what the best way would be. The unified "directory-unique" ID strikes me as being useful, but this is problematic in that there are edge cases to worry about.

Roughly speaking, I think the plan of implementation needs to consider the following:
1. Something unified for all nsIAbItems, i.e. top-level directories, contained collections (tags and mailing lists), and cards.
2. Interaction with synchronization, both palmsync and weave (along with anything that comes along in the future).
3. For cards especially, what happens if I...
3a. Move it between address books?
3b. Copy a card?
3c. Create a new card not from a directory API?
3d. Get a card from external synchronization source?

(directory, per-directory ID) is the approach I personally favor for non-synchronization use, but it's something that would need to be well-defined, to the point where the documentation starts sounding like an RFC. Which isn't necessarily a bad thing, though...

[1] Disclaimer: I was (and for the most part still am) knowledgeable only about LDAP and MDB implementations wrt the address book. That said, I have been looking at system address book APIs to increase my knowledge.

Comment 6

[Dan Mosedale (:dmosedale, :dmose)]

(In reply to comment #5)
> (directory, per-directory ID) is the approach I personally favor for
> non-synchronization use,

Can you elaborate on why you prefer using a tuple rather than something like the calendar's hashId?  The nice property of the hash ID is that the individual provider-types can arbitrarily choose to include stuff other than just those two items (like calendar does with recurrence-ID).

Calendar folks, what has your experience been with the hashId scheme so far?  Any notable advantages or disadvantages worth calling out here?

> but it's something that would need to be well-defined,
> to the point where the documentation starts sounding like an RFC. Which isn't
> necessarily a bad thing, though...

I don't understand why this would need to be more than a small number of paragraphs.  Where do you think the complexity would come in?

> Roughly speaking, I think the plan of implementation needs to consider the
> following:
> 1. Something unified for all nsIAbItems, i.e. top-level directories, contained
> collections (tags and mailing lists), and cards.

Not sure what needs to be considered here; you and I have put a couple of proposals on the table already.

> 2. Interaction with synchronization, both palmsync and weave (along with
> anything that comes along in the future).

Agreed, whatever scheme we end up with should be reviewed by folks who have some level expertise in such things (I might be able to stand in for Weave, though its API just got completely re-written, so...).

> 3. For cards especially, what happens if I...
> 3a. Move it between address books?
> 3b. Copy a card?
> 3c. Create a new card not from a directory API?
> 3d. Get a card from external synchronization source?

These are good questions.  One wrinkle that's likely to make things more interesting is there is a possible impedance mismatch.  vCard claims that its ID is the ID of the thing being represented by a given card (eg a person), meaning that you could reasonably have multiple vCards about the same person with different sets of information at the same time (eg if a URI exposes different amounts of info depending on whether you're authenticated).  Portable Contacts current draft, OTOH, claims that its ID is the ID of the card itself.  I'm not sure what semantic claims other providers (LDAP, Google, ...) make about this.  I also haven't entirely figured out how much this difference is likely to matter.

Comment 7

[Leni]

Some data points on Google's <id> and a thought balloon around rfc4122-style UUIDs.

Google

Google's id is based on atom:id, see:
ftp://ftp.rfc-editor.org/in-notes/rfc4287.txt
http://www.atomenabled.org/developers/syndication/atom-format-spec.php#element.id

atom:id's are unique within a given <feed> ie an addressbook at a given point in time.

Google's REST Api adds some constraints:
- username is encoded into <id> which makes <id>'s unique across
  all users
- placeholders for deleted contacts are retained for 30 days
http://code.google.com/apis/contacts/developers_guide_protocol.html#Deleting
  So after 30 days, there's no guarantee that a Google <id> won't be reused.

rfc4122-style UUIDs

I'm not entirely sure what the objective of this exercise is.  But for the sake of floating a thought balloon, it's fun to imagine all cards in every thunderbird across the planet and across time having a unique id that could be referenced by URI.  What one would do with such a namespace I'm not sure.

That sort of a UUID sits within this definition of UUID:
ftp://ftp.rfc-editor.org/in-notes/rfc4122.txt

To explore this idea a bit further, it would mean that:
- each and every card, regardless of which directory it's in,
  has an rfc4122-style id.
- a card moved from one addressbook/directory to another retains it's
  UUID and UI drag+drop can be treated as a move, rather than as
  delete+add.    These ids are persistent across addressbooks which is nice.
- for non-local addressbooks (ie LDAP, system, web-based)
  the mapping between a card's UUID and the non-local unique id
  isn't stable, ie: each time a directory got repopulated it's
  cards would get a brand new set of UUIDs.  So no good for sync
  of non-local addressbooks.

I know it's not the preferred approach, just tossing it in as an idea in case it's "universal" nature has an appeal.

Comment 8

[Günter]

I second Leni's comment and reference to the RFC as a very valuable approach here!

Not dipping into the current TB/card implementation too much I'm asking what an unequivocal id for a TB/card stands for. Let my try to figure out here:

And -- at least for me -- it's much the same as we see it with the 'UID' used with 'iCalendar' (RFC2445) .. and we have it already with 'vCard' (RFC 2426)!

Why referring to 'iCalendar'?
An event is generated with one UID, and that is NEVER changed! The event gets modified/updated, send for invitation and resend with attendees comment/ack ... but ALWAYS it has the very first UID. That related iCalendar has some mechanism to show versioning 'SEQUENCE' , update timestamp 'DTSTAMP'. So different states of the event can be handled in the right sequence etc.

Livetime of  a TB/card (vCard)
All that UUID discussion is around usage of a card.  And that's about local and remote usage.
Local:
- a card has to be 'unequivocal' in my addressbook. One 'unequivocal' card with one UID in one place/ one directory (you name it)
- a card can be moved to another place/directory, but it holds the UID
- a card can be copied, and here we have two different situations:
  --  I want to 'hold' the current status fe. as an backup etc. Here we should hold the UID, but use similar mechanism like 'iCalendar'
  --  I want to use to current card as a template for an new card. Here we set a NEW UID

Remote
Is 'remote' the same for 'sync' with a device (PAD eg. with FinchSync) and 'sync' with a remote system (Google, Zindus, etc)?
Not speaking about the individual implementation those services have, FMPOV it's very much the same as we have with 'local'.
A single card/address needs to have a very 'unequivocal' UID, which should not change, not ever, not with aging, and should NEVER be reused (may be I misunderstand what Leni wrote "So after 30 days, there's no guarantee that a Google <id> won't be reused.")

Consequently once a card/address has got it's UID by one 'partner', all others have to use that UID .. and update a 'local' card/address or save it to it's storage.

How does that match with current TB/AB?
-- the current implementation of an UID equivalent uses the directory? I'm not sure here, but if it's the case that isn't a 'general' UID!
-- we have a 'Last-modified'. This changes with some TB/AB actions like moving from one dir to another, or importing with LDIF. That's not wise. And with some action the change is a reset to '0Z'.
-- multiple copies of a card can exist in a given TB/ab, without a differentiation about the status (not using things like 'SEQUENCE' or 'DTSTAMP'.)


So before an implementation of an  UUID as shown with rfc2122, is the "Livetime of  a TB/card (vCard)" and it's consequences I mentioned what we want to achive?

Comment 9

[Joshua Cranmer [:jcranmer]]

(In reply to comment #6)
> (In reply to comment #5)
> > (directory, per-directory ID) is the approach I personally favor for
> > non-synchronization use,
> 
> Can you elaborate on why you prefer using a tuple rather than something like
> the calendar's hashId?  The nice property of the hash ID is that the individual
> provider-types can arbitrarily choose to include stuff other than just those
> two items (like calendar does with recurrence-ID).

My opinions on this precise topic seem to fluctuate rapidly... I've changed my mind just writing this comment.

> > but it's something that would need to be well-defined,
> > to the point where the documentation starts sounding like an RFC. Which isn't
> > necessarily a bad thing, though...
> 
> I don't understand why this would need to be more than a small number of
> paragraphs.  Where do you think the complexity would come in?

The points I mentioned later. Not that any other scheme wouldn't have similar amounts of complexity.

> > 3. For cards especially, what happens if I...
> > 3a. Move it between address books?
> > 3b. Copy a card?
> > 3c. Create a new card not from a directory API?
> > 3d. Get a card from external synchronization source?

Adding onto this list some more questions:
3e. Import or export a card (or directories for that matter)? Suppose I export an address book and import it into another--or maybe the same, in a few edge cases--installation. How would other tools relying on address book UUIDs act in such situations? How should they act?

3f. How will UUIDs interact with non-editable sources, like a read-only LDAP server? Specifically, if we generate our own UUIDs and something blows away the storage of wherever the UUIDs are stored, what ramifications should/could/would it have?

I'll comment on the other comments later, but I have to run now. I'm sure I have some more requirements on the UUID end that aren't coming to mind right now as well.

Comment 10

[Joshua Cranmer [:jcranmer]]

(In reply to comment #7)
> Some data points on Google's <id> and a thought balloon around rfc4122-style
> UUIDs.
> 
> Google
> - username is encoded into <id> which makes <id>'s unique across
>   all users
> - placeholders for deleted contacts are retained for 30 days
> http://code.google.com/apis/contacts/developers_guide_protocol.html#Deleting
>   So after 30 days, there's no guarantee that a Google <id> won't be reused.

So it's guaranteed unique among everything now, but not among everything from all times.

> That sort of a UUID sits within this definition of UUID:
> ftp://ftp.rfc-editor.org/in-notes/rfc4122.txt

http://tools.ietf.org/html/rfc4122, for those of you who prefer the HTML RFCs.

> To explore this idea a bit further, it would mean that:

Essentially we're kind of screwed if the address book is read-only (AFAICT).

(In reply to comment #8)
> And -- at least for me -- it's much the same as we see it with the 'UID' used
> with 'iCalendar' (RFC2445) .. and we have it already with 'vCard' (RFC 2426)!

Having it with vCard doesn't help when we're not using vCard... like with LDAP servers.

> - a card can be copied, and here we have two different situations:
>   --  I want to 'hold' the current status fe. as an backup etc. Here we should
> hold the UID, but use similar mechanism like 'iCalendar'
>   --  I want to use to current card as a template for an new card. Here we set
> a NEW UID

Sounds almost like Message-ID. It's unique in theory but in practice would not be.

> Remote
> Is 'remote' the same for 'sync' with a device (PAD eg. with FinchSync) and
> 'sync' with a remote system (Google, Zindus, etc)?

There are different cases to consider in terms of address books.
1. Mozilla address books: we have complete control over what we do here.
2. System address books: we have to work with possibly several applications doing stuff. No control over schemata.
3. LDAP: likely read-only and also rather global.
4. Synchronization. This includes small-device synchronization (e.g., PDAs), as well as Weave-like synchronization or web-service synchronization (e.g., Google Contacts). In particular, note that the issue of the same other end synchronizing to different profiles.

> How does that match with current TB/AB?
> -- the current implementation of an UID equivalent uses the directory? I'm not
> sure here, but if it's the case that isn't a 'general' UID!

We don't really have a current UID-like property; LDAP has DN and MDB directories have DbRowID. I don't know what OS X and Outlook address books do (looks like RDF-related URIs ?)... This is actually the point of this bug--to find a uuid for address book items in general.

Comment 11

[Leni]

(In reply to comment #10)
> Essentially we're kind of screwed if the address book is
> read-only (AFAICT).

How so?  Wouldn't it be the case that when a card comes into existence it has a UUID?  Including cards in read-only addressbooks?

> This is actually the point of this bug--to
> find a uuid for address book items in general.

I wonder whether there's two parallel discussions going on here, around:
1. a UUID for mozilla address books
2. a UUID abstraction for cards in all addressbooks accessible
   from Thunderbird.

For (2), I'm not sure what can be usefully generalised - perhaps only "uniqueness at a given moment in time".  In which case it's an LUID, not a UUID.

For (1), the UUID could be more along the lines of entryUUID - ie uniqueness across all mozilla addressbooks and across time.  Here's some text from the LDAP entryUUID rfc:
http://tools.ietf.org/html/rfc4530

   Servers SHALL generate and assign a new UUID to each entry upon its
   addition to the directory and provide that UUID as the value of the
   'entryUUID' operational attribute.  An entry's UUID is immutable.

   UUID are to be generated in accordance with Section 4 of [RFC4122].
   In particular, servers MUST ensure that each generated UUID is unique
   in space and time.

Comment 12

[Joshua Cranmer [:jcranmer]]

(In reply to comment #11)
> I wonder whether there's two parallel discussions going on here, around:
> 1. a UUID for mozilla address books
> 2. a UUID abstraction for cards in all addressbooks accessible
>    from Thunderbird.

nsIAbItem::uuid--the scope of this bug--is a uuid that will exist for all cards in all address books, whether they be Mozilla address books, LDAP, or system address books.

So the discussion should (at least for the present) should be focusing more on item 2 than item 1.

Comment 13

[Joey Minta]

Apologies in advance for the length of this comment. I'm trying to tease apart a few issues here...

First, I think the vCard spec's claim (as describe in comment #6) is misguided in two respects, and Thunderbird should avoid it. There are two different identification questions involved here. (1) A program asks "Is this the same {card} object I had before?" (2) A user asks "Do these two 'card's refer to the same person?" The vCard spec collapses these two questions, and first creates the fundamental problem impls are required to have knowledge of each other's data (in order to maintain the person<->uuid relationship). Second, the spec tries to force users to make identifications in ways they normally don't. To the user, the (mostly) unique identifier of a card is the display name. No (sane) app would display the UUID to the user, but the vCard approach requires that users implicitly manipulate this data that is otherwise meaningless to them.

Thus, to me, the UUID should only be used internally, and only as a tracking device for keeping track of various card-datastructures.

Second, embedded in my first comment is the notion that we can't require various datasources to react to each other. As jcranmer mentioned, read-only sources are especially problematic here. Thus, the notion that ids (even if asserted to be globally unique), should only be treated as unique-within-datasource, is one a conservative program should make.

Along these lines, the tuple of (directory id, per-directory ID) is a good choice for a mozilla-unique id. Because directory-ids are likely handed out by a central service (the notion that a directory can assign itself a unique id fails under the same analysis as asserted globally unique card ids), this central service is likely the best place to perform the hashing of this tuple. Scattering the tuple computations creates more danger of incorrect hashing.

Finally, the above means that the concerns discussed in comment #5 and comment #6 about copying/duplicate cards in multiple datasources are *not* a problem of UUIDs, nor is it one UUIDs should be used to solve. (This is where vCard went wrong.) Rather, the fact that the same card exists or moves datasources is only relevant when displaying the data to the user. As I said before, however, the user doesn't care what the card's UUID is, but rather what a person's name is. Thus, the front-end should be responsible for properly displaying these multi-copy cards, it should not be embedded in the data-model itself.

I should include a note here about institutional databases, where the large numbers of cards actually creates a realistic danger of non-unique display names, since that is the obvious retort to the above. First, the user *must* distinguish between these non-unique names in some way, based on the other information. This is again a problem of displaying the data to allow the user to make this distinction, rather than a problem of storing the data. For now I'll simply point out that appending an index to the names ("Tom Smith (2)") or including other differentiating info ("Tom Smith (Wash. D.C.)") should generally be sufficient.

In summary:
(1) UUIDs should be independent of the object the card describes
(1.1) The problem of "duplicate" cards is one of data-display, not data storage
(2) UUIDs should be assumed to be only directory-unique
(3) A tuple of (directory, card-id) is then a unique identifier
(4) A central service should hash these IDs, since that is the only place that has knowledge of all existing datasources.

Comment 14

[Günter]

(In reply to comment #12)
> (In reply to comment #11)
> > I wonder whether there's two parallel discussions going on here, around:
> > 1. a UUID for mozilla address books
> > 2. a UUID abstraction for cards in all addressbooks accessible
> >    from Thunderbird.
> 
> nsIAbItem::uuid--the scope of this bug--is a uuid that will exist for all cards
> in all address books, whether they be Mozilla address books, LDAP, or system
> address books.
> 
> So the discussion should (at least for the present) should be focusing more on
> item 2 than item 1.

FMPOV we have a discussion about a "data set" stored to a given system. And the need is to make that data set UNIQUE. It's not about stamping a 'person' (data) to be unique. And that's said with comment #6. 

From all of this: if  TB/AB needs to generate a nsIAbItem::uuid is goes with the [RFC4122] -- see  comment #10.
Is this the result of the disscussion?

The remaining questions are (and answers FMPOV):
1. when TB/AB needs to generate a NEW nsIAbItem::uuid?? 
   A: in any case TB/AB has to build a new card.
2. In which cases TB/AB would NOT generate a nsIAbItem::uuid??  
   A: Always when working with 'remote' apps which already have an equivalent of an 'UUID'. (see [1]) 
3. How about 'remote' apps with read-only??  
   A: TB/AB read access has no prob, but if the user wants to hold a 'local' copy this is getting a new 'unique' card .. and has to generate an UUID. Here the original remoteUUID could help to get an connection for updates etc. And for that the remoteUUID is a different item than our nsIAbItem::uuid!

Any open point left? 
Why is a 'remote' side different from a 'small-device' if it cames to discuss UUID?
(I don't see a difference!)


[1] examples:
vCARD: 'UID', LDAP: 'DN' etc. 
For MS Outlook have a look here: http://msdn.microsoft.com/en-us/library/bb220287.aspx
"Solutions should not depend on the EntryID property to be unique unless items will not be moved. The EntryID property returns a MAPI long-term Entry ID."

Comment 15

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

(In reply to comment #6)
> Calendar folks, what has your experience been with the hashId scheme so far? 
> Any notable advantages or disadvantages worth calling out here?
Sorry I haven't read all of this bug, but just to answer to the above. So far I think we have been doing quite good with hashId. It was very good to have something that is unique across all calendars, but I think the combination of something application-unique and something calendar unique (item UID) is the best solution since it opens more possibilities for comparison.

The following model was once suggested to handle same items in different calendars: if two events from different calendars with the same UID show up, then they could be shown as one event, with colorful boxes to show that the event is in different calendars. Calendars can be marked as "your own", so that the main color is taken from your calendar instead of the one you subscribed to.

Given the hashid/uid model:
> > 3. For cards especially, what happens if I...
> > 3a. Move it between address books?
New hashid, same UID

> > 3b. Copy a card?
same addressbook: new UID
other addressbook: basically a move: new hashid, same uid

> > 3c. Create a new card not from a directory API?
new UID is needed.

> > 3d. Get a card from external synchronization source?
if external card just has a UID, then generate hashId.
if external card has no UID, its  hard to synchronize it.

Comment 16

[Andrew Sutherland [:asuth] (he/him)]

Perhaps go with the (directory id, card id) tuple, but allow directories to provide an indication of the 'namespace' in which their card unique ID's are allocated, presumably as a URI.

A directory provider which has had to make up its own unique ID's from scratch returns NULL to indicate its ID's are not comparable with anyone else's.  A readable LDAP directory and a writable LDAP directory that end up talking to the same underlying datastore would return a URI that indicates that the ID's come form the same pool and are comparable.

From gloda's perspective, this is ideal.  If we index a card and want to find it again, we have a unique identifier for the directory and the card, making it trivial.  But when it comes to indexing and user-display ramifications, we can use the namespace functionality to eliminate duplicates which would otherwise complicate our lives.

Comment 17

[Leni]

Just to restate that a card dragged+dropped between two local mozilla addressbooks ought to be trackable as a move - rather than delete+add.  If something like the namespace approach suggested in comment #16 enables that, great.

Also, the properties of the uid component of the tuple should be clarified.  In particular: can uid's be reused over time?  If no ==> then no issue.  If yes ==> then clients need to know how soon reuse can happen.  If reuse can happen immediately then how does deletion detection work?  The answer isn't "observers" - deletes ought to be able to be identified purely on the basis of state.

This last question leads me to wonder whether a UUID that is generalisable (read: lowest common denominator) across any conceivable system/remote addressbook gives the best result for the mozilla-managed addressbook(s).

Comment 18

[Günter]

From the discussion I read two different mentality about what that the nsIAbItem::uuid stands for.
Is it an <id> for the person the card describes, or
is it an <id> for a data set and it's describing a specfic view onto a given person.

The wording 'nsIAbItem' leads me to the second. It's a dataset. And it has to be unique, and in the world where we handle datasets between systems (TB/AB, LDAP, PAD, etc ...) I have to count on one specific representation of the data. For this POV it's "totally" irrelevant where the dataset is stored in the TB/AB, and for that a UUID should NOT have an relation to that. 

The comment #16 
"But when it comes to indexing and user-display ramifications, we can
use the namespace functionality to eliminate duplicates which would otherwise
complicate our lives." 
shows a problem when using that [RFC4122] based UUID, saying not including that 'tuple' info (whatever 'tuple' holds) or directory info. If the application (TB/AB) needs additional items to make life easier, OK ... but that can NOT be part of nsIAbItem. If thats the case forget about a 'unequivocal' item with the card.

The Comment #15 shows how LG/SB uses an UIID with calendar iCal. TB/AB cards are exactly the same situation, two dataset may have different status, may come from different source, but having ONE UIID you have to drill down to one dataset.

And for <card> you have to have items to track changes as with iCal/invitations. You have to update items  SEQUENCE or DTSTAMP to show changes to certain iCal items (Title, dates etc).
With <card> we have 'Last-modified' ... a good start! See above ;-)
Why not an additional item ... very TB/AB specific?

Comment 20

[Joshua Cranmer [:jcranmer]]

*Mutters about bugzilla needing a rewrap button.*

Grand Synthesis time!

The first consensus I see is that we have a (directory, card) for the unique part.

nsIAbItem::uuid would be some identifier for address book directories (the preference name or its URI are good choices). For cards and other sub-directory structures, it would be the equivalent to the hashId of calIItemBase.

The sub-directory structers (nsIAbCard/nsIAbGroup) would gain a localId, referring to their ID within the directory containing them, and directoryId, the uuid of their directory. All of these properties are of type AUTF8String, for maximum flexibility. An empty string means that the property hasn't been assigned.

This arrangement will give us the property that getting the uuid, no matter the type, will give us a unique string.

The questions:
(In reply to comment #5)
> 3. For cards especially, what happens if I...
> 3a. Move it between address books?

Reset uuid, keep same localId if reasonable.

> 3b. Copy a card?

nsIAbCard::copy will clear out the strings.

> 3c. Create a new card not from a directory API?

Empty strings (unassigned values)

> 3d. Get a card from external synchronization source?

directoryId is empty, uuid/localId set to whatever the external source has.

(In reply to comment #9)
> 3e. Import or export a card (or directories for that matter)?

Import would act like simply adding new cards. Export... I don't know yet.

> 3f. How will UUIDs interact with non-editable sources, like a read-only LDAP
> server?

The directory will synthesize its UUID from the local ID and the directory ID, so this is not an issue, except for...

3g. How persistent are UUIDs? Not all external DBs guarantee universal uniqueness for their IDs.

I'll get back to you on this one.

3h. How best to denote logical equality between two cards from different directories?

I think having a property like "ID_of_<directory.uuid>" storing the card's localId for the given directory works the best.

3i. How to get an item for its uuid?

nsIAbManager::getDirectoryForId(in AUTF8String uuid);
nsIAbManager::getItemForUUID(in AUTF8String uuid);
nsIAbDirectory::getItemForID(in AUTF8String localId);

Responses to later comments:
(In reply to comment #13)
> Along these lines, the tuple of (directory id, per-directory ID) is a good
> choice for a mozilla-unique id. Because directory-ids are likely handed out by
> a central service (the notion that a directory can assign itself a unique id
> fails under the same analysis as asserted globally unique card ids), this
> central service is likely the best place to perform the hashing of this tuple.
> Scattering the tuple computations creates more danger of incorrect hashing.

This would seem to call for
nsIAbManager::generateUUID(in AUTF8String directoryId, in AUTF8String localID)

> Thus, the front-end should be responsible for properly displaying these
> multi-copy cards, it should not be embedded in the data-model itself.

I somewhat disagree. A standardized way to denote the relation is needed, IMO, but actually creating these relations should be left up to consumers, not implementations.

(In reply to comment #14)
> From all of this: if  TB/AB needs to generate a nsIAbItem::uuid is goes with
> the [RFC4122] -- see  comment #10.
> Is this the result of the disscussion?

The consensus is that uuid is a tuple, where directories are expected to provide the necessary localIds.

> Why is a 'remote' side different from a 'small-device' if it cames to discuss
> UUID?

For remote directories, the content (including UUID) is authoritative, so there is only one thing to worry about. Any change made on our end is reflected in the remote end.

In synchronization cases, there are two places where changes can happen (in general, neither is authoritative), so that our changes are independent of those made on the remote end.

(In reply to comment #15)
> So far I
> think we have been doing quite good with hashId. It was very good to have
> something that is unique across all calendars, but I think the combination of
> something application-unique and something calendar unique (item UID) is the
> best solution since it opens more possibilities for comparison.

I made uuid the equivalent of hashId, under the assumption that keeping universal uniqueness embedded in nsIAbItem::uuid is more ideal.

> The following model was once suggested to handle same items in different
> calendars: if two events from different calendars with the same UID show up,
> then they could be shown as one event, with colorful boxes to show that the
> event is in different calendars.

"Was once"? Does that mean it was rejected? If so, why?

Address books have the problem that it's difficult--if not impossible--to have an authoritative localId scheme, so preserving these localIds across directories is probably unfeasible. At best, I can put a note in to "keep the same localId if feasible."

(In reply to comment #16)
> A
> readable LDAP directory and a writable LDAP directory that end up talking to
> the same underlying datastore would return a URI that indicates that the ID's
> come form the same pool and are comparable.

I'm not sure how common having two AB directories representing the same data would be; the only reasonably common case I can come up with is LDAP AB replication. Possibly, synchronization might in effect produce the same occurrence, but this is a separate can of worms, I think.

> But when it comes to indexing and user-display ramifications, we can
> use the namespace functionality to eliminate duplicates which would otherwise
> complicate our lives.

Would having a standardized mechanism to indicate duplication via properties satisfy you?

(In reply to comment #17)
> can uid's be reused over time?  If no ==> then no issue.  If yes
> ==> then clients need to know how soon reuse can happen.  If reuse can happen
> immediately then how does deletion detection work?

I don't know the answer to this. Ideally, they shouldn't be reusable; in practice, that could be difficult to guarantee.

> This last question leads me to wonder whether a UUID that is generalisable
> (read: lowest common denominator) across any conceivable system/remote
> addressbook gives the best result for the mozilla-managed addressbook(s).

I believe LDAP's version starts as "a generic string" while Outlook uses a 32-bit unsigned integer....

(In reply to comment #18)
> From the discussion I read two different mentality about what that the
> nsIAbItem::uuid stands for.
> is it an <id> for a data set and it's describing a specfic view onto a given
> person.

Yes.

> For this POV it's "totally" irrelevant where the dataset is stored in the
> TB/AB, and for that a UUID should NOT have an relation to that. 

Not quite. For a UUID to be useful, we have to be able to get the item that it represents. So we have to be able to map to a directory and further the directory has to be able to map to a card.

> The Comment #15 shows how LG/SB uses an UIID with calendar iCal. TB/AB cards
> are exactly the same situation, two dataset may have different status, may come
> from different source, but having ONE UIID you have to drill down to one
> dataset.

It sounds like people want to have some notion that "if and only if x is true, A and B are the same card" for some value of "same." I think the best that can be done is to denote a mechanism but let others use that mechanism.

Some parting notes:
1. At this point, how our very own address books generate UUIDs is still open, but I'm not going to try to close it now as it's not important until we agree on what UUIDs look like.
2. Uniqueness across all time (or all reasonable time) of UUIDs is an open question that needs resolving. LDAP and Google are big ones I'm thinking of now (although it looks like pi is going with a synchronization model as opposed to a new backend model).
3. Equality-like relationships between cards is an important question to consider, although I would also like to refer people to bug 444091. I guess the better question to ask for now is what causes equal cards to be created.

Congratulations for reading this far in the comment! That's all the replies and comments I have for now. *Mutters about bugzilla's textbox being too small.*

Comment 21

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

> > The following model was once suggested to handle same items in different
> > calendars: if two events from different calendars with the same UID show up,
> > then they could be shown as one event, with colorful boxes to show that the
> > event is in different calendars.
> 
> "Was once"? Does that mean it was rejected? If so, why?

Bad wording, sorry. All I meant was that some time ago it was suggested (and then implemented).

Comment 22

[Leni]

(In reply to comment #20)
> > 3. For cards especially, what happens if I...
> > 3a. Move it between address books?
> Reset uuid, keep same localId if reasonable.

I guess 'keep same localId if reasonable' is a nod towards retaining some history when a card moves between local addressbooks.

But even if localId can be preserved (doubtful), I wonder how clients will be able to distinguish whether that happened by accident or by design?

Other ideas (each has drawbacks):
- a multivalued backreference card property, eg. "uuid_prior_to_move" - populated when a card is moved between directories.
- the manager service could offer a separate interfaces for "uuid history of cards recently moved".

(In reply to comment #12)
> nsIAbItem::uuid--the scope of this bug--is a uuid that will exist for all cards
> in all address books, whether they be Mozilla address books, LDAP, or system
> address books.
> 
> So the discussion should (at least for the present) should be focusing more on
> item 2 than item 1.

To restate my observation (comment #17) - it'd be a pity if a UUID for the general case delivers a suboptimal result for mozilla addressbooks.  Thinking particularly about UUID reuse and deletion detection here.  If sync clients have to maintain their own luids on cards to support deletion detection, this UUID isn't much use - other than for O(1) lookups.  A UUID just for mozilla addressbooks would be so very much simpler - particularly if implemented against an SQL backend.

To be honest, I still feel a bit at sea in understanding exactly what problem this bug is intended to address.

Comment 23

[Henrik Kaare Poulsen]

I am sorry if I am totally off topic here, but, coming from a world of synchronization, my point of view if this:

We need two different identifiers:
1) One id globally identifying the physical person / event whatever
2) One id identifying a specific instance's view on this person / event.

So simple clients (like sync) would in the first place work on (2): If my peer's impression of the person/event has changed, so should mine.

If you move or copy a card/event, it gets a new (2), but retains (1).

More advanced clients could work on (1) if they have the capability.

So, my suggestion:
Let's have two identifiers on each card/event/...

This would also make implementation very simple:
If you get a new / copied / moved / whatever entry, give it a new local (2) ID, and preserve the global (1) ID.

Obviously the problem remains: if the user changes 100 properties of the card/event, is it still the same global (1) card/event? I would think so, but in any case it would still be the same local (2) card/event...

/Henrik

Comment 24

[Joshua Cranmer [:jcranmer]]

Attachment: Patch, version 0.5

This is not fully working yet, but this is a WIP. UUIDs aren't fully useful (there's no unified way to get the objects a UUID represents yet), and I haven't fully hooked up the tests yet. The tests also probably need to test search results.

I'm uncertain about the actual generateUUID delimiter as well.

Comment 25

[Mark Banner (:standard8)]

Setting target of beta 3, if we can bring this forward to beta 2 that would be good, but I don't believe its vital to do that at the moment.

Comment 26

[Joshua Cranmer [:jcranmer]]

I have discovered a few problems that arose while testing a more modern version of this patch:

1. nsIAbManager::createNewDirectory doesn't seem to work for outlook directories,
2. some of mork's assertions only happen on Windows and Macs (o_O), and
3. nsIAbDirectory::dirPrefId is not set for all directories.

Asides from these issues, the UUID work appears to work for Outlook and MDB directories.

Comment 27

[Joshua Cranmer [:jcranmer]]

Attachment: Patch, version 0.8

Updated patch: this should work for all types, although it yet needs verification on OS X address books and LDAP address books, so version 0.8 until then.

Comment 28

[Mark Banner (:standard8)]

(In reply to comment #27)
> Created an attachment (id=373648) [details]
> Patch, version 0.8
> 
> Updated patch: this should work for all types, although it yet needs
> verification on OS X address books and LDAP address books, so version 0.8 until
> then.

There's a bit of LDAP bitrot due to recent check-ins.

You also need to do (I think):

-    card->SetDirectoryUuid(ourUuid);
+    card->SetDirectoryId(ourUuid);

in nsAbOSXDirectory::Init

test_uuid.js is also broken due to the do_get_file changes.

Once I fix those, this patch compiles & passes unit test on os x.

Comment 29

[Joshua Cranmer [:jcranmer]]

Attachment: Patch, version 1

Thanks to Standard8 for verifying this on OS X for me. This also should apply to the recent trunk stuff.

Comment 30

[Joshua Cranmer [:jcranmer]]

Comment on attachment 375052 [details] [diff] [review]
Patch, version 1

Putting on dmose's radar.

Comment 31

[Mark Banner (:standard8)]

Unless I'm mistaken at a quick look, there's not a possibility for obtaining a card directly via its uuid. What are the plans for that? Is it necessary to make this useful?

Comment 32

[Joshua Cranmer [:jcranmer]]

(In reply to comment #31)
> Unless I'm mistaken at a quick look, there's not a possibility for obtaining a
> card directly via its uuid. What are the plans for that? Is it necessary to
> make this useful?

Having UUIDs alone should be useful; getting a card or directory from a UUID will be in a separate patch.

Comment 33

[Mark Banner (:standard8)]

Comment on attachment 375052 [details] [diff] [review]
Patch, version 1

>+   * Consumers of this interface outside of directory implementations SHOULD
>+   * NOT, in general, modify this property.
>+   */
>+  attribute AUTF8String directoryId;
>+  /**
>+   * The per-directory ID of this card.

nit: blank line after the attribute line please.

>+/**
>+ * A top-level address book directory.
>+ *
>+ * The UUID of an nsIAbDirectory is its preference ID.
>+ */

I was thinking why not use the URI, but I guess it doesn't matter either way really.

>+  nsCOMPtr<nsIAbManager> manager = do_GetService(NS_ABMANAGER_CONTRACTID);

You should be checking for do_GetService failure here.

>+NS_IMETHODIMP nsAbDirProperty::GetUuid(nsACString &uuid)
>+{
>+  // XXX: not all directories have a dirPrefId...

That would then seem to me that we should use the URI, or is the URI not unique in all cases?

Saving & pausing review at nsAbLDAPDirectory.cpp

Comment 34

[Mark Banner (:standard8)]

Comment on attachment 375052 [details] [diff] [review]
Patch, version 1

>+NS_IMETHODIMP
>+nsAbManager::GenerateUUID(const nsACString &aDirectoryId,
>+                          const nsACString &aLocalId, nsACString &uuid)
>+{
>+  uuid.Assign(aDirectoryId);
>+  uuid.AppendLiteral("#");

This is better as uuid.Append('#');

> NS_IMETHODIMP nsAddrDatabase::CreateNewCardAndAddToDB(nsIAbCard *aNewCard, PRBool aNotify /* = FALSE */, nsIAbDirectory *aParent)
...
>+  rowId.mOid_Id = id.ToInteger(reinterpret_cast<PRInt32*>(&rv), 10);

Please make this compatible with the frozen api, nsMsgIncomingServer.cpp has some existing examples.


In summary, looking good, but I'd just like to take another quick look once you've updated the patch.

Comment 35

[Joshua Cranmer [:jcranmer]]

Attachment: Patch, version 2

This should address all of Standard8's nits.

Comment 36

[Dan Mosedale (:dmosedale, :dmose)]

We wouldn't hold b3 for this; moving out to b4.

Comment 37

[Mark Banner (:standard8)]

The Thunderbird drivers wish to release Thunderbird 3 as soon as possible. As a
result, we feel that this bug shouldn't stand in the way of all the other good
work getting into the hands of users sooner rather than later. Therefore we are
retargeting it for 3.1. See http://ccgi.standard8.plus.com/blog/archives/242
for more details. The 3.1 release is expected to be a quick release soon after
Thunderbird 3.

Comment 38

[Igor Velkov]

To allow weave to use uids ti sync, there should be mechanism to create object with already-known ids, and to change id of already-existed object (it used when similar objects on different computers found)

Comment 39

[Günter]

Moving from beta1+ to beta2+ does that mean there are already final decisions about the card UUID, to hold it with move/copy. Is the UUID for or the 'person' etc How about the UUID for the directory.
I understood these points are under discussion, see also comments up to #23.
What is the 'final' concept decision here ?

Comment 40

[Dan Mosedale (:dmosedale, :dmose)]

We're resetting the blocking flag for 3.1 on this bug and instead setting the wanted-thunderbird+ flag. We have too many blocking-3.1 bugs, to the point where it doesn't mean much, and managing the list is making it hard to actually work on closing bugs, which helps no one.

Thunderbird 3.1's primary purpose is to allow us to offer a prompted major update to Thunderbird 2 users, to ensure their continued ability to safely use Thunderbird.  Thunderbird 2 is built on an outdated version of Gecko, and our long-term ability to maintain the users' safety for Thunderbird 2 users is limited.

If you think this bug meets the requirements below, please renominate with a detailed explanation of how it meets the following two criteria, and we will reconsider.  To qualify, this bug must either:

a) make the upgrade experience from TB2 very painful for a large number of users

or

b) be a new, reproducible, severe quality issue (eg dataloss, frequent crashes)

Just because this bug doesn't block TB3.1 doesn't mean it can't or won't make the release.  Once they're done with their blockers (if any), we encourage developers to keep working on non-blocking bugs, and to try to land them as early in the cycle as possible, as non-blocking bugs will become increasingly difficult to land in the later stages of the cycle.

Comment 41

[Mark Banner (:standard8)]

Comment on attachment 384769 [details] [diff] [review]
Patch, version 2

I spoke to dmose recently about this and he was happy for me to take his sr and grant it.

Therefore sr=Standard8. There's a little bit of bitrot, but it looks simple, and I'm not going to re-review it as I've reviewed it already before.

Comment 42

[Mark Banner (:standard8)]

And please accept my apologies for this taking so long to happen.

Comment 43

[Joshua Cranmer [:jcranmer]]

Pushed as changeset 600b8c098a82.