153228 - Loading of large LDAP replicated address book make client unresponsive during offline lookup

Status: NEW

(MailNews Core :: LDAP Integration, defect)

Description

[Rajiv Dayal]

Judson Valeski wrote:

Scott,

Just wanted to give you some usability feedback re: LDAP in offline mode. I've
replicated the directory locally for offline use, however, it's not usable
offline due to performance problems. Perhaps our ldap directory isn't a good
test case because of it's size, but, the client becomes unresponsive (for
minutes at a time on my 500mhz 256ram machine) when trying to do lookups.

Not sure if this is a known issue or not; either way, it should probably be
release noted or resolved.

Jud

Comment 1

[Rajiv Dayal]

Judson Valeski wrote:

we might want to consider turning it off by default in offline mode if it's
greater than a certain size. I've had to start disabling ldap lookups when in
offline mode.

Jud

Scott Putterman wrote:
It's a limitation of our db.  We read everything into memory and large AB's
suffer because of it.  We didn't have the ability to try to optimize this during
Mach V.  I'm not an expert at ldap queries.  I'd think we could limit the search
query to people who still work at Netscape (ignoring phone number = XXXX).  I've
cc'd Rajiv to see if he knows of a way to do this.

Scott

Comment 2

[Rajiv Dayal]

We can go for a temporay solution as suggested by Jud. But the question is what
is the cutoff size for the AB beyond which we should diable the offline lookup ?

To decide this we can check user's available system memory by making systems
calls, and in offline mode disable the AB search / lookups for truly large
Address Books that can take up more than x% (guesswork - 50%) of the available
system memory. But this too is a solution which maynot work for all cases, if
the machine has large available system memory but a slow CPU, Mork could still
take a long time to load a large AB file and do the search/lookup.

Besides if we want to disable the user to do the lookup for large Adress Books,
we should put restriction at the LDAP replication stage itself, mscott suggested. 
But in this case also there is an issue, since we donot know during LDAP
replication the total number of records we might have to let the user first
download 'n' entries based on the user's system memory and then tell him/her
"that's it" !

In offline mode what we do is ABDirectory query and not really LDAP query. We
use Mork as our database to store Address Book, and Mork is an in-memory
database. During an offline search the Mork's FindRow method is called which
needs to load the Mork file associated with the AB and being in-memory database
it loads the complete file which takes a long time. Also Mork is not thread safe
thus search cannot be started along with loading the data.

So the right solution is that Address Books should use a database that supports
paging and multithreading. Either Mork should be modified to support paging and
multithreading or 'Berkley DB' (small footprint, fast and open source) could be
a good candidate for use with AB instead of Mork. However this solution cannot
be thought of for Mach V, but we can shoot for this in the next release.

Comment 3

[Rajiv Dayal]

Users/Administrators should be instructed to restrict the replicated entries for
making this work till we can truly fix the problem, as suggested by Scott
Putterman above in comment #1. 

This could be done by defining a restrictive Base DN in Directory Server
Properties dialog, General Tab. This could be taken care easily by the
administrators.

Replicated entries can be further restricted by defining search filter in the
Directory Server Properties dialog, Advanced Tab. However this could be
difficult since it requires knowing the syntax for the filter. 

Maybe we should add some instructions for all these in the user guide.

Comment 4

[James Andrewartha]

This is somewhat related to bug 117230.

Comment 5

[David Rees]

I'm not sure what "large" is considered, but I am seeing this with a 100K offline LDAP directory. I agree the user should at least be warned rather than having Thunderbird lockup when offline.

That said, I would think a simple 100K email lookup should be quite doable on modern laptops with many gigs of RAM.

Also see bug 332169 which requests for LDAP offline to be turned on when LDAP times out. Which would be great if not for this issue.