1277195 - LDAP query fails since update to 45.1.1. Decouple ldap search format from local query format

Status: NEW

(Thunderbird :: Address Book, defect)

Description

[Joey Crockett]

Attachment: 2016-06-01 10_21_22-ldap_trace.pcapng.png

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0
Build ID: 20160210153822

Steps to reproduce:

Using the search bar on the top right of the address book.

Thunderbird is configured with an LDAP address book, that uses a regular, unencrypted connection on port 389 with simple Login method without any credentials. All other options have the default settings. Offline replication of the address book works fine.


Actual results:

No results were shown in the result window.

Capturing the network traffic reveals that searching from the search bar produces a very long filter query. The server then responds with a constraintViolation and no results are shown. The same error happens, when searching from the contacts sidebar inside the write message window. 
Using the advanced address book search shows results, as it performs a query with a simple CN= filter. Autocomplete does also work.


Expected results:

Results should have been shown from all search fields.

Comment 1

[Joey Crockett]

Three other colleagues (2x Windows, 1x Mac) have confirmed this error. As soon as they upgraded to 45.1.1, LDAP stopped working.

Comment 2

[Jorg K (CEST = GMT+2)]

Kent, who does LDAP? Only peer: Joshua.

Comment 3

[Wayne Mery (:wsmwk)]

(In reply to Joey Crockett from comment #1)
> Three other colleagues (2x Windows, 1x Mac) have confirmed this error. As
> soon as they upgraded to 45.1.1, LDAP stopped working.

what happens if you install 45.1.1 from download of http://getthunderbird.com/

Comment 4

[:aceman]

I've dived in some LDAP search stuff in another bug.
I can confirm the AB search looks into more fields than e.g. the compose autocomplete search. So there are some differences.

Joey, do you use multiple words in the search string? What does "very long filter query" mean? Is it long in the number of characters sent? Can you maybe paste the query here (with some junk search keyword).
Do you use some special characters in the search string?

Comment 5

[Joey Crockett]

Attachment: Search Query

This is a screenshot of the captured ldap package.

Comment 6

[Joey Crockett]

@Wayne:
During testing, I have deinstalled and reinstalled the current version.
I have also deleted the Thunderbird folders in appdata and localappdata and recreated my profile.
No changes.

@:aceman:
I attached a screenshot. It shows Thunderbird searching for my name.

Comment 7

[Wayne Mery (:wsmwk)]

perhaps compare an ldap log from a version that works against a version that does not work
See https://wiki.mozilla.org/MailNews:Logging   use module setting of  ldap:5

Comment 8

[Joey Crockett]

Attachment: LDAP Log 38.8.0

Comment 9

[Joey Crockett]

Attachment: LDAP Log 45.1.1

Comment 10

[Joey Crockett]

These two files contain the ldap logs of searching for the string "crock" in their respective versions.
It is clearly visible, that the search filter variable changed:

38.8.0:
aFilter = '(&(|(mail=*crock*)(cn=*crock*)(givenName=*crock*)(sn=*crock*)))';

45.1.1
aFilter = '(&(|(cn=*crock*)(givenName=*crock*)(sn=*crock*)(mozillaNickname=*crock*)(mail=*crock*)(mozillaSecondEmail=*crock*)(&(description=*crock*))(o=*crock*)(ou=*crock*)(title=*crock*)(mozillaWorkUrl=*crock*)(mozillaHomeUrl=*crock*)))';


Imho, this needs to be reverted.
I have no idea, why attributes like mozillaNickname and the like should be inside a query towards a directory server.

Comment 11

[:aceman]

Can you try reducing the value in the pref mail.addr_book.quicksearchquery.format ?
Currently it is at (or(DisplayName,c,@V)(FirstName,c,@V)(LastName,c,@V)(NickName,c,@V)(PrimaryEmail,c,@V)(SecondEmail,c,@V)(and(IsMailList,=,TRUE)(Notes,c,@V))(Company,c,@V)(Department,c,@V)(JobTitle,c,@V)(WebPage1,c,@V)(WebPage2,c,@V)) . Try removing the fields you do not want.

Comment 12

[Joey Crockett]

Did that and tested different combinations. It boils down to this:

Values like NickName or SecondEmail seem to correspond to filters with the prefix mozilla(...).
As soon as one of these is present, the server throws a constraintViolation.

I currently have it at
(or(DisplayName,c,@V)(FirstName,c,@V)(LastName,c,@V)(PrimaryEmail,c,@V)(Company,c,@V)(Department,c,@V)(JobTitle,c,@V)(and(IsMailList,=,TRUE)(Notes,c,@V)))
and it works fine.

Thank you very much.


PS: Might be worth changing the default setting for this again. I can imagine more people running into it.

Comment 13

[:aceman]

(In reply to Joey Crockett from comment #10)
> 45.1.1
> aFilter =
> '(&(|(cn=*crock*)(givenName=*crock*)(sn=*crock*)(mozillaNickname=*crock*)(mai
> l=*crock*)(mozillaSecondEmail=*crock*)(&(description=*crock*))(o=*crock*)(ou=
> *crock*)(title=*crock*)(mozillaWorkUrl=*crock*)(mozillaHomeUrl=*crock*)))';
> 
> 
> Imho, this needs to be reverted.
> I have no idea, why attributes like mozillaNickname and the like should be
> inside a query towards a directory server.

It is because that same query is run also on local addressbooks and there people want to search over these fields.

Yes, so we need to think if we decouple the query template for the LDAP from the query for local addressbooks. Not sure if it is easily possible.

Comment 14

[Kent James (:rkent)]

(In reply to Jorg K (GMT+2, PTO during summer, NI me) from comment #2)
> Kent, who does LDAP? Only peer: Joshua.

jcranmer is the only person I know who has looked at LDAP at all.

Comment 15

[Wayne Mery (:wsmwk)]

(In reply to :aceman from comment #13)
> (In reply to Joey Crockett from comment #10)
> ...
> > Imho, this needs to be reverted.
> > I have no idea, why attributes like mozillaNickname and the like should be
> > inside a query towards a directory server.
> 
> It is because that same query is run also on local addressbooks and there
> people want to search over these fields.
> 
> Yes, so we need to think if we decouple the query template for the LDAP from
> the query for local addressbooks. Not sure if it is easily possible.

Which bug is this a regression of?

Comment 16

[:aceman]

Bug 118624.

Comment 17

[eugene.tsuno]

I don't think the fix that is in 118624 fixes this problem.  That one just customizes internal searches, it doesn't address external ldap searches or decouples the search strings

Comment 18

[eugene.tsuno]

BTW, this also affects 45.2.  Unless you test against an LDAP server that does referrals or does not like "unknown" attributes, you will not see a problem.

Comment 19

[Martin Bremer]

Searching for attributes that are not in your LDAP schema will produce quite a lot of warnings or errors in your logs. All the mozilla-related attributes are non-default, therefore most people will be faced with a search string that will generate errors or warnings in the LDAP logs.

With an LDAP addressbook that is used by a lot of users, this makes the error logs near unusable due to the high amount of warnings. I solved this issue on the LDAP server by modifying the schema and added the extra attributes. But this is just a workaround for the problem, not a real solution. Many people have no access to their corporate addressbook schema.

A split between an internal search and an external LDAP search is an elegant solution.

Comment 20

[Martin Bremer]

This bug has been sitting here for nearly a year, without any progress. Bug reports are quite clear what is happening and what needs to be done to resolve it. It would be nice if someone would start working on this.

Comment 22

[Sven Giermann]

I'm glad, I finally found this bug...
I was reported this exact problem: search does not work, but address completion does.
And with the hints above, I was able to find a working config.

BUT: For some reason I had to strip off much more than Joey in comment #12.
It only works for the 3 main fields for me:
user_pref("mail.addr_book.quicksearchquery.format", "(or(DisplayName,c,@V)(FirstName,c,@V)(LastName,c,@V)(PrimaryEmail,c,@V))");

But even worse: using this, I get 25% CPU usage after the first search (4 core CPU, meaning 1 thread hanging at 100%) for about 1 minute. Doing the second search makes thunderbird.exe using 80-100% of my CPUs, maybe caused by more search results (searching for 'a') - leaving the PC unsable for one or two minutes.
Something's very strange with this LDAP update - I did not see anything else with prior versions...
But this fault keeps me from deploying the workaround setting to all our client machines!

Comment 23

[:aceman]

Thomas, would you like to take this?

Comment 24

[Thomas D. (:thomas8)]

Unfortunately I don't have time currently...

Comment 25

[Wayne Mery (:wsmwk)]

I see several users able to reproduce this issue...

Can you create a new profile https://support.mozilla.org/en-US/kb/using-multiple-profiles to see if this reproduces with beta https://archive.mozilla.org/pub/thunderbird/releases/90.0b2/ ?

beta implements new ldap code https://www-stage.thunderbird.net/en-US/thunderbird/90.0beta/releasenotes/

Comment 26

[YG]

I tested LDAP with a secured connection over SSL in Thunderbird stable v102 and Beta v112 and can confirm that LDAP search works for me. This bug may still be valid for unsecured connections.

Comment 27

[hujq]

In thunderbird 102 the default value of "mail.addr_book.quicksearchquery.format" is still "(or(DisplayName,c,@V)(FirstName,c,@V)(LastName,c,@V)(NickName,c,@V)(PrimaryEmail,c,@V)(SecondEmail,c,@V)(and(IsMailList,=,TRUE)(Notes,c,@V))(Company,c,@V)(Department,c,@V)(JobTitle,c,@V)(WebPage1,c,@V)(WebPage2,c,@V))" and used for LDAP search, and it still doesn't work for me. Changing it to "(or(DisplayName,c,@V)(FirstName,c,@V)(LastName,c,@V)(PrimaryEmail,c,@V))" works.

Comment 28

[Joey Crockett]

Can confirm, this is still an issue in 115.2.2.
No results whatsoever until the format is changed.