853393 - Searching messages is STILL NOT working (IMAP online search. IMAP server returns nothing to 'uid SEARCH UNDELETED BODY "ftp"' when mail is multipart or when text part under multipart is base64 encoded)

Status: UNCONFIRMED

(Thunderbird :: Search, defect)

Description

[hawran]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:19.0) Gecko/20100101 Firefox/19.0
Build ID: 20130307164200

Steps to reproduce:

1. Clicked on a folder with messages (intentionally not on a root of my folders).
2. Right mouse / Search messages ...
3. Choosing "Run search on server" does not matter.
4. Match all of the following.
5. Body / Contains / <a phrase to find>
6. Search


Actual results:

Nothing has been found.
Despite the fact I'd chosen a phrase I copied from one particular message.


Expected results:

At least that one message found.

Comment 1

[hawran]

Name  Thunderbird
Version 17.0.4
User Agent  Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130308 Thunderbird/17.0.4
Profile Directory  (Local drive)
Application Build ID  20130308141847

Comment 2

[hawran]

A snippet of the message's source:

Content-Type: multipart/related;
	boundary="_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_";
	type="text/html"
MIME-Version: 1.0

--_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64

PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt
YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj
...
aHRtbD4=

--_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_
Content-Type: image/jpeg; name="image001.jpg"
Content-Description: image001.jpg
Content-Disposition: inline; filename="image001.jpg"; size=9925;
	creation-date="Mon, 11 Mar 2013 14:47:49 GMT";
	modification-date="Mon, 11 Mar 2013 14:47:49 GMT"
Content-ID: <image001.jpg@01CE1E67.6604D2E0>
Content-Transfer-Encoding: base64

/9j/4AAQSkZJRgABAQEARwBHAAD/2wBDAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEB
AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQH/2wBDAQEBAQEBAQEBAQEBAQEBAQEBAQEB
AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQH/wAARCABLAK8DASIA
...
IyxQOygGuqoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigD/2Q==

--_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_
Content-Type: image/jpeg; name="image002.jpg"
Content-Description: image002.jpg
Content-Disposition: inline; filename="image002.jpg"; size=1821;
	creation-date="Mon, 11 Mar 2013 14:47:50 GMT";
	modification-date="Mon, 11 Mar 2013 14:47:50 GMT"
Content-ID: <image002.jpg@01CE1E67.6604D2E0>
Content-Transfer-Encoding: base64

/9j/4AAQSkZJRgABAQEAYABgAAD/4QBoRXhpZgAATU0AKgAAAAgABAEaAAUAAAABAAAAPgEbAAUA
AAABAAAARgEoAAMAAAABAAIAAAExAAIAAAASAAAATgAAAAAAAABgAAAAAQAAAGAAAAABUGFpbnQu
...
Z02GaGVdrxOtrGGVgehBBBHtRRXyfGWMnQp0lBLVvfyse3kFGNWc+bokf//Z

--_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_
Content-Type: image/jpeg; name="image003.jpg"
Content-Description: image003.jpg
Content-Disposition: inline; filename="image003.jpg"; size=1605;
	creation-date="Mon, 11 Mar 2013 14:47:50 GMT";
	modification-date="Mon, 11 Mar 2013 14:47:50 GMT"
Content-ID: <image003.jpg@01CE1E67.6604D2E0>
Content-Transfer-Encoding: base64

/9j/4AAQSkZJRgABAQEAYABgAAD/4QBoRXhpZgAATU0AKgAAAAgABAEaAAUAAAABAAAAPgEbAAUA
AAABAAAARgEoAAMAAAABAAIAAAExAAIAAAASAAAATgAAAAAAAABgAAAAAQAAAGAAAAABUGFpbnQu
...
Hb+JtehijHyoLgsq46dQa3FOra/GF1LV9T1KMNuKTyFkz/u9P0oor5/FUacXzKKv6H2OCxFaUVCU
212u7GsixabbZb5FHGSOv+Jqf+y9e/6F/Vf+/LUUV5des6aTte/f5HuYXDqrJxbatba3W/dPsf/Z

--_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_
Content-Type: image/jpeg; name="image004.jpg"
Content-Description: image004.jpg
Content-Disposition: inline; filename="image004.jpg"; size=1924;
	creation-date="Mon, 11 Mar 2013 14:47:50 GMT";
	modification-date="Mon, 11 Mar 2013 14:47:50 GMT"
Content-ID: <image004.jpg@01CE1E67.6604D2E0>
Content-Transfer-Encoding: base64

/9j/4AAQSkZJRgABAQEAYABgAAD/4QBoRXhpZgAATU0AKgAAAAgABAEaAAUAAAABAAAAPgEbAAUA
AAABAAAARgEoAAMAAAABAAIAAAExAAIAAAASAAAATgAAAAAAAABgAAAAAQAAAGAAAAABUGFpbnQu
...
j90vUY4NFFfDca4mWEcKNFJRu/0Pv+BaKxCqV6jfNof/2Q==

--_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_
Content-Type: image/jpeg; name="image005.jpg"
Content-Description: image005.jpg
Content-Disposition: inline; filename="image005.jpg"; size=1987;
	creation-date="Mon, 11 Mar 2013 14:47:50 GMT";
	modification-date="Mon, 11 Mar 2013 14:47:50 GMT"
Content-ID: <image005.jpg@01CE1E67.6604D2E0>
Content-Transfer-Encoding: base64

/9j/4AAQSkZJRgABAQEAYABgAAD/4QBoRXhpZgAATU0AKgAAAAgABAEaAAUAAAABAAAAPgEbAAUA
AAABAAAARgEoAAMAAAABAAIAAAExAAIAAAASAAAATgAAAAAAAABgAAAAAQAAAGAAAAABUGFpbnQu
...
WieL/EXiD4hQeP8Axx4vlim1G5huTcPIYwViRByVAB6cV083wl8b+IpW1CLS8RX5NwgKEHa/zD9D
XI/Bb/ketB/67ivuyD/UJ/uivVoUo04JI/O8yx1fMMRLEYmV5H//2Q==

--_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_--

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to hawran from comment #0)
> 2. Right mouse / Search messages ...
> 3. Choosing "Run search on server" does not matter.

Because IMAP Online Search, two things are relevant;
- IMAP command Tb generated for MAP Online Search
- Search executed by IMAP server by the IMAP command
Can you get and check IMAP log for your IMAP Online Search request?
See bug 402793 comment #28 for getting IAP log.

What mail was not found even though the mail should be found by your search? (false nagative)
What mail was found even though the mail should not be found by your search? (false positive)

(In reply to hawran from comment #0)
> 2. Right mouse / Search messages ...
> 3. Choosing "Run search on server" does not matter.
>(snip)
> 5. Body / Contains / <a phrase to find>
(In reply to hawran from comment #2)
> A snippet of the message's source:
> 
> Content-Type: multipart/related;
> 	boundary="_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_";
> 	type="text/html"
> MIME-Version: 1.0
> 
> --_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_
> Content-Type: text/html; charset="utf-8"
> Content-Transfer-Encoding: base64
> 
> PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
>(snip)
> aHRtbD4=
> 
> --_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_
> Content-Type: image/jpeg; name="image001.jpg"
> Content-Description: image001.jpg
>(snip)
> --_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_--

Because multipart/related without start=, "Message Body" for ordinal mailer/mail server is "first text/html part with no data line" in the mail, isn't it?
(all other parts is "part which is not referred by message body of multipart/related in multipart/related". some mailers ignore them because they are not used by message body, but some mailers show them as if attacment because they are not shown in message body as part of html message body.)

Comment 5

[hawran]

(In reply to WADA from comment #4)
> (In reply to hawran from comment #0)
> > ...
> > PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
> >(snip)
> > aHRtbD4=
> > 
> > --_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_
> > Content-Type: image/jpeg; name="image001.jpg"
> > Content-Description: image001.jpg
> >(snip)
> > --_006_B4F41F4D532B3E429C7DEFCCE8ABE74F068A9CE94Bcznagmbx01int_--
> 
> Because multipart/related without start=, "Message Body" for ordinal
> mailer/mail server is "first text/html part with no data line" in the mail,
> isn't it?
> (all other parts is "part which is not referred by message body of
> multipart/related in multipart/related". some mailers ignore them because
> they are not used by message body, but some mailers show them as if
> attacment because they are not shown in message body as part of html message
> body.)

Wada, I am NOT a messaging/imap/... guru, I have no idea what are you talking about.

Comment 6

[hawran]

(In reply to WADA from comment #4)
> (In reply to hawran from comment #0)
> > 2. Right mouse / Search messages ...
> > 3. Choosing "Run search on server" does not matter.
> 
> Because IMAP Online Search, two things are relevant;
> - IMAP command Tb generated for MAP Online Search
> - Search executed by IMAP server by the IMAP command
> Can you get and check IMAP log for your IMAP Online Search request?
> See bug 402793 comment #28 for getting IAP log.

I'm here to add it's still not working, however I've performed a couple of tests trying to get some imap logs (NSPR_LOG_MODULES=timestamp,imap:5,pop3:5,nntp:5,smtp:5):

Searching an email for the "ftp" phrase within the body.
At least the body should be found (a snippet of visible body):
-----------------------------------------
...
The archive is also uploaded on ftp server. Use the following credentials
directory: ...
on: ftp. ...
...
-----------------------------------------

The message source snippet:
-----------------------------------------
Content-Type: multipart/mixed;
	boundary="_002_CALg32w9WxrfxPHcNfu644rVuX3XeWA52Q36iVwz2y0ySdszQmailgm_"
MIME-Version: 1.0

--_002_CALg32w9WxrfxPHcNfu644rVuX3XeWA52Q36iVwz2y0ySdszQmailgm_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Hi,<br><br>Hope you&#39;ll receive this ma=
il. Attached are the core dump and log.<br><br></div><div>The archive is al=
so uploaded on ftp server. Use the following credentials<br><br></d=
iv>
<div>directory: ...<br>on: <a href=3D"http:/=
/ftp.acision.com">ftp. ....</a><br>user: ....<br>password:...<br></div><div><br></div>Cheers,<br></div>...<br></div>

--_002_CALg32w9WxrfxPHcNfu644rVuX3XeWA52Q36iVwz2y0ySdszQmailgm_
Content-Type: application/x-gzip; name=...
...
-----------------------------------------

1. Searching on a server, within a folder with the message:
It was unexpectedly quick, NOT FOUND.
Grepping log:
$ egrep -i 'ftp|search' imap.130520-105738.searching-on-server--within-folder.log
$

2. Quick filter, within a folder with the message:
Rather longer, however NOT FOUND
Grepping log:
$ egrep -i 'ftp|search' imap.130520-105828.quick-searching--within-folder.log
$

3. Searching on a server, within a parent folder of the folder with the message:
Interesting results (please, see below).
Greppig log:
$ egrep -i 'ftp|search' imap.130520-112214.searching-on-server--parent-folder.log
2013-05-20 09:22:45.790267 UTC - -1543505040[a4008e20]: a4230c00:imap.intinfra.com:S-INBOX/acision/BT/PRs:ProcessCurrentURL:imap://[e-mail]:143/search%3EUID%3E/INBOX/acision/charging/pokus_druhy/CRs_after_cp18/00023343%3ESEARCH%20UNDELETED%20BODY%20%22ftp%22:  = currentUrl
2013-05-20 09:22:45.888407 UTC - -1543505040[a4008e20]: a4230c00:imap.intinfra.com:S-INBOX/acision/charging/pokus_druhy/CRs_after_cp18/00023343:SendData: 73 uid SEARCH UNDELETED BODY "ftp"
2013-05-20 09:22:45.917501 UTC - -1543505040[a4008e20]: a4230c00:imap.intinfra.com:S-INBOX/acision/charging/pokus_druhy/CRs_after_cp18/00023343:CreateNewLineFromSocket: * SEARCH
2013-05-20 09:22:45.917561 UTC - -1543505040[a4008e20]: a4230c00:imap.intinfra.com:S-INBOX/acision/charging/pokus_druhy/CRs_after_cp18/00023343:CreateNewLineFromSocket: 73 OK SEARCH completed.
$

- Interesting, I would expect the 'CRs_after_cp18' as a parent of the search as that was the highlighted folder I was in.

- One of the FOUND was a message only because of this source (a line snippet):
uSdKRJT7faxqVwpMdX2ce9Gw2tEFTP0DUDnH2fMcOqXxWgtX/pqdEY/1d0ZEPcX8guy9uzk+VyTy

And more similar lines. Yes, base64.

- One of the correct FOUND:
...
<p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" face=3DCalibri><span =
lang=3DEN-US
style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>They are avail=
able
on <a href=3D"https://ftp.....
1282/">https://ftp....</=
a>
...

- However, the message I was looking for was NOT FOUND.



> 
> What mail was not found even though the mail should be found by your search?
> (false nagative)
> What mail was found even though the mail should not be found by your search?
> (false positive)

Wada, neither of them.
It's simple, it just found NOTHING.

Comment 7

[hawran]

One note: as I'm not familiar with this parts, if here's whatever private/confidental/..., feel free and remove/edit my previous post, please.
Thank you.

Comment 8

[WADA:World Anti-bad-Duping Agency]

> 73 uid SEARCH UNDELETED BODY "ftp"
> * SEARCH
> 73 OK SEARCH completed.

Search command used by Tb is pretty normal one, and nothing is returned from your IMAP server.
What can IMAP mail client do?
( See RFC 3501 for serach command/reponse)
(   http://tools.ietf.org/html/rfc3501#section-6.4.4 )
(   http://tools.ietf.org/html/rfc3501#section-7.2.5 )

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to hawran from comment #6)
> (NSPR_LOG_MODULES=timestamp,imap:5,pop3:5,nntp:5,smtp:5):

Because your problem is IMAP problem, remove ",pop3:5,nntp:5,smtp:5" in order to exclude log for needless protocol. And, stop any "automatic new mail check" whe you get IMAP log, in order to excluded log for irrelevant IMAP folders and mails.

Comment 10

[Ivan]

IMAP search doesn't work here too, there simply no way to convince TB to REALLY search on the server.

On a new installation where local storage is off, search will simply find nothing.
No matter if it is Ctrl-K search or Right click->Search Messages.
No matter if Global Indexer is on or off.
No matter if apparently TB knows the hierarchy of your folders.
No matter if Run search on server and Search in subfolder are checked.
No matter if check_all_folders_for_new is true.

Turning on local storage on LARGE IMAP archive may simply freez TB and could be nonsensical on a netbook connected to a 50Gb mail archive.

The only way to get results back is to visit ALL folders of your IMAP and just then TB seems to get aware of where to search.
And still I haven't been able to check if TB ALWAYS use server side search if explicitly asked and if it really return consistent results.

There should be somewhere a cache of IMAP folders (or... well just download it one more time if a "search on server" is set) and TB should truly recursively SELECT and SEARCH in each folder.

This has been raised elsewhere... if you're searching in bodies of messages IMAP search can be pretty slow if the server doesn't have FTS capability.
Still slow is better than nothing and it is less puzling for users.
Users that have disabled local storage should generally know what they are doing and won't get surprised if search is slow.

At a later point some improvement in the UI could make it clearer that you could search just in headers or body too (at a cost) but at least make IMAP SEARCH work.

Comment 12

[Wayne Mery (:wsmwk)]

It's also possible for somone with online search problem to be seeing bug 404255

Comment 13

[wdehoog]

It looks like searches for 'To:' are only performed on locally cached folders and 'search runs on server' is only being done when searching in 'Body:'.

We have a 25G email collection. Opening more folders (browsing the collection) result in finding more emails when searching for 'To:'. For 'Body:' this does not seem to happen.

(windows 10, thunderbird 68.11.0, dovecot 2.3.5+solr)