92219 - (advanced) Search Messages missing hits (imap) on both UW-IMAP and Cyrus

Status: RESOLVED INCOMPLETE

(MailNews Core :: Search, defect)

Description

[Ben Bucksch (:BenB)]

I searched through my Sent folder on my UW-IMAP folder for a message that
matches "To or CC contains heise". I get no results, although there are several
hits, e.g.
To: [e-mail]
CC: [e-mail]
. I only found the messages by doing a grep on the files on the server. Now that
I found them, I do see them in the thread pane.

Let me know, what details about the message/setup you need.

Comment 1

[Christopher Aillon (sabbatical, not receiving bugmail)]

> Let me know, what details about the message/setup you need.

How about starting with a build ID?  :-)

Comment 2

[Ben Bucksch (:BenB)]

0.9.2 branch, Jul 10.

> Now that I found them, I do see them in the thread pane.

Search still doesn't find them.

Comment 3

[Mauro Innocenti]

I've found similar problem searching messages
("Search Messages" dialog window) on my folders.
I try to find messages with some text contained
in the Body, but I get no results.
The same text was found by "grep" directly in
the mozilla's filesystem.

Mozilla 0.9.5 (Build ID: 2001101201)
(Mozilla 0.9.4 too)

Comment 4

[Waldemar Horwat]

Nominating for the next release.  This is a basic and important functionality bug.

I confirmed this bug with a recent trunk build under MacOS 9.2.1.  I searched a
large folder for a word in the body of a message and found no hits.  I had to
use BBEdit to manually open the mail file to find the messages I was looking
for.  The messages were otherwise OK and displayed correctly if opened manually.

Comment 5

[Waldemar Horwat]

Additional note:  I am using POP and keep the messages in the Mail folder that
Mozilla created on my Mac's hard disk.

This bug was originally reported under Linux.  I changed the platform to All.

Comment 6

[esther]

I tried the basic scenario for the UW-IMAP server Sent folder.  Search found all
the To or CC for the text string I entered.  (original scenario)
I tried a basic search on an IMAP server for Body contains xyz and Search found
all the messages in that mail account with xyz in the body.  (scenario dated
10-16-2001)
I tried a basic search on a large POP folder on my Mac OS 9.1 for Body Contains
xyz and the search found all instances of the message which I copied to folders,
subfolders and 3rd level subfolders.   

If any of you can reproduce the missing hits with current builds (I used
12-20-2001 builds on win, mac os9.1 and linux), please update this bug and give
steps that include 1.) exact criteria options: all of/any of; search subfolders
unchecked or checked; if more than one criteria give details;  2.) sequence of
steps (i.e. messages were in the folders originally or just moved to the folder
or filtered into the folder;  3.)type of folder: user created, special folders,
subfolder (which level), pop or imap.  4.) again confirm what type of mail
account: IMAP, POP, UW-IMAP.  Thanks.

Comment 7

[Ben Bucksch (:BenB)]

Sorry for the late answer.

> If any of you can reproduce the missing hits with current builds (I used
> 12-20-2001 builds on win, mac os9.1 and linux), please update this bug

Yes, I could reproduce the bug on a recent (~2 days ago) CVS trunk build of my
own (gcc 2.95.3 on Debian, -O2 ppro, stripped). I got the same (faulty) results
each time I did the same query during one session.

I then downloaded the latest nightly to exclude local horkage, but couldn't
reproduce the bug with the same steps. Back to my own build, I couldn't
reproduce it there anymore either.

I just tried another query using the nightly and get faulty results again.

> 1.) exact criteria options: all of/any of; search subfolders
> unchecked or checked; if more than one criteria give details;

1.
See original description.

2.
All of
Sender contains benb
Body contains we
Search in folder unsorted
Search subfolders checked

3.
Any of
Body contains Stefan
Search in folder Implementation
Search subfolders checked

Results:
2.
Several results, but not all of them. maybe 1/2 or 1/3.
3.
None (should be dozens)

> 2.) sequence of steps (i.e. messages were in the folders originally or just
> moved to the folder or filtered into the folder;

1., 2.
Messages were stored using the fcc feature (copy of sent mail to folder). The
folder has been selected via "Other".

3.
manually sorted

> 3.)type of folder: user created, special folders,
> subfolder (which level), pop or imap.

1.
Sent folder (in the original description) is a top-level folder.
2., 3.
unsorted and Implementation are a subfolders of level 3 (if top-level folders
are level 1).

Sent and unsorted are folders used to copy sent mail to. (i.e. special "Sent"
folders.)
IMAP on UW-IMAP.

> 4.) again confirm what type of mail
> account: IMAP, POP, UW-IMAP. 

IMAP on UW-IMAP.

Comment 8

[Jaime Rodriguez, Jr.]

Is this really a nsbeta1+, P2? If yes, then we need to try and targeted to a
milestone M1.0 or earlier to make the beta.

Comment 9

[Navin Gupta]

Can somebody list the reproducible steps correctly ?

Comment 10

[Navin Gupta]

I meant

1...
2..
3...

Comment 11

[Ben Bucksch (:BenB)]

1. Open Mailnews
2. Open IMAP account
3. Search|Search Messages...
4. Enter criteria similar to those mentioned above
5. Click Search

Comment 12

[Navin Gupta]

Can you elaborate on 3 & 4. Please list them again because esther could not 
reproduce this bug in general so I need more specific test cases.

3.1
3.2

4.1
4.2

Comment 13

[Ben Bucksch (:BenB)]

naving, I can't say more than I said in comment 7. I found no simple, reliable
reprodution. It did happen open enough, though.

Comment 14

[Navin Gupta]

moving to 1.0.

Comment 15

[(not reading, please use [e-mail] instead)]

mass re-assign.

Comment 16

[Wayne Mery (:wsmwk)]

(In reply to comment #13)
> naving, I can't say more than I said in comment 7. I found no simple, reliable
> reprodution. It did happen open enough, though.

Ben, Is this still a problem?  It seems to me unlikely that this would not have been fixed in some other bug.

Comment 17

[Ben Bucksch (:BenB)]

Yes, I've heard similar reports from others, using Moz 1.7, IIRC.

Comment 18

[Ben Bucksch (:BenB)]

They are using Cyrus, FWIW

Comment 19

[David :Bienvenu]

IMAP search runs on the server, unless you say to run it locally. Thus it's highly likely that it's the server that's not returning the right results.

Comment 20

[Ben Bucksch (:BenB)]

Then why am I seeing this on 2 different server implementations (UW-IMAP and Cyrus)?

Comment 21

[David :Bienvenu]

feel free to generate an imap protocol log and check for yourself if we're showing all the results the server is returning:

http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

Comment 22

[Wayne Mery (:wsmwk)]

> feel free to generate an imap protocol log and check for yourself if we're
> showing all the results the server is returning:
> 
> http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

Ben, can you supply a log?
This is probably the only why the issue will get resolved.

Comment 23

[Wayne Mery (:wsmwk)]

=> incomplete

Comment 24

[Ben Bucksch (:BenB)]

The report is not imcomplete. I just don't have time to do debugging including logs. You can reproduce it yourself.

Comment 25

[David :Bienvenu]

actually, no, search works just fine on the various imap servers I have access to.

Comment 26

[Wayne Mery (:wsmwk)]

(In reply to comment #24)
> You can reproduce it yourself.

for closure from my end...no I can't. which is why protocol log would seem to be the obvious means of achieving diagnosis and solution, and why I offered to help.

I've seen only two cases of missing hits :
* when subject spanned multiple lines - seen in Thunderbird, tried in SM against same message store but did not fail, and was not able to reproduce after restarting TB; one such case was formatted thusly 
   Subject:
       <text of subject>
* in some types of attachments

Comment 27

[Ben Bucksch (:BenB)]

OK. If I find some time, I'll try to get a log, but I can't promise it.
Note that even for me, the bug was not 100% reproducable.
It is not a local problem, though, because I saw it on different servers, different clients (4 years in between), and other people saw the same problem.

Comment 28

[Wayne Mery (:wsmwk)]

Thanks Ben. I should have commented, that I leave imap trace turned on all the time (via windows environment) to facilitate capturing difficult or unexpected imap problems. You might try that.

I've had it on so long that it has no noticeable impact on the performance of my PC, or at least none that bothers me.  Some times I even forget I have it on (as I now lament that I obviously had a trace of my missing subject hits, but didn't keep it)

Comment 29

[Wayne Mery (:wsmwk)]

Ben, what are your thoughts about the possibility of reproducing or closing this?   

I'm not finding related open bugs (there is bug 500272 but it is gmail specific).  Last significant comment here is 7 years ago, we don't have server version numbers, plus Thunderbird search has signficantly changed in version 3.x.

Comment 30

[Wayne Mery (:wsmwk)]

correction, we do have the blockers to meta bug 519202, and almost all are very specific as to the nature of their respective failure mode - https://bugzilla.mozilla.org/buglist.cgi?bug_id=1101474%2C576994%2C721167%2C404255%2C667854%2C379988%2C569009%2C460737%2C176694%2C700541%2C733852%2C476341%2C270232%2C37031%2C586729%2C338761%2C481616%2C92219%2C628098%2C507819%2C546925%2C500272&list_id=12073710&order=bug_severity - I don't know if any of them are applicable to your report

Comment 31

[Wayne Mery (:wsmwk)]

Thanks for the detail.  At this point I think this is *very* likely WFM, and in any event 1) not actionable for the reasons cited, 2) there are better bugs that have more info than such an old bug.