476341 - Local "Body Search for name used in localpart or domainpart of mail address" searches message headers, giving false positives. But searched message headers is never message headers in "payload of message/rfc822 part"(==mail data stream of attached mail).

Status: NEW

(MailNews Core :: Search, defect)

Description

[Mitra Ardron]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-GB; rv:1.9.0.6) Gecko/2009011912 Firefox/3.0.6
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3pre) Gecko/20090123 Shredder/3.0b2pre

Another bug in searching is that searching by "Body" searches Headers as well. 

Reproducible: Always

Steps to Reproduce:
1.Go to an Account
2.Search Messages
3.Choose "Body"
4.Enter text appearing in the header (though not the Subject), but NOT in the body of one of the messages.

Actual Results:  
Messages are found that shouldn't be. 


This is relevant because there are reasons for searching the body, and not the headers. For example, I have a filter that SHOULD search incoming mailing lists for my name, or the names of my company and adds a Star. The trouble is that my name turns up in the header so the filter finds EVERY message. 

This is related to the very old Bug 271222, which points out that "Entire Message" searches the body, but not the headers. 

Searching is a significant weak spot in TB, and some things to make it behave closer to what is expected should be easy to do as they seem to be simple UI issues as to what gets searched when what gets selected and its often counter-intuitive.

Comment 1

[Wayne Mery (:wsmwk)]

Mitra I'm pretty sure this is a duplicate. The duplicate may have sev=ENH

Comment 2

[Mitra Ardron]

Wayne - its similar to Bug 271222, and to numerous other bugs relating to the really poor search UI in TB, but I didn't find a duplicate.

Comment 4

[[:Aureliano Buendía]]

Cleanup *dupeme* whiteboard flag from bugs that are marked as Resolved Duplicate!

Comment 5

[WADA:World Anti-bad-Duping Agency]

Re-opening, because bug 379988 looks for "search of attachment part" and looks message/rfc822 part case, and because all duped bugs to that bugs except this bug is for "search of attachment part"(not message/rfc822), and because there are two kinds of "wrong or unwatd search" if problem of "message header is searched" case.

Known issues of "header is searched".
(a) Tb searches "message body data of message/rfc822 part" == a mail data stream
(b) bug 697021 on multipart mail. (Tb searches message header of next mail)

Mitra Ardron(bug opener), what header did you search? which issue is your case?
Because you say next, I guess your case is bug 697021.
> The trouble is that my name turns up in the header so the filter finds EVERY message.

Comment 6

[WADA:World Anti-bad-Duping Agency]

As both (a) & (b) actually exists, confirming.

Comment 7

[Mitra Ardron]

Hi Wada - I didn't search a header,  I searched  "Body" "contains" "mitra" and it found messages where Mitra was in the header - 

These messages did not contain forwarded messages or other headers inside their bodies, nor did they contain attachments. 

This is Not Bug #697021 which refers to the search continuing into an attachment, this is a search of the header fields in the message itself, it relates to where the search STARTS searching from.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to Mitra Ardron from comment #7)
> These messages did not contain forwarded messages or other headers inside their bodies, 
> nor did they contain attachments. 

Does it mean "the found mail was not multipart mail", "the found mail was text/html or text/plain mail only"?

Can you find mail which don't have "mitra" in his message body data but has mitra in his message header lines, by "Body contains mitra, AND, Content-Type doesn't contain multipart"?
(Custom header of Content-Type should be added)

> Hi Wada - I didn't search a header,  I searched  "Body" "contains" "mitra"
> and it found messages where Mitra was in the header - 

I never refer "Search for header by user". I refer "Search for Body by user" only.
I'm suspecting "Body search of Tb additionally searches headers of other mail(next mail) even though body search is requested by user", so I refer to string in message header, and the string is "string in other mail's header(next mail in mail folder)". 

> this is a search of the header fields in the message itself,
> it relates to where the search STARTS searching from.

If found mail by "Body contains mitra" has mitra in his own message header, can you produce your problem with "the wrongly found mail only in a mail folder(after Compact)"?

If found mail by "Body contains mitra" has mitra in his own message header, can you produce your problem by next?
  First mail in folder  : (Offset=0, value of "Order Received" column=0)
    A multipart mail in which string of mitra doesn't exist at any place.
  Second mail in folder : (Offset=>0, value of "Order Received" column>0)
    The mail found by your search of "Body contains Mitra".
    In message source of the mail, mitara exists in message header line.
  No other mail. Two mail only.
Bug 697021 is for problem of "first mail matches by Body contains mitra" in this case, when mitra is placed at top part of message header lines of second mail.
Bug 697021 is for "at where body search ENDS".
Please note that Received: header at top part of mail frequently contains phrase of "for mail-address of final recipient".

> This is Not Bug #697021 which refers to the search continuing into an attachment

Bug 697021 is never for "search of attachment data by Tb" nor "search of header of attachment part by Tb".
Bug 697021 is for "Body search of Tb searches message header of NEXT message, if mail is maltipart mail".
Do you read bug 697021 comment #7? (see test mails attached by the comment)

I never say "your problem is bug 697021". I merely want to know which is your case.
(a) Body search of Tb searches message/rfc822 part of a mail. This is a variant of "Body search of Tb searches attachment part".
(b) Body search of Tb searches message headers of other mail(next mail in folder), if multipart mail. This is bug 697021.
(c) Other than (a) and (b).
And, if (c), whether you problem is actually "it relates to where the search STARTS searching from.".
In other words, please surely rule out case (a) and case (b) from your problem.
And, "your case is never (a)" is already apparent by your answer.

Because I tested with Tb 7 last month in 2011, problem I could see/problem I cant's see may be different from orginal comment #0 reported in 2009. Comment #0 may be problem of "Body search of Tb around 2009 searched entire message including his own message headers".
So, my questions also imply "can you see your problem with recent Tb 7"?

Comment 9

[Mitra Ardron]

OK - so maybe this is Bug #697021 - I am unclear to setup the tests you want, but this bug is really easy to reproduce, so I assume you can test it easily for the tests you want.

Comment 10

[WADA:World Anti-bad-Duping Agency]

I'm asking very simple tests, although checking of test results is slightly hard...

(Test-1) To know "false positive" occurs on multipart mail only or not.
(1) Add custom header of Content-Type at Search(Search of context menu of folder)
(2) Execute Search
(2-a) Body contains Mitra => Your problem occurs
(2-b) Body contains Mitra, AND, Content-Type contains multipart
(2-c) Body contains Mitra, AND, Content-Type doesn't contain multipart

(2-a) == (2-b) + (2-c), if "multipart or not" is primary condition.
Can you see "false positive" in (2-c)?
"false positive" in this case: even though no string of "mitra" in message, message is returned by (2-c).
If required, other condition such as "Subject contains", to reduce number of hits, for ease of test result checking.

(Test-2) To know "mitra at where" can be searhed by Tb in your case.
(1) Create a local folder(call folderX),
    Show "Order Received" column(offset in mail folder file)
(2) Copy a multipart mail which doesn't have string of "mitra"(no quote) in message source, to folderX(call mail-0).
(3) Copy a mail which doesn't have string of "mitra"(no quote) in message source, to folderX(call mail-1)
(4) Compact folderX
(5) Search for "string in message header of MAIL-1 only"(no such string in message source of mail-0).
    Create Tb's tag named mitra.
    mail-0 : Remove any tag    => X-Mozilla-Keys:
    mail-1 : Add tag mitra     => X-Mozilla-Keys: mitra
    (5-a) Search: Body contains mitra
    (5-b) Search: Body contains X-Mozilla-Keys: mitra
(6) Search for "string in message header of mail-0 only".
    mail-0 : Remove any tag, Add tag mitra => X-Mozilla-Keys: mitra
    mail-1 : Reemove any tag               => X-Mozilla-Keys:
    (6-a) Search: Body contains mitra
    (6-b) Search: Body contains X-Mozilla-Keys: mitra
Which mail is retturned by these Body Search?
 
Replace string of "mitra" by appropriate one when you test, for ease of testing. please.

If a mail is multipart mail(mail-0), and mail at next of the mail(mail-1) has Received: header, Reply-To: header etc. which has "mitra"(no quote) in it, Body Search returns mail-0, as seen in (Test-2).
Mails of "false positive" at (Test-1)/(2-b) is such case only?
Or different case is involved?

To see it, next text is probably helpful.
(0) Disable auto-compact.
(1) Create two local folders(call folderX, folderY). 
(2) Call local folder on which your problem occurs "folderP". 
    At folderP, show "Order Received" column, sort by the column, in ascending.  
    Compact, do body search: Body contains Mitra => Your problem occurs
(3) Copy all mails in folderP to folderX.
    At folderX, do body search: Body contains Mitra => Your problem occurs
(4) At folderX, do search:
       Received contains mitra,
       OR, Retun-Path contains mitra,
       OR, Delivered-To contains mitra,
       (if required, add other headers at top part of mail) 
    Move all found mails to folderY.
(5) At folderX, do body search: Body contains Mitra
    => Your problem probably still occurs
(6) At folderX, Compact
(7) At folderX, do body search: Body contains Mitra
    => Does your "false positive" problem occur?

Comment 11

[Mitra Ardron]

I'm sorry Wada, but I don't have time to do these detailed tests. This bug is trivial to reproduce, so whoever is going to fix it can run whatever tests they need to.

Comment 12

[WADA:World Anti-bad-Duping Agency]

Because Bug 697021 is the prime suspect("never bug 700541 for message/rfc822" is apparent), xref to that bug, for ease of tracking.

Comment 13

[WADA:World Anti-bad-Duping Agency]

Changing bug summary to explain that your this bug is apparently report of "false positive in Body Search due to search of message headers of mail or mails in local mail folder file" and the searched message headers is never "message headers in payload data of message/rfc822 part".
Mitra(bug opener), please correct bug summary if it's wrong or not appropriate.