721167 - Body search of Gmail IMAP includes headers / If condition contains "body", Tb always executes Online Search for IMAP folders regardless of "Search Online" option of Search Folder

Status: NEW

(Thunderbird :: Search, defect)

Description

[John Hopkins (:jhopkins)]

1) Send yourself an email:
   cc: [e-mail]
   body: nothing to see here

2) Once the email arrives, create a Saved Search as follows:
   Body Contains testing123

Expected: no matches
Actual: the email we sent ourselves matches on the To: header.

In practice, this makes filtering on my own name in the body of an email difficult (if not impossible) because the To: header matches even if my name doesn't occur in the body text.

Comment 1

[WADA:World Anti-bad-Duping Agency]

There are many bug reports for phenomenon of "body search searches message header", but currently clear case is next only.
(i) Actual false positive. Message header of next mail is searched.
  bug 697021 : if multipart mail, message headers of next mail is searched.
(ii) True positive for current body search, but false positive for user.
  bug 700541 : message header in message/rfc822(attached mail) is searched.
               this is "data in body of mail" in current implementation
               as data in attached text file is "data in body of mail".
In case (i), if both (a) false positive mail(multipart mail), and (b) next mail, has same header(e.g. To: aaa a@a.a.a, and search for aaa), it looks for user that Tb searched message header of (a), so user's bug report is usualy 
one like "Tb searched message header of the mail==(a)" and user never provides data of next mail in such case, problem analysis is very hard(nearly impossible).
   
Which case? Or different case?

Comment 2

[John Hopkins (:jhopkins)]

I created a second mail account in Thunderbird (pointing to my Gmail account), deleted all emails, emptied the trash, and ran through the test in my bug description.  I had the same outcome occur.
The email was not a multipart and there are no other emails so (i) doesn't apply.
I don't believe (ii) applies either.

Here is the raw source of the email:

------8<-------
MIME-Version: 1.0
Received: by 10.52.24.173 with HTTP; Wed, 25 Jan 2012 18:26:20 -0800 (PST)
Date: Wed, 25 Jan 2012 21:26:20 -0500
Delivered-To: [e-mail]
Message-ID: <CAEeq63ps7z91zP7VhSPumGUNx0QZohywp5wNbLisDJAwTqd=[e-mail]>
Subject: this is a test
From: John Hopkins <[e-mail]>
To: [e-mail]
Cc: [e-mail]
Content-Type: text/plain; charset=ISO-8859-1

nothing to see here
----->8--------

Are you able to reproduce this?

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to John Hopkins (:jhopkins) from comment #2)
> I created a second mail account in Thunderbird (pointing to my Gmail account)

Gmail IMAP?
If IMAP, Local search? Online search?

Following is questions for "false negative"(not found) case. So it's irrelevant to your "false positive" case, unless problem is "hits with any string if mail data is not downloaded yet" like one.
If IMAP, offline-use=on folder? offline-use=off folder? (Folder Properties/Synchronization)
If offline-use=on folder and local body search, is all mail data auto-sync'ed? (In activity manager panel, <account> is now update messsage)
If offline-use=folder, is mail data downloaded by "Download Now"?

> there are no other emails so (i) doesn't apply.

Needless to say, because non-mailtipart mail, I believe your problem is not that bug. I couldn't observe problem of that bug on non-multipart mail.
However, if local mail folder(POP3 or Local Folders), that bug searches "mail data placed at next of the mail" regardless of active/deleted(flagged as deleted) mail. And, even if only one active mail, deleted mail can exist after the mail. So, result can be different between "before Compact" and "after Compact".

> Expected: no matches
> Actual: the email we sent ourselves matches on the To: header.

When "Body contains testing123", why it can hit with "To: [e-mail]" header of the mail?
"Search Folder" can have multiple search target folders. Single search target folder? Multiple search target folders?

> Version: 10

It my be regression in Tb 10. IIRC, "Body search(name might be different)" was "full message data search including headers" in the past, and was corrected to "full message data search excluding headers".
Do you see similar result with "Body Cotains johnnycobol", "Body Cotains Hopkins" etc.?
Do you see your problem in Tb 9 or Tb trunk daily build(Tb12.0a1)?

Any way, to make it sure, please test after "delete all mails except one mail" and "Compact folder". In both IMAP folder case and local mail folder case, data in mail folder file is surely data of one mail only after Compact. (check mail folder file content, Inbox not Inbox.msf if Inbox, after Compact, please.)
And, please attach (a) Search Folder's definition(an entry in virtualFolders.dat for the search folder), (b) full message source. Please don't paste them, save to file and attach file to this bug.

Comment 4

[WADA:World Anti-bad-Duping Agency]

Do you see your problem with Edit/Find/Search Messages(or Context menu of folder/Search) and "Filter these messages..." in Quick Filter Bar/with Body only?

Comment 5

[WADA:World Anti-bad-Duping Agency]

Attachment: IMAP log for Online "Body Search" by Gmail IMAP

Gmail IMAP of Inbox has mail of UID=12099 only.
Gmail IMAP returns "* SEARCH 12099" to '37 uid SEARCH UNDELETED BODY "soarex"'.
As seen in data fetched by "51 UID fetch 12099 (UID RFC822.SIZE BODY[])", message body is "IMAP mail" line only.

If you are talking about this phenomenon with Gmail IMAP, with Online Body Search by Tb, it's Gmail's current design/implementation, and IMAP mail client can do nothing on it.
You need to execute local search of Tb instead of online search, if you want to search message body only.

Comment 6

[John Hopkins (:jhopkins)]

I don't have time to answer all the questions in depth right now, but I did do a local folder test and was not able to reproduce the problem.  I'm using IMAP.
I tried "Search online" checked and unchecked but had the same problem.  Inbox is selected for Offline downloading.

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to John Hopkins (:jhopkins) from comment #6)
> I'm using IMAP.
> I tried "Search online" checked and unchecked but had the same problem. 

I could see following with Tb 3.0b2pre(final build), Tb7.0a1((final build), Tb 9.0.1, Tb12.0a1(2012-01-19 build).
- If search target folders of Search Folder(Virtual Folder) is IMAP folder only,
  (i.e. no local mail folder is included in search target folder)
  Online Search is always executed(SEARCH command isalways issued),
  regardless of "Search Online" option setting of the Search Folder.
- If search target folders of Search Folder(Virtual Folder) is mixed,
  (i.e. IMAP folder(s) and local folder(s) is included in search target folder)
  "Search Online" option setting of the Search Folder works on IMAP folder.
  Note: "Local Search is actually done for local folders or not
         when Search Online is checed or not" is not unknown yet.

FYI.
- Edit/Find/Search Messages(and Context menu of folder/Search) :
    "Search Online" option works as expected.
- "Filter these messages..." in Quick Filter Bar/with Body only :
    Local Search is always executed even when IMAP folder.
    There is no option to execute Online Search. (known issue)

Because phenomenon is seen with Tb 3.0b2pre(final build), it looks old issue.
Confirming.

Comment 8

[WADA:World Anti-bad-Duping Agency]

Offline-use=On/Off, mixing of Local mail folders etc. was irrelevant.
Existence of "Body" only was relevant.

virtualFolders.dat content.
> scope=imap://...offline-use=Off-folder...|imap://...offline-use=On-folder...|mailbox://nobody@Local%20Folders/Inbox
> terms=OR ("received-spf",contains,google.com) OR (body,contains,soarex) OR (subject,contains,xyz)
> searchOnline=false
Following command was issued for any IMAP folder, even though searchOnline=false.
> 105 uid SEARCH UNDELETED (OR (OR HEADER "received-spf" "google.com" BODY "soarex") SUBJECT "xyz")

Phenomenon of this bug was:
(A) If "body" is conained in condition, searchOnline=false is ignored,
    and Online Search is always executed for IMAP folder by Search Folder.
(B) Body Search of Gmail IMAP searches message header too.

Comment 10

[Leprechaun]

Hi, I can confirm the "phenomenon" (either bug or bad feature) in version 3.1 Linux for any "regular", POP3 folders for any Mail Provider. It is a general issue, non specifically tied to IMAP nor Gmail. 

Searching for a <word> in "Body" of the message detects any message having <word> also in the "To:" field, at least. Not tried with other fields like Cc, Bcc, From, etc.

It seems that "search in Body" effectively searches also in the header of the message.

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to Leprechaun from comment #10)
> Hi, I can confirm the "phenomenon" (either bug or bad feature)

What have you confirmed?
This bug is for issue of (a) "Online Search is always executed if following conditions met, regardless of online search/local search request",
  - Search Folder(Virtual Folder)
  - Search target folder of the Search Folder is IMAP folder
  - Search fo "Body" is requested
  - Tb is Working Online mode
and for issue of (b) Gmail IMAP's online search searches for message headers such as To:, CC: and some other IMAP servers also behave similarly.
This bug is never for generic "search result is different from your thought".
Please open separate bug for your different problem from this bug, with detailed problem description and sufficient data for problem analysis, after search bugzilla.mozilla.org well for already opened bugs for same problem as yours, please.
As for issue of "actual message header search by local Body Search of Tb", actual Tb bugs I know is following only.
  Bug 721167 : Searches message header of next mail
  Bug 697021 : Searches message header of attached mail(message/rfc822 part)

Comment 12

[WADA:World Anti-bad-Duping Agency]

Correction of typo. Sorryfor spam.
  Bug 697021 : Searches message header of next mail
  Bug 700541 : Searches message header of attached mail(message/rfc822 part)