569009 - "Message body filter" misses most manually moved messages (IMAP folder only,All Folders view,not virtual folder,Body search of Quick Filter Bar at move target folder)

Status: UNCONFIRMED

(Thunderbird :: Search, defect)

Description

[William Throwe]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.3) Gecko/20100409 Gentoo IceCat/3.6.3 (like Firefox/3.6.3)
Build Identifier: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.9) Gecko/20100509 Shredder/3.0.4

Also filed at Gentoo as http://bugs.gentoo.org/show_bug.cgi?id=316399 about a month ago with no activity since.

The "Message body filter" search box at the top of the message list pane fails to find most manually moved (by dragging to a different folder) messages when text in their bodies is searched for.  Instead, these messages match the string "x-mozilla-status2: 00000000" (without the quotes), despite not containing that string.

This affects most, but not all, manually moved messages.  It doesn't seem to affect messages moved by filters or messages that are left in the inbox.

Reproducible: Always

Comment 1

[jeff.rubin]

This bug is a really bad one for me as I manually copy messages from
my inbox to a saved folder and now I cannot search for specific messages
by content in that saved folder.

Comment 2

[Wayne Mery (:wsmwk)]

Jeff, William, (I just realized you are the gentoo reporter)

do you see this in newer version?

Comment 3

[William Throwe]

I have switched away from thunderbird to a different mail client.  I'll try to install a new version of thunderbird and test this bug, but it might take me a few days to get around to it.

Comment 4

[William Throwe]

I still see this on 6.0, at least with messages I manually moved before I switched clients.  (I still have my profile directory in place.)  I haven't tried moving a message with 6.0 to see if this happens with newly moved messages because I am only running in offline mode because I don't want to let thunderbird try to do anything with my mail.

Comment 5

[Wayne Mery (:wsmwk)]

John, does this fail for you also?

Comment 6

[John L. Galt]

Don't have any *nix installed right now.  Not occurring in Windows, though, not on 10a, nor 7.x

I do have VBox installed, and I can try to get a working 64bit *nix installed, probably Gentoo, but not sure as of yet.

Also, another thing I just thought about - what sort of accounts, POP specific, IMAP specific, or protocol independent?

Finally, @William, did you perchance build a brand new profile for Thunderbird when performing the last test with 6.x, or use an old established profile?

Comment 7

[William Throwe]

I have only tested using IMAP.  I haven't used POP for any of my accounts for a long time.

My test was with an established profile.  I will try it with a new profile when I get a chance, and then report back.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to William Throwe from comment #7)
> I have only tested using IMAP.

(Q1) With which folder pane view? "All Folders"? "Unified Folders"? Or other?
If "Unified Folders", from which folder to which folder did you move mails?
 (a) Unified folder shown as Inbox at folder pane
 (b) Real IMAP folder shown as account name when unified folder such as
     Inbox, Drafts, Templates, Sent, Archives, Junk, Trash, is expanded.
 (c) Real IMAP folder shown under account name when account is expanded.
 (d) Saved Search Folder shown under account name when account is expanded.

Assume you moved mails from FolderA(move source) to FolderB(move target).
(Q2) At which folder did you execute the "Message body filter" via search box at the top of the message list pane?
As you say "move by drag&drop of mails", focus is usually stays at FolderA(move source) after move operation. If real mail folder, search at FolderA(move source) for mails already moved to other folder is usually meaningless. However, if unified folder or saved search folder, search at FolderA(move source) is possible. And, if IMAP delete model of "Just mark it as deleted", search at real FolderA(move source) for mails already moved to other folder(marked as deleted) has meaning.

(Q3) Does you problem persistent once problem occurs? Or your problem disappears after some events such as restart of Tb, foldeer switch, account switch?

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to William Throwe from comment #7)
> Instead, these messages match the string "x-mozilla-status2: 00000000" (without the quotes), despite not containing that string.

I could see it, with only two multipart/mixed message held in local mail folder in Tb, using Tb 7.0.1.
  Create new profile and dummy POP3 account, 
  and import only two multipart/mixed mails to a local mail folder.
Body only search showed first mail in local mail folder for search string of x-moz, -mozilla-, -status, and also for From:, To:, Subject: and so on, even though both mail doesn't have such string in message body.
IIRC, such phenomenon of "body search searches message header" is already reported in other bug.
If multiple multipart/mixed mails are held in a local mail folder, above search for header string returned all mails except last mail in mail folder file.
This false positive by header string doesn't look to occur on simple text/plain mail.
"Gloda is enabled/disabled" looks irrelevant. Even after disable of Gloda and delete of global-messages-db.sqlite, the false positive was observed.  

If small text/plain mails are added after the large multipart/mixed mail,
   mail-0(large multipart/mixed, no "?" in header nor body),   
   mail-1(small text/plain, ?-001 in body),
body search for ?-001 returned both mail-0 and mail-1.
If multiple small text/plain mails are added after the large multipart/mixed mail,
   mail-0(large multipart/mixed, no "?" in header nor body),   
   mail-1(small text/plain, ?-001 in body),
   mail-2(small text/plain, ?-002 in body),
   mail-3(small text/plain, ?-003 in body),
body search result for ?-001/?-002/?-003 depended on next mail to the multipart/mixed mail in mail folder file.
  (1) Initially, body search for ?-001 returns mail-0 and mail-1.
  (2) Shift+Delete of mail-1 => body search for ?-001 returns mail-0.
      (needless to say, mail-1 shouldn't be returned, as it's already deleted)
  (3) Compact folder
      => body search for ?-001 returns nothing
         body search for ?-002 returns mail-0 and mail-2
  (4) Shift+Delete of mail-2 => body search for ?-002 returns mail-0.
      (needless to say, mail-2 shouldn't be returned, as it's already deleted)
  (5) Compact folder
      => body search for ?-002 returns nothing
         body search for ?-003 returns mail-0 and mail-3

Possibly phenomenon is one like next when multipart mail.
  Search doesn't stop at "close boundary" or end of the mail.
  Gloda doesn't stop indexing for a mail at "close boundary" or end of the mail.
  It seems to stop at mid of next mail and/or end of mail folder file.

Comment 10

[WADA:World Anti-bad-Duping Agency]

FYI. I've opened bug 697021 for phenomena of comment #9.

Comment 11

[Wayne Mery (:wsmwk)]

William, et al,  please reply to the questions for you in comment 8.

Comment 12

[William Throwe]

(In reply to WADA from comment #8)
> (Q1) With which folder pane view? "All Folders"? "Unified Folders"? Or other?
> If "Unified Folders", from which folder to which folder did you move mails?
>  (a) Unified folder shown as Inbox at folder pane
>  (b) Real IMAP folder shown as account name when unified folder such as
>      Inbox, Drafts, Templates, Sent, Archives, Junk, Trash, is expanded.
>  (c) Real IMAP folder shown under account name when account is expanded.
>  (d) Saved Search Folder shown under account name when account is expanded.

Just the normal "All Folders" setup.  Everything was done with actual IMAP folders.

> 
> Assume you moved mails from FolderA(move source) to FolderB(move target).
> (Q2) At which folder did you execute the "Message body filter" via search
> box at the top of the message list pane?
> As you say "move by drag&drop of mails", focus is usually stays at
> FolderA(move source) after move operation. If real mail folder, search at
> FolderA(move source) for mails already moved to other folder is usually
> meaningless. However, if unified folder or saved search folder, search at
> FolderA(move source) is possible. And, if IMAP delete model of "Just mark it
> as deleted", search at real FolderA(move source) for mails already moved to
> other folder(marked as deleted) has meaning.

Not sure I quite follow this.  I searched for the mail in the folder I moved it to (FolderB), which is the folder it is displayed in when there is no filter applied.

> 
> (Q3) Does you problem persistent once problem occurs? Or your problem
> disappears after some events such as restart of Tb, foldeer switch, account
> switch?

Completely persistent.  It still occurs on mail I moved years ago.

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to William Throwe from comment #12)

Thanks for your(bug opener's) reply.
I could rule out "search folder" case, POP3 folder case, which are referred by some comments, from your original case.
Original case is;
  All Folders view, IMAP folder only, not search folder(not virtual folder),
  Move from folderA to folderB, Search at folderB(move target folder)  
  Problem is completely persistent. It still occurs on mail I moved years ago.

I could see funny many false positives and false negative on same mail in different mail folder by next quick test;
  Gmail IMAP, folderX: offline-use=off, folderY: offline-use=on,
  Global Search and Indexer: enabled, 
  Body search via "Filter These Messages:" in Quick Filter Bar, 
  Search for Japanse characters,
  Mail should be found: utf-8, quoted-printable body(multipart/alternative),
  folderX and folderY contains same mail which should be found.
folderX: many false positives by Body Search for Japanese text
folderY: false negative. Mail is not found by Body Search for Japanese text.

If Search of Folder context menu(or Edit/Find/Search Messafes),
(i) If offline-use=off folder, Body is not shown as criteria unless "online search" is requested. (known issue, current spec/implementation, workaround of many complaints like "body search finds nothing").
When "online search" is requested, Body is shown as criteria.
(ii) If offline-use=on folder, Body is shown as as criteria, and both online search and offline search is possible.
Body search result was same as Body Search at Quck Filter Bar. When folderY(offline-use=on), same in online search and offline search.

It looks next;
(a) "Body search via Filter These Messages: in Quick Filter Bar" perhaps executes "online search" if offline-use=off folder.
(b) "False positives & false negative in online search if Japanese characters" is perhaps Gmail IMAP problem in online search.
(c) "False negative in offline search if Japanese characters" may be caused by quoted-printable mail and url in message body in my case. (see bug 667854)

(Q1) Do you enable Gloda(Global Search and Indexer)?
(Q2) What is yout offline-use setting for folderA and folderB?
     (Folder Properties/Synchronization)
(Q3) What is your search text? Ascii characters? Or non-ascii characters?
(Q4) Does your problem occur by next?
     Create root-level folders(call folderX, folderY, no subfolder),
     Show "Order Received" column(to see UID of mail),
     Show Locatin column(to see found folder/subfolder when multi-folder search) 
     (1) Copy a mail to folderX from folderA, Body Search at folderX.
     (2) Move the mail to folderY from folderA, Body Search at folderY.
     (3) Body search at folderA for the mail.
     (4) After move back of the mail to folderA, Body search at folderA.
(Q5) If you can see problem, does "Compact Folder" affect on phenomenon?
(Q6) If you can see problem, is there any special characteristics in mail?
     (base64/quoted-printable encoded message body, url in message body, ...)

Comment 14

[William Throwe]

(In reply to WADA from comment #13)
> (Q1) Do you enable Gloda(Global Search and Indexer)?
Yes.

> (Q2) What is yout offline-use setting for folderA and folderB?
>      (Folder Properties/Synchronization)
Both are selected for offline use.  (This is the default.)

> (Q3) What is your search text? Ascii characters? Or non-ascii characters?
Just ascii.

> (Q4) Does your problem occur by next?
>      Create root-level folders(call folderX, folderY, no subfolder),
I had to create the folders as subfolders of INBOX, because the server won't let me create folders outside of that.

>      Show "Order Received" column(to see UID of mail),
>      Show Locatin column(to see found folder/subfolder when multi-folder
> search) 
>      (1) Copy a mail to folderX from folderA, Body Search at folderX.
Doesn't match.

>      (2) Move the mail to folderY from folderA, Body Search at folderY.
Doesn't match.

>      (3) Body search at folderA for the mail.
Doesn't match (correctly).

>      (4) After move back of the mail to folderA, Body search at folderA.
Doesn't match.

> (Q5) If you can see problem, does "Compact Folder" affect on phenomenon?
I compacted folderX.  This had no effect.

> (Q6) If you can see problem, is there any special characteristics in mail?
>      (base64/quoted-printable encoded message body, url in message body, ...)
Nothing obvious.
Content-Type: text/plain; charset=us-ascii
Nothing special in the body.

Comment 15

[WADA:World Anti-bad-Duping Agency]

It sounds for me "false positive" at folderA before move, and "true negative" at folderX, folderY, and "true negative" at folderA after move back.
For example, phenomenon of "false positive" like bug 697021 although that bug doesn't look to occur if IMAP folder.
If "false negative" at copy/move target folder, phenomenon such as bug 667854, and other "false positive" at foler before move.

Was folder sync'ed at copy/move target folder when you did "Body Search" after copy/move?
(sync'ed : In Activity manager, imap.account is up to date, total number of messages download: N is shown afer copy/move).  

Can you reproduce the "false negative after copy/move" at local mail folder?
Create a local mail folder, copy the mail in IMAP folder to the new local mail folder, then Body Search.

Comment 16

[William Throwe]

(In reply to WADA from comment #15)
> Was folder sync'ed at copy/move target folder when you did "Body Search"
> after copy/move?
> (sync'ed : In Activity manager, imap.account is up to date, total number of
> messages download: N is shown afer copy/move).  
No.  The only message from after I told it to copy says "Copied 1 message from folderA to folderB".

> 
> Can you reproduce the "false negative after copy/move" at local mail folder?
> Create a local mail folder, copy the mail in IMAP folder to the new local
> mail folder, then Body Search.
I cannot reproduce this with a local folder.  The search successfully finds the message.  It even works if I copy a message that the search cannot find in the IMAP folder.

Comment 17

[ripngo]

While this is a super old bug, I can confirm that I'm running into the same issue, for some messages, after they're moved into other folders. I'm willing to send one such message if that helps.

• If the message is in the inbox, the quick filter finds the string "3.10" that's inside the message body.
• If I move the message to a subfolder, the same quick filter on that folder fails to find the same string.

This happens only for some messages (I have two messages from ebay, very similar, one has the issue the other does not).

I'm running 91.11.0 (64-bit) on windows 10.

Comment 18

[ripngo]

In addition:
I confirm that if the message is moved by a message filter, the problem does not occur.

• I created a filter that moves the message based on the body containing the string "3.10".
• I ran that filter on the inbox
• The message was correctly moved to my "purchase" subfolder
• The quick-filter on the purchase subfolder now correctly finds the message based on the "3.10" search of the body.

So I confirm that this happens with some messages, and apparently only when they're moved manually.

Comment 19

[ripngo]

I tried to save the offending message (to attach it to this bug), then load it into thunderbird, but in that case, the problem disappears. If I forward it to myself from thunderebird itself, the problem does occur. This is not going to help debugging :(

Comment 20

[ripngo]

Attachment: Email showing the quick filter problem. Look for "3.10" in the body.

This emails shows the quick filter issue. Drag/drop it into an inbox, then do a quick filter on "3.10"
Even though the string is present in the email, quick filter does not find it.

For this email, you don't have to move the email manually to show the issue. Drag/dropping it into an inbox is enough.

Comment 21

[RobL]

This exact same issue happens for me 100% of the time. Every time I move an email to a folder, that email will NEVER be found by the "Filter these messages" box (aka, the quick filter).

It is very easy to reproduce. Choose an email in some folder A that contains some memorable text FOO. Now drag that email from folder A to folder B. Now select folder B. Now type FOO into the "Filter these messages" box, make sure "Body" is selected, and hit enter. Your FOO email will not be found.

This bug is literally 100% reproducible: It happens EVERY time I drag an email to a folder, regardless of the version of Thunderbird, which OS I'm using, or who is hosting the email account. I have reproduced this on Windows 7 with various older versions of Thunderbird with version numbers in the 90s. I have reproduced it on Lubuntu with a version in the 90s and the latest version. I am currently using Linux Mint and Thunderbird version 91, and it happens here too. I have reproduced it with a gmail hosted IMAP account, as well as a rackspace hosted IMAP account.

This should definitely be prioritized higher for several reasons:

1. the problem widget is right on the main window, and thus will be heavily utilized, and people are going to rely on it to work every single time
2. people who don't know about the bug will quick filter using this widget and will receive a false negative and assume it is correct and base important decisions off of that false negative
3. people who DO know about this bug are going to be incredibly annoyed every time it happens
4. I suspect the fix is going to be relatively simple. I would not be surprised if the function that does the indexing is not being called whatsoever when an email is dragged and dropped into a different folder, so it may be as simple as adding one line of code to call that function.
5. Competing email client Evolution does not have this issue, and it's annoying enough to make at least one person (me) switch.

Comment 22

[RobL]

This bug has been reported separately a number of times, dating back at least 11 years. I have been able to find these so far:
https://bugzilla.mozilla.org/show_bug.cgi?id=569009
https://bugzilla.mozilla.org/show_bug.cgi?id=1777443
https://bugzilla.mozilla.org/show_bug.cgi?id=1245532