1777443 - Quick "Filter these messages" by body on IMAP fails to find matching messages w/o "run search on server" or "Repair folder"

Status: UNCONFIRMED

(Thunderbird :: Search, defect)

Description

[M Lopez-Ibanez]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/101.0

Steps to reproduce:

Moved a message to an IMAP folder.
Tried searching for a string of text using "Filter these messages" box and selecting "Body" => No result.

Tried searching with Gobal search => cannot find the message.

Tried searching with Search messages without enabling "Run search on server" => cannot find the message.

Tried searching with Search messages enabling "Run search on server" => NOW it can find the message.

Clicked in "Repair folder" => NOW it can find the message.

Expected results:

This time I knew where the message was but in general it would be impossible to know which folder needs repairing.

There is something broken somewhere that breaks indexing and search when moving messages.

Comment 1

[RobL]

I've been having the same issues for a long time. Some details that might help:

OS: This most recent test I did just now is Lubuntu 22.04. However, I've seen it on Windows 7 for a number of years now, as well as older versions of Lubuntu going back to 18.04.
Thunderbird Version: I reproduced this with 91 just a few minutes ago, then I grabbed the latest, version 102.1.2. Same issue.
Email Host: I've reproduced this with a rackspace.com-hosted email account, as well as a gmail account. Both are IMAP.
Steps to reproduce:
(I am putting every step here, no matter how innocuous or obvious it may seem -- even including my steps in setting up gmail -- to aid in reproducing this error):

(1) Create new gmail account.
(2) In gmail, click the gear (aka "settings") icon, then click "see all settings"
(3) On the Forwarding and POP/IMAP tab in gmail, select the "Enable IMAP" radio button and click "Save Changes".
[ I've included these 3 steps to ensure everyone is on the same page regarding how the gmail account is set up ]

(4) Open Thunderbird, and go to File | New | Existing Mail Account ...
(5) Enter your name, gmail address, and gmail password from step 1 into the boxes and hit "Continue" and then "Done" on the next screen, and then "Finish".
(6) Now in the left pane, under "Inbox" right click on the italicized "[Gmail]" folder and select "New Subfolder ...".
(7) Give the new folder a name and click "Create Folder".
(8) Now click the "Write" button at the top and type an email to someone. Be sure to put an easily distinguishable word such as "sasquatch" or "lemmings" in the body of the email.
(9) After entering the recipient, subject, and the body (again, with a nice easily distinguishable word in it), hit "Send".
(10) Select the "Sent Mail" folder and be sure you see your email you just sent in step (9).
(11) Just for fun, try typing in your easily disinguishable word from step (8) into the "Filter these messages <Ctrl+Shift+K>" box (be sure "Body" is selected). This should work just fine.
(12) Now click and drag your email from the sent box to the new folder you created in step (7).
(13) Now select your new folder in the left pane. Note that the email you just dragged is there now.
(14) In the "Filter these messages <Ctrl+Shift+K>" box at the top, type in your easily distinguishable word (and again, be sure "Body" is selected).
(15) Note that your email is not found! Even though it's right there! Whammo, there's the bug.

This is 100% reproducible for me.

Note: This bug has been around I would say at least 2 years, if not more. And again, on both Windows 7 and Lubuntu. My work around that always fixes the issue is to go to the [randomcharacters].default-release/ImapMail/imap.gmail.com/[Gmail].sbd directory on my machine and delete the files for the email folder in question (ex, MyFolderName and MyFolderName.msf). Once I do that and then go back to Thunderbird and do File|Compact Folders (twice!) then the little quick-filter works again in that folder. The work around is a big pain though and really annoying, so I'm trying to be as thorough as possible here to aid in the reproduction and elimination of this bug.

I would not be surprised if this bug also occurs when an email is moved by one of the filtering rules (for example if you set up a rule to move every email from your manager to a special folder as soon as it arrives). Someone should probably check to be sure. Might also be worth checking if the same problem happens with things other than Body (Sender, Recipients, Subject).

As the OP stated, something is likely wrong with updating the index when emails are moved to different folders. Or perhaps the index isn't even being updated at all when emails are moved.

Comment 2

[RobL]

Update: I have now alsoreproduced this on Linux Mint with Thunderbird version 91.11.0

Again, it is 100% reproducible for me regardless of Thunderbird Version, which OS I'm using, or who is hosting the IMAP email account (gmail or rackspace).

This should definitely be prioritized higher for several reasons:

1. the problem widget is right on the main window, and thus will be heavily utilized, and people are going to rely on it to work every single time
2. people who don't know about the bug will quick filter using this widget and will receive a false negative and assume it is correct and base important decisions off of that false negative
3. people who DO know about this bug are going to be incredibly annoyed every time it happens
4. I suspect the fix is going to be relatively simple. I would not be surprised if the function that does the indexing is not being called whatsoever when an email is dragged and dropped into a different folder, so it may be as simple as adding one line of code to call that function.
5. Competing email client Evolution does not have this issue, and it's annoying enough to make at least one person (me) switch.

This bug has been reported separately a number of times, dating back at least 11 years. I have been able to find these so far:
https://bugzilla.mozilla.org/show_bug.cgi?id=569009
https://bugzilla.mozilla.org/show_bug.cgi?id=1777443
https://bugzilla.mozilla.org/show_bug.cgi?id=1245532

It does not appear that anyone has yet attempted to verify or reproduce this bug.

Comment 3

[M Lopez-Ibanez]

Thanks RobL for this analysis. Hopefully the new direction announced for Thunderbird (https://thunderbird.topicbox.com/groups/planning/Tc404795356dcf7e9/update-from-thunderbird-product-manager) will mean that these long-standing bugs will get addressed.

The quality of Thunderbird is important. Unless you are self-employed, you have been, or will be soon, forced to use Teams and Outlook by your employer with the argument that open-source software is too unreliable and they cannot support it. This means that we are going back to the dark ages of Internet Explorer supremacy (ending that supremacy was one of the reasons why Mozilla was born), but this time in email and office software: Outlook/Office will drop open standards in favour of closed and proprietary ones. We are already seeing this with the push to use Exchange servers for email and calendar, while dropping support for IMAP and CalDAV features.