1517313 - Filter of IMAP Sent Items misses text in Body

Status: NEW

(Thunderbird :: Search, defect)

Description

[John Avery]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0

Steps to reproduce:

Using the Quick Filter Bar with Body selected, filtering my Sent Items folder for a word turned up nothing.  After I ran Repair Folder, the expected messages showed up.  However after sending 3 more test messages containing the word and filtering again, those new test messages did not show up.

"Sent Items > Properties > Include messages in this folder in Global Search results" is checked.
"Sent Items > Properties > When getting new messages for this account, always check this folder" is checked.
"Account Settings > (the account) > Synchronisation & Storage > Message Synchronising > Keep messags in all folders ..." is checked, and nothing is unchecked in "Advanced..."

Version: 60.2.1
Build ID: 20181010142617
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1
OS: Linux 4.15.0-43-generic

Sent Items folder: Number of messages: 8986, Size on disc: 85.6 MB


Actual results:

Using Quick Filter with Body selected did not find messages in the Sent Items filter until I ran Repair Folder.


Expected results:

Quick Filter / Body of Sent Items should have found the messages containing the search text.  A Quick Filter / Body of Inbox for the same text found all the expected messages.  Any Repair Folder of Inbox happened a long time ago, well before those messages arrived.

Comment 1

[Jorg K (CEST = GMT+2)]

Body search was riddled with problems which should all be fixed now if the folder is a local folder, not an IMAP folder. Your problem it with IMAP folder which appears to be synchronised for offline use. That's best checked in the folder properties:
Properties, Synchronisation tab: Select this folder for offline use.
Then a "local" folder should be created that contains all the messages.

As you confirmed, messages are found once you repair the folder. So when you repair "Sent Items" again, are the messages found now?

Gene, how does this synchronisation work. Is it possible that freshly sent messages are not yet available for offline use in the local copy but a repair will fetch them?

Comment 2

[gene smith]

I assume that since the folder has local storage for offline use that the quick filter just searches the mbox file for the body text. In that case, yes a sync has to occur before the body text can be found. However, clicking from Inbox to Sent should cause the sync to occur. It will also occur on tb startup (assuming autosync is enable, that is by default) or when new message check occur (if Sent is set to be checked for new messages).

So the reporter should try this:

Click on Inbox and send a message.
Click on Sent and verify the message is present
Do the quick search on something in the message. Does it now show?

If that doesn't work, try this:

Click get messages
Click Sent (if not already there)
Do the quick search on something in the message. Does it now show?

If that still doesn't work try this:

Restart tb
Wait 1m for autosync to occur (not sure of exact time but 1m should be enough)
Click Sent (if not already there)
Do the quick search on something in the message. Does it now show?

Report back with the results. Thanks.

Comment 3

[John Avery]

Click on Inbox and send a message.
Click on Sent and verify the message is present
Do the quick search on something in the message. Does it now show?

No.

If that doesn't work, try this:

Click get messages
Click Sent (if not already there)
Do the quick search on something in the message. Does it now show?

No.

If that still doesn't work try this:

Restart tb
Wait 1m for autosync to occur (not sure of exact time but 1m should be enough)
Click Sent (if not already there)
Do the quick search on something in the message. Does it now show?

No.

FWIW, this test was performed on my travel computer. Thunderbird had not been run for quite a while, before I had started sending the messages with the word I was searching for. However, by the time I received this Bugzilla request for more information, Thunderbird had been running long enough to sync all my folders, from what I noticed. When I quick-searched the body of the messages in Sent, all the messages with the word appeared. However, as indicated in my answers above, new messages that I sent with the word did not show up.

I also tried another similar test: I sent a message on my phone, using the same account. Back on my travel computer, the message appeared in the Sent folder and it also appeared in my quick-search results.

Comment 4

[gene smith]

Well, I think you are right, this is a problem that I have now duplicated.

Also it appears that a very similar issue was reported a while back but was deemed invalid because repairing the folder fixed it for that reporter (see Bug 891305).

On this laptop I run without offline store due to very limited flash disk. But I temporarily made a small folder having offline storage and set it as where sent messages go. I sent a message, selected the new "sent" folder and quick searched and the word wasn't found. Looking at the mbox file, the word was there. Then after repair the word could be found with quick search. (The right-click on folder "advanced" search has the same problem.)

I see the same problem when I just copy a random messages to my new "sent" folder so I don't think that the sending action has anything to do with this.

There seem to be other problems with search on imap folders that are not affected by "repair folder" or messages that were just sent or copied. Search for words that are definitely there (in body) are sometimes not found. Also, search finds messages that don't have the target word at all in the body or in the complete messages based on when source is viewed and a find is performed.

Comment 5

[Jorg K (CEST = GMT+2)]

(In reply to gene smith from comment #4)

Search for words that are definitely there (in body) are sometimes not found. Also, search finds messages that don't have the target word at all in the body or in the complete messages based on when source is viewed and a find is performed.

Well, the search depends on locating the message (body) in the raw data file by indexing into it from the MSF file. If those pointers aren't right (and maybe corrected via repair), then the search hit will be associated with a message that doesn't contain it. Like if the search hit is at line 100 of the overall file, and the MSF says that line 100 belongs to message 2, that message 2 will be delivered as result, even if line 100 in reality belongs to message 4.

Comment 6

[gene smith]

By "raw data file" I assume you mean the mbox file. I saw the problem (false search hits) with a "Sent" folder that had no mbox file. In that case I assume the search uses the imap SEARCH command on the body. I haven't looked at the imap log or network trace or at the code that does this.
Also, I just did a repair on the folder before I noticed this. (This was my main "Sent" folder that went back many years and took quite a while to repair.)

Comment 7

[gene smith]

Network trace shows imap BODY search returns no message UIDs for words that are only in message BODY. BODY search only returns message UID when search term is NOT in the body (i.e., in the SUBJECT). Weird. So seems to be a server problem. This is a proprietary server (in Poland I think). Will check this with other servers. (This is not really the reporter's issue.)

Comment 8

[gene smith]

I don't see the reporter's problem when there is no offline store. In that case the search is done via IMAP. Other servers I have tried work fairly well searching via imap (dovecot and openwave). But openwave doesn't find words that are embedded in mime parts but simple text-only messages have no problems with finding words in the BODY.

Apparently, per the reporter, the folder msf file repairs itself given some time and then the search works and finds the words. At this time I have no idea how that occurs nor have I seen it occur.

(In reply to Jorg K (GMT+1) from comment #1)

Gene, how does this synchronisation work. Is it possible that freshly sent messages are not yet available for offline use in the local copy but a repair will fetch them?

Not sure if I completely answered this. It looks like the freshly sent message is immediately appearing in the mbox file but it's not immediately found in a search. So the .msf file is not "pointing" to it completely, at least for what is needed to do the search. Maybe the .msf gets updated later (maybe by autosync or "gloda"?) and then the search works. Of course, the repair immediately fixes it but I don't think it changes the mbox file (which is just a concatenation of all the messages in the folder).

Of course, for a big folder repair can take a long time, so not really a 100% acceptable workaround.

This also sounds like a duplicate of Bug 569009; the user was just copying mails to other imap folder and the search failed at the destination folder. (But not sure about finding "x-mozilla-status2: 00000000" that is mentioned.) Also, no mention of the problem becoming OK on its own.

(In reply to John Avery from comment #3)

I also tried another similar test: I sent a message on my phone, using the same account. Back on my travel computer, the message appeared in the Sent folder and it also appeared in my quick-search results.

When I copy a message from a different account to the problem Sent folder, I can immediately find words in that messages's body using search. So this means that a message that is imap appended works OK but when imap copied it messes up the .msf file internally for search but not for normal display.

Hopefully this is not related to this can 'o worms: Bug 1430480
But Bug 1430480 comment 102 provides a possible workaround. Delete and then undelete the problem messages in Sent folder. You have to move off of Sent and then click back to it to forces the required message fetch that "repairs" the search problem. It worked for me and avoids the full repair.

Comment 9

[gene smith]

Hopefully this is not related to this can 'o worms: Bug 1430480

I reverted the patch for Bug 1430480 and it didn't affect this bug. So it's not a regression.

Comment 10

[Eugene Savitsky]

Attachment: Screenshot of Search Messages - no Body filter in IMAP Sent folder

There is no Body filter in full "Search Messages" IMAP Sent folder as well, see screenshot.

Seems the same problem?

Comment 11

[gene smith]

"Body" appears only if you select "Run search on server" when the folder has no offline store.
If the folder does have offline store, the "Body" item appears without checking "Run search on server".
At least that's what I'm seeing.