Closed Bug 1831475 Opened 3 years ago Closed 1 year ago

Messages with multiple authors or no author are not indexed for global search.

Categories

(Thunderbird :: Search, defect)

Product:
Thunderbird
Component:
Search
Version:
Thunderbird 102
Type:
defect

Tracking

(thunderbird_esr115 wontfix, thunderbird_esr128 wontfix, thunderbird138 wontfix, thunderbird139 wontfix, thunderbird140 fixed, thunderbird141 fixed)

Status:
RESOLVED FIXED
Milestone:
141 Branch
Tracking Flags:
Tracking Status
thunderbird_esr115 --- wontfix
thunderbird_esr128 --- wontfix
thunderbird138 --- wontfix
thunderbird139 --- wontfix
thunderbird140 --- fixed
thunderbird141 --- fixed

People

(Reporter: Alexander, Assigned: welpy-cw)

References

Details

(Whiteboard: [datalossy])

Attachments

(3 files)

Screenshot 2023-07-24 at 7.35.38 AM.png
137.57 KB, image/png
Details
screenshot_g_id.png
12.99 KB, image/png
Details
Bug 1831475 - Enable GloDa indexing of messages from multiple authors. r=#thunderbird-reviewers
48 bytes, text/x-phabricator-request
wsmwk
: approval-comm-beta+
Details | Review

Steps to reproduce:

I was searching for a term consisting of only upper and lower case ASCII characters (no umlauts or other special characters) in a local subfolder (not part of any e-mail account). There is a message in that folder which contains the search term as a separate, stand-alone word (not a substring). The term occurs both in the message subject and the body.

Actual results:

The global full-text search (Ctrl-K) finds several other messages in other folders, matching the search criterion, but not the message I was describing above. I even repaired the folder and watched in the activity window how the folder was subsequently reindexed successfully for global search. So I am 100% sure that the messages in that folder must be in the global search index.

Expected results:

The message should have been found.

Feel free to contact me on a private channel. I will be happy to show what I am doing in a desktop sharing session to any developer who cares to see it reproduced live. I can also provide all kinds of logs, provided they are not readable publicly and do not disclose too many private data.

BTW, this does not look like a duplicate of bug 598605, because that old bug is about substring matching. Here, we are dealing with matching full stand-alone terms.

(In reply to Alexander Kriegisch from comment #1)

BTW, this does not look like a duplicate of bug 598605, because that old bug is about substring matching. Here, we are dealing with matching full stand-alone terms.

Please provide some example search terms

Flags: needinfo?(Alexander)

Seriously? After 3 months, this is your question? A search term, without any context, i.e. without you seeing my mailbox? Fine: The search term is "intellidatalink" (without the double quotes), an old product name by one of my ex employers. The term matches 4 messages, if I use regular folder search in the target folder. But when using full-text search, it only finds messages in other folders, not in that particular one.

Flags: needinfo?(Alexander)

Hypothetical scenarios without concrete examples tend to result in unproductive guesswork. Thank you for the example. And thank you for the offer to provide logs. Unfortunately there is no logging for search functionality.

The two primary search functions were built at different times, use different code, and therefore at times produce different results. The one that sometimes doesn't find items is global search. There are plans to improve this area by unify the functions in the future, but that doesn't help you now.

Global search sometimes doesn't find search terms in html, and by design only indexes the first 20k or so of a message. I suspect you are seeing an example of this.

It is also possible the message in question did not get indexed. You can also delete global-messages-db.sqlite in your thunderbird profile to allow the search index to rebuild.

Flags: needinfo?(Alexander)

Sorry to disappoint you. The messages are all short, plain 8-bit text, no HTML or other rich text format. The search term occurs in the subject for all 4 messages, in 1 of them also in the body. It might really be the case that the messages were not indexed for some reason. They originate in an archived "Gesendet" (German for "Sent") folder. It looks as if nothing in that folder is being found, no matter what I search for. How can whole folders not be indexed? I did not specify any exceptions, and like I said before: I watched the indexing process and waited until it was finished. That was 3 months ago.

Flags: needinfo?(Alexander)

It looks as if nothing in that folder is being found, no matter what I search for.

And folder properties (or in account sync settings) the folder is set to download for offline use / not being excluded from message download? ?

Like I wrote before, it is a local folder, not an IMAP one or so. There is no need to sync anything. The folder is also marked to be indexed by global search in its properties.

I also just - for the 5th time or so - clicked "repair folder", watched the activity view for the folder to be reindexed, which was done in 25 seconds for the 1.700+ messages in that folder, but still the messages in that folder are nowhere to be found in global search.

Repair is unlikely to ever help global search.
We will need to determine whether indexing of the folder stopped at some flawed message, or the entire folder isn't indexed. There is an add-on which will help - I'll try to find it.

Flags: needinfo?(vseerror)
Keywords: dupeme
Summary: Full-text search not finding content in indexed folder → Full-text search not finding content in (presumably) indexed local folder

Repair is unlikely to ever help global search.

Oh, OK.

We will need to determine whether indexing of the folder stopped at some flawed message,

In that case, whatever "flawed" might mean to TB, indexing ought to just skipt that message and continue with the next one instead of not indexing the whole folder. I wonder, however, how a message could be regarded as flawed, if normal folder search finds it and I can display it.

or the entire folder isn't indexed.

The indexing protocol in the activities view contradicts that. So either that cannot be the case, or activity logging does not tell the truth.

There is an add-on which will help - I'll try to find it.

Thanks, I will be glad to help you debug this, using said tool.

I have "rescued" that old addon to current TB versions.
You are welcome to forward the email to buecher at optosolar dot com, then I can see whether it will be indexed in my TB test version.
Or by your preference, I can sent the addon to you so you can check in your installation.

Klaus

Hi Klaus.
The messages are very old, but still contain internal information from a former employer. Therefore, I would prefer to try the plugin locally and only send the messages elsewhere as a last resort. In my StackOverflow profile, you find ways to contact me.
Xander

Alexander, how did you do with the add-on?

Severity: -- → S3
Flags: needinfo?(vseerror) → needinfo?(Alexander)

Wayne, I do not think that anyone tried to contact me regarding the plugin.

Klaus, is this still on your radar?

Flags: needinfo?(Alexander)
Flags: needinfo?(buecher)
Flags: needinfo?(buecher)

(In reply to Alexander Kriegisch from comment #14)

Wayne, I do not think that anyone tried to contact me regarding the plugin.

Klaus, is this still on your radar?

The add-on I'm thinking of is https://addons.thunderbird.net/en-us/thunderbird/addon/glodaquilla-search-indexing-en/

The last correspondence I have with Klaus on the subject is Jan 2023

Flags: needinfo?(buecher)

I am working on this for 128, should come out end of October. The columnpart is working already, the indexed folder picking part not yet, as it is not shown in the UI

Flags: needinfo?(buecher)

(In reply to klaus from comment #16)

I am working on this for 128, should come out end of October. The columnpart is working already, the indexed folder picking part not yet, as it is not shown in the UI

klaus,
Have you had good results?

Flags: needinfo?(buecher)
Summary: Full-text search not finding content in (presumably) indexed local folder → Full-text global search not finding content in (presumably) indexed local folder
Whiteboard: [datalossy]
Blocks: 1348662
Blocks: 1531446

Alexander, please try https://addons.thunderbird.net/en-us/thunderbird/addon/glodaquilla-search-indexing-en/

Flags: needinfo?(buecher) → needinfo?(Alexander)

Wayne, what exactly are you advising me to do with an add-on which was last updated in 2019 and is said in comments not to be working with more recent versions of TB anymore?

Flags: needinfo?(Alexander)

Wayne, do you maybe mean this?

https://addons.thunderbird.net/de/thunderbird/addon/glodaquilla-ng-index-ondisk/

Again, what are you advising me to do, watch and report?

yes, this
https://addons.thunderbird.net/de/thunderbird/addon/glodaquilla-ng-index-ondisk/

is the one he means.

Go to the folder with the suspect messages, allow to display the columns gloda-id, gloda-dirty, ondisk (or similiar names).

If gloda-id is 2, it cannot be indexed. >2 gives the id of the indexing results. Gloda-dirty column: should be indexed/indexing underway (Wayne will know which of the two), ondisk: message has been downloaded.

If you repair the folder, it will also re-index, which you can see in the activity monitor and in the three columns.

Klaus

In one of my folders, I see both messages with g_id == 2 and others, see the attached screenshot. Why the former cannot be indexed, is beyond my comprehension. The messages are not particularly complex and even text-only.

When using global search, the ones with g_id != 2 are found, the others are not.

Attached image screenshot_g_id.png —

The non-indexable ones seem to be native Outlook messages (not SMPT) which I had imported into TB many years ago. The headers look look like this:

To: Some-Name-Without-Email-Address
Date: Fri, 20 Jul 2001 15:53:29 +0100
Subject: Foo Bar Zot
From: Doe, John
Content-type: text/plain; charset=windows-1252; format=flowed
MIME-Version: 1.0
Content-transfer-encoding: 8bit

This is the body text.

Can missing e-mail addresses like doe@acme.org in To and From derail global search indexing so much that the whole message is not indexed at all?

I would be glad if somebody responded to my latest message. I think, finally after 2 years we are onto something. I do wish to find out which bug or quirk stops TB from indexing my Outlook-imported messages for full-text search, while normal search finds all text inside those messages as expected. This ought to be fixed, because it means that thousands of imported messages are outside of my reach regarding global full-text search.

yes, if there is no real email address in from, it is at least likely that indexing fails.

If I have possibly from = <name>email, I would expect it to index, also, when from = email, but if from is from = <name> it might well fail. Because email would be unique, but <name> could exist for multiple email addresses.

If it is only a few emails not being indexed, there is a workaround:
install header tools lite addon, highlight the email, in context menu choose edit lite (or a similiar entry, don't remember exactly), then edit the from header and add the email address.
Then reindex the folder/message.

@wayne: if that is the case, one could append indexing for such a case to set an email unknown@unknown.com, which then might enable indexing.

@klaus, thanks for the suggestion, but changing headers manually is not an option for thousands of messages. It would just be a workaround anyway. What really should be done is to fix gloda indexing to be a bit more lenient. Normal search in mail bodies and headers also works without the assumption that there must be an actual e-mail address like john@doe.org instead of just "John Doe" in From, To, CC, BCC fields. I find that assumption too strong for a full-text indexer. I also do not think that there should be a dummy address added to the index, because this is not what I want to search for or match but the names that are actually in those header fields.

No longer blocks: 1348662

(In reply to Alexander Kriegisch from comment #24)

The non-indexable ones seem to be native Outlook messages (not SMPT) which I had imported into TB many years ago. The headers look look like this:

[...]
From: Doe, John
[...]

Can missing e-mail addresses like doe@acme.org in To and From derail global search indexing so much that the whole message is not indexed at all?

It's the From header. For some reason, Gloda only indexes messages with exactly one author, see source code and corresponding unit test.

Since there is no such constraint in RFC 5322, this for sure is a valid bug. While there is an abandonded work in progress for this in bug 1526985, I don't see this getting fixed for Gloda anytime soon, if at all. Maybe you'll have more luck with Panorama…

Status: UNCONFIRMED → NEW
status-thunderbird138: --- → wontfix
status-thunderbird139: --- → wontfix
status-thunderbird_esr115: --- → wontfix
status-thunderbird_esr128: --- → wontfix
Depends on: 1526985
Ever confirmed: true
Keywords: dupeme
Summary: Full-text global search not finding content in (presumably) indexed local folder → Messages with multiple authors or no author are not indexed for global search.
No longer blocks: 1531446

(In reply to Hartmut Welpmann [:welpy-cw] from comment #29)

It's the From header. For some reason, Gloda only indexes messages with exactly one author, see source code and corresponding unit test.

Since there is no such constraint in RFC 5322, this for sure is a valid bug. While there is an abandonded work in progress for this in bug 1526985, I don't see this getting fixed for Gloda anytime soon, if at all. Maybe you'll have more luck with Panorama…

I am surprised that this will not get fixed. Waiting for a future global message database implementation is less than a consolation prize, because it leaves me with nothing after having had this problem for many years, having waited for replies here, investigated, provided input and finally found the root cause together. The potential fix does not look so complex - if I was a C++ and not a Java guy, I might even create a PR. In German, we would call this "Elfmeter mit Anlauf", in English I would call it a rather low-hanging fruit. You know the root cause, the place in the source code and I am sure you have ideas how to fix it. I wish one of the contributors here would change his/her mind. Why is Gloda declared clinically dead, just because Panorama in on the way?

Instead of silently throwing an error and not indexing such messages, just use the first author provided in the 'From:' header.

Assignee: nobody → h.w.forms
Status: NEW → ASSIGNED

Hartmut, will this also work in case the header value is not "Tweedle Dee <dee@example.com>" or "dee@example.com>" but just "Tweedle Dee"?

(In reply to Hartmut Welpmann [:welpy-cw] from comment #31)

Created attachment 9489615 [details]
Bug 1831475 - Enable GloDa indexing of messages from multiple authors. r=#thunderbird-reviewers

Instead of silently throwing an error and not indexing such messages, just use the first author provided in the 'From:' header.

(In reply to Alexander Kriegisch from comment #32)

Hartmut, will this also work in case the header value is not "Tweedle Dee <dee@example.com>" or "dee@example.com>" but just "Tweedle Dee"?

Yes, but the correct representation of correspondents without email adresses in the faceted view depends on bug 1223951. For 'From:' headers containing names in the form "Doe, John" only "Doe" is considered.

Depends on: 1223951
No longer depends on: 1526985
See Also: → 1526985

That is certainly not ideal, but if at least messages get indexed as expected, it will be a substantial improvement compared to the status quo.

(In reply to Hartmut Welpmann [:welpy-cw] from comment #33)

(In reply to Alexander Kriegisch from comment #32)
the correct representation of correspondents without email adresses in the faceted view depends on bug 1223951. For 'From:' headers containing names in the form "Doe, John" only "Doe" is considered.

(In reply to Alexander Kriegisch from comment #34)

That is certainly not ideal, but if at least messages get indexed as expected, it will be a substantial improvement compared to the status quo.

There is no way to solve this for "Doe, John" without breaking this for headers conforming to RFC 5322, since the comma separates different authors.

(leave-open for blocking bug bug 1223951)

status-thunderbird140: --- → affected
status-thunderbird141: --- → affected
Keywords: checkin-needed-tb, leave-open
Target Milestone: --- → 141 Branch

Pushed by geoff@darktrojan.net:
https://hg.mozilla.org/comm-central/rev/bac1ea56e147
Enable GloDa indexing of messages from multiple authors. r=john.bieling,mkmelin

Keywords: checkin-needed-tb

Sorry for maybe asking a stupid question, but I am not knowledgeable concerning Thunderbird release channels and cadences. Does branch 141 imply that the fix will not be in the upcoming ESR release? I am on the ESR channel and was even hoping for a backport, because it is a very old problem and the corresponding part of the code is unlikely to have changed sinde 128, so a backport should be easy.

(In reply to Alexander Kriegisch from comment #38)

Sorry for maybe asking a stupid question, but I am not knowledgeable concerning Thunderbird release channels and cadences. Does branch 141 imply that the fix will not be in the upcoming ESR release? I am on the ESR channel and was even hoping for a backport, because it is a very old problem and the corresponding part of the code is unlikely to have changed sinde 128, so a backport should be easy.

This will be decided after ESR 140 had been released, but I can't say what the outcome will be. However, Thunderbird 141 will have this. If you want to try this now, feel free to download 141 Daily and test it with a copy/part of your current profile.

Closing this now that bug 1223951 has been fixed as well.

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED

I have a specific usage question: Wishing to stay on ESR, but also willing to test and provide feedback, would the following scenario work?

  1. Install a TB 141 development or final version.
  2. Reindex my old e-mail archives containing Outlook imports with the improved gloda indexer.
  3. Switch back to 128 ESR and test full-text search from there.

The reason for this scenario is that only my old Outlook archives are affected by the problem described in this issue. The folders in question should not need further re-indexing under 128. This way, I could buy time until there will be a 141+ ESR version containing the improvement.

Flags: needinfo?(h.w.forms)

Hartmut, what about uplifting this to 140 (coming esr)?

(In reply to Magnus Melin [:mkmelin] from comment #41)

Hartmut, what about uplifting this to 140 (coming esr)?

I would support that. But bug 1223951 is currently marked as wontfix for 140. If it these are to get into 140.0 esr, it would need to be on beta ASAP.

status-thunderbird140: wontfix → affected
Flags: needinfo?(h.w.forms)

Comment on attachment 9489615 [details]
Bug 1831475 - Enable GloDa indexing of messages from multiple authors. r=#thunderbird-reviewers

Uplift Approval Request

  • Please state case for uplift consideration and ensure bug severity is set: Messages from multiple authors are not indexed at all.
  • User impact if declined: see above.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Daily?: Yes
  • Has the fix been verified in Beta?: No
  • Needs manual test from QA?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: Bug 1223951
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Simple fix covered by automated tests.
  • String changes made/needed: None
Attachment #9489615 - Flags: approval-comm-beta?

Magnus, Wayne, thanks for the feedback. I'm totally okay with getting this into 140!

Comment on attachment 9489615 [details]
Bug 1831475 - Enable GloDa indexing of messages from multiple authors. r=#thunderbird-reviewers

[Triage Comment]
Approved for beta
Dovetails to bug 1223951

Attachment #9489615 - Flags: approval-comm-beta? → approval-comm-beta+

It is looking good in 140 ESR. My imported Outlook archive e-mails with funny headers are now being gloda-indexed and searchable as expected. Thank you so much for the backport. ☺️🙏

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: