1763729 - Slow CPU- and storage-consuming messages indexing --> Add more fine-grained settings for gloda index

Status: RESOLVED DUPLICATE of bug 1553439

(Thunderbird :: Search, defect)

Description

[Ulf Zibis]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0

Steps to reproduce:

Have 14 GB of messages in profile.

Actual results:

• Moving ~ 5000 messages slows down machine for plenty hours.
• Complete (re-)build of messages index takes 40 % CPU over 5 days to complete.
• global-messages-db.sqlite consumes 1.4 GB

Expected results:

Indexing could be much faster and less storage consuming.
Add settings for indexing which only scan on headers but on complete message content.
Eventually exclude special headers such as X-abcd.
Eventually provide 3 levels: all headers + body / only headers / only important headers like From:, To:, CC:, BCC:, Subject:, etc.
Add those preferences globally / per account / per folder.

Comment 1

[Ulf Zibis]

Now, with update from TB 91 to TB 102 the gloda index was completely build again. It took again 4 days to complete with ~ 40 % CPU usage.
global-messages-db.sqlite now consumes 1.9 instead 1.4 GB.

Comment 2

[Wayne Mery (:wsmwk)]

Complete (re-)build of messages index takes 40 % CPU over 5 days to complete.

Remind us please, on what kind of hardware?

Comment 3

[Ulf Zibis]

Intel® Core™2 Duo CPU P8600 @ 2.40GHz × 2

Comment 4

[Wayne Mery (:wsmwk)]

(In reply to Ulf Zibis from comment #3)

Intel® Core™2 Duo CPU P8600 @ 2.40GHz × 2

This in combination with a non-SSD disk and 14gb profile is indeed going to be slow for a long time, although 5 days seems excessive.
Does 5 days include downloading the messages? Or just rebuilding from deleting global-messages-db.sqlite?

Add settings for indexing which only scan on headers but on complete message content....

bug 1553439. But I seriously doubt that will be implemented, and thus may be closed.

If you want better performance today, your options are

• check whether your computer has something slowing down the indexing process (going into low power or sleep for example - you don't describe your actual hardware)
• reduce the size of your profile
• set some folders to not synchronize (folder/account settings)
• set some folders to not be indexed (account settings)
• install SSD
• system with faster processor - core duo is roughly 15 years old

Comment 5

[Ulf Zibis]

(In reply to Wayne Mery (:wsmwk) from comment #4)

Please continue here: bug 1553439 comment #22