1744453 - Thunderbird often hangs while updating the global index, using server side filters for cron logfile emails

Status: RESOLVED WORKSFORME

(MailNews Core :: Database, defect)

Description

[Herman van Rink]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0

Steps to reproduce:

Regular work like composing an email.

Version: 78.14.0+build1-0ubuntu0.18.04.1 and also 91.3.2 (64-bits) flatpak 1.0

System: Intel i7-3770, 32GB memory, nvme storage

Actual results:

The Thunderbird interface sometimes freezes. New key presses are not visible until the freeze is over, which can anywhere take between 10 and 30 seconds.

I've been having this issue for a long time but recently it's become much more noticeable/annoying.

I do have a fairly large set of mailboxes.

When I have the Activity manager window open I see that it's always 'Indexing messages' when all Thunderbird windows freeze.

But it does not always freeze when indexing.

Last week I deleted the global-messages-db.sqlite file to start over, before that it was 12GB. After letting work for a few days it seems to be finished again and it's now 5GB.

While it was pretty much constantly indexing it didn't seem to freeze up notisably.

The directory with Thunderbirds data files is already stored on an nvme ssd.
I tried looking with tools like dstat and iotop to see it the freezes correlated to spikes in IO but could not find such a correlation.

I updated from the Ubuntu packages(78.x) to flatpak which ships version 91, no effect. I also tried with Add-ons Disabled which had no effect.

Expected results:

Updating the global index should never freeze the interface.

Could it be that while most indexing operations are done in a background thread that there is some sqlite operation that blocks the main thread?

Anything I can try? Besides placing the index file in tmpfs/ram drive or upgrading to a faster CPU?

Is there a way to pause indexing without removing the possibility to search completely like the current settings checkbox for the Global Index feature does? That could be a workaround, letting it update the index during off-hours.

Comment 1

[Wayne Mery (:wsmwk)]

(In reply to Herman van Rink from comment #0)

...
Last week I deleted the global-messages-db.sqlite file to start over, before that it was 12GB. After letting work for a few days it seems to be finished again and it's now 5GB.

5gb is quite large, even for a large message store. What is the size of your Thunderbird profile directory? Does the sum of your Thunderbird message folders approximately equal what your mail server reports as your storage size?

While it was pretty much constantly indexing it didn't seem to freeze up notisably.

So performance now better with the 5gb size?

Could it be that while most indexing operations are done in a background thread that there is some sqlite operation that blocks the main thread?

Gloda is very complex. For architectural reasons, Indexing is done on main thread in chunks, mostly during idle.

Anything I can try? Besides placing the index file in tmpfs/ram drive or upgrading to a faster CPU?

Options are limited. Hopefully your profile is on local disk, otherwise there's not much you can do regarding improving IO responsiveness - which I doubt is the problem anyway. The main issue will be main thread contention caused by relentless indexing CPU. This is normally caused by a bad message or bad folder - the former difficult to address, the later helped by repairing folders. But sometimes the cause is messages being downloaded multiple times - which is why asked above about comparing Thunderbird profile size to server size.

Is there a way to pause indexing without removing the possibility to search completely like the current settings checkbox for the Global Index feature does? That could be a workaround, letting it update the index during off-hours.

Your performance will continue to worsen from the time you deleted the global index. So my advice is first attempt to find the cause.

Comment 2

[Herman van Rink]

5gb is quite large, even for a large message store. What is the size of your Thunderbird profile directory? Does the sum of your Thunderbird message folders approximately equal what your mail server reports as your storage size?

Thunderbird's profile dir is 38GB, the data dirs for the various imap accounts are a bit larger then what's on the servers.

While it was pretty much constantly indexing it didn't seem to freeze up notisably.

So performance now better with the 5gb size?

No not really.

Could it be that while most indexing operations are done in a background thread that there is some sqlite operation that blocks the main thread?

Gloda is very complex. For architectural reasons, Indexing is done on main thread in chunks, mostly during idle.

:(

Anything I can try? Besides placing the index file in tmpfs/ram drive or upgrading to a faster CPU?

Options are limited. Hopefully your profile is on local disk, otherwise there's not much you can do regarding improving IO responsiveness - which I doubt is the problem anyway.

It's on local nvme.

The main issue will be main thread contention caused by relentless indexing CPU. This is normally caused by a bad message or bad folder - the former difficult to address, the later helped by repairing folders. But sometimes the cause is messages being downloaded multiple times - which is why asked above about comparing Thunderbird profile size to server size.

Is there a way to 'auto repair' all folders? or script the repair via cli commands? I've now manually done all inboxes. It would be nice if the Error console would log some hint as to which folder was having issues.

After repairing the inbox for all accounts it did list the error below 22 times:

gloda.index_msg:
            Exception
              while attempting to mark message with gloda state afterdb
              commit
Exception { name: "NS_ERROR_ILLEGAL_VALUE", message: "Component returned failure code:
                    0x80070057 (NS_ERROR_ILLEGAL_VALUE)
                    [nsIMsgDBHdr.getUint32Property]", result: 2147942487, filename: "resource:///modules/gloda/IndexMsg.jsm",
                  lineNumber: 151, columnNumber: 0, data: null, stack: "_commitCallback@resource:///modules/gloda/IndexMsg.jsm:151:33\nhandleCompletion@resource:///modules/gloda/GlodaDatastore.jsm:64:11\n",
                  location: XPCWrappedNative_NoHelper }
columnNumber: 0
data: null
filename: "resource:///modules/gloda/IndexMsg.jsm"
lineNumber: 151
location: XPCWrappedNative_NoHelper { QueryInterface: QueryInterface(),
                  filename: Getter, name: Getter, … }
message: "Component returned
                  failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE)
                  [nsIMsgDBHdr.getUint32Property]"
name: "NS_ERROR_ILLEGAL_VALUE"
result: 2147942487
stack: "_commitCallback@resource:///modules/gloda/IndexMsg.jsm:151:33\nhandleCompletion@resource:///modules/gloda/GlodaDatastore.jsm:64:11\n"
<prototype>: ExceptionPrototype { toString: toString(),
                    name: Getter,
                    message: Getter,
                    … }
           IndexMsg.jsm:170:30
    _commitCallback resource:///modules/gloda/IndexMsg.jsm:170
    handleCompletion resource:///modules/gloda/GlodaDatastore.jsm:64

As repairing didn't work I started to disable indexing for the inboxes one by one. And after disabling one certain account the indexing issues stopped.

That particular account is mainly used for high traffic cron messages(a few 100 a day), and archived from the server side each night.

As indexing that particular mailbox is not very useful anyway I'm happy with the solution.

One thing that could help with future issues is for the Activity manager to show from which account it's indexing (the Inbox) from.

Thanks,

Herman

Comment 3

[Wayne Mery (:wsmwk)]

Thanks for the added detail. Your description matches to bug 589310

Comment 4

[Wayne Mery (:wsmwk)]

Please create a performance profile using version 102 from https://www.thunderbird.net/en-US/ and the procedure at https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance

Comment 5

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #4)

Please create a performance profile using version 102 from https://www.thunderbird.net/en-US/ and the procedure at https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance

Or with version 115

Comment 6

[Herman van Rink]

I've not seen the issue recently :)