1819895 - Daily unusable. High CPU activity in morkParser::ReadRow and nsImapMailFolder::GetDatabase, high memory usage, then crashes OOM during Autosync (fastmail with server-side filters).

Status: NEW

(MailNews Core :: Database, defect)

Description

[Emilio Cobos Álvarez (:emilio)]

It hangs constantly. It seems to be doing a ton of main-thread I/O.

Perf profile: https://share.firefox.dev/3ZfQdrY

Comment 1

[Emilio Cobos Álvarez (:emilio)]

This is:

CodeName=Thunderbird Daily
Version=112.0a1
BuildID=20230302101845

I'd use a firefox profiler profile, but I can't because I can't even pull up the inspector.

Comment 2

[Wayne Mery (:wsmwk)]

Emilio's profile https:///share.firefox.dev/3ZfQdrY

Comment 3

[Ikey Doherty]

Follow up from Matrix, Emilio is gonna get some coredump backtracery my way so I can have a gander and
see what's going on :)

Comment 4

[Wayne Mery (:wsmwk)]

(Let's assume not supernova related, until proved otherwise)

Comment 5

[Emilio Cobos Álvarez (:emilio)]

Yeah, I can't provide much more than the profile above unfortunately because removing the mail db "fixed" it, it seems. I have this bigger profile from daily in https://crisal.io/tmp/daily-perf.txt which could be useful?

Comment 6

[Wayne Mery (:wsmwk)]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #0)

It hangs constantly. It seems to be doing a ton of main-thread I/O.

Perf profile: https://share.firefox.dev/3ZfQdrY

Stack (partial):
nsAutoSyncState::ProcessExistingHeaders
...
nsImapMailFolder::GetDatabase
...
morkParser::ReadRow
morkBuilder::OnRowEnd
morkRow::TakeCells
morkPool::AddRowCells
morkZone::ZoneNewRun
morkZone::zone_new_chip
morkZone::zone_grow_at
orkinHeap::Alloc
replace_malloc
Allocator::malloc
BaseAllocator::malloc
arena_t::AllocRun
nsProfileLock::FatalSignalHandler

Possible matches :

• bug 588952 - autosync should use new async folder opening to avoid extremely bad performance at startup or when reading messages with very big IMAP folders (like gmail All Mail)
• bug 729504 - mail.check_all_imap_folders_for_new enabled and low value for "check for new messages every" yields high mork activity and frequent short hangs when writing mail.
• bug 561272 - UI unresponsive for several minutes with high CPU while .msf files are thrashed with reads and lseeks. global indexing is disabled (linux?)
• bug 1242046 - Removing unnecessary |Seek| that caused the C-C TB to operate slowly in terms of I/O
• bug 585429 - Global indexing slows because of high system I/O rate, slow I/O, or lack of multicore support (antivirus, slow disk, disk errors, etc) [was Significantly speed up global indexing (gloda)]

Comment 7

[Wayne Mery (:wsmwk)]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #5)

Yeah, I can't provide much more than the profile above unfortunately because removing the mail db "fixed" it, it seems.

The Inbox.msf file ?
Do you have server side filter? (short list of server side issues https://mzl.la/3Z2mH9S )
Non-gmail backed account?

Comment 8

[Emilio Cobos Álvarez (:emilio)]

I do have various server-side filters to manage bugmail, yeah. This is a fastmail account, effectively.

Comment 9

[Wayne Mery (:wsmwk)]

You might try setting mail.db.max_open to higher values - >=100. Default is 30.

Ref: bug 561272, bug 1305207

Comment 10

[Wayne Mery (:wsmwk)]

bug 1869260 has a performance profile. Perhaps we return to this after that bug is resolved.