Closed Bug 1863957 Opened 2 years ago Closed 2 years ago

Thunderbird 115 very slow when opening .EML files

Categories

(Thunderbird :: Account Manager, defect)

Product:
Thunderbird
Component:
Account Manager
Version:
Thunderbird 115
Type:
defect

Tracking

(thunderbird_esr115 verified, thunderbird123 verified)

Status:
VERIFIED FIXED
Milestone:
124 Branch
Tracking Flags:
Tracking Status
thunderbird_esr115 --- verified
thunderbird123 --- verified

People

(Reporter: rapettif, Assigned: mkmelin)

References

Details

(Keywords: perf, regression)

Attachments

(2 files)

screenshot from profiler.png
40.62 KB, image/png
Details
Bug 1863957 - Fix slow .eml opening on profiles with many folders. r=BenC
48 bytes, text/x-phabricator-request
wsmwk
: approval-comm-beta+
wsmwk
: approval-comm-esr115+
Details | Review

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/119.0

Steps to reproduce:

Double click on .EML files in Windows Explorer, or File open dialogue of Thunderbird.

Actual results:

The .EML seemingly opens with no content. Thunderbird acts very sluggish, and CPU use is exactly 25% on a quad-core CPU. After 50 seconds, the content of the EML finally displayed. It was a 800 KB EML file!

Expected results:

The file should have opened "in the blink of an eye". As happened in every previous release.
Can i recommend to implement "stable" and "testing" channels? Some people value stability a lot more than to be "on the edge" of new releases. Something like what Libreoffice did with their "Stable" and "Fresh" versions.

Do you have a sample .eml file you can attach here?

(In reply to Magnus Melin [:mkmelin] from comment #1)

Do you have a sample .eml file you can attach here?

Sorry, it's a work account, and that particular message has a lot of confidential information. I don't see anything out of the usual for a work e-mail. It's a quite long mail chain, with a lot of signatures with images and links to enterprise website, social network accounts, etc.

Is there some kind of debugging tool that i can use to send you thunderbird's behaviour without revealing the confidential stuff in the message?

It's also worth mentioning, that the message is one .EML that i drag-and-dropped from thunderbird into a folder, to have it at hand. But if i watch the original message that is listed inside thunderbird, it loads very fast. It's only when i load the saved .EML file that it takes a lot to load.

(In reply to Magnus Melin [:mkmelin] from comment #3)

Try https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance

Hi, i just ran the profiler. The result was uploaded here from Thuderbird's profiler app: https://share.firefox.dev/47zXK97

I also saved the file locally on my computer, but it's supposed to be the same that was uploaded by the profiler to share.firefox.dev
Tell me if you need anything else.

Thanks in advance.
Regards.

Looks like a lot/slow I/O activity.
Is it a local drive?
Maybe antivirus/firewall is affecting it?

(In reply to Magnus Melin [:mkmelin] from comment #5)

Looks like a lot/slow I/O activity.
Is it a local drive?
Maybe antivirus/firewall is affecting it?

It's a local SSD. Thunderbird and Windows are installed on this SSD, and the Thunderbird profile files are there too. However, opening the same mail from the list of mails in Thunderbird is fast. The same mail saved to an .EML file on a folder inside of Windows 10's desktop is slow.
I just disabled the antivirus (ESET Endpoint Security 10.1.2058.0) to check, and opening the .EML remains slow.
Also, the .EML file is only 1.17 MB

Maybe have a look at this bug report from October: https://bugzilla.mozilla.org/show_bug.cgi?id=1856922

It seems that the problems are somehow related. In my case, archiving old mails helped a bit. The archive still is located on the mail server. For a colleague of mine, opening eml-files is still awfully slow, even though he archived also his old emails. There seems a connection also to the Thunderbird profile size. And it all started with the Supernova Update.

Hello,
I just have the same issue with almost all my users ( about 50 mailboxes on Thunderbird).
We work all the time with EML files archived on our file server.
Before the Supernova update EML files opened in seconds, but since Supernova it sometimes takes minutes to open even a tiny littel mail with no attachments.
If users wants to do something in the wain Thunderbird window they can't because Thunderbird goes sluggish and almost crash.
It is crippling users worktime, and as IT Admin/Technician I'm being constantly questionned about this problems.

Technical details :

  • all PCs are DELL 1-5 years old max.
  • They all have Thunderbird with profile (default path) on the same SSD (SATA or NVMe depending of the model of PC)
  • The problem occurs either on Windows 10 Pro and Windows 11 Pro
  • Antivirus is Sophos. I saw in an other thread it can cause this, but I can reproduce the problem on PCs without Sophos.
  • Randomly, I can't explain why, some users are not affected.
  • The problem is present whatever the size of the mailbox (from 1 to 40 Go), or the size of the EML opened.
See Also: → 1856922

Is long paths enabled on the affected computers? (IIRC someone wrote something about that elsewhere).
Or could there be some other premission (ish?) attributes on the files that makes opening them slow.

Hi,
yeah i saw the post 1856922 before posting here, but in my case profiles are all on the main drive ( C: ) along with Thunderbird.
For Sophos as I said it doesn't matter if it is here or not ( i tested with several "problematic" TB profile on two computers : one with Sophos and one with fresh Windows install without Sophos = same result ).

It is very complicated to ask a user with +12000 mail (typically a project manager, a commercial, or the CEO ) to ask him to lose 2-3 hour to sort mails before/after 2023 and copy them on local folders, so I cannot test that "solution".

On all my pc the paths are the default one, when I install TB i let all settings by default.
I didn't do anything special about long path on Windows, we don't do anything special about Windows, except the base debloating and activate sharing of the Firefox and Thunderbird user folder for the save once a day.

The eml files open slow equally when we open from the servers or the Windows desktop on the pc, so I don't think there is a permission problem. I agree it could be the case if the only eml concerned where those on the servers. But in my case I don't think so.

(sorry for my bad english I'm french ^^)

From the profile in comment 4, it's GetShortPathNameW being slow. I found bug 394486. Florian, anything come to mind?

See Also: → 394486

Hello any idea of what is causing this bug ?

I found a workaround : if the user close Thunderbird, then open the .eml file, so Thunderbird just opened a window with the eml file quite quicker.

But with the very last version, if we do so Thunderbird open the main windows, and only after the windows with the eml (3-4 solid minutes for a hundred Ko mail !). So my workaroud dont work anymore...

My colleagues seriously consider go Outlook because they lose too much time waiting for their offline saved emails to open :-(

We in the IT department dont want to, but it become very difficult to find valid reasons if users lost 1-2h per day just waiting...

110 KB HTML e-mail of which 47 KB are an attachment opens in about 2 seconds, takes about 4 seconds to render completely.

Comment #1 asked for a sample file, but none has been supplied so far.
Comment #8 says: Randomly, I can't explain why, some users are not affected. So that sounds more like a problem of the system. In bug 1856922 the antivirus was at fault.
The profile from comment #4 shows an awful lot of time spent in Windows system calls that access files:
ntdll.dll, KERNELBASE.dll and KERNEL32.DLL (see screenshot).

Maybe someone can do another profile to confirm.

Hi. My colleague just confirmed that the workaround closing TB and then opening the .eml quicker really does not work any more since the latest update. On my PC, archiving the mails (on server archive folder) helped. My colleague also tried this but with no effect.

Comment #1 asked for a sample file, but none has been supplied so far.

=> I represent a factory with 3 locations and +/- 200 users ans we receive hundreds of mail comming from intern (Thunderbird) or supplyers and customers (mainly Outlook). All kind of mail will take long to open when downloaded.

Comment #8 says: Randomly, I can't explain why, some users are not affected. So that sounds more like a problem of the system. In bug 1856922 the antivirus was at fault.

=> To me it is more likely profile related, because when I copy the same profile on different machines (Win10 / Win11 / with or without antivirus) I have the same problem. It seem that the users not affected (1-2% of all my users) have newer profiles with few mails.
Maybe the bug is caused by precedent majors updates that had consequent databases changes ?

For myself I am an the Linux (Mint) version and have no bug at all.
Today I will try one "buggy" profile with my version to see if it is only Windows related or not.

(In reply to [:Testnick] from comment #14)

Hi. My colleague just confirmed that the workaround closing TB and then opening the .eml quicker really does not work any more since the latest update. On my PC, archiving the mails (on server archive folder) helped. My colleague also tried this but with no effect.

Hello what antivirus have you and your colleague ?

(In reply to EB [:Shybear] from comment #16)

(In reply to [:Testnick] from comment #14)

Hi. My colleague just confirmed that the workaround closing TB and then opening the .eml quicker really does not work any more since the latest update. On my PC, archiving the mails (on server archive folder) helped. My colleague also tried this but with no effect.

Hello what antivirus have you and your colleague ?

Our company uses Sophos on all clients. We once uninstalled Sophos on my machine and the problem then went away. But it is company policy to run Antivirus so we had to reinstall it. In the end, I am pretty sure that it is not directly related to Sophos. As you wrote above, and since problems only started with the "Supernova Update", I think it is not related to Antivirus, Windows-Permissions etc.

On my machine TB at the moment works just fine. Why, I can't really tell. Archiving somehow did it for me. My colleagues really struggle as opening .eml from the server or also locally simply does not work.

(In reply to [:Testnick] from comment #17)

(In reply to EB [:Shybear] from comment #16)

(In reply to [:Testnick] from comment #14)

Hi. My colleague just confirmed that the workaround closing TB and then opening the .eml quicker really does not work any more since the latest update. On my PC, archiving the mails (on server archive folder) helped. My colleague also tried this but with no effect.

Hello what antivirus have you and your colleague ?

Our company uses Sophos on all clients. We once uninstalled Sophos on my machine and the problem then went away. But it is company policy to run Antivirus so we had to reinstall it. In the end, I am pretty sure that it is not directly related to Sophos. As you wrote above, and since problems only started with the "Supernova Update", I think it is not related to Antivirus, Windows-Permissions etc.

On my machine TB at the moment works just fine. Why, I can't really tell. Archiving somehow did it for me. My colleagues really struggle as opening .eml from the server or also locally simply does not work.

Ok thanks. We use Sophos as well, but before devs take the "not our fault" route, I confirm the bug persists on freshly formatted PCs or VMs which have never had Sophos installed, and even if I deactivate the Windows AV (anyway, nobody wants to run a PC with no antivirus, so even if it is the culprit Mozilla must adapt I think).
I confirm also that the version before "Supernova" worked like a charm. I don't hate this version that much, but as an IT technician I can tell it's an improvement considering the bugs and the feedback of my users.
Please don't make us go Outlook :-(

"but as an IT technician I can tell it's an improvement"
read "I can NOT tell" of course, sorry ;-)

So I did some tests with a "problematic" profile on the Linux (Mint, Ubuntu based) version and all works very fine : eml files opened very instantly as they should.

I can pretty much confirm this problem occurs only in Windows version (again with or without Sophos).

Could be interesting if someone who use an another AV can testify.

How .eml from files are handled is implemented was more or less rewritten for 115, so it may stem from that. But I don't see the issue with that code.

I will continue tests with my offline Windows 10 VM (without Sophos) and the same "problematic" sample profile.
I will post updates if i find out something new.
From my experience it could as well come from a shitty Windows parameter or somewhat...
At least the Linux version works pretty fine. Too bad I can't install Linux on all my computer ;-)

(In reply to Magnus Melin [:mkmelin] from comment #21)

How .eml from files are handled is implemented was more or less rewritten for 115, so it may stem from that. But I don't see the issue with that code.

Shouldn't that then be of major interest in tracking down the problem? I mean seriously: if this particular portion of code was completely rewritten, it seams the obvious place to search for the bug first. Of course users can provide example .eml files and create performance tests etc. - but in my humble opinion I would urge the developers to have another look. Thank you.

After some tests on Windows 10 here's some results :

System : freshly installed Win10 Pro on an offline PC without antivirus (except the Windows one).
The system is on one Samsung 860 SSD drive and the file I try to open is on the same user desktop (random 32 Mo mail I saved from the test profile).
The test profile is IMAP (17 Go mails copied on the disk) and have a Local Folder with many subfolders that takes +/- 50 Go.
I configured a brand new mailbox for tests, which is empty.
I switch between them by deleting / copying profile files directly from a save.

  • Big profile with 50 Go Local Folder = eml slow
  • Big profile with NO Local folder = OK, opens instantly
  • New profile with 50 Go Local Folder = eml slow
  • New profile with NO Local folders = OK, opens instantly

So it seems the slowness of opening eml is directly correlated with the size and/or number of subfolders and/or number of mails of the "Local Folder", not the size of the IMAP mailbox even if the mails are saved on local drive.
Does TB execute some indexation when it opens an eml file ? If not what the Local Folders have to do with a file I open from an other location ?

I forgot to say that the "big" IMAP mailbox has also many subfolders and sub-subfolders.
I remember of something about the "long path" but it seem to cause no problems with the mailbox, only with Local Folders.

Ask me if you want other tests, now I have my benchmark ;-)

I just received an email from the "Betterbird"-project claiming that they solved the issue. I hope the Thunderbird team will also be able to sort this problem out. Thanks!

Me too, but the P.S. make me careful :
"P.S.: Please don't mention this communication on Bugzilla. You can mention that Betterbird fixed the issue."

If somebody says you "don't say it's me" or "don't say i talked to you", generally something will go nasty :-/

I'll take a look at it. Note to self: https://github.com/Betterbird/thunderbird-patches/blob/main/115/bugs/1863957-fix-slow-eml.patch

Assignee: nobody → mkmelin+mozilla
Status: UNCONFIRMED → NEW
Component: Untriaged → Account Manager
Ever confirmed: true

To open an .eml file, nsMailboxUrl::GetUri() gets called tree times.
Each time the system compares the filename to all the folder filenames
in the system. Clearly, no folder filename ever matches.
Especially on Windows, the operation of filename comparison is expensive.

Based on https://github.com/Betterbird/thunderbird-patches/blob/main/115/bugs/1863957-fix-slow-eml.patch

Status: NEW → ASSIGNED
status-thunderbird123: --- → affected
status-thunderbird_esr115: --- → affected
Keywords: checkin-needed-tb
Target Milestone: --- → 124 Branch

Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/4351a8f7c072
Fix slow .eml opening on profiles with many folders. r=BenC

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Keywords: checkin-needed-tb
Resolution: --- → FIXED

Comment on attachment 9377617 [details]
Bug 1863957 - Fix slow .eml opening on profiles with many folders. r=BenC

[Approval Request Comment]
Regression caused by (bug #): unclear, probably supernova just hitting a different code path due to changes in how .eml files are handled
User impact if declined: can get super slow opening of .eml files
Testing completed (on c-c, etc.): c-c
Risk to taking this patch (and alternatives if risky): safe

Attachment #9377617 - Flags: approval-comm-esr115?
Attachment #9377617 - Flags: approval-comm-beta?
Keywords: regression

Comment on attachment 9377617 [details]
Bug 1863957 - Fix slow .eml opening on profiles with many folders. r=BenC

[Triage Comment]
Approved for beta

Attachment #9377617 - Flags: approval-comm-beta? → approval-comm-beta+

Comment on attachment 9377617 [details]
Bug 1863957 - Fix slow .eml opening on profiles with many folders. r=BenC

[Triage Comment]
Approved for esr115

Attachment #9377617 - Flags: approval-comm-esr115? → approval-comm-esr115+

I wanted to inform that opening the .EML files is now fast again. Thanks for fixing it.

Regards

rapettif, Thanks for the update

status-thunderbird_esr115: fixed → verified

Can confirm this as Verified.
Tested on OS WIN(64)
Version: 123.0a1
Build: 20240111105733

Version: 124.0a1
Build: 20240215104311

Mac 14
Version: 125.0b2
Build: 20240325163039

Status: RESOLVED → VERIFIED
status-thunderbird123: fixed → verified
Keywords: perf
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: