Open Bug 1512974 Opened 6 years ago Updated 1 year ago

very slow to read complicated mail with images

Categories

(Thunderbird :: Message Reader UI, defect)

Product:
Thunderbird
Component:
Message Reader UI
Type:
defect

Tracking

(Not tracked)

People

(Reporter: llrraa, Unassigned)

References

(Depends on 1 open bug)

Details

(Keywords: perf, testcase, Whiteboard: [dupme])

Attachments

(2 files)

test.eml
1.98 MB, message/rfc822
Details
test2.eml
6.82 MB, application/octet-stream
Details
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36 Steps to reproduce: My inbox is about the size of 2GB. According to my observe, if the mail is very complicated(for example there is a 10X10 table ), it will be very slow to show that mail. When I click the mail, thunderbird will be no response, and starts to read the harddrive heavily. After several minutes, thunderbird would be ok again and shows the mail. But if the mail is simple, thunderbird is fine. Actual results: if the mail is very complicated(for example there is a 10X10 table ), it will be very slow to show that mail. When I click the mail, thunderbird will be no response, and starts to read the harddrive heavily. After several minutes, thunderbird would be ok again and shows the mail. But if the mail is simple, thunderbird is fine. Expected results: All mails should be read quickly. In the post "[Maildev] thunderbird near term focus" by Magnus Melin("http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2018-October/001317.html"), he says the following about "Address UI slowness": Address UI slowness * analyze UI slowness to and identify pain points * create tests to automate measuring of UI operational performance * perform architectural changes to support smoother operations * rewrites leveraging web technologies (especially moving load to workers where possible) * multi-process Thunderbird, using Frame scripts etc. UI jank of various kinds is a top complaint from our users. We should pinpoint the causes of these and fix them. I think having automated tests so that we can measure progress would be important to see things go in the right direction. I am quite interested in helping this since I'm facing very terrible slowness issues. I had to switch to another mail client at least temporarily due to the slowness. My inbox is about the size of 2GB. According to my observe, if the mail is very complicated(for example there is a 10X10 table ), it will be very slow to show that mail. When I click the mail, thunderbird will be no response, and starts to read the harddrive heavily. After several minutes, thunderbird would be ok again and shows the mail. But if the mail is simple, thunderbird is fine. So my guess is that thunderbird's storage is too complicated to get the content of a complicated mail, maybe a simpler storage will help on the performance. Also I observe another mail client named foxmail, which is quite popular in China. its storage makes good use of the file system. It seems that every mail is stored singly as a file. To avoid too many files in one folder, which may also lead to slowness, the files are stored in two level folders. So a mail may be stored as file "Mails\10\17\554" . Inspired by foxmail's storage, I think maybe thunderbird can have a new storage, one mail one file with folders leveled by time, such as "mails\yyyy\mmm\dd\hh\mm\ss\nnn", so every mail would be read in one single file, then hopefully performance would be ensured. Hope the Thunderbird Engineering Council would like this idea, do some confirmation test and work on it. I have some program experience. If needed, I might help on this , and just let me know what to help. If no one would do this, I will do it. But some help is needed to make the progress faster, since I'm not so familiar with the thunderbird source code. Help needed: 1. How to comfirm my guess is right or not? 2. If we want to change the storage, what documents and source code should I read?
The most relevant item to this report is "if the mail is very complicated(for example there is a 10X10 table ), it will be very slow to show that mail. When I click the mail, thunderbird will be no response, and starts to read the harddrive heavily. After several minutes, thunderbird would be ok again and shows the mail. But if the mail is simple, thunderbird is fine." So 1. Please use "Attach File" to attach a sample message to this bug report. 2. Please start Windows 7 in safe mode so that we know to what extent, if any, other software impacts the display of these messages and please report your results 3. Roughly how long has this been happening - by version number or number of weeks? 4. Which build date of version 65 are you using?
Flags: needinfo?(llrraa)
Keywords: perf
Keywords: testcase-wanted
Whiteboard: [closeme 2018-12-20]
Flags: needinfo?(llrraa)
(In reply to llrraa from comment #2) > Created attachment 9030688 [details] > a sample message to this bug report. > > please keep the attach file from public. Thanks. > The sample message to this bug report is simplified for business reason, so > it's not as slow as the true one. The true message is more complicated with > pictures and a lot of text. > > 2. Please start Windows 7 in safe mode so that we know to what extent, if > any, other software impacts the display of these messages and please report > your results > A: If Windows 7 in safe mode, thunderbird is working fine. But I don't know > how to figure out what extent, if any, other software impacts the display of > these messages. I haven't install any add-on. If it works in safe mode then some other software is causing a problem. Antivirus SW is a good place to start. Perhaps you have multiple AV/Malware programs installed? > 4. Which build date of version 65 are you using? > I'm using daily build 65 0al. Your build date/buildid is shown at Help > troubleshooting
build date/buildid：20181121100042 I test in Windows 7 in safe mode again, and find that some emails are still with slowness. I obverse the CPU usage, when thunderbird try to show the mail, it uses all the CPU. There's one email even worse. 65 is no response and uses all CPU and about 7G memory. In fact, that mail is only about 7MB. 56.0b3 will crash soon when reading this mail. So there's definitely some issue insight thunderbird. Is it possible that I get some log for you? I can send you the exact message for business reason.
For your infomation, those slow email are working fine in foxmail.
1. Do you have message bodies synced to Thunderbird (the default)? 2. Is this Inbox in an imap account, pop account, or local folder? 3. If imap, is still a problem if you copy the message to a local folder and open from there? 4. If still a problem start *Windows'* safe mode with networking enabled. Then still In Windows safe mode, start thunderbird in safe mode - https://support.mozilla.org/kb/safe-mode-thunderbird Does problem go away or reduce?
(In reply to Wayne Mery (:wsmwk) from comment #6) > 1. Do you have message bodies synced to Thunderbird (the default)? > > 2. Is this Inbox in an imap account, pop account, or local folder? > > 3. If imap, is still a problem if you copy the message to a local folder and > open from there? > > 4. If still a problem start *Windows'* safe mode with networking enabled. > Then still In Windows safe mode, start thunderbird in safe mode - > https://support.mozilla.org/kb/safe-mode-thunderbird Does problem go away > or reduce? 1.yes 2. The inbox is local folder 3. not imap 4.The problem is still the same. When these several messages are being shown, CPU are all used, and 600+ to 800+MB memory are used. After the messages are shown, the CPU usage goes down to almost 0, and memory about 150MB. There's one worst message, when it's being shown, 7GB+ memory is used, and it's never shown in the end after several hours. I have to terminate it manually.
(In reply to llrraa from comment #4) > I can send you the exact message for business reason. Please do that, or even better attach it to this bug (if possible, you can remove personal info).
(In reply to Magnus Melin [:mkmelin] from comment #8) > (In reply to llrraa from comment #4) > > I can send you the exact message for business reason. > > Please do that, or even better attach it to this bug (if possible, you can > remove personal info). That's a mistake. I mean I can't send you the exact message for business reason, expectialy when the attach file is public. Is it possible that you send me a private mail to me, then I can send you the mail.
The 7mb email provided to me contained 10,000 images, what appears to be a zip file, and 35,000 lines of html in QP format! The performance issue/hang is seen with two modified cases - just the images, or just the QP. That later issue is something like bug 289644, but I'm reasonable certain there are other bug reports. If someone wanted to create a Denial of Service message, this would be a classic. This is not related to folder size, or background activity, or general UI performance. Who is sending you this stuff? And why are there 10,000 images? (all small, but still)
Flags: needinfo?(llrraa)
Keywords: testcase-wanted
That's daily report. I get it every day. In fact they are not images. They are several tables. Foxmail can successfully show it. I think thunderbird has update for this issue. I can see it today in thunderbird, but only a small part of it. And so are all the other daily report mails. That's too bad for me.:)
Flags: needinfo?(llrraa)
It takes also quite long time for foxmail to show this particular mail, but it's still sucessfully shown. Foxmail can show other daily report mails easily and completely.
(In reply to llrraa from comment #12) > It takes also quite long time for foxmail to show this particular mail, That's not terribly specific. How many minutes is a long time? > In fact they are not images. They are several tables. Foxmail can successfully show it. How long have you been using Thunderbird? Has it every displayed these messages fast(er)? Please email me a smaller one that is same format and does not take so long to display - 1-2 minutes for example.
Flags: needinfo?(llrraa)
(In reply to Wayne Mery (:wsmwk) from comment #13) > (In reply to llrraa from comment #12) > > It takes also quite long time for foxmail to show this particular mail, > > That's not terribly specific. How many minutes is a long time? 7 minutes. > > > In fact they are not images. They are several tables. Foxmail can successfully show it. > > How long have you been using Thunderbird? > Has it every displayed these messages fast(er)? > Please email me a smaller one that is same format and does not take so long > to display - 1-2 minutes for example. I'm using thunderbird since 2014. Before October 2018, they were sending mails in different format, in which the tables are really pictures. Since October 2018, the messages are sent as tables with text inside. I think they just copy the tables from MicroSoft excel. I think thunderbird never displayed these messages fast(er).
Flags: needinfo?(llrraa)
Whiteboard: [closeme 2018-12-20]
Any update on this bug? I have some program experience. If needed, I might help on this , and just let me know what to help. If no one would do this, I will do it. But some help is needed to make the progress faster, since I'm not so familiar with the thunderbird source code. Help needed: What documents and source code should I read for this bug?
We first need to identify the cause. A profiler will point us to the hot code, but the patch still pending in bug 1507709
Severity: normal → major
Status: UNCONFIRMED → NEW
Depends on: 1507709
Ever confirmed: true

llrraa: If you want to help, please be specific. Your reports so far have been very vague. "It's slow" and "large mail" and "large table" is not very helpful.

You could start with creating minimal test cases. First, modify your email by removing all sensitive data from it, and attach it. Then, try to find which part of the email causes the slowness. Then, create an artificial email that has only this one feature (e.g. only a table, or only images, nothing else), but still shows the bug.

Make sure that the test case is realistic. Sure you can make all programs slow, if you just throw large enough data at it.

But frankly, from what the little you have said here, I think the problem is simply your email: 10000 images in an email are really over the top. You should put the report on a webpage and look at it here. And use faster HTML, not 10000 images. If it's third party software, please contact the vendor of that software.

Attached file test.eml —

test mail for this bug. It contains 16*187 table in it. The content is copied directly from excel in foxmail(a popular mail client in China).

(In reply to Ben Bucksch (:BenB) from comment #17)

llrraa: If you want to help, please be specific. Your reports so far have been very vague. "It's slow" and "large mail" and "large table" is not very helpful.

You could start with creating minimal test cases. First, modify your email by removing all sensitive data from it, and attach it. Then, try to find which part of the email causes the slowness. Then, create an artificial email that has only this one feature (e.g. only a table, or only images, nothing else), but still shows the bug.

Make sure that the test case is realistic. Sure you can make all programs slow, if you just throw large enough data at it.

But frankly, from what the little you have said here, I think the problem is simply your email: 10000 images in an email are really over the top. You should put the report on a webpage and look at it here. And use faster HTML, not 10000 images. If it's third party software, please contact the vendor of that software.

You are right about the test case. I have attach test.eml for test.
test.eml contains a 16*187 table copied directly from excel in foxmail( a popular mail client in China.)
To make the test case realistic, you can install foxmail at https://www.foxmail.com/win/en/, then try to open test.eml in foxmail and thunderbird, then you will know it's realistic.

I see no real delay opening the EML file at all. No more than a few seconds from when I double click on in on the desktop where I saved it until it is displayed.

(In reply to Matt from comment #20)

I see no real delay opening the EML file at all. No more than a few seconds from when I double click on in on the desktop where I saved it until it is displayed.
I have attached 2 files, please make sure you open the second one which is larger.
If you do open the right file, can you please share your thunderbird version. I will try it.

(In reply to llrraa from comment #21)

I have attached 2 files, please make sure you open the second one which is larger.
If you do open the right file, can you please share your thunderbird version. I will try it.
I see only one attachment to this bug. Test.eml. You posted it 21 hours ago. So if you want me to open something else, please attach it.

(In reply to Matt from comment #22)

I see only one attachment to this bug. Test.eml. You posted it 21 hours ago. So if you want me to open something else, please attach it.

Did you see the whole content? A table with 1 to 16 in the first line, 1-187 in the first column, and 1 in all the other cells.
If yes, can you please share your thunderbird version. I will try it.

It take time to load the first time as my slow disk takes an age to deliver the data but still it is only noticeable as delayed because I am watching, we are talking a few second perhaps 5. Opening that same file a second time where the data is cached by windows takes about 2 seconds.

Name Thunderbird
Version 60.4.0
Build ID 20181217101353
Update Channel release
User Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0
OS Windows_NT 10.0

(In reply to Matt from comment #24)

It take time to load the first time as my slow disk takes an age to deliver the data but still it is only noticeable as delayed because I am watching, we are talking a few second perhaps 5. Opening that same file a second time where the data is cached by windows takes about 2 seconds.

Name Thunderbird
Version 60.4.0
Build ID 20181217101353
Update Channel release
User Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0
OS Windows_NT 10.0

Just to make sure, did you see the numbers of the EML files?
I try 60.4.0 in windows 7 safe mode and thunderbird safe mode, but see only blank page, no table and numbers at all.

happens at least as far back as version 59, so setting version accordingly

Keywords: testcase
Version: 65 → 59
Severity: major → normal

(In reply to llrraa from comment #4)

Is it possible that I get some log for you? I can send you the exact message for business reason.

Please run the performance profiler:

  1. You must be using Thunderbird 68 or newer - betas from https://www.thunderbird.net/en-US/channel/ or current nightly build from https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/
  2. Install profiler add-on into thunderbird 68 (or newer https://www.thunderbird.net/en-US/channel/ ) - get the add-on file from https://github.com/firefox-devtools/Gecko-Profiler-Addon/blob/master/gecko_profiler.xpi?raw=true and in Tools > add-ons click the gear to install add-on from file
  3. Follow instructions at https://profiler.firefox.com/ (videos BASED ON FIREFOX at https://profiler.firefox.com/docs/#/./videos-intro )
  4. Create a profiler URL and post it here.
Flags: needinfo?(llrraa)

I'm working on it for days. But since I'm quite busy these days, and I have to learn the performance profiler, it may still take me some days. If possible, someone who is good at the performance profiler can actually try to use thunderbird to open test.eml with the performance profiler started, so as to faster the progress.

And thank you for following up, Mr. Wayne Mery (:wsmwk). I was afraid that you already forgot this issue. :)

Today I try the start the performance profiler, but failed to Capture Profile, and ctrl+shift+2 is not working either. Please help.

Application Basics

Name: Thunderbird
Version: 72.0a1
Build ID: 20191104093451

Update Channel: nightly
User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:72.0) Gecko/20100101 Thunderbird/72.0a1
OS: Windows_NT 6.1

Launcher Process: Enabled
Multiprocess Windows: 0/0
          Disabled
Remote Processes: 0
Enterprise Policies: Inactive
Google Location Service Key: Missing
Google Safebrowsing Key: Missing
Mozilla Location Service Key: Missing
Safe Mode: false

(In reply to Wayne Mery (:wsmwk) from comment #27)

(In reply to llrraa from comment #4)

Is it possible that I get some log for you? I can send you the exact message for business reason.

Please run the performance profiler:

  1. You must be using Thunderbird 68 or newer - betas from https://www.thunderbird.net/en-US/channel/ or current nightly build from https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/
  2. Install profiler add-on into thunderbird 68 (or newer https://www.thunderbird.net/en-US/channel/ ) - get the add-on file from https://github.com/firefox-devtools/Gecko-Profiler-Addon/blob/master/gecko_profiler.xpi?raw=true and in Tools > add-ons click the gear to install add-on from file
  3. Follow instructions at https://profiler.firefox.com/ (videos BASED ON FIREFOX at https://profiler.firefox.com/docs/#/./videos-intro )
  4. Create a profiler URL and post it here.

I try to start the Gecko Profiler(buffer size: 90MB), and Capture Profile without doing anything, it does open a new tab and try to grab the profile, but it fails with the following hint:
We were unable to connect to the Gecko profiler add-on within thirty seconds. This might be because the profile is big or your machine is slower than usual. Still waiting...

Should I report a new bug for this?

Someone is working on the profiler. You might find a match for your issue in this list https://mzl.la/2ABZJ2q but there are several more.
That said, I am not able to reproduce this issue using current beta 78 build on small Macbook Air.

Flags: needinfo?(llrraa)
Attached file test2.eml —

(In reply to Wayne Mery (:wsmwk) from comment #32)

Someone is working on the profiler. You might find a match for your issue in this list https://mzl.la/2ABZJ2q but there are several more.
That said, I am not able to reproduce this issue using current beta 78 build on small Macbook Air.

Yes, now beta 78 can open test.eml quickly, and that's a really breakthrough.
But there's still some issue with some more complicated mail, such as my new attached mail file: test2.eml, which size is about 7MB.
When I try to open it with thunderbird, at first, the I/0 rate reaches more 300MB/s， and memory usage increases quickly, and finally reaches about 9GB. I wait 30 minutes, but the content doesn't show up yet, so I give up.
For foxmail, it only takes about 3 minutes to open this file, and use only about 200MB memory, and I/0 rate is about 100KB/S. But foxmail fails to forward or reply this mail .
Hope the community can continue to improve the performance of thunderbird, so that it can open, forward or reply test2.eml quickly with low resource.

test2 is 7mb with about 15 thousand table elements and 11k image/png. If image/png item are removed the test is 3mb and the message loads quickly.

That probably makes this dependent on bug 563677

Depends on: 563677
No longer depends on: 1507709
Summary: very slow to read complicated mail → very slow to read complicated mail with images
Whiteboard: [dupme]
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: