Closed Bug 380897 Opened 17 years ago Closed 15 years ago

FOUND SOLUTION: Problem Thunderbird hanging on large e-mails

Categories

(Thunderbird :: Message Reader UI, defect)

x86
Windows 2000
defect
Not set
major

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: heliophorus, Unassigned)

Details

(Keywords: perf, Whiteboard: [waiting on reporter][closeme 2009-03-15])

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; nl; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3
Build Identifier: 2.0.0.0 (20070326)

You see a few tickets about large e-mails.  Thunderbird hangs on large e-mails
using IMAP, or viewing large emails in local folders.  Older versions do crash. Version 1.5.x and v2.0 runs in a time consuming loop.

More ! Normally Thunderbird use 20.000K memory (v1.5.x), 45.000K memory (v2.0) When reading the mail for display, it runs up to 570.000K memory !!!

What do You do ?  You cancel the process in taskmanager (windows)

Short term solution:
In case of large e-mails:

   Close MESSAGE PREVIEW PANE (F8) 
   And You can handle the e-mail for removal etc.

Long term solution for developers:

   a) Message-view should load and display only a part of the mail.
   Much like an editor handling large mails with a paging feature.
   And it should display arrows to browse through the large email.
   Much like paging.

   b) Memory.  Thunderbird should not use so much memory !!!
   At least it may not raise to 570.000K of mem. Due to script, or due
   to internal mechanism.  This is asking for trouble anyway.

   c) (Qmail) Maildir storage would be a better option to remove large mails
      You have to remove ONE file.  Currently You need to remove the large
      mail from Your mailbox and recompress folder to remove wasted space
      and to shrink the inbox file.
      
I deserve a signed Thunderbird T-Shirt from the dev team for this :-D 
Don't I :-D


Reproducible: Always

Steps to Reproduce:
1. produce a large e-mail, around 50-100MB in TEXT
2. Use IMAP to view the message
3. Put the message in local folders.  Same results when viewing
4. Disable with F8 preview.  you can handle the e-mail
5. When loading preview look to memory consumption by task manager
Actual Results:  
Crash or loop.

Expected Results:  
Streamline memory usage
Loading large messages in paging (block)

Problem and cure.
Can be used as D.O.S. !
Last but not least.

Besides (qmail) Maildir.

SQL Storage would be welcome to avoid problems with large e-mail folders.
ODBC connection (windows) or Mysql support with DBmail(.org)
Not a security bug (and not really a bug at all, as far as I can see, but I'll leave that to Thunderbird triagers).
Group: security
This IS A SECURITY bug.
You can perform denail of service (DOS) with this.
What does the attack scenario look like? Sending out hundred megabyte emails isn't really a realistic attack vector, and even if it was, DoS issues don't need to be kept secret. I think it's pretty well known that dealing with (opening, displaying, etc) files that approach the size of installed RAM can cause performance issues.
Sometimes internal mail can be oversized.  Logs and reports
from servers.  In a spam attack to a server, You can have
a mail from the root or postmaster of Your own server(s), 
reporting all abusive attacks.  You can have e-mails which 
can be pretty large in textual form. Sometimes logs are not 
compressed. System logs etc.

In our case.  Our servers are located in the USA. But we retrieve
mail with IMAP TLS. You need to make scripts on the server, to filter
large e-mails, because thunderbird can't handle them and runs
in a time consuming loop.

If You are an IT, You can handle those problems.  However, if You are
a regular user of thunderbird, You will think thunderbird is not working
in a good manner.

Anyway the problem is flagged a few times on bugzilla.  I have the solution
to the problem.

Qmail is bug 58308.

Don't seem much of a bug here. 50-100 MB text will be slow and eat a lot of memory, nomatter what. And few servers even accept such sizes...

->INVALID
Status: UNCONFIRMED → RESOLVED
Closed: 17 years ago
Resolution: --- → INVALID
A mail client running up to 570.000K memory !
And You flag this bug as invalid.
What a wonderful programmers You are.
Status: RESOLVED → UNCONFIRMED
Resolution: INVALID → ---
Related to cpu usage:
https://bugzilla.mozilla.org/show_bug.cgi?id=296453
-> why You need to tune memory and cpu usage

Related to large mail folders:
https://bugzilla.mozilla.org/show_bug.cgi?id=347665
-> why You should consider SQL storage on Mysql and ODBC

Related to large e-mails:
https://bugzilla.mozilla.org/show_bug.cgi?id=298297
https://bugzilla.mozilla.org/show_bug.cgi?id=334358
https://bugzilla.mozilla.org/show_bug.cgi?id=358427
https://bugzilla.mozilla.org/show_bug.cgi?id=373821
https://bugzilla.mozilla.org/show_bug.cgi?id=379195

So before You flag this invalid again.

Setup a testcase on a remote server on a large distance in terms
of hops. load 6000 e-mails in Your imap folder.  Some of them,
pretty large.

Tell me if You can work on Your mail account by IMAP.

In our case this is not possible.
Connection breaks and red flags on CPU and memory usage.

So.  you can easely mailbomb an account, exploiting this
'feature'.
(In reply to comment #8)
> Setup a testcase on a remote server on a large distance in terms
> of hops. load 6000 e-mails in Your imap folder.  

Helios. Is this the real issue for you?  If so, under what scenarios? Is the typical scenario when thunderbird starts up?  Or just when you view a large message?  At what size does it become a problem?  Is it all large messages, or just some?

(Please let's stick to just one issue and try to keep that issue as simple as simple as possible. The more issues you introduce the more difficult it becomes to find the solution, eg I don't see bug 298297 or bug 296453 being related to your steps to reproduce in comment 0)

FWIW, iirc some progress is being made in the "large text messages" issue(s)
Severity: critical → major
I don't think it's a security issue as such. You'd have to be mailbombed for this to have any real effect, so your email box would probably fall over first.

But if not a bug, it is most definitely a performance issue. For example, I've just been sent an XML file of just 1MB (not zipped, .xml). When I highlight this message, Thunderbird freezes up for 30 seconds or so. Eventually the message does appear, complete with 1MB of XML underneath it. But during this period, Thunderbird is completely frozen, and the whole machine pretty much grinds to a halt (Windows XP SP2, Pentium 4, 3.42 GHz with 1GB RAM).

Although the preview feature is handy, it should not cause the program to freeze up in this way. It was just a 1MB file, not the 50-100MB mentioned earlier (that many email systems would reject anyway). Imagine what 5MB would do?

Since the preview pane is exactly that, a preview, then it should just display the first 50 lines of text from any attached text document. That way:

1) The email package does not freeze up for long periods when you just happen to select this mail
2) The user can still get a brief preview of attached text which should be enough to see what it is
3) The user can download any attachment for further investigation in the appropriate programme, without having to wait for Thunderbird to come back to life first

Sending around text documents of 1MB in size is not unusual or extreme usage of Thunderbird, yet it causes severe performance issues.
We're probably examining and linkifying urls in the message. Are there a lot of things that look like URL's in the text?

With 2.0, we also spend a lot of time in the code that figures out if any of the urls are phishing urls. 
David, yes that sounds like it - it's a froogle feed so has a URL every few lines of XML. I guess for security you'd need to check every URL in the displayed text, which might be another good reason to limit the amount of text shown in a preview? Or at least have a toggle for big previews "Show the whole of this attached text". At least that way I wouldn't have to tread carefully around this mail while it's in my inbox (it sinks TB for a good time if I accidentally select it).
> Helios. Is this the real issue for you?  If so, under what scenarios? Is the
> typical scenario when thunderbird starts up?  Or just when you view a large
> message?  At what size does it become a problem?  Is it all large messages, or
> just some?

The problem araise for ALL emails.
If You use the massage preview pane, this window tries
to load the full E-mail for reading.

If the e-mail does have a large text. Thunderbird locks the system down
while reading the full message.

Similar case if You open a large textfile in an texteditor. Some editors
do provide a method to edit very large text files. They load only a part 
of the message. You can define the amount of lines in a preference field.
When you scroll down, the editor loads the next part. But the editor does
not open the full message at once. 

You can create an e-mail with a size of 5MB. Measure loading. Increase this 
file to 50MB.  See what happens.

There are servers, which do not have quota protection on the size of e-mails.
In my case, there was no quota restriction on a postmaster account.  This account was receiving the statistics and logs of the server in textual form.
A few weeks ago, there was a major attack on the server.  The server did protect itself very well, however, the machine returned (automatically) a very large e-mail with a description of the attack, etc.

I do not blame Thunderbird at all. I tried several other e-mail clients.  They have the same problem.  They lock down the system at the moment they want to provide a preview of the mail, the memory is increasing during load and in some cases, You need to reboot Your system.

If You have several large e-mails, You can not scroll to other e-mails, because 
they want to preview those large e-mails, so the client 'hangs' on every large e-mail You have.

As said, if You disable the 3th screen (F8) You do not have this problem.
The cure is to code the same procedures as You find into text editors.
Loading text in blocks of lines.
oeps, 
i did't read the mail of captain_flack before posting.
But he makes the same conclusion, the same point.
(In reply to comment #13)
> > Helios. Is this the real issue for you?  If so, under what scenarios? Is the
> > typical scenario when thunderbird starts up?  Or just when you view a large
> > message?  At what size does it become a problem?  Is it all large messages, or
> > just some?
> 
> The problem araise for ALL emails.
>...

if you see the problem even for small messages then it's not limited to large messages :)

do you see the problem if you keep address book open while viewing message?
It has not to do with Phonebook it has to
do (correction) with all LARGE emails.

quote: If You have several large e-mails, You can not scroll to other e-mails, because they want to preview those large e-mails, so the client 'hangs' on every large e-mail You have.

This is the solution to implement:
Since the preview pane is exactly that, a preview, then it should just display
the first 50 lines of text from any attached text document. (short review)

note:  Currently I have switched to DBmail to store all messages to MySQL.
       I run over 500.000 emails and large mailboxes on the old IMAp sofware
       needed to split into sub-boxes.  With MySQL I don't have this problem.
       I have less impact of this issue on DBmail but it still exists.

note2: The future of mailstorage is definetly inside SQL tables. Maildir
       creates too many filehandles, and the regular storage to a file is
       dangerous for corruption and slow with big mailboxes... I guess
       Thunderbird should tune in into this evolution.



The problem is the preview pane, because the problem solves if You close the preview pane :-) So there is a need to limit the rows You display into the preview plane, three dots on the end and put a button to view the full message...
Helios, When you are willing to test what is suggested (eg comment 15) feel free to cc: me back on the bug.  preview pane is the symptom, not the issue. 

There may not be just one issue. But you're going to have pick one message that causes a problem and run with it - this query [1] lists some, but please don't spam them with comments unless you have a testcase message to contribute that matches the symptoms.


[1] https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&product=Core&product=Mozilla+Application+Suite&product=Thunderbird&long_desc_type=anywordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&resolution=WONTFIX&resolution=---&bug_severity=blocker&bug_severity=critical&bug_severity=major&bug_severity=enhancement&emailassigned_to1=1&emailtype1=exact&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailtype2=exact&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=short_desc&type0-0-0=nowordssubstr&value0-0-0=chang+crash+reply&field1-0-0=component&type1-0-0=anywordssubstr&value1-0-0=mailnews%3A+main+front&field1-0-1=product&type1-0-1=substring&value1-0-1=thunder&field1-0-2=short_desc&type1-0-2=anywordssubstr&value1-0-2=mail+message&field2-0-0=short_desc&type2-0-0=anywordssubstr&value2-0-0=hang+slow+freez+froz+long&field2-0-1=keywords&type2-0-1=substring&value2-0-1=perf&field2-1-0=keywords&type2-1-0=substring&value2-1-0=&field3-0-0=short_desc&type3-0-0=anywordssubstr&value3-0-0=big+large+load++html+link
I would not register a bug in the system, if I did not test the issue by myself.
I write software for two decades.  i known how to set up a test case.

Take any imap server, fill it with 10K e-mails, some of them large (unzipped logs
in plaintext. Scroll through the messages.  And You will see what happens on
each large e-mail, if the preview pane is activated. If You disable it, speed
turns back.
Is this issue being addressed, or has it been left alone because the workaround (disabling the preview-pane) was found?

Comments #10, #11 and #12 seems to be spot on, according to my observations in Thunderbird 2.0.0.9 (20071031) on Windows XP.

We get commit-mails from perforce, with links to diffs of each file committed. 

Lately we've made some rather large changes, and we have had commit-mails of around 3-4MB of text, with URLs on almost every line. If you should happen to click on one of those, TB grinds to a halt. If you leave it for a while, it finishes whatever it was doing and displays the mail.
Morten: that's bug 383718, you can turn off scam detection to check. (Maybe all of this bug is...?)
Magnus: You are right, turning off the scam detection removes the problem for me. My bad I guess... 
Reporter, does this still occur with the latest supported 2.0.0.x / trunk nightlies? (You can test with http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk/ builds for the latest trunk nightlies)
Assignee: mscott → nobody
=> incomplete due to no response to comment 23

please reopen if you still see this using current beta release
 http://www.mozillamessaging.com/en-US/thunderbird/early_releases/
Status: UNCONFIRMED → RESOLVED
Closed: 17 years ago15 years ago
Keywords: perf
Resolution: --- → INCOMPLETE
I will perform soon some tests.
Status: RESOLVED → UNCONFIRMED
Resolution: INCOMPLETE → ---
you will want to test with beta 2.
please quantify times.

please describe details of which other mail apps have the same problem where you say in comment 13 "I do not blame Thunderbird at all. I tried several other e-mail clients.  They have the same problem.  They lock down the system at the moment they want to provide a preview of the mail," ... so,
Component: Mail Window Front End → Message Reader UI
QA Contact: front-end → message-reader
Whiteboard: [waiting on reporter][closeme 2009-03-15]
RESO INCO due to lack of response to last question. If you feel this change was made in error, please respond to the bug with your reasons why.
Status: UNCONFIRMED → RESOLVED
Closed: 15 years ago15 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.