619493 - Message with too many email addresses in TO: or CC: field hangs TB. Need to sanity check display of addresses.

Status: NEW

(Thunderbird :: Message Reader UI, defect)

Description

[patrick_bugzilla]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 ( .NET CLR 3.5.30729; .NET4.0E)
Build Identifier: 3.1.7

I received a spam email that had more than 20.000 email address in the TO: field!!!

TB hangs then to create the .msf file. I can't then access to anything. Erasing or reparing the msf file does not solve the issue as when TB start recreating it the problem occurs again. Only a manual edition of the inbox file to erase the too many email addressee solve it.

I can't even understand how an email server can let these email propagate anyway. But TB should be smarter to avoid hanging as it could be used for malicious purpose.

Reproducible: Always




Was happening on different OS: Windows XP, Vista and 7x64

Comment 1

[Wayne Mery (:wsmwk)]

patrick you might be seeing bug 460605. Did it hang only when you clicked on the message?  If so, in the future you can probably right-click on the message and pick delete.

asuth, a thought - does gloda sanity check against "too many addresses"?

Comment 2

[patrick_bugzilla]

No, the problem is I couldn't even do anything within TB as it hangs as soon as I go to the message list (rebuilding the index).

My only option (after 3 hours trying to understand what was bugging, trying switching between imap and pop, using webmail etc...) was to go with a text editor in the inbox file to find the guilty message to manually erase it.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Patrick, any chance you saved a copy of the message?  We can obviously synthesize a message with 20,000 addresses, but it's possible there could be other intentionally bad stuff in there that might be even more concerning.

Wayne, gloda does not defend against too many addresses.  Its processing is sufficiently expensive and does not yield much on the address processing step that I could see it playing the largest part of any apparent lockup.

I would somewhat expect the .msf file to get successfully written if gloda is involved though, so it's also conceivable that the badness happens before things get to gloda.

Comment 4

[Ludovic Hirlimann [:Usul]]

No need to keep this secret - it's only a DOS.

Comment 5

[patrick_bugzilla]

I looked at the email, there is nothing else bad in it. I tested it by removing manually only the too many addresses and the message was handled properly. Just a few lines of basic html, without any embedded code or attachments.

Comment 6

[Wayne Mery (:wsmwk)]

Thanks Andrew. Your wide-ranging expertise in this area confirms my thinking. 

I took an interest in this because I am familiar with the other bugs if this ilk, both of which are long fixed but it is certainly possible for unfixed cases to exist.
Bug 460605 - Message Reader: header pane right-click context menu popup takes too long for contacts in long list with lots of addresses
Bug 415135 - Slow/Hang/lock-up on long From lines in header with high cpu

bug 415135 has a testcase which could probably be easily modified to test this bug.

Comment 7

[WADA:World Anti-bad-Duping Agency]

20000 addresses was not sufficient in my environment, so checked with 60000 addreseses.
  To: <a-000000@a.b.c>, <a-000001@a.b.c>, ... , <a-060000@a.b.c>  
  Simple text/mail.
  This mail only in a local mail folder.

(1) Rebuld-index took long and it took long to show thread pane, but not so severe.
(2) Clink mail at thread pane. 
Mail display took long. CPU 100%(CPU 50% if dual core) is kept for long time. But mail was shown like "to: a-000000@a.b.c, ... a-000005@a.b.c 55595 more" at header pane. "Unresponsive script" dialog might be shown.
(3) Click "55595 more" at header pane.
CPU 100%(CPU 50% if dual core) kept for very long time. "Unresponsive script" dialog was shown multiple times.
As header was not expanded and it took too long, I killed Tb.

Above was observed with both Tb 3.1.7 and trunk nightly on Win-XP, but (1) and (2) was not so critical in trunk nightly as Wayne says. If trunk nightly, I could wait. (1) and (2) was improved by trunk even though it's not still sufficient.
But (3) was still critical even with trunk nightly. I killed Tb again.

Cinfirming.

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

WADA, thanks very much for the detailed reproduction.  Did you have gloda enabled in your test?

Comment 9

[WADA:World Anti-bad-Duping Agency]

I disabeled Gloda as usual in my test, because it's not intentional test of Gloda.

Comment 10

[WADA:World Anti-bad-Duping Agency]

Attachment: HTML/JavaScript to generate long To: header text

Comment 12

[Ben Bucksch (:BenB)]

From bug 653024 comment 4:
*If* we were to fix this, with the current implementation, I would just count
the number of addresses (i.e. array length) and if there're more than
CrazyNumber (e.g. 20), use only the first CrazyNumber for calculation and show
the others as "and 5623 more".

Comment 13

[Wayne Mery (:wsmwk)]

(the testcases cited might be rare, but are certainly at least sev=major)

Comment 14

[rsx11m]

(In reply to comment #12)
> CrazyNumber (e.g. 20), use only the first CrazyNumber for calculation and show

That can't be a constant though given that a user may have set the preference mailnews.headers.show_n_lines_before_more to show more than one line when opening a message. It sure could be a multiple of that value though.

Comment 15

[rsx11m]

Looking at http://mxr.mozilla.org/comm-central/ident?i=fillAddressesNode it appears to me that the filling of the address nodes is stopped now already once the maximum number of displayable addresses is reached, thus whatever causes that delay for long headers seems to happen before the width-calculation step.

Comment 16

[Hendrik Brummermann]

This does not only happen with SPAM messages but also in companies with about 300 people.

When creating an email, Outlook offers an easy way to expand an email-list. This is frequently used to exclude one member (for example to organizes birthday presents).

Workaround: Delete the message unread and ask an Outlook user to forward it.

Comment 18

[:aceman]

Is this still a problem today? I think rsx11m fixed a problem with caching the nodes displaying the recipients.

Comment 19

[rsx11m]

Bug 732144 was about caching the DOM nodes for display, but comment #15 should still apply here (specifically the task of parsing of long To/Cc headers).

Comment 22

[Jim Porter (:squib)]

Taking this to profile it.

Comment 24

[Bob]

I experience this problem with just 76 email addresses (just counted them). TB hangs for about 2 to 3 seconds in this case.

The assumption so far is that the user wants to delete these messages. That's not true. I get many of these valid work messages that I need to read along with everybody's replies, and they all have the same problems. The more people the bigger the lag in opening messages, and the more people reply to all...

Also, I keep the preview pane on, and move between messages by arrow. This will automatically trigger this problem to happen.  I tried to work without the preview pane, but it is not as efficient, and the problem still happens when opening the email in a separate tab. I don't know ahead of time which emails will have a large number of email addresses. So, I hit this annoyance every day many many times a day.

Just wanted to share my experience. Thanks!

Comment 25

[WADA:World Anti-bad-Duping Agency]

(In reply to Bob from comment #24)
> I experience this problem with just 76 email addresses (just counted them).
> TB hangs for about 2 to 3 seconds in this case.

This bug is originally for "severe performance problem when very many mail addresses are placed in message header", which is mainly caused by slowness in "very long string handling", slowness in "generating very many per-mail-addr XUL element".
See attached HTML. You can generate 10000, 100000, 1000000, ..., mail addresses in To:/CC: :-)

"A part of mail address display" is Address Book Search for "Show only display name for people in my address book", and if found in address book, "reading address book entry" and "replacing string by address book entry" are added.
(Tools/Options/Advanced/Reading&Display,Display, if Win. IIRC, enabled by default...)
Do you see difference between "the option Enabled" and "the option Disabled"?

Comment 26

[Bob]

(In reply to WADA from comment #25)
> (In reply to Bob from comment #24)
> > I experience this problem with just 76 email addresses (just counted them).
> > TB hangs for about 2 to 3 seconds in this case.
> 
> This bug is originally for "severe performance problem when very many mail
> addresses are placed in message header", which is mainly caused by slowness
> in "very long string handling", slowness in "generating very many
> per-mail-addr XUL element".

Thanks for your answer Wada. I did open a more specific Bug 977854 after searching through bugzilla and not finding exactly my problem. However, it was marked as duplicate of this one. So here I am. :)


> Do you see difference between "the option Enabled" and "the option Disabled"?

I have tried this option before and I tried it now on the 76 email addresses email. I don't feel a difference. I can't really time it accurately, but it is in the seconds whether that flag is set or not.

I had also gone through the Config Editor and searched for anything related to addresses and toggled it, nothing worked. 

I purged my address book. That didn't help the problem either.

Thank you for your time!

Comment 27

[Bob]

By the way, Viewing the Message Source (Ctrl + U) works in a snap. So it is about the graphical interface somehow.

Comment 28

[WADA:World Anti-bad-Duping Agency]

(In reply to Bob from comment #26)
> > Do you see difference between "the option Enabled" and "the option Disabled"?
> I have tried this option before and I tried it now on the 76 email addresses
> email. I don't feel a difference. I can't really time it accurately, but it
> is in the seconds whether that flag is set or not.

Do you see your problem with pretty simple mail but many To: or CC:?
1. Specify 10000 at attached HTML
   Setect 10000 mail addresses after "To:", Copy
2. Compose a text mail, put a mail address in To:, paste 10000 mail address in CC:, Send Later.
3. Move generated mail in Outbox of "Local Folders" to a local mail folder.
4. View the mail.
What is difference among 100 CC;. 1000 CC;, 10000 CC:, ... ? 

Do you see your problem with -safe-mode of Tb? (...\thunderbird.exe -safe-mode)

Comment 29

[WADA:World Anti-bad-Duping Agency]

Oh, you were opener of bug 977854 ...
Symptom of this bug is : Phenomenon written in comment #7.
"Same bug summary by user" doesn't always mean "same problem"...
I've re-opened bug 977854 for analysis of your case.

Comment 30

[Bob]

(In reply to WADA from comment #28)
> Do you see your problem with pretty simple mail but many To: or CC:?
Almost all the emails these days have formatting in them. Nobody sends plan text anymore. That's a good question though.

> 1. Specify 10000 at attached HTML
>    Setect 10000 mail addresses after "To:", Copy
> 2. Compose a text mail, put a mail address in To:, paste 10000 mail address
> in CC:, Send Later.

Thunderbird is stuck at this step now for the last ~10 minutes. I am going to sleep (It is 2am, PST time)

> 3. Move generated mail in Outbox of "Local Folders" to a local mail folder.
> 4. View the mail.
> What is difference among 100 CC;. 1000 CC;, 10000 CC:, ... ? 

I should have started with 100 :)

> Do you see your problem with -safe-mode of Tb? (...\thunderbird.exe
> -safe-mode)

Will try tomorrow. Hoping that I don't have to kill the Thunderbird process.

Btw, my machine is running an Intel i5-2540M @2.6 GHz, 4 GB RAM, with a Win 7 Enterprise SP1 64 bits.

Comment 31

[Bob]

It took about 10 minutes to process those 10000 addresses in the edit window. After that, saving the email took less time (seconds).

(In reply to Bob from comment #30)
> > 3. Move generated mail in Outbox of "Local Folders" to a local mail folder.
> > 4. View the mail.
It took around 10~12 seconds to open the email with 10000. 
I tried the same steps from scratch with 100 emails. There was no lag while editing the email. Opening it however in step 4 took around 4 seconds. (did this one in safe mode)

> > Do you see your problem with -safe-mode of Tb? (...\thunderbird.exe
> > -safe-mode)
Yes, same response time for both the 10000 addresses (text email) and the 76 addresses (html style)as without safe mode. The first takes about 10-12 seconds, the second takes around 2-3 seconds each time a message is opened either in the preview pane or in a separate window.

I forwarded the 76-address email to myself through outlook (I need it for its calendar function), the forwarded email opens in a snap in Thunderbird.

Thanks again!

Comment 32

[Ben Bucksch (:BenB)]

Bob, thanks for sharing this genuine story in comment 24, and for your kind tone. Very exemplary, and much appreciated.

The bug is clearly in Thunderbird's header pane email address display routine. That routine is very complicated, probably too complicated.

A fix/workaround is probably simple, see comment 12 and 14.
Any takers?

Comment 33

[Bob]

(In reply to Ben Bucksch (:BenB) from comment #32)
> Bob, thanks for sharing this genuine story in comment 24, and for your kind
> tone. Very exemplary, and much appreciated.
Welcome. I learned the hard way :)
 
> The bug is clearly in Thunderbird's header pane email address display
> routine. That routine is very complicated, probably too complicated.
Not to derail this thread and I am sure there has been a lot of discussion on this, but why was this format of header pane adopted? The code is complicated, and it is very inconvenient.
I tried installing this add-on which reverts to a simple editable list that can be copied between to: and cc:
https://addons.mozilla.org/en-us/thunderbird/addon/mrc-compose/
but it resulted in many crashes, so I disabled it for now. Ideally that's how the interface should have been.

> A fix/workaround is probably simple, see comment 12 and 14.
> Any takers?
I wish I could.

Comment 34

[WADA:World Anti-bad-Duping Agency]

"Critical performane problem" is usually observed only when "NNN more" is clicked, if NNN is not so huge.
i.e. "non-rendered XUL element generation for all mail addresses" is not critically slow.
     "rendered XUL element generation for all mail addresses" and "rendering(load)" takes very long.

Bob, do you use View/Headers/Normal mode? Or View/Headers/All mode?
If you use View/Headers/All mode, do you see next problem with View/Headers/Normal mode?
> with just 76 email addresses (just counted them). TB hangs for about 2 to 3 seconds in this case.
Note: "CrazyNumber in comment #12" is "Header Pane box width" currently.

Comment 35

[Bob]

First, thank you! But, please read on:

I was using the Normal mode.

(In reply to WADA from comment #34)
> If you use View/Headers/All mode, do you see next problem with View/Headers/Normal mode?
Actually, when I switched to "View All", that message was opening faster! There is still some lag, but it is way faster.

I then tried to open the emails that I created yesterday, and got conflicting/confusing results:
With Normal mode:
    100 emails: ~3 seconds in preview pane or in a separate tab (tried both).
  10000 emails: ~11 seconds in preview pane or in a separate tab (tried both).

With All mode:
    100 emails: ~1 second (comfortable time between two clicks on a chronometer, ie it is closer to 1 second than ~zero, but much better than with the Normal mode)
  10000 emails: ~45 seconds in the preview pane and ~41 seconds in a separate tab ?!!

So given that I don't get usually get 10000 emails, I think I am going to keep viewing "All" lines and see how that goes. It comes with an inconvenience that is easy to workaround: When forwarding an email in viewing All mode, all the headers are shown in a huge table, which is a pain to clean up (and should clean up when sending a business email). Fortunately, there is an add-on for that:
http://smarttemplate4.mozdev.org/index.html
(<rant> that's another of TB's usability issues, that should have been done as per this addon. It really took me a lot of time to find the 20+ addons that make Thunderbird more usable. It is a shame development has been halted. </rant>)

> Note: "CrazyNumber in comment #12" is "Header Pane box width" currently.
It seems to me that there are two problems that show up differently in the two modes. I hope you can figure it/them out. 

Thanks again!

Comment 36

[WADA:World Anti-bad-Duping Agency]

(In reply to Bob from comment #35)
> With Normal mode:
>   10000 emails: ~11 seconds in preview pane or in a separate tab (tried both).
> With All mode:
>   10000 emails: ~45 seconds in the preview pane and ~41 seconds in a separate tab ?!!

This difference is "this bug".

> With Normal mode:
>     100 emails: ~3 seconds in preview pane or in a separate tab (tried both).
> With All mode:
>     100 emails: ~1 second (comfortable time between two clicks on a chronometer,
>     ie it is closer to 1 second than ~zero, but much better than with the Normal mode)

Funny, in my environment(with local mail folder on local HDD):
> With All mode:
>     100 emails: N seconds (approximately 1 to 1.5 seconds)
> With Normal mode:
>     100 emails: mail instantly shown by mail click.
                  When "94 more" is clicked, it takes "N seconds"(==same as With All mode).

When "9x more" is clicked with Normal mode, what is observed?
Do you see same result with -safe-mode of Tb, with the crafted/simple test mail?

Do you see same difference on following mail?
> with just 76 email addresses (just counted them). TB hangs for about 2 to 3 seconds in this case.

1. Create local folder named Folder1, Folder2, copy the mail to Folder1/Folder2.
2. Edit ...\Folder1(not Folder1.msf) by Text Editor, change Subject to Test1,
   change to 10 email addresses, Save.
   At Folder pane, "Repair Folder" of Folder1.
3. Edit ...\Folder2(not Folder2.msf) by Text Editor, change Subject to Test2,
   change to 200 email addresses, Save.
   At Folder pane, "Repair Folder" of Folder2.
4. Create Folder0, copy original mail(76 email address) to Folder0, 
   copy Test1 in Folder1 & Test2 in Folder2 to Folder0.
5. Now you have three mails in Folder0.
     Test1    :  10 email addresses
     Original :  76 email addresses
     Test2    : 200 email addresses
What difference is observed on three mails by Normal mode/All mode?
With Normal mode which you are using in daily use, "delay in mail display"(freeze for a while) is linear to number of emails in message header?
(i.e. Majority of your "TB hangs for about 2 to 3 seconds" is actually problem of this bug?)

Comment 37

[Jim Porter (:squib)]

Unassigning myself from this, since I never made much progress here.

Comment 39

[:aceman]

I'll try to look at this. I worked with thousands of recipients in bug 1075398.
Corporations may need to address 100s and 1000s of recipients (their server may be configured to allow that).

(In reply to WADA from comment #28)
> Do you see your problem with pretty simple mail but many To: or CC:?
> 1. Specify 10000 at attached HTML
>    Setect 10000 mail addresses after "To:", Copy
> 2. Compose a text mail, put a mail address in To:, paste 10000 mail address
> in CC:, Send Later.

After pasting into To: or CC: is there a press on Enter? Then TB converts the recipients into 10000 rows.

This took about 10 minutes for me on current trunk (which is already optimized from bug 1292097) until it failed with 
"too much recursion - addressingWidgetOverlay.js:1158:1"

I'll make a new bug for this part.

Comment 40

[Bob]

Thanks aceman for taking a shot.
And Thanks Jim for attempting earlier; I am sorry for not replying on this thread anymore. Life has been busy.
But, I do still hit this problem every day, many times. As described in previous messages, I couldn't find a workaround.
Bob

Comment 41

[Bob]

Short of fixing the bug itself, how difficult is it to switch the order of rendering the message window vs the header window?
At least then we could start reading the message, while waiting for the header to show up. 
Not ideal, but, if it is quickly feasible, the end user feasibility is enhanced, especially when moving between emails of the same thread, the list of addresse doesn't change - it is just shuffled around.
Thanks for considering this!
Bob

Comment 42

[Bob]

Also it would be good to allow the pre-emption of this rendering, when the user tries to move to the next message, or wants to just delete the message.
But, that might be a bigger change probably?
Bob


(In reply to Bob from comment #41)
> Short of fixing the bug itself, how difficult is it to switch the order of
> rendering the message window vs the header window?
> At least then we could start reading the message, while waiting for the
> header to show up. 
> Not ideal, but, if it is quickly feasible, the end user feasibility is
> enhanced, especially when moving between emails of the same thread, the list
> of addresse doesn't change - it is just shuffled around.
> Thanks for considering this!
> Bob

Comment 43

[:aceman]

(In reply to Ben Bucksch (:BenB) from comment #32)
> The bug is clearly in Thunderbird's header pane email address display
> routine. That routine is very complicated, probably too complicated.

I have looked into this now an made some timings and it ruled out that the bottleneck is in fillAddressesNode(). This function runs for 20-40ms whether there are 300 or 4000 recipients. Visually I can see the rendering to take several seconds. So I need to find out where that time is used.

Any ideas where to look?

Comment 44

[Jim Porter (:squib)]

Hmm, could you use the devtools to record a performance profile and see where we're spending the most time? Other than that I couldn't really say where the bottleneck is...

Comment 46

[EarlgreyTea]

Attachment: generate-many-recipients.html

HTML (script) to generate a test message with many recipients.

Comment 47

[EarlgreyTea]

This bug has been open for a long time, but I would like to request a fix. And, I report a new case.

The following case was posted on the MozillaZine.jp forum.
https://forums.mozillazine.jp/viewtopic.php?f=3&t=20187

• After receiving a spam email, Thunderbird was stuck in "(Not Responding)" state for several hours.
• The spam email contained about 20 lines of text and no attachments, but contained about 900,000 addresses in To: field.

I think it's a serious problem that Thunderbird becomes unusable just by receiving one spam email.

(1) Performance issues such as displaying recipients should be fixed when email headers contain a large number of recipients.
(2) But if that can't be done quickly, I think Thunderbird needs to check the number of email recipients and work around it before it hangs if there are too many.

(1) Performance issues such as displaying recipients should be fixed when email headers contain a large number of recipients.

I did some performance testing.
(a)
I generated a test email using the attached generate-many-recipients.html, imported it into Thunderbird 102.2.0, opened the profiler and got the profile from opening the test email to displaying it.

(a-1)

• mailnews.database.global.indexer.enabled: false
• mail.show_headers: 1
• recipients: 200,000
• result: https://share.firefox.dev/3AgiaEN
• Gmail: It took 25 seconds for the email to appear. But then the Gmail page stopped.

(a-2)

• mailnews.database.global.indexer.enabled: false
• mail.show_headers: 2
• recipients: 10,000
• result: https://share.firefox.dev/3R8BkTM
• Gmail: It took 13 seconds for the email to appear. I also made it list all recipients, but the Gmail page didn't stop.

(b)
Imported a test email into Thunderbird and profiled until the "Not Responding" state went away.

(b-1)

• mailnews.database.global.indexer.enabled: true
• recipients: 200,000
• result: I waited 10 minutes but it didn't come back from "Not Responding" and I couldn't capture profiling.

(b-2)

• mailnews.database.global.indexer.enabled: true
• recipients: 10,000
• result: https://share.firefox.dev/3R3Lx43

(2) But if that can't be done quickly, I think Thunderbird needs to check the number of email recipients and work around it before it hangs if there are too many.

I used the extension "FiltaQuilla" to add JavaScript conditions to Message Filters, and tried setting the following script to perform actions according to the number of recipients.

const MAX_ADDRS = 10000;
let recipients = message.getStringProperty('recipients').split(",", MAX_ADDRS);
(recipients.length >= MAX_ADDRS);

This allowed me to do things like "Set Junk Status to" and "Move Message to".
I would like to see such a check made available as an option.