Open Bug 383718 Opened 17 years ago Updated 19 days ago

plaintext message with many URLs locks Thunderbird due to slow scam detection

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

People

(Reporter: schmidi2, Unassigned)

References

(
URL
)

Details

(Keywords: hang, perf, Whiteboard: [duptome])

Attachments

(2 files)

Zipped-up profile
671.21 KB, application/zip
Details
Screenshow with proposed solution
32.24 KB, image/png
Details
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.1) Gecko/20061208 Firefox/2.0.0.1 Build Identifier: 20070326 System-Mails send from a cronjob of the mnogosearch indexer causes Thunderbird to crash (unable to response interaction of user) immediately after opening the mail. CPU is use 100% Action after opening the mail: $ killall thunderbird-bin This problem is NEW in Thunderbird 2.0. Previous version (1.5) asked if you want to stop the script. By clicking on "Stop Script" you were able to read the full message. Mail details: Lines: 7739 Size: 1.2 MB Body: http://projekte2.t-p.com/benjamin/tmp The full eMail you can find at: http://projekte2.t-p.com/benjamin/tmp Reproducible: Always Steps to Reproduce: 1. 2. 3.
Severity: critical → major
Keywords: hang
Summary: Huge mail brings thunderbird to crash → Huge (large) mail brings thunderbird to hang
Please give the talkback crash ids. <http://kb.mozillazine.org/Talkback>
Ah, sorry, realized you said hang, not crash.
(In reply to comment #0) > CPU use 100% > Lines: 7739 > The full eMail you can find at: http://projekte2.t-p.com/benjamin/tmp It was not hang in my environment. It was simply huge memory consumption, and lo---ng time of process, and CPU usage as if loop, when many (more than 7600) linklify's of string in text mail. Tested with Thunderbird 2.0.0.0(Win-XP SP2, New profile,No extension). (Case-1) Mail data after changing "http://www" to "http!//www" 1-1. Render ended at a glance. (Case-2) Original mail data 2-0. Start Tb => VM size=14MB, open mail of case-1 => VM size=17MB, open mail 2-1. Top part was rendered and scroll bar was set properly after a while, (same display as case-1), then became no response. 2-2. Virtual memory size was increasing(2MB per 5 sec.) till 140MB during no response state. CPU utilization by Tb was 55% (core2 duo). (Note: if killed Tb at this stage, crash of Tb occurred) 2-3. After a several minutes, virtual memosy size increase stopped at 140 MB, then decreased to 125MB and CPU utilization becomes 0, and Tb came back. To Benjamin Schmidt(bug opener): Really hang in your environment?
CC-ing to Boris. In mail window, plain text mail with linkify is displayed using <BODY> <DIV> <PRE> <A>... internally. (checked by DOM inspector of Seamonkey) And this is very similar to test URL in Bug 216418 Comment #0. Is this bug simply DUP of Bug 216418?
Can changing from "<pre>all of <A></A's></pre>" to N*"<pre>several <A><A>'s</pre>" be a solution? (Splitting of huge <pre> to multiple <pre>'s is technique already used by View/Source)
So what exactly are the steps to reproduce? Please be sure to specify whether HTML or plaintext composition is used if something needs to be mailed, and exactly what needs to be done if something is just placed in the profile directory. > Is this bug simply DUP of Bug 216418? No idea until I have a profile. > Splitting of huge <pre> to multiple <pre>'s Breaks bidi resolution; we live with that in view-source, but it's not desirable at all in e-mail.
(In reply to comment #6) > So what exactly are the steps to reproduce? Oh sorry. I've forgotten that not all people is familiar with Unix Mbox file and spec of Thunderbird/Seamonkey who uses Unix Mbox file. (steps to reproduce the problem) 1. Save file of http://projekte2.t-p.com/benjamin/tmp thru "Save Link As" in context menu ("Save Link target As" when Seamonkey). 2. Copy the file to a file under mail directory of your Tunderbird or Seamonky. (Say file name of "Bug-383718"). You will see mail folder of "Bug-383718" after restart of Tb or Seamonkey. If you copied the file into a directory such as TEST.sbd, you can see mail folder of "Bug-383718" as subfolder of mail folder named "TEST". (Tb/Sm uses <folder_name>&<folder_name>.msf for a mail folder, and uses) (directory of <folder_name>.sbd to hold files of <sub_folder> & ) (<sub_folder>.msf for mail folder of <sub_folder> under <folder_name>. ) Note: After restart, mail count of "???" or invalid count is possibly displayed. This is already known and fixed on trunk bug. Re-open the mail folder(click other mail folder then click again) will correct the count display problem. 3. Open a(and only) mail in mail folder named "Bug-383718". (steps to confirm "no problem when no "http://www...") 1. Copy file of "Bug-383718" to "Bug-383718-altered" 2. Edit "Bug-383718-altered" by text editor, and change all string of "http://www" to "http!//www", and save it. 3. Restart Thunderbird or Seamonky. 4. Open mail folder named "Bug-383718-altered", and open a(onld only) mail
(In reply to comment #6) > Please be sure to specify whether HTML or plaintext composition is used This bug is problem when simplest mail - very usual mail headers(Content-Type: text/plain;) + null line + very simple mail payload lines for Content-Type: text/plain; (7bit us-ascii text lines). A special in this case : string of "http://www...." in almost all of more than 7000 plain&simple 7bit us-ascii text lines. See mail source, please.
So in a current trunk build of SeaMonkey I can scroll fine as the mail loads (normal layout of large page), until a point when it starts hogging CPU... then I get a slow script dialog, as described in comment 0. Note that the timeout for the slow script dialog increased since Thunderbird 1.5, so that may be what's going on. I do see us spending a good bit of time in some script called from nsMsgStatusFeedback::OnStateChange. At a guess, this is the OnEndMsgDownload() which calls onEndMsgDownload in msgHdrViewOverlay.js, which calls OnMsgParsed, which calls gPhishingDetector.analyzeMsgForPhishingURLs, which walks the list of all links in the message, analyzing all the URLs. This is probably the hang issue right here. I'll attach the profile, but it just shows us spending 75% of our time in JS. roc, the remaining 25% seems to be in textframe stuff. Not sure whether you're interested in the details, esp. since this is on trunk, not with all your local changes. But in case you are...
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: blocking-thunderbird3?
Attachment #267982 - Attachment is patch: false
Attachment #267982 - Attachment mime type: text/plain → application/zip
Component: Mail Window Front End → MailNews: Backend
OS: Linux → All
Product: Thunderbird → Core
QA Contact: front-end → backend
Version: unspecified → Trunk
Summary: Huge (large) mail brings thunderbird to hang → More than 7000 "http://www..." in text/plain mail brings thunderbird to hang
Bug 401090 might be a dupe, says disabling scam detection helps...
From bug 401090: - recommend changing the subject of this bug to "message with hundreds of URLs locks Thunderbird due to slow scam detection". You don't even need 7000 to hang TB, just a few hundred to get a huge performance hit. - messages on an IMAP server exhibit this problem even more - indeed, disabling TB's scam detection solve the problem
Summary: More than 7000 "http://www..." in text/plain mail brings thunderbird to hang → plaintext message with many URLs locks Thunderbird due to slow scam detection
A (simple) solution would be to create a user preference: [X] Don't scan email larger than ______ KB for scams Then add one of those yellow warning headers on messages larger than the above limit with a button linking to the UI for the preference setting (see attachment for idea).
Product: Core → MailNews Core
I don't think we'd block on this. Seems pretty unclear to me what we actually should do here. Perhaps give up scanning for scams after a few 100 links? Or what is the proposed solution? I don't think we want UI for setting how many links/KB/whatever, it should Just Work.
Flags: wanted-thunderbird3+
Flags: blocking-thunderbird3?
Flags: blocking-thunderbird3-
Priority: -- → P3
Hardware: PC → All
Target Milestone: --- → Thunderbird 3.0rc1
It seems the problem is linked to messages with too many URLs, not the actual message size, so my vote would be to not scan at all if message contains > 100 links. I also think it's important that users know the message wasn't scanned.
You could scan async so you don't lock up the UI....
Severity: major → critical
Keywords: perf
Whiteboard: [duptome]
(In reply to comment #17) > I don't think we'd block on this. > > Seems pretty unclear to me what we actually should do here. Perhaps give up > scanning for scams after a few 100 links? Or what is the proposed solution? I > don't think we want UI for setting how many links/KB/whatever, it should Just > Work. magnus, what do you propose indicating after stopping at NNN links? (resetting TM)
Priority: P3 → --
Target Milestone: Thunderbird 3.0rc1 → ---
Blocks: 1322203
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: