Closed Bug 392728 Opened 18 years ago Closed 15 years ago

big performance degrade for saving attachments (when very large downloads.rdf)

Categories

(Thunderbird :: Mail Window Front End, defect)

Product:
Thunderbird
Component:
Mail Window Front End
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: hanno.falk, Unassigned)

References

Details

Attachments

(1 file)

"big" thunderbird logfile
98.32 KB, application/zip
Details
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 Build Identifier: 2.0.0.6 Time needed for saving attachments has increased dramatically. A "download manager" window appears (24% of a file saved ...). Reproducible: Always Steps to Reproduce: 1. get an email with many big attachments 2. press "save all" 3. select folder, confirm AND WAIT AND WAIT... Actual Results: When I save my attachments (many big jpegs) the current version needs at least 10 times longer than the previous while a completely unnecessary and useless download manager window appears. Expected Results: Should not take more than some seconds to save 10 images as before the update. This is really boring and annoying. First time I thought firefox hangs.
(Q1) IMAP? POP3? (Q2) How many garbages do you keep in your Temp directory? Go Command Prompt, and DIR %TEMP% (Q3) How long will next take in your environment? Go Command Prompt (where server of \\XXX doesn't exist) DIR \\XXX\YYY NET VIEW \\XXX
(A1) I am using pop3 (gmx.net). (A2) Not much, cleaning up periodically. Currently 9 Folders and about 150 files. (A3) DIR \\XXX\YYY needed 8 seconds to fail in first try, 0 seconds in retry. NET VIEW \\XXX needs 4 seconds to fail in first try, 0 seconds in retry. Your previous version saved about 1 attached jpg per second, sometimes even more. Some massage box was flickering near the bottom. Now it takes about 10 seconds for each 150kb-jpg while both CPUs run at 80%. For every single file after several seconds some download manager appears and disappears near the top left with long breaks where nothing visible happens. Got an HP Compaq nx6325 (2*2GHz, 2Gb ram, 80Gb disk, XP, DSL6000, WLAN, no problems) Thx for your attention!
Are you running a virus checker? It sounds like an other process is interfering with the saving of attachments.
Of course, I use Avira. But I have no performance problems at all - except with your new algorithm to save attachments. There seems to be a giant overhead. I can't imagine other people not having the same degradation. Other programs I use, can save dozens of small files per second or write 1Gb to disk in a few seconds (e. g. VirtualDub).
There isn't a new algorithm to save attachments.
There has never been a big square "download" window before! It always appears near the top left and looks like the "download manager" of firefox. It typically gets a bit stuck at "24%" and then completes. After several seconds it reappears with the next attachment and so on. In the previous release there were only some small flashes of activity by a little window near the lower right and everything was done. Can I, some how, install the old release - for testing? And the new release wastes so much screen area with the unnecessary icons on top of the attachments - I hate that. Sorry, I don't know how to disable those unwanted icons (see my other bug-report I posted today).
I'm not sure, but I think the download window is only appearing because the saving of attachments is taking so long... mailnews.attachments.display.largeView - try using the config editor to toggle that to false - at least it will make the icons smaller. There might be a pref to hide them completely - I don't know for sure.
(In reply to comment #2) > Now it takes about 10 seconds for each 150kb-jpg while both CPUs run at 80%. WORKSFORME (Core2 Duo T5500 / 1.66GHz 1GB, MS Win-XP SP2, Thunder bird 2.0.0.6) "Save all" for 10 jpeg attachment(200KB each) completed within 10 sec (less than 1 sec per 200KB attachment) > For every single file after several seconds some download manager appears and > disappears near the top left with long breaks where nothing visible happens. CONFIRMED (Thunderbird 2.0.0.6) To Johannes Falk(bug opener): (Q4) To where you tried to save attachments? Local hard drive? Network drive? USB drive? (Q5) If you'll try "save all" again for same mail and save target directory, dialog of "replace or not" will probably appear for each attachment. (option setting via Config Editor may be required) In this case, will saving of 150KB jpeg take 10 sec for each too? (Q6) What is file size of your downloads.rdf in profile directory?
(A4) local hard drive (A5) I tried 3 jgp-files with 100kb each, and did this several times: 1st dialog appears 0 seconds after selection of destination folder 2nd ok to overwrite appears always after 15 seconds 3rd ok to overwrite always took another 27 seconds(!) the end of the 3rd download is not clearly visible, probably about 20 sec (A6) 2373 kb
(In reply to comment #9) > (A6) 2373 kb Try "RENAME downloads.rdf downloads.rdf.BKUP" and "Restart of Thunderbird". Same result as your (A5)?
Thunderbird didn't restart after closing it's gui. I found to processes named thunderbird.exe still running, one with 0 seconds and one used 120Mb and had 12 minutes CPU-time. I killed this one and both vanished. I renamed the downloads.rdf and restarted thunderbird. *MY problem is solved*: Now it saves those three files in less than a second! (YOURS may be to find out, why there where to orphan thunderbird zombies left over) But now there is no feedback at all, it feels as if I accidently press "cancel" instead of "OK". Would be nice if it displays something in the bottom line like "attachments saved succesfully". (Sorry that I am never content, now it is the opposite bug than yesterday ;-)) Thanks a lot for your assistance! Sorry, but I can't tell whether the remaining processes where survivors from the previous version or not. The Task Manager has no column "start time" and as a unix-admin I leave my notbook up and running or in suspend for weeks without reboot (the idle-task has 60 hours by now). Thanks and regards! Hanno Falk from Flensburg/Germany
(In reply to comment #11) > *MY problem is solved*: > Now it saves those three files in less than a second! (Q7) What about "some download manager appears and disappears"? FYI. Performance problem when large download.rdf part is DUP of Bug 159107. > Bug 159107 page saving/downloads takes too much time (is slow) ('marooned' entries in downloads.rdf)
FYI I am not sure whether it was really the huge download.rdf Problem may just have been caused by the two zombies. If you want me to, I can undo the rename for testing. (A7) Nothing at all appears by now. As I told you: <ok> feels like <cancel> and it seems not to last at all. As you told me: It appears only if it takes too much time.
(In reply to comment #13) > I am not sure whether it was really the huge download.rdf > If you want me to, I can undo the rename for testing. I want you to do it. We need to know what was the problem and the cause.
the logfile that degraded time for saving my attachments dramatically
BTW it's correct name is downloadS.rdf (it took windows some minutes *not* to find it under the wrong name). This file IS the reason for the bad performance, time to save again 15+25+20 seconds after renaming the old to .rdf. And the "download window" was visible too (showing 24% in its header and 100% in the windows task bar). I had to zip it for being able sent it as attachment to you for debuging. While undoing the renaming the same thing happened as before: Exiting the front-end did *not* stop thunderbird.exe. (Again with the big downloads.rdf. At first, with the small, one it stopped immediately.) Rechecked performance - OK, thx. ------------------------------------- THAT IS REALLY SILLY, your tool said: Some one else (Johannes Falk) changed this ... (of course, because I myself sent the attachment to you, while writing the above text! (Yank, "throw away" and paste helped)). You have the following choices: * This will cause all of the above changes to be overwritten, except for the added comment(s). * Throw away my changes, and revisit bug 392728
I've understood what happened on you, after getting next test results. (1) With your 2MB downloads.rdf (Tb 2.0.0.6) 1-1) "Save As" => Download manager appeared, then disappeared 1-2) Shutdown Thunderbird 50% CPU continues. (Dual-Core case. 100% if single CPU) Not so large virtual/real memory increase was observed. No I/O count increase was observed during CPU 50%. 1-3) After several minutes(more than a few, but less than 5), process of thunderbird.exe disappeared. 1-4) Size of downloads.rdf became 1K. No entry in downloads.rdf. (2) With your 2MB downloads.rdf (Firefox 2.0.0.6 + Thunderbird 2.0.0.6) 2-1) "Save As" 2-2) Open Download Manager => slightly slow, but not so heavily slow. 2-3) Shutdown Firefox => Size of downloads.rdf is 2.x MB. 2-4) Copy downloads.rdf of Firefox to Thunderbird 2-5) Same test as (1) (Entry at 2-1) was seen when download manager panel was opened) => Same result as (1) My Conclusion: Thunderbird 2.0.0.x(at least 2.0.0.6) clears downloads.rdf during termination. (I think this is to avoid severe performance problem due to huge downloads.rdf) Then you experienced phenomenon of Bug 306190 (DUP'ed to Bug 273224). > Bug 306190 : Firefox.exe hangs with 99% CPU usage when clearing Download Manager. And, you didn't permit Thuderbird to complete termination, problem of both Bug 159107 and Bug 306190 occurred repeatedly.
One more test result: (3) With cleared downloads.rdf (Tb 2.0.0.6) 3-1) "Save As" several times => entries were added in downloads.rdf 3-2) Terminate Tb => Size of downloads.rdf became 1K. No entry in downloads.rdf.
I understand your test only partly, because I thought that I had a Thunderbird problem only. Though Firexfox was running, as far as I know, it was neither really involved into my problem nor my tests. I never copied anything from Ff to Tb or vice versa. With my big Tb/downloads.rdf Tb never truncated it any more, maybe because Tb did not exit completely - even after hours. Now, after having once renamed the downloads.rdf, it grows slowly and gets truncated to 1kb -like yours- on exit of Tb. Some hints about my unusual usage of Tb: I probably had Tb running continuously for one or two weeks when the upgrade to 2.0.0.6 happened. Around 2007/07/13 I had at least one session, where I several times received over 300 emails with 30 pictures each. (Some thousands of jpegs all together, to create some videos).
(In reply to comment #19) > Though Firexfox was running, as far as I know, it was neither really involved into my problem nor my tests. > I never copied anything from Ff to Tb or vice versa. Oh, sorry for my unclear description. My test with Firefox was: (1) Copy your 2MB downloads.rdf to Firefox's profile directory (2) Start Firefox, "Save As", open Download Manager, ... This is to see whether your downloads.rdf is corrupted or not. And to see difference of behavior of Download Manager between Fx & Tb. => Your downloads.rdf is not corrupted => Clear of downloads.rdf after termination of Tb is not caused by corrupted downloads.rdf (3) Copy Firefox's downloads.rdf to Thunderbird => Same result as test with your 2MB downloads.rdf So I checked Tb's behavior when clean downloads.rdf, then I saw that downloads.rdf was updated during active but was always cleared after termination. > With my big Tb/downloads.rdf Tb never truncated it any more, maybe because Tb > did not exit completely - even after hours. I experienced similar phenomenon once(only once) during test with your 2MB downloads.rdf. - CPU 50% continued. No I/O count increase - Virtual memory size(then real memory size) increased continuously (never stopped inreasing when I experienced) VM size easily exceeded 100MB, although max 40 MB to 50 MB when termination of Thunderbird completes within 5 minutes - I watched for 15 minutes, and when virtual memory size exceeded 200MB, I killed thunderbird.exe But I couldn't re-create this phenomenon by my simple test procedure. So I can't know whether the termination will complete or not in this case. Something wrong really exists in downloads.rdf handling by Download Manager. If you think problem of "never ending clearing of downloads.rdf" really exists, and if you think it is critical problem, open separate bug after search bugzilla.mozilla.org well, please. But attempt to change of downloads.rdf use is in progress... > Bug 161783 Need a more scaleable implementation for download data storage
Correction. Bug 306190 was DUP'ed to Bug 273244. Sorry for spam. To Johannes Falk(bug opener): Your problem is combination of 2 bugs. (1) Bug 159107 (slow "Save As" due to big downloads.rdf) (2) Bug 273244 (This bug looks to involve your "not terminate even after hours") (Bug 306190 when my test. Clear of big Tb's downloads.rdf takes very long) Your bug summary is for (1). Do you think DUPing to Bug 159107 is appropriate?
I suggest, rather than duping, that this be instead confirmed and made dependent on both bug 161783 (which bug Bug 159107 is dependent on) and bug 159107. The reason for my suggestion is thunderbird bug filers probably won't find this if the only open bug is in Firefox. I haven't read this full account so I don't know if it's applicable, but you might want to check out bug 380978
Yes, ok, thanks. To me it is a Tb-bug too, not a Ff-bug. One final comment: I have been working with Ff and Tb for many years now, but I have never really understood, what the download manager and it's history may be useful for. Especially as a beginner it's surprising "remove" and "cleanup" buttons looked very suspicious to me. By now I know it simply deletes the entry in the list, not my downloaded file, as I initially expected. The only nice thing I think may be it's "execute" or "open" button. But if you just delete or disable it, I wouldn't miss it.
Status: UNCONFIRMED → NEW
Depends on: 159107, 161783, 273244
Ever confirmed: true
Summary: big performance degrade for saving attachments → big performance degrade for saving attachments (when very large downloads.rdf)
Wada, at what size roughly does download.rdf become a performance problem?
Probably worth noting that the solution to this is to migrate from an RDF-backed system to an SQLite-backed one, which is already done on the trunk, so this bug is 2.0.0.x-only, and almost certain to not have a fix safe enough to land on 2.0's branch at this point.
Version: unspecified → 2.0
(In reply to comment #24) > at what size roughly does downloads.rdf become a performance problem? Sorry but I don't know. However, I can say that "deletion of downloads.rdf when any size" will never produce any problem when thunderbird, because Thunderbird(at least 2.0.0.6) always clears downloads.rdf during termination.
Isn't this FIXED in Thunderbird 3.0 at least?
should be WFM per comment 25
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: