231923 - Close filterlog.html and junklog.html after use (those files corrupted after crash)

Status: NEW

(MailNews Core :: Filters, defect)

Description

[:aceman]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.6b) Gecko/20031216 Firebird/0.7+
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.6b) Gecko/20031216 Firebird/0.7+

The file mentioned in the summary is open for a very long time while Mozilla is
running, even though it may not be used often. If Mozilla or the OS crashes (or
the computer locks up even on a ultra stable OS), this file may get corrupted.
When this happens, data in this file is lost (partly because Mozilla recreates
corrupt files from scratch). And users may not able to access their profile
(data) anymore.

The point of this bug is a request to close the file after each usage. Ideally
the file should be closed immediatelly, when the operation is done. For
performance, it would be better to close it only after several seconds, but that
may be difficult to code. Maybe it can be left on the OS cache to handle the
frequent opening and closing of the file.

We can try it now in the 1.7 alpha stage.

See bug 231606 for further details on the general problem.

I don't know of any case of corruption of these 2 files. However, they
are used only when new mail arrives or the "show log dialog" is open. They are
not used most of the time, so we should close them to free some resources (file
handles).

Reproducible: Always

Steps to Reproduce:

Comment 1

[Michiel van Leeuwen (email: mvl+moz@)]

*** Bug 231922 has been marked as a duplicate of this bug. ***

Comment 2

[:aceman]

I take back my last paragraph. Recently, I got hit by this. My PC locked up. On
boot, scandisk repaired the junklog.html file by extending it a bit. Binary
garbage was added to the file. Successive log entries were added after this crap.
I didn't try to view the log in it's dialog window and it might not cause any
problems there. But it would be ugly.

Comment 3

[Daniel Kies]

Recently, I have been hit by this bug too during the last few weeks, running
WinXP SP1 with successive trunk nightlies each day.  The bug started for me
around 20040609, but I can't verify that.  Mozilla would hang (meaning, Moz
would take 10-20 minutes to "process" new mail after an initial start of the
program and downloading new messages off the server).  Mozilla would also hang
(again, taking 10-20 minutes to close completely and release the parent.lock
file) after closing the program.  

During the "hangs," Mozilla would eat 90-99% of system resources and make the
whole system sluggish or unresponsive. Simply waiting until Moz or the OS was
finished with the process would return the system's responsiveness.

Just today I discovered that renaming junklog.html to junklog.old solved my
problem.  Moz and/or the OS remains responsive once again after downloading mail
and after closing the program.  I do not have filter logging enabled, so my
filterlog.html file is 0kb.  My (old) junklog file is 11,791kb.

I would be happy to zip my old (corrupted?) junklog and mail it to anyone if it
could help resolve this issue for others.

Comment 4

[:aceman]

This is interesting, but not subject to this bug, if the file was not damaged
due to a OS/Mozilla crash. 11MB is too much to send to me. Can you look into the
file and confirm there is some corruption? Binary garbage not resembling real logs?

a) if not, try to put the file back and check whether the long hang comes back.
If yes, you can file a bug about this saying that just the size of the log file
causes drastic slowdown. Because appending to the log should be an instant
operation regardless of the file size.

b) if yes, could you try to remove the garbage in a plain text editor and put
the file in a consistent state? Then put it back into the mail account and
continue at point a):) And of course, tell us how much garbage (in MB) there was.

Only if there is garbage caused by a premature Mozilla stop AND only this
garbage causes the long hangs, we might have a case for this bug.

Comment 5

[Daniel Kies]

Re: comment #4.  I scrolled through the junklog.html file looking for binary
garbage that didn't belong.  However, I found nothing unusual.

I did discover though that the real source of my problem with that file was not
related to Mozilla at all.  It was my anti-virus program, scanning the large
HTML file, that caused the system to "hang" for 10-20 minutes each time the file
was accessed.  I discovered this by simply trying to open the junklog with a
text editor.  Once again the system became unresponsive, and Mozilla was not
running at all.  Writing an exclusion rule for the anti-virus program solved the
problem for all applications.

Apologies for the spam, and sorry that I could not produce a test case for the bug.

Comment 6

[Gervase Markham [:gerv]]

This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

Comment 7

[Gervase Markham [:gerv]]

This bug has been automatically resolved after a period of inactivity (see above
comment). If anyone thinks this is incorrect, they should feel free to reopen it.

Comment 8

[:aceman]

Still there in Seamonkey 1.0.2.

Comment 9

[Serge Gautherie (:sgautherie)]

Could be related/duplicate to bug 361865 ?

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As I already described in bug 361865 Thunderbird also blocks itself when you are trying to empty the junklog.html file from within the Junk Mail Log dialog. Following js error appears:

Error: uncaught exception: [Exception... "Component returned failure code:
0x80520015 (NS_ERROR_FILE_ACCESS_DENIED) [nsIFile.remove]"  nsresult:
"0x80520015 (NS_ERROR_FILE_ACCESS_DENIED)"  location: "JS frame ::
chrome://messenger/content/preferences/junkLog.js :: clearLog :: line 29" 
data: no]

Comment 12

[timeless]

most likely the file is open. you should be able to use process explorer (procexp) from www.sysinternals.com to verify that thunderbird has a handle to it.

Comment 13

[(not reading, please use [e-mail] instead)]

sorry for the spam.  making bugzilla reflect reality as I'm not working on these bugs.  filter on FOOBARCHEESE to remove these in bulk.

Comment 14

[Wayne Mery (:wsmwk)]

[notacrash]

Comment 15

[:aceman]

Ironically, after almost 9 years I can maybe fix this :)

Rkent, does this belong into filters? If yes, can you decide what to do here? I proposed to close the file after a timeout but that may be hard. If it is OK to close the log file after each output message then I could probably code that.

Comment 16

[Wayne Mery (:wsmwk)]

(In reply to :aceman from comment #15)
> Ironically, after almost 9 years I can maybe fix this :)
> 
> Rkent, does this belong into filters? If yes, can you decide what to do
> here? I proposed to close the file after a timeout but that may be hard. If
> it is OK to close the log file after each output message then I could
> probably code that.

Comment 17

[Kent James (:rkent)]

In general, I don't think that it is a problem to open and close a file each time that a message is processed for an optional feature like a log. But once per batch would be better. Opening and closing multiple times per message is probably excessive.

I traced through the issues far enough with the junk log file to determine that it is being called in code that has message batch management (::OnMessageClassified), so what you should really do is to add a call to spamSettings to flush and close the junk log file (with lazy re-opening) so that you can close one per batch, and not once per message. I think something similar is also possible with the filter log.

So try to close once per batch if possible.

Comment 18

[Wayne Mery (:wsmwk)]

Note also, disabling filter logging does not cause file to close. The file currently is never closed until thunderbird shuts down.