479285 - compact local folder fails because we couldn't remove the .msf file of original folder after compact finished. ForceDBClosed not successfully closing the db?

Status: RESOLVED WORKSFORME

(MailNews Core :: Database, defect)

Description

[David :Bienvenu]

I'm seeing compaction of some local folders failing because we couldn't remove the .msf file of the original folder after the compaction process is finished. To me, this sounds like ForceDBClosed is not successfully closing the db in some situations. It's possible that the underlying cause is the same as the assertion we see on shutdown about db's left open.

This results in nstmp and nstmp-N folders getting left in the folder pane. I'll investigate.  This is a pretty visible issue that I think is worth fixing for b2.

Comment 1

[David :Bienvenu]

I haven't been able to reproduce this again...I'll keep trying.

Comment 2

[Robert Roessler]

Have you examined bug 442588?

This plagued me for over 6 months, and then mysteriously disappeared with the last 2.0a3pre build on the 18th...

Comment 3

[Kent James (:rkent)]

It would be good to have a more robust way to ForceDBClosed than the current method that only works if we successfully remove all references.

Comment 4

[David :Bienvenu]

ForceDBClosed could close the underlying mdb to release the .msf file - the danger is that anyone holding onto the db or msg hdrs could crash by accessing freed memory. Maybe there would be a way to get Mork to release the file handle without freeing up all the memory - I'll look into that.

Comment 5

[Saul Tannenbaum]

The "failure to compact" is not limited to local folders, I've seen it on server
side folders as well.

See: https://bugzilla.mozilla.org/show_bug.cgi?id=467305

This can be difficult to reproduce, but from protocol logs, it's clear that Thunderbird is issuing EXPUNGEs, but whatever's cached client-side isn't getting updated. Forcing an index rebuild fixes the problem.

As a user, I believe this has to be considered a blocking bug. As someone involved in my organization's email infrastructure, I know that the support call volume this bug will generate are going to be unacceptable.

Comment 6

[David :Bienvenu]

I haven't seen this again (this bug was about local folder compaction failing, not imap).

Comment 7

[David :Bienvenu]

I'm going to minus this unless we can figure out a way to reproduce this.

Comment 8

[Wayne Mery (:wsmwk)]

David, is the regression range close to the date of bug report?


Bienvenu in comment #4
> ForceDBClosed could close the underlying mdb to release the .msf file - the
> danger is that anyone holding onto the db or msg hdrs could crash by accessing
> freed memory. Maybe there would be a way to get Mork to release the file handle
> without freeing up all the memory - I'll look into that.

does this deserve a separate bug?

Comment 9

[David :Bienvenu]

I'm not sure of the regression range since I don't compact often - it might have happened earlier as well.

Comment 10

[David :Bienvenu]

reworking the file handling of mork in that way might be a bit scary, but we could file a bug on it, I guess.

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> This results in nstmp and nstmp-N folders getting left in the folder pane.

I couldn't reproduce "left nstmp and nstmp-N" for local mail folder by "outdated msf". (I probably saw it when simultaneous compact & rebuild-index test, but I'm not sure.) Instead, I saw Bug 492344 by test with intentional "outdated msf" condition.
David Bienvenu, Bug 492344 is relevant to this bug?

Comment 12

[David :Bienvenu]

some part of it may be, though it's hard to tell.

Comment 13

[WADA:World Anti-bad-Duping Agency]

External software(auto-backup, file scan by anti-virus software) can interfere "compact folder" by Tb.
Putting in dependency tree for meta Bug 498274, for ease of tracking.

Comment 14

[Wayne Mery (:wsmwk)]

bienvenu, still seeing this 2 years later?  :)

(In reply to comment #0)
> I'm seeing compaction of some local folders failing because we couldn't remove
> the .msf file of the original folder after the compaction process is finished.
> To me, this sounds like ForceDBClosed is not successfully closing the db in
> some situations. It's possible that the underlying cause is the same as the
> assertion we see on shutdown about db's left open.
> 
> This results in nstmp and nstmp-N folders getting left in the folder pane. I'll
> investigate.  This is a pretty visible issue that I think is worth fixing for
> b2.

Comment 15

[David :Bienvenu]

I have not seen this in a long time.

Comment 16

[Wayne Mery (:wsmwk)]

(In reply to David :Bienvenu from comment #15)
> I have not seen this in a long time.

=> WFM then. one more nail