Set temp folder to provide space for compaction

RESOLVED WONTFIX

Status

MailNews Core
Database
--
enhancement
RESOLVED WONTFIX
15 years ago
6 years ago

People

(Reporter: Jon Roland, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

15 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.3b) Gecko/20030221
Build Identifier: Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.3b) Gecko/20030221

Using filter to archive messages to removable drive. Messages do get copied to
target file, and removed from the message list of the Inbox folder, but the file
size of Inbox does not get reduced, and am running out of space on the drive.
Yes, I can set preferences to compact folders, but messages moved out of Inbox
using a filter should be physically removed from the Inbox file to make more
space available.

I started with 32000 messages and an Inbox file size of about 700 MB. First used
the filter to move about 16000 messages to folder in Local Files, which is on a
removable drive. Next used it to move another 7000 messages to another Local
Files folder, leaving 9000 messages in message list, but Inbox file now at 800
MB, only 26 MB free space left on drive, and getting messages that I am running
out of space on that drive. How do I get that Inbox file size reduced so that it
contains only the 9000 messages in the message list? This subject does not seem
to be covered in the help files, and an entry on it should be made.

Reproducible: Always

Steps to Reproduce:
1.
2.
3.
(Reporter)

Comment 1

15 years ago
Just got an error message when tried to compact Inbox:

The folder 'Inbox" could not be compacted because writing to folder failed.
Verify that you have enough disk space, and that you have write privileges to
the file system, then try again.

So can't compact, can't delete messages, and can't free up any more disk space
on that drive. What are the options remaining? 
(Reporter)

Comment 2

15 years ago
After further investigation I have discovered that what is happening is that
messages are not being physically removed from folder files. It doesn't matter
how one tries to remove them. Moving messages to another folder, or deleting
them from the Mozilla interface, removes them from the message list, but not
from the physical file, which just keeps growing, and growing, and growing,
until disk space is exhausted.

I have found a workaround: Move all the messages in a folder to a folder on an
external disk, then delete (or otherwise remove) the original folder file,
together with its msf file, then move all the messages back again.

Before I did this, the Inbox file on my most active account (300-400 messages a
day) had grown to 820,438 KB, and I had only 20 MB remaining. After "moving" the
messages to a folder in the Local Files account, which is physically located on
an external removable drive (a 120 GB Western Digital 1394 HD), removing the
original Inbox file, and then "moving" the messages back to the Inbox folder,
its physical file size was only 222,342 KB. Fortunately, no messages appear to
have been lost during these transfers (but I saved the files just in case).

Needless to say, if one has to do this kind of thing on every folder of every
account to free up disk space when it becomes scarce, then it is a critical bug,
and I advocate upgrading it to that level.

Compacting is all very well and good, but compacting doesn't work if disk space
is insufficient, and what is really needed is to physically purge messages
folders of removed messages.
Severity: normal → critical
(Reporter)

Comment 3

15 years ago
One more thing that should be added. It doesn't work to try to move too many
messages at once. On my system (128 MB RAM) I can't move more than about 3000
messages at a time. For those who may have similar limits, and if you want to
preserve the order in which messages are received (and many of us do, especially
if our senders' system clocks are not set accurately), you will need to first
order the messages by order received. Then estimate how far along the scroll bar
will give you about, say, 2000 messages, select them, and move them, starting
with the oldest messages remaining. If disk space is limited, the transfer
folder will need to be on another drive with plenty of space. Each move will
take a long, long time, so be patient. Good time to take a break. After all the
messages are moved, remove both the messages and msf files, before moving the
messages back. They first batch you move will also trigger the creation of a new
summary (msf) file. Set the view order for all folders to view by order received
for convenience and to make sure the messages are in the same order when this is
completed.
(Reporter)

Updated

15 years ago
Summary: Filter does not reduce size of Inbox file → Move does not reduce size of folder file
(Reporter)

Comment 4

15 years ago
I am changing the summary line to more accurately reflect what I am requesting.
The basic problem is that compaction fails when there is not enough free space
available on the drive where the message folders are located. The obvious
solution is to allow the user to define a temp folder on another drive to
provide enough space, perhaps by copying the unremoved messages to a folder
there, then deleting the original folder file, and copying the messages back
from the temp directory.

I also propose the option of immediate removal of messages from the folder file,
a kind of compaction of just the one removed message, rather than searching for
and removing all of them in a single batch operation.

The help function should also advise the user to set the trigger for initiating
compaction to be something like 10,000 KB so the prompt to compact messages
won't come too often, interrupting normal usage.
Summary: Move does not reduce size of folder file → Set temp folder to provide space for compaction
(Reporter)

Comment 5

15 years ago
Any further action on this? It is still a priority for me, as it is a constant
chore.
Product: MailNews → Core

Comment 6

13 years ago
(In reply to comment #5)
> Any further action on this? It is still a priority for me, as it is a constant
> chore.

I can't believe this is still classed as Unconfirmed.  I can confirm it is a
problem and it has totally stuffed my mail.
I foolishly tried to move my (large) messages around to tidy up my inbox, I then
found that I was running very low on disk space.  I tried to compact the
messages but to no avail (as mentioned above).  I then deleted a lot of the
messages only to find that no disk space was freed up at all.  It seems that the
only file which genuinely deletes messages is the Trash file.
I now have no disk space and none of my other drives are large enough to use the
workaround listed above.
I for one would be very grateful if this bug could be fixed soon, otherwise I
will have to use another mail client, which would be a shame as I've been very
impressed with everything else about Thunderbird.
This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

Comment 8

13 years ago
This bug should definitely be kept open and fixed. The fix should be easy, and
the lack of comments doesn't mean it isn't important to people, or won't become
important as the product is used in more commercial environments that get a lot
of email.

It is a good design practice to always offer the user a choice of setting temp
folders for anything. No application should ever assume sufficient disk space on
a given drive.
Is this still a problem? This bug is listed as pertaining to Windows 98, which is no longer supported by Microsoft.

If this bug cannot be reproduced, feel free to close the bug as WORKSFORME.
(Reporter)

Comment 10

12 years ago
The last post fails completely to understand the problem, which is that the user can't select a temp folder on a filesystem with enough space, having an unchangeable default to a filesystem which may not have enough space. It has nothing whatsover to do with the operating system. Same problem exists for all operating systems. 

What might additionally be needed would be a way to do the processing when there isn't enough space on any filesystem, by partitioning the mailboxes and doing them in phases, making as much use of available space as possible without overflowing.
<- OS to All based on comment 10.
OS: Windows 98 → All

Comment 12

11 years ago
Jon, please list a short set of steps to reproduce
 1. blah
 2. blah
 3. ...
actual results: blah
expected results: blah

bear in mind, mailnews is not likely to gain the smarts of a filesystem or get predictive, which may have some bearing on why this bug has not been confirmed.

Comment 13

11 years ago
(In reply to comment #4)
> I am changing the summary line to more accurately reflect what I am requesting.
> The basic problem is that compaction fails when there is not enough free space
> available on the drive where the message folders are located. The obvious
> solution is to allow the user to define a temp folder on another drive to
> provide enough space, perhaps by copying the unremoved messages to a folder
> there, then deleting the original folder file, and copying the messages back
> from the temp directory.

Changing severity to enhancement for this part of the bug. 


> I also propose the option of immediate removal of messages from the folder file,
> a kind of compaction of just the one removed message, rather than searching for
> and removing all of them in a single batch operation.
> 
> The help function should also advise the user to set the trigger for initiating
> compaction to be something like 10,000 KB so the prompt to compact messages
> won't come too often, interrupting normal usage.

you will want to address these other issues in separate bugs.
Assignee: naving → nobody
Severity: critical → enhancement
Component: MailNews: Filters → General
QA Contact: laurel → general
Hardware: PC → All

Comment 14

8 years ago
I have had similar problems with running out of space to compact. Having the 'auto-compact' option on has helped somewhat, but if I forget to delete a bunch of large attachments then I am completely hooped and have to go through all the shenanigans that Jon Roland has detailed.

I think the temp folder idea is a good one and probably dead simple to implement.

Another suggestion is this: I have noticed that the compaction routine fails as soon as it can't compact a particularly large folder as there is not sufficient space on the drive for the compaction routine to run. However, there are often other folders that are smaller that could be compacted - leaving more space to compact the large ones. I know this because I quite often manually have to do this to these smaller folders before going back to my bigger folders. My suggestion is that the compaction routine not instantly fail with the message:

 "The folder 'Blah' could not be compacted because writing to folder failed. Verify that you have enough disk space, and that you have write privileges to
the file system, then try again."

But instead keep on trying to compact other folders. It should only return this error message when it has tried to compact every single folder, but there is not sufficient space to compact any of them. And in that case, a temp folder would be required... However, for my usage, a smarter compaction algorithm would likely solve 99% of my space issues.

Comment 15

8 years ago
While probably helpful for some people, are there other advantages (performance, etc) to allowing for a temp space that is not within the user's profile or in the standard windows defined temp space?  

If not, this is just a workaround for a combination of 
a) primarily insufficient disk space resources caused by the user, and 
b) secondarily not great compaction smarts or improvements that could be made on the part of Thunderbird in several areas, and 
c) in some cases unwillingness on the part of the user to delete messages that are no longer needed, or lack of knowledge that not deleting messages has significant impact on disk space and performance

compact bugs which mention space - https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc=Compact&longdesc=space&short_desc_type=allwordssubstr&type0-0-0=nowords&value0-0-0=count%20counts&resolution=---&product=MailNews%20Core&product=Thunderbird&longdesc_type=allwordssubstr

(moving back to mailnews land)
Component: General → Database
Product: Core → MailNews Core
QA Contact: general → database

Updated

7 years ago
Duplicate of this bug: 413955

Comment 17

7 years ago
hmm, no response to comment 15.

bienvenu, I can't see this ever being worth the code and testing cost. Plus we aren't in the business of making thunderbird run for users who can't maintain drive health. 

FWIW, the dead easy solutions for a user are: a) move the profile do a different drive b) free enough space space on the current drive
Whiteboard: [wontfix?]

Comment 18

7 years ago
yeah, I'd agree with wontfix. diskspace is cheap and very few users would ever be able to take advantage of this.

Updated

7 years ago
Status: UNCONFIRMED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → WONTFIX
Whiteboard: [wontfix?]

Comment 19

6 years ago
I'd have to disagree with your assumption(s).

I've been using Thunderbird for many versions from the begining after I switched from Netscape > Mozilla > Firefox + Thunderbird and I seldom had any problems that couldn't be rectified.

However, I've been using Thunderbird for U3 for a few years now and I have been having the same problems as described above. As one might guess my thumbdrives are mainly for use outside the office and have files on it that are needed in those situations and I find it inconvenient to have to swap off a bunch of files to get my e-mail client to function properly. My SanDisk Gator 8GB [7.47GB] shows me how much free space I have on it. I have had between 650MB and 1GB+ of free space and run into the dreaded message that it cannot compact folder because there is either not enough free space or I don't have administrator privileges. I do compact my folders but about a 1/3 of the time I run into this issue. And it usually seems to fail when ~80-90% finished if the progress bar is to be believed to be accurate. Why not compact the folders individually? Or default to finishing compacting the folders it has found sufficient space for and then continuing on with the remaining when more space has been opened up? How about splitting that error box into two separate errors and telling the end user when they fail to complete compacting it was because a folder of X size was encountered, please free up at least X amount of room and retry. At least then the user has some idea of what to look to to resolve the issue.

For most of the suggestions listed above I can't even access the folders to copy the files out to another location. I also can't export the profile which I could then conceivably import it on another machine such as the laptop or desktop with plenty of room to allow compaction to complete and then re-export and re-import it back to my U3 drive.

I've run into this issue on my Windows XP Pro SP3, Vista Home Premium and Windows 7 Home Premium OSs on both laptops and desktops.
You need to log in before you can comment on or make changes to this bug.