Files saved by any version of Mozilla are extremely fragmented on NTFS volumes

RESOLVED WONTFIX

Status

Core Graveyard
File Handling
--
minor
RESOLVED WONTFIX
14 years ago
a year ago

People

(Reporter: Maxim Masiutin, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

14 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a2) Gecko/20040714
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a2) Gecko/20040714

No matter from which place does Mozilla save files to disk, e.g. during a
download or "Save Image As", the files are saved to an NTFS volume are extremely
fragmented. For those files which size is known in advance, Mozilla should
reserve a continous space on the disk. This should done by a combination of
Win32 API calls SetFilePointer and SetEndOfFile immediately after CreateFile.

There is a code sample at
http://www.mit.edu/afs/sipb.mit.edu/user/zacheiss/zip/unzip-5.50/win32/win32.c
Look for the implementation of SetFileSize.

An article "Disk Subsystem Performance Analysis for Windows" at
http://download.microsoft.com/download/e/b/a/eba1050f-a31d-436b-9281-92cdfeae4b45/subsys_perf.doc.
also described this technique.

Even for the small files, SetFilePointer/SetEndOfFile are useful, because when
you save a small file and didn't report the exact file size in advance, Win32
allocates extra free space after each file for probable file growth. This saving
series of small files creates series of free spaces on disk, which leads to
non-optimal space usage and further fragmentation.

Reproducible: Always
Steps to Reproduce:
1. Defragment your hard disk using built-in Windows XP disk defragmenter.
2. Download a large file (e.g. 600 MB)
3. Run Windows XP disk defragmenter in "Analyze Mode" and view the report. The
downloaded file will have about three thousand fragments.

Actual Results:  
Huge file system fragmentation on disk.

Expected Results:  
Under Win32, reserve disk space in advance in cases where file size is known, by
calling SetFilePointer/SetEndOfFile.
interesting suggestion. I think I like it. this doesn't seem very hard to do.
nsIFile::SetFileSize should do it, ideally after the move of the file to the
target dir since %temp% may not have enough free space
Status: UNCONFIRMED → NEW
Ever confirmed: true
OS: Windows NT → All
Hardware: PC → All
hmmmm. on the other hand, if the connection is interrupted the file should
probably only have the size that it actually received
(Reporter)

Comment 3

14 years ago
(In reply to comment #2)
> hmmmm. on the other hand, if the connection is interrupted the file should
> probably only have the size that it actually received

Thank you for the prompt reply. If the connection is interrupted you can
SetFilePointer to the size actually received and call SetEndOfFile, to truncate
the file to the size originally received.

The only problem may arise if the Mozilla.exe process is forcefully terminated
by TerminateProcess() API call. In this case, the contents of the file between
the actually received data and the end of file may be undefined. But if a file
if first downloaded to a temporary name or a temporary directory and renamed
after the successfully completed download, it won't cause any trouble.
Why are we working around bad implementation decisions by the filesystem
authors, exactly?

In any case, if this happens I feel it should happen in XPCOM (when the file
stream is closed).
doing it in Close would be missing the point, since the point is this would need
to happen before all the data is written to it
Oh, I see.  That's pretty silly....

Given all the filesystems out there that achieve minimal fragmentation under the
same conditions, I'm not really sure we want/should do anything about this.
(Reporter)

Comment 7

14 years ago
(In reply to comment #4)
> Why are we working around bad implementation decisions by the filesystem
> authors, exactly?

Whichever file system to be, if we suggest the final file size to the file
system when creating a file, it would help the file system alot!
The file APIs in Mozilla are frozen, so we'd need to write platform-specific
code in all the places that save files for each OS we want to "suggest" to
(since as you pointed out the suggestions are not POSIX).

That's a lot of work for very dubious benefit.

Comment 9

14 years ago
Perhaps we should do some perf testing.  Like, maybe this would be a valuable
optimization to use in the disk cache on Win32 builds.
(Reporter)

Comment 10

13 years ago
We can improve Mozilla before "file system suggestions" will be defined by
"POSIX" ;-)

There will be a function that under Win32 will call "SetFilePointer and
SetEndOfFile", and under other systems merely return.

In future, when POSIX will ratify "file system suggestions", we will insert
POSIX-calls to this dummy function.
The point is that the file apis in Mozilla are _frozen_.  So this function would
have to be added in all places that save to files, since it can't be added to
the file api.
actually... I'd hate downloads to have a wrong size when mozilla crashes, making
any reliable resuming impossible. so I'll mark this WONTFIX.
Status: NEW → RESOLVED
Last Resolved: 13 years ago
Resolution: --- → WONTFIX
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.