251876 - Files saved by any version of Mozilla are extremely fragmented on NTFS volumes

Status: RESOLVED WONTFIX

Description

[Maxim Masiutin]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a2) Gecko/20040714
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a2) Gecko/20040714

No matter from which place does Mozilla save files to disk, e.g. during a
download or "Save Image As", the files are saved to an NTFS volume are extremely
fragmented. For those files which size is known in advance, Mozilla should
reserve a continous space on the disk. This should done by a combination of
Win32 API calls SetFilePointer and SetEndOfFile immediately after CreateFile.

There is a code sample at
http://www.mit.edu/afs/sipb.mit.edu/user/zacheiss/zip/unzip-5.50/win32/win32.c
Look for the implementation of SetFileSize.

An article "Disk Subsystem Performance Analysis for Windows" at
http://download.microsoft.com/download/e/b/a/eba1050f-a31d-436b-9281-92cdfeae4b45/subsys_perf.doc.
also described this technique.

Even for the small files, SetFilePointer/SetEndOfFile are useful, because when
you save a small file and didn't report the exact file size in advance, Win32
allocates extra free space after each file for probable file growth. This saving
series of small files creates series of free spaces on disk, which leads to
non-optimal space usage and further fragmentation.

Reproducible: Always
Steps to Reproduce:
1. Defragment your hard disk using built-in Windows XP disk defragmenter.
2. Download a large file (e.g. 600 MB)
3. Run Windows XP disk defragmenter in "Analyze Mode" and view the report. The
downloaded file will have about three thousand fragments.

Actual Results:  
Huge file system fragmentation on disk.

Expected Results:  
Under Win32, reserve disk space in advance in cases where file size is known, by
calling SetFilePointer/SetEndOfFile.

Comment 1

[Christian :Biesinger (don't email me, ping me on IRC)]

interesting suggestion. I think I like it. this doesn't seem very hard to do.
nsIFile::SetFileSize should do it, ideally after the move of the file to the
target dir since %temp% may not have enough free space

Comment 2

[Christian :Biesinger (don't email me, ping me on IRC)]

hmmmm. on the other hand, if the connection is interrupted the file should
probably only have the size that it actually received

Comment 3

[Maxim Masiutin]

(In reply to comment #2)
> hmmmm. on the other hand, if the connection is interrupted the file should
> probably only have the size that it actually received

Thank you for the prompt reply. If the connection is interrupted you can
SetFilePointer to the size actually received and call SetEndOfFile, to truncate
the file to the size originally received.

The only problem may arise if the Mozilla.exe process is forcefully terminated
by TerminateProcess() API call. In this case, the contents of the file between
the actually received data and the end of file may be undefined. But if a file
if first downloaded to a temporary name or a temporary directory and renamed
after the successfully completed download, it won't cause any trouble.

Comment 4

[Boris Zbarsky [:bzbarsky, bz on IRC]]

Why are we working around bad implementation decisions by the filesystem
authors, exactly?

In any case, if this happens I feel it should happen in XPCOM (when the file
stream is closed).

Comment 5

[Christian :Biesinger (don't email me, ping me on IRC)]

doing it in Close would be missing the point, since the point is this would need
to happen before all the data is written to it

Comment 6

[Boris Zbarsky [:bzbarsky, bz on IRC]]

Oh, I see.  That's pretty silly....

Given all the filesystems out there that achieve minimal fragmentation under the
same conditions, I'm not really sure we want/should do anything about this.

Comment 7

[Maxim Masiutin]

(In reply to comment #4)
> Why are we working around bad implementation decisions by the filesystem
> authors, exactly?

Whichever file system to be, if we suggest the final file size to the file
system when creating a file, it would help the file system alot!

Comment 8

[Boris Zbarsky [:bzbarsky, bz on IRC]]

The file APIs in Mozilla are frozen, so we'd need to write platform-specific
code in all the places that save files for each OS we want to "suggest" to
(since as you pointed out the suggestions are not POSIX).

That's a lot of work for very dubious benefit.

Comment 9

[Darin Fisher]

Perhaps we should do some perf testing.  Like, maybe this would be a valuable
optimization to use in the disk cache on Win32 builds.

Comment 10

[Maxim Masiutin]

We can improve Mozilla before "file system suggestions" will be defined by
"POSIX" ;-)

There will be a function that under Win32 will call "SetFilePointer and
SetEndOfFile", and under other systems merely return.

In future, when POSIX will ratify "file system suggestions", we will insert
POSIX-calls to this dummy function.

Comment 11

[Boris Zbarsky [:bzbarsky, bz on IRC]]

The point is that the file apis in Mozilla are _frozen_.  So this function would
have to be added in all places that save to files, since it can't be added to
the file api.

Comment 12

[Christian :Biesinger (don't email me, ping me on IRC)]

actually... I'd hate downloads to have a wrong size when mozilla crashes, making
any reliable resuming impossible. so I'll mark this WONTFIX.