Closed Bug 251876 Opened 21 years ago Closed 21 years ago

Files saved by any version of Mozilla are extremely fragmented on NTFS volumes

Categories

(Core Graveyard :: File Handling, defect)

defect
Not set
minor

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: max, Unassigned)

Details

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a2) Gecko/20040714 Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a2) Gecko/20040714 No matter from which place does Mozilla save files to disk, e.g. during a download or "Save Image As", the files are saved to an NTFS volume are extremely fragmented. For those files which size is known in advance, Mozilla should reserve a continous space on the disk. This should done by a combination of Win32 API calls SetFilePointer and SetEndOfFile immediately after CreateFile. There is a code sample at http://www.mit.edu/afs/sipb.mit.edu/user/zacheiss/zip/unzip-5.50/win32/win32.c Look for the implementation of SetFileSize. An article "Disk Subsystem Performance Analysis for Windows" at http://download.microsoft.com/download/e/b/a/eba1050f-a31d-436b-9281-92cdfeae4b45/subsys_perf.doc. also described this technique. Even for the small files, SetFilePointer/SetEndOfFile are useful, because when you save a small file and didn't report the exact file size in advance, Win32 allocates extra free space after each file for probable file growth. This saving series of small files creates series of free spaces on disk, which leads to non-optimal space usage and further fragmentation. Reproducible: Always Steps to Reproduce: 1. Defragment your hard disk using built-in Windows XP disk defragmenter. 2. Download a large file (e.g. 600 MB) 3. Run Windows XP disk defragmenter in "Analyze Mode" and view the report. The downloaded file will have about three thousand fragments. Actual Results: Huge file system fragmentation on disk. Expected Results: Under Win32, reserve disk space in advance in cases where file size is known, by calling SetFilePointer/SetEndOfFile.
interesting suggestion. I think I like it. this doesn't seem very hard to do. nsIFile::SetFileSize should do it, ideally after the move of the file to the target dir since %temp% may not have enough free space
Status: UNCONFIRMED → NEW
Ever confirmed: true
OS: Windows NT → All
Hardware: PC → All
hmmmm. on the other hand, if the connection is interrupted the file should probably only have the size that it actually received
(In reply to comment #2) > hmmmm. on the other hand, if the connection is interrupted the file should > probably only have the size that it actually received Thank you for the prompt reply. If the connection is interrupted you can SetFilePointer to the size actually received and call SetEndOfFile, to truncate the file to the size originally received. The only problem may arise if the Mozilla.exe process is forcefully terminated by TerminateProcess() API call. In this case, the contents of the file between the actually received data and the end of file may be undefined. But if a file if first downloaded to a temporary name or a temporary directory and renamed after the successfully completed download, it won't cause any trouble.
Why are we working around bad implementation decisions by the filesystem authors, exactly? In any case, if this happens I feel it should happen in XPCOM (when the file stream is closed).
doing it in Close would be missing the point, since the point is this would need to happen before all the data is written to it
Oh, I see. That's pretty silly.... Given all the filesystems out there that achieve minimal fragmentation under the same conditions, I'm not really sure we want/should do anything about this.
(In reply to comment #4) > Why are we working around bad implementation decisions by the filesystem > authors, exactly? Whichever file system to be, if we suggest the final file size to the file system when creating a file, it would help the file system alot!
The file APIs in Mozilla are frozen, so we'd need to write platform-specific code in all the places that save files for each OS we want to "suggest" to (since as you pointed out the suggestions are not POSIX). That's a lot of work for very dubious benefit.
Perhaps we should do some perf testing. Like, maybe this would be a valuable optimization to use in the disk cache on Win32 builds.
We can improve Mozilla before "file system suggestions" will be defined by "POSIX" ;-) There will be a function that under Win32 will call "SetFilePointer and SetEndOfFile", and under other systems merely return. In future, when POSIX will ratify "file system suggestions", we will insert POSIX-calls to this dummy function.
The point is that the file apis in Mozilla are _frozen_. So this function would have to be added in all places that save to files, since it can't be added to the file api.
actually... I'd hate downloads to have a wrong size when mozilla crashes, making any reliable resuming impossible. so I'll mark this WONTFIX.
Status: NEW → RESOLVED
Closed: 21 years ago
Resolution: --- → WONTFIX
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.