Closed Bug 593349 Opened 9 years ago Closed 9 years ago

meta: Startup cache writing algorithm causes severe fragmentation

Categories

(Core :: XPCOM, defect)

All
Windows 7
defect
Not set

Tracking

()

RESOLVED FIXED
Tracking Status
blocking2.0 --- final+

People

(Reporter: khronos, Assigned: taras.mozilla)

References

Details

(Keywords: perf, regression, Whiteboard: [softblocker])

Attachments

(1 file, 3 obsolete files)

User-Agent:       Mozilla/5.0 (Windows NT 6.1; rv:2.0b6pre) Gecko/20100903 Firefox/4.0b6pre
Build Identifier: Mozilla/5.0 (Windows NT 6.1; rv:2.0b6pre) Gecko/20100903 Firefox/4.0b6pre

The way startup cache file gets written to disk causes severe fragmentation, that gets worse and worse over time.

Reproducible: Always

Steps to Reproduce:
1. delete startup cache (profile\startupCache\startupCache.4.little
2. restart Firefox
3. run analyze in some defragmentation program that shows file fragmentation
Actual Results:  
Startup cache is severely fragmented, always 128 fragments when freshly created, and gets worse over time - 238 fragments (and that's for a 2,6MB file only) after 8 hours of usage and several Firefox restarts right now.

Expected Results:  
The writing algo should be optimized to get at least some reasonable levels of fragmentation.

System: Windows 7. Profile folder on a local, NTFS formated, fully defragmented, non-system disk with 400GB of contiguous free space. No system restore, shadow copies or any other unmovable files that could cause higher fragmentation.
Thanks for the excellent report.
Depends on: 594172
Assignee: nobody → bhsieh
Note (mostly) to self: Taras's patch in bug 594172 could help with this.
However, the main thing is to have a better timer for the initial write. Either extend the length of the timer, or (better) reset the timer when the user does some action.
This is a major issue on my system. I can run Firefox, close Firefox, open Diskeeper, see the startup cache is in 2000 or 3000 fragments. So I defrag. Reboot, run Firefox, close Firefox, open Diskeeper and guess what, it's back to being fragmented. Not as bad, but even so, it shouldn't be fragmented at all. Win7 Pro x64.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Hardware: x86 → All
Version: unspecified → Trunk
(In reply to comment #4)
> This is a major issue on my system. I can run Firefox, close Firefox, open
> Diskeeper, see the startup cache is in 2000 or 3000 fragments. So I defrag.
> Reboot, run Firefox, close Firefox, open Diskeeper and guess what, it's back to
> being fragmented. Not as bad, but even so, it shouldn't be fragmented at all.
> Win7 Pro x64.

what about other files in the profile directory? How much free space do you have on the drive with the startupcache?
Fragmentation on the other files in the profile directory are about normal when compared with other directories on the drive. I do defrag daily. As far as space, this is a RAID10 (4x250GB drives), 500GB total, 216GB free.
(In reply to comment #6)
> Fragmentation on the other files in the profile directory are about normal when
> compared with other directories on the drive. I do defrag daily. As far as
> space, this is a RAID10 (4x250GB drives), 500GB total, 216GB free.

You can try a test build from http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/tglek@mozilla.com-a754436412b3/tryserver-win32/firefox-4.0b5pre.en-US.win32.zip

To see if it reduces your fragmentation. Delete your startup cache first.
Taras did some jar work to reduce fragmentation, next step is probably just to bump the timer timeout to something like 10 minutes.
Attached patch extends buffer timer (obsolete) — Splinter Review
Currently, the write algorithm buffers all the writes that happen during startup and writes them all out at once. This patch just extends that window to catch more writes. It does seem to help a bit, especially for the first-run fragmentation.

I did not anticipate that each write with the zipWriter API would cause so much fragmentation, though, so it's worth thinking about whether we can get the zipwriter to allocate some fixed amount of space or other strategies to reduce this fragmentation.
It's a lot better with all the changes. Right now I get about 40 fragments when creating a new cache and it doesn't climb that much during usage.
Let's do this first, since the fragmentation of the first-run write will affect perf for all other runs as well. Long-term, we should occasionally / periodically defragment, perhaps by copying to a new file.
Attachment #479951 - Attachment is obsolete: true
Attachment #480204 - Flags: review?(dwitte)
Comment on attachment 480204 [details] [diff] [review]
extends buffer timer, plus some testing convenience

>diff --git a/startupcache/StartupCache.cpp b/startupcache/StartupCache.cpp

>   // Wait for 10 seconds, then write out the cache.
>-  rv = mTimer->InitWithFuncCallback(StartupCache::WriteTimeout, this, 10000,
>+  rv = mTimer->InitWithFuncCallback(StartupCache::WriteTimeout, this, 600000,
>                                     nsITimer::TYPE_ONE_SHOT);

Stale comment.

r=dwitte
Attachment #480204 - Flags: review?(dwitte) → review+
blocking2.0: --- → ?
Not sure how to mark this, but will also help Bug 601186 which is a Fennec blocker.
Attachment #480204 - Flags: approval2.0?
Ready to go.
Attachment #480204 - Attachment is obsolete: true
Attachment #481913 - Attachment is obsolete: true
Attachment #480204 - Flags: approval2.0?
I think Taras had some ideas on when we can copy the startupcache file to reduce fragmentation further.
Assignee: bhsieh → tglek
Comment on attachment 481916 [details] [diff] [review]
extends buffer timer, plus some testing convenience

>   // Wait for 10 seconds, then write out the cache.
>-  rv = mTimer->InitWithFuncCallback(StartupCache::WriteTimeout, this, 10000,
>+  rv = mTimer->InitWithFuncCallback(StartupCache::WriteTimeout, this, 600000,
>                                     nsITimer::TYPE_ONE_SHOT);

You might want to also change the comment to match the result: 10 seconds should be 10 minutes now.
the zipwriter fix reduced fragmentation by 10x, bunching up writes better in cache-on-thread bug should correct the rest.
Depends on: 586859
Summary: Startup cache writing algorithm causes severe fragmentation → meta: Startup cache writing algorithm causes severe fragmentation
Duplicate of this bug: 616441
Keywords: perf, regression
Whiteboard: [softblocker]
Landed part of 586859 which takes care of abnormally high fragmentation.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.