Last Comment Bug 627591 - preload dlls on windows
: preload dlls on windows
Status: RESOLVED FIXED
[ts]
: perf
Product: Core
Classification: Components
Component: XPCOM (show other bugs)
: unspecified
: x86 All
: -- normal with 46 votes (vote)
: ---
Assigned To: (dormant account)
:
: Nathan Froyd [:froydnj]
Mentors:
Depends on: 481815 554421 632404
Blocks:
  Show dependency treegraph
 
Reported: 2011-01-20 17:32 PST by (dormant account)
Modified: 2013-12-27 14:30 PST (History)
133 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
-


Attachments
wip (967 bytes, patch)
2011-01-20 17:32 PST, (dormant account)
shaver: review-
Details | Diff | Splinter Review
new patch (2.73 KB, patch)
2011-01-26 15:25 PST, (dormant account)
no flags Details | Diff | Splinter Review
ready to go (2.34 KB, patch)
2011-01-26 16:36 PST, (dormant account)
shaver: review-
Details | Diff | Splinter Review
wrapper.exe source (3.32 KB, text/plain)
2011-02-09 16:15 PST, (dormant account)
no flags Details
detect when prefetch is off(for whatever reason) and preload our way (3.35 KB, patch)
2011-05-04 12:17 PDT, (dormant account)
no flags Details | Diff | Splinter Review
detect when prefetch is off(for whatever reason) and preload our way (3.25 KB, patch)
2011-05-04 12:35 PDT, (dormant account)
no flags Details | Diff | Splinter Review
style fixes (3.21 KB, patch)
2011-05-04 14:52 PDT, (dormant account)
shaver: review+
Details | Diff | Splinter Review
telemetry probes (2.64 KB, patch)
2011-05-06 12:09 PDT, (dormant account)
no flags Details | Diff | Splinter Review

Description (dormant account) 2011-01-20 17:32:23 PST
Created attachment 505620 [details] [diff] [review]
wip

We page in most of xul.dll and mozjs.dll. So i figured why not try lazy-linking libxul and preloading it. Turned out lazylinking didnt make any difference, but preloading seems to consistently shave up 2seconds of startup on my profile(40%). 
I'm going to doublecheck on my favourite user's slow computer. Here is the patch in meantime.

Basic idea is that the sequential flag + bullshit read tricks windows into reading xul in 2mb chunks instead of stupid 32k(or smaller) ones. Have to do it this way because there is no fadvise() on Windows(that I know of)
A big sequential read cuts down on a lot of seeks.
Comment 1 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-01-20 18:12:19 PST
I love you a little.
Comment 2 Ben Turner (not reading bugmail, use the needinfo flag!) 2011-01-20 18:49:22 PST
Only a little?!
Comment 3 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-01-20 18:54:55 PST
Maybe a moderate amount. Fine.
Comment 4 Alex Limi (:limi) — Firefox UX Team 2011-01-20 22:34:18 PST
<3
Comment 5 (dormant account) 2011-01-22 14:29:14 PST
Comment on attachment 505620 [details] [diff] [review]
wip

I just confirmed, this also works wonders on slow harddrives. Shaved around 50% off cold startup on my reference slow system. This patch works better than I expected. Not only does it trade random xul.dll io for faster sequential io. It also reduces the amount of seeks later on(ie no need to seek for xul bits), which significantly speeds up io on other files.

There are some tricky refinements that can be done here, such as trying to count pagefaults in order to detect warm startup and checking the amount of ram available, but I think this patch as is is simple and low risk so the rest can be done as followups. 

@shaver, I guess there is a fine line between hatred and love :) https://bugzilla.mozilla.org/show_bug.cgi?id=554421#c11 was regarding a similar trick on linux(the improvement there wasn't as huge on linux as it is on windows)
Comment 6 u88484 2011-01-22 14:39:49 PST
This patch greatly reduced my cold startup time so requesting blocking.  See below for the cold startup numbers with facebook and gmail app tabs loaded along with the about:home mozilla google search page loaded from previous session:

Taras build with patch applied:

main	        5070
sessionRestored	7083
firstPaint	7239

main	        5428
sessionRestored	6958
firstPaint	7067

main	        5382
sessionRestored	6881
firstPaint	7021

Latest trunk:

main	        2294
sessionRestored	16179
firstPaint	16787

main	        2325
firstPaint	16646
sessionRestored	17614

main	        2777
firstPaint	18331
sessionRestored	19455
Comment 7 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-01-22 16:06:30 PST
Comment on attachment 505620 [details] [diff] [review]
wip

Would it work better to preload separately in two threads?  I'm of two minds: on the one hand you're making the drive seek more, given that we defragment on update now, but on the other you can sometimes win by keeping the I/O queue full so that the head can pick up data more often during its rotation.  Maybe worth trying as an experiment later, though as I type this I think the first hand is the right hand.

{ /* nothing */ } instead of the hard-to-see semi, and let's land this bad boy for Monday's nightlies.
Comment 8 (dormant account) 2011-01-22 16:36:11 PST
Actually there is one issue here, the preload happens in firefox and plugin-container. What's the best way to only run this for firefox? Is there a define or a global variable I can use?
Comment 9 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-01-22 16:37:03 PST
I thought it was intentional to do it in plugin-container too, oh.
Comment 10 Ben Turner (not reading bugmail, use the needinfo flag!) 2011-01-22 16:52:20 PST
XRE_GetProcessType is probably what you want?
Comment 11 Brendan Eich [:brendan] 2011-01-22 20:25:48 PST
Comment on attachment 505620 [details] [diff] [review]
wip

Shaver's flirting and { /* nothing */ } got me into drive-by-nit mode.

>+static void preload(const char *dll) {

{ on new line.

>+  HANDLE fd = CreateFileA(dll, GENERIC_READ, FILE_SHARE_READ,
>+                          NULL, OPEN_EXISTING, FILE_FLAG_SEQUENTIAL_SCAN, NULL);
>+  if (fd == INVALID_HANDLE_VALUE)
>+    return;
>+  
>+  char buf[64*1024];
>+  DWORD dwBytesRead;
>+  
>+  while(ReadFile(fd, buf, sizeof(buf), &dwBytesRead, NULL) && dwBytesRead == sizeof(buf));

Rather than braces and /* nothing */, just

..    continue;

on its own line.

>+   CreateThread( NULL, 0, preload_thread, NULL, 0, NULL); 

Leading space before first NULL.

/be
Comment 12 Bjørnar Snoksrud 2011-01-23 04:34:34 PST
Small issue: CreateThread is returning a handle that is not freed until the process dies. You should CloseHandle on it. (MSVS2008 at least should pick this up with code analysis)
Comment 14 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-01-24 05:12:41 PST
Comment 13 hidden, due to suspicious executable attachment.
Comment 16 Mike Beltzner [:beltzner, not reading bugmail] 2011-01-24 08:15:29 PST
While this isn't a blocker, it's approved and I would be sadfaces if we neglected to check it in and make go faster now please k thx bye.
Comment 17 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-01-24 08:17:48 PST
Comment on attachment 505620 [details] [diff] [review]
wip

withdrawing review given the desire to not have it affect plugin-container, and I guess the thread handle leak too I suppose maybe.
Comment 18 Paul [pwd] 2011-01-24 08:28:15 PST
Did I miss where it was explained why we don't want this to affect the plugin-container?
Comment 19 Pablo Greco 2011-01-24 08:47:04 PST
I don't think it was explained, but the way I undrestand it, plugin-container.exe is launched by firefox, and if firefox is already running, the dlls are already preloaded, so it would be counterproductive to "preload" them again
Comment 20 Jim Jeffery not reading bug-mail 1/2/11 2011-01-24 09:51:51 PST
Couple of notes:  If in ccomment #17 the review was revoked, shouldn't the keyword 'check-in needed' be removed ? 

In respect to comment #19, the plugin-container.exe is not launched when Firefox starts if you only open a blank tab.  Plugin-container.exe only seems to start when there is say something on the page that uses a plugin, Flash of course being the predominate user.  So, unless the dll's are loaded even though the 'container' is not running, I don't understand.  

If Firefox is 'already running' - then the whole preload stuff is moot anyhow, no?
Comment 21 Jim Jeffery not reading bug-mail 1/2/11 2011-01-24 09:52:37 PST
oops, also the 'whiteboard' needs to be changed I would think.
Comment 22 Scott A. 2011-01-24 15:00:41 PST
What if the homepage contains plugins, would having the plugin-container included in the preload be a good thing in this instance?
Comment 23 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-01-24 15:03:58 PST
No, once it's loaded it's off disk, and reading it again isn't going to help.  Also, plugin-container uses much less of libxul than the main process.  We should just do it for the main process for now.
Comment 25 Justin Lebar (not reading bugmail) 2011-01-24 21:40:58 PST
Not to drive by, but as I understand from [1] and [2], the patch currently in the bug would help only on Vista and newer.  Is that right?

[1] http://infrequently.org/2011/01/on-the-care-and-feeding-of-spinning-disks/
[2] Search for function PreReadImage in 
http://src.chromium.org/viewvc/chrome/trunk/src/base/file_util_win.cc?view=markup
Comment 28 bobito 2011-01-24 23:50:11 PST
I may be totally wrong, but wouldn't the patch fail to preload if the current directory isn't the Firefox directory?

This could cause problems when external applications launch Firefox, for example. The user would see inconsistent startup performance and would have no idea why.
Comment 29 faravirusicom 2011-01-25 09:04:53 PST
How can I apply the patch in the attachement in order to test it ?
Comment 30 Csaba Kozák [:WonderCsabo] 2011-01-25 09:13:09 PST
apply:   https://developer.mozilla.org/en/Mercurial_FAQ
compile: https://developer.mozilla.org/en/build_documentation

BTW, you don't have to to this if there is a try build available. So, to the others: is there a try build somewhere with this patch?
Comment 31 (dormant account) 2011-01-25 09:56:57 PST
(In reply to comment #30)
> apply:   https://developer.mozilla.org/en/Mercurial_FAQ
> compile: https://developer.mozilla.org/en/build_documentation
> 
> BTW, you don't have to to this if there is a try build available. So, to the
> others: is there a try build somewhere with this patch?

Nothing you'd want to use yet.


Thanks for all of the drive-by reviews. Turned a large part of the speed up was depending on remnants in my builds from my previous experiment. I'm reworking the patch at the moment. I'll post a better one soon.
Comment 32 (dormant account) 2011-01-26 15:25:06 PST
Created attachment 507271 [details] [diff] [review]
new patch

This took me way too long. Spent half a week rebooting my windows laptop to find what kind of a change can cut our startup by 50% vs slowing us down by 50%(the difference in code is as little as one line of code). 
As I understand it, the key to good io perf is to win the race to open xul.dll with sequential_io flag and then preload the file. Normally the linker beats us to that and performance goes to the crapper.
Took me a while to realize that linking against xpcom.dll was dragging in xul.dll early.
I'm running this patch through try now. Hopefully I got my linker magic right.

Oh and turned out preloading on a thread was a bad idea. That gets us into a nondeterministic race with the linker as to who opens xul.dll first.

This patch also fixes relative paths, handle leakage. I haven't looked at how this behaves under xp. Had a hard enough time getting numbers out of my win7 box. My approach is different than that of chrome preload, they seem to prefer loading via sequential page faults, whereas file io seems to perform better for me.
Comment 33 (dormant account) 2011-01-26 16:36:00 PST
Created attachment 507298 [details] [diff] [review]
ready to go

Got rid of debug leftovers. Tested, this is indeed 2x faster to start.
Comment 35 IU 2011-01-26 21:48:31 PST
(In reply to comment #34)
> here is a try build
> http://stage.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/tglek@mozilla.com-eb7ffaf8ef72/try-w32/firefox-4.0b10pre.en-US.win32.zip

A total bust on XP.  My numbers:

UNPATCHED
---------
1. about:startup
main: 1344
SessionRestored: 10813
firstPaint: 13719

2. manual timing (stop watch)
chrome appears: 9 sec
chrome painted: 13 sec

PATCHED
-------
1. about:startup
main: 2265
SessionRestored: 10093
firstPaint: 11453

2. manual timing (stop watch)
chrome appears: 9 sec
chrome painted: 12 sec
Comment 36 IU 2011-01-26 21:56:07 PST
Just to be clear.  In my previous post:

Unpatched = January 26 nightly build

Patched = Try server build of comment 34

All timings are for cold startup after system restarted and hard drive allowed to settle (no more indication of post windows startup activity).
Comment 37 Csaba Kozák [:WonderCsabo] 2011-01-27 00:33:54 PST
I see no improvement :(

Jan 26 build:

main	3400
sessionRestored	7130
firstPaint	8534

Patched build:

main	3448
firstPaint	7458
sessionRestored	7770
Comment 38 Justin Lebar (not reading bugmail) 2011-01-27 00:35:23 PST
Csaba, what version of Windows are you running?
Comment 39 Jake 2011-01-27 00:38:36 PST
Just put it in nightly, so everybody can test.
Comment 40 Csaba Kozák [:WonderCsabo] 2011-01-27 00:57:58 PST
(In reply to comment #38)
> Csaba, what version of Windows are you running?

Windows 7 x64.
Comment 41 CorZ 2011-01-27 01:09:44 PST
(In reply to comment #34)
> here is a try build
> http://stage.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/tglek@mozilla.com-eb7ffaf8ef72/try-w32/firefox-4.0b10pre.en-US.win32.zip

windows xp sp3 / core 2 duo E6550, I don't see any different on my machine.
Comment 42 Zibi Braniecki [:gandalf][:zibi] 2011-01-27 01:38:04 PST
Windows 7 64bit, a bit slow HDD, latest nightly vs. try build

Cold start (latest nightly):
main	2937
sessionRestored	6421
firstPaint	6909

main	2284
sessionRestored	6239
firstPaint	6507

main	2193
sessionRestored	5071
firstPaint	5330

main	1662
sessionRestored	3499
firstPaint	3800

main	1989
sessionRestored	3393
firstPaint	3506


Cold start (try build):


main	1700
firstPaint	5612
sessionRestored	9993

main	4015
sessionRestored	5189
firstPaint	5562

main	3274
sessionRestored	4154
firstPaint	4275

main	3457
sessionRestored	4367
firstPaint	4492

main	3575
sessionRestored	4771
firstPaint	4880

main	3400
sessionRestored	4780
firstPaint	4900

Warm start (nightly):

main	10
sessionRestored	531
firstPaint	635

main	16
sessionRestored	498
firstPaint	589

main	16
sessionRestored	489
firstPaint	601

main	11
sessionRestored	503
firstPaint	616

main	33
sessionRestored	523
firstPaint	602


Warm start (try build):

main	56
sessionRestored	1360
firstPaint	1473

main	58
sessionRestored	797
firstPaint	905

main	27
sessionRestored	495
firstPaint	596

main	37
sessionRestored	613
firstPaint	725

main	39
sessionRestored	494
firstPaint	600
Comment 43 Eddward 2011-01-27 01:51:12 PST
Netbook - Win 7 Pro - tested 2x
nightly
main    2324
sessionRestored    19033
firstPaint    19392

build with patch
main    3011
sessionRestored    14681
firstPaint    15086 (still very bad, Chrome's cold startup time is ~6s)

Desktop - Win 7 Ultimate - tested 2x
nightly
main    2300
sessionRestored 3821
firstPaint     3911

build with patch
main    2530
sessionRestored 4201
firstPaint     4281
Comment 44 Csaba Kozák [:WonderCsabo] 2011-01-27 09:35:33 PST
Some other numbers from MozillaZine: http://forums.mozillazine.org/viewtopic.php?p=10357709#p10357709
Comment 45 rob64rock 2011-01-27 09:50:54 PST
I can confirm

XP SP3 32bit - zip builds - old profile - no session restore - home=(default)about:home - 4 tests average - cold start:

Nightly Build 20110127
main	1422
sessionRestored	18235
firstPaint	19094

Try Build
main	2531
sessionRestored	18640
firstPaint	19125

No real Cold start improvement seen for WinXp or Win7/Vista users that I can see.
Comment 46 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-01-27 09:51:40 PST
Comment on attachment 507298 [details] [diff] [review]
ready to go

Seems like we should only do this on Vista and up, like Chrome found too -- can you add that check, Taras?

> // we want to use the DLL blocklist if possible
> #define XRE_WANT_DLL_BLOCKLIST
>+#define XRE_PRELOAD_XUL

Comment here.

>+static void preload(LPCWSTR dll) {
>+  HANDLE fd = CreateFileW(dll, GENERIC_READ, FILE_SHARE_READ,
>+                          NULL, OPEN_EXISTING, FILE_FLAG_SEQUENTIAL_SCAN, NULL);
>+  char buf[64*1024];
> 
>+  if (fd == INVALID_HANDLE_VALUE) {
>+    return;
>+  }
>+  
>+  DWORD dwBytesRead;
>+  DWORD total = 0;
>+  while(ReadFile(fd, buf, sizeof(buf), &dwBytesRead, NULL)
>+        && dwBytesRead == sizeof(buf)) {
>+	total+= dwBytesRead;
>+  }

Why do we want to stop on a short read?

>+//Populate the windows page cache
>+static DWORD WINAPI preload_libs(LPCWSTR appPath) 
>+{
>+  wchar_t msg[MAX_PATH];
>+  size_t pathlen = wcslen(appPath);
>+  for(;pathlen && appPath[pathlen] != L'\\'; pathlen--);
>+  if (pathlen)
>+    pathlen++;
>+
>+  if (pathlen + 10 > MAX_PATH) {
>+    return 0;
>+  }
>+
>+  wcsncpy(msg, appPath, pathlen);
>+  size_t remaining = sizeof(msg)/sizeof(msg[0]) - pathlen - 1;
>+  wcsncpy(msg + pathlen, L"xul.dll", remaining);
>+  preload(msg);
>+  wcsncpy(msg + pathlen, L"mozjs.dll", remaining);
>+  preload(msg);
>+  return 0;
>+}
>+#endif
> int wmain(int argc, WCHAR **argv)

Blank line after #endif

> {
>+#ifdef XRE_PRELOAD_XUL
>+  preload_libs(argv[0]);
>+#endif
>+
> #ifndef XRE_DONT_PROTECT_DLL_LOAD
>   mozilla::NS_SetDllDirectory(L"");
> #endif
Comment 47 (dormant account) 2011-01-27 12:11:30 PST
Thanks for the feedback. I'm guessing the optimization gains are cancelled out by losses due to directwrite/d2d/etc. 
Please try setting:
gfx.direct2d.disabled -> true
gfx.font_rendering.directwrite.enabled -> false
in your normal build. 
then verify that when you go to about:support direct2d and directwrite are disabled.

Then compare this build vs nigtlies in that setting. Sorry, this isn't a magic bullet that can fix all startup issues, this fix is aimed solely at eliminating the overhead of loading our libraries(which is the slowest part of startup unless we are running into d2d bugs)

See bug 602792 for the directwrite-related slowdown info. I'm investigating other direct* slowdowns.
Comment 48 Csaba Kozák [:WonderCsabo] 2011-01-27 12:33:47 PST
(In reply to comment #47)

I've already logged the DW fontcache loading times as described in Bug 602792 Comment 110. I'm not hitting the DW fontcache slowdown. But i'll test with DW disabled to be sure.
Comment 49 IU 2011-01-27 12:59:18 PST
(In reply to comment #47)
> I'm guessing the optimization gains are cancelled out
> by losses due to directwrite/d2d/etc. 

Not on XP.  I've long had hardware acceleration disabled.
Comment 50 Csaba Kozák [:WonderCsabo] 2011-01-27 13:31:04 PST
(In reply to comment #48)
> (In reply to comment #47)
> 
> I've already logged the DW fontcache loading times as described in Bug 602792
> Comment 110. I'm not hitting the DW fontcache slowdown. But i'll test with DW
> disabled to be sure.

Okay, here are the new numbers. I disabled D2D and DW, and waited much longer after reboot to let *every* I/O get finished.

Cold startups, Win7 x64, 5 tabs.

Jan 27:

main	3135
sessionRestored	5523
firstPaint	6256

Patched:

main	2761
sessionRestored	6038
firstPaint	6459
Comment 51 (dormant account) 2011-01-27 13:37:01 PST
(In reply to comment #50)
> (In reply to comment #48)
> > (In reply to comment #47)
> > 
> > I've already logged the DW fontcache loading times as described in Bug 602792
> > Comment 110. I'm not hitting the DW fontcache slowdown. But i'll test with DW
> > disabled to be sure.
> 
> Okay, here are the new numbers. I disabled D2D and DW, and waited much longer
> after reboot to let *every* I/O get finished.
> 
> Cold startups, Win7 x64, 5 tabs.
> 
> Jan 27:
> 
> main    3135

Your main numbers are beyond bad, esp for win7. It would be very helpful if you could do an xperf profile of your nightly cold startup and email it to me. See instructions in.
http://blog.mozilla.com/tglek/2010/10/04/diagnosing-slow-startup/ 

Sounds like something is hurting your startup more than xul loading.
Comment 52 rob64rock 2011-01-27 15:40:55 PST
(In reply to comment #51)
> (In reply to comment #50)
> > (In reply to comment #48)
> > > (In reply to comment #47)
> > > 
> > > I've already logged the DW fontcache loading times as described in Bug 602792
> > > Comment 110. I'm not hitting the DW fontcache slowdown. But i'll test with DW
> > > disabled to be sure.
> > 
> > Okay, here are the new numbers. I disabled D2D and DW, and waited much longer
> > after reboot to let *every* I/O get finished.
> > 
> > Cold startups, Win7 x64, 5 tabs.
> > 
> > Jan 27:
> > 
> > main    3135
> 
> Your main numbers are beyond bad, esp for win7. It would be very helpful if you
> could do an xperf profile of your nightly cold startup and email it to me. See
> instructions in.
> http://blog.mozilla.com/tglek/2010/10/04/diagnosing-slow-startup/ 
> 
> Sounds like something is hurting your startup more than xul loading.

It would be interesting to see if the results from others using Win7/Vista following Csaba Kozák [:WonderCsabo] testing process with disabling DirectWrite or HWA with both todays Nightly Build and the TryServer Build. To just to rule out bug 602792 has the culprit of small improvements seen by bug 627591.
Comment 53 CorZ 2011-01-28 01:35:46 PST
(In reply to comment #47)
> Thanks for the feedback. I'm guessing the optimization gains are cancelled out
> by losses due to directwrite/d2d/etc. 
> Please try setting:
> gfx.direct2d.disabled -> true
> gfx.font_rendering.directwrite.enabled -> false
> in your normal build. 
> then verify that when you go to about:support direct2d and directwrite are
> disabled.
> 
> Then compare this build vs nigtlies in that setting. Sorry, this isn't a magic
> bullet that can fix all startup issues, this fix is aimed solely at eliminating
> the overhead of loading our libraries(which is the slowest part of startup
> unless we are running into d2d bugs)
> 
> See bug 602792 for the directwrite-related slowdown info. I'm investigating
> other direct* slowdowns.

tried again with these settings:

patched:
main	4844
sessionRestored	21453
firstPaint	22657

unpatched: (better)
main	2281
sessionRestored	16906
firstPaint	18015

I don't know why my firefox is so slow..
Comment 54 u88484 2011-01-28 05:48:05 PST
I'm on Windows 7 64bit and my numbers are very close to my original comment 4.  I think some of you might not be testing this right so here is detailed description on how to test this.

1) Install latest nightly and start it up a few times (Make sure about:startup is not loaded when you shutdown firefox...causes slow startup times for some reason)
2) Shutdown the system and then restart
3) Wait a minute, two or three minutes after windows starts then start Firefox (Time it so you start firefox at the same point for the test build)
4) load about:Startup and save the numbers somewhere

5) Get the build that Taras linked to in comment 34
6) Install it and start the build a few times with the same profile as before VERY IMPORTANT
7) repeat steps 2-4 (Again make sure you don't have about:Startup loaded before rebooting
Comment 55 Zibi Braniecki [:gandalf][:zibi] 2011-01-28 06:07:42 PST
I spent some time with Taras yesterday profiling my startup with xperf and testing his variants of the patch.
I was consistently getting rather poor results (~3500-4500 for main) no matter from which build (nightly or the one from taras).

Since the profile was showing a lot of loading from c:\$MFT I did some googling and found out that it's a Master File Table that stores data on all files on the hard drive (sec privs, settings, etc.). Which probably means that the more files we load, the more MFT is accessed.

I did several things:
* I checked my hard drive for errors - none found
* I defragmented it
* I installed Puran Defrag Free which has an on-boot defrag option that also defrags MFT table.
* I booted from Samsung HD tools CD, checked it (one again, it's all fine), and tuned it to be less noisy.

Since then I'm getting significantly better results. My average main dropped to ~1000-1500ms.

My total Disk utilization time before the defrag (numbers are from launching try build):
Service time: 4702288
Size: 18524672

After:
Service time: 3354041
Size: 14980608

Chrome Canary build:
Service time: 2897334
Size: 11442176

So I do see a major improvement and the perceived startup time of chrome and fx is now very close.

I'll try to compare 5 cold starts of try vs nightly now.

Also, during testing, we had a problem to trigger readahead and it only started working after Taras suggested me to delete data from c:\windows\prefetch.
Comment 56 Zibi Braniecki [:gandalf][:zibi] 2011-01-28 07:19:54 PST
I did another round of tests. Windows 7 64bit, Direct2d turned on (yeah, forgot to turn it off).

I decided to follow the path of prefetch and test both builds with and without prefetch. First scenario (removed prefetch) means that I deleted a content of the c:\windows\prefetch directory before each reboot. In the second scenario, I did not touch the prefetch directory at all.

Normal nightly:

Prefetch deleted:

main	729
sessionRestored	3398
firstPaint	4091

main	754
sessionRestored	3476
firstPaint	4127

main	680
sessionRestored	3331
firstPaint	3946

Prefetch preserved:

main	1476
sessionRestored	2063
firstPaint	2207

main	1830
sessionRestored	2390
firstPaint	2536

main	1577
sessionRestored	2162
firstPaint	2297

===============

Latest build from Taras (from his people.mozilla.org directory):

Prefetch deleted:

main	573
sessionRestored	2950
firstPaint	3315

main	644
firstPaint	3212
sessionRestored	3245

main	680
firstPaint	3183
sessionRestored	3224

main	619
firstPaint	4123
sessionRestored	4165

Prefetch preserved:

main	2090
sessionRestored	3264
firstPaint	3411

main	2257
sessionRestored	2928
firstPaint	3071

main	2336
sessionRestored	3032
firstPaint	3175
Comment 57 Scott A. 2011-01-31 09:44:57 PST
So what's going on with this, is it still the plan to get this working for the majority and get it in?
Comment 58 (dormant account) 2011-01-31 10:27:14 PST
(In reply to comment #57)
> So what's going on with this, is it still the plan to get this working for the
> majority and get it in?

Good question. I'm a bit stuck. Been analyzing user profiles for the past week. Seems that this is a significant win for people with non-broken harddrives. Ie if a harddrive can read at > 20mb/s(most modern non-broken drives should start at 50mb/s), then this is a win. Otherwise it is a regression. 

Seems like the easiest thing to do is to add a commandline -preload option and have users opt-in to that, but that clearly isn't the correct way to go.

I could also benchmark startup times and measure preloading throughput, but then that data has to be stored somewhere and there is a risk of measuring wrong (ie mistaking 1 case of slow startup due to an abnormally high disk load). 

There is also the option of landing this as is and hoping that more users benefit than those who regress from this. But this has the downside of increasing the rift between well-performing firefox and slow firefox. 

Perhaps the best option would be an escape hatch that measures amount of time spent in the preload() and aborts after a second or two.
Comment 59 Ed Morley [:emorley] 2011-01-31 11:11:33 PST
(In reply to comment #58)
> I could also benchmark startup times and measure preloading throughput, but
> then that data has to be stored somewhere and there is a risk of measuring
> wrong (ie mistaking 1 case of slow startup due to an abnormally high disk
> load). 

As unhelpful as the Windows Experience Index the rest of the time, how about utilising it in this instance, on just the first run of Fx4, to set a preload=true flag for subsequent instances, if say the primary hard disc score was above whatever value equates to 20mb/s ?

It would avoid the issues of Fx trying to benchmark on first run (which as you say, might lead to a false positive if disk load was abnormally high at that moment), as well as the complexities of trying to come up with an accurate benchmarking system. If no WEI scores had been calculated yet, then the true/false flag could say just be rechecked in X days time, once the user has run the OS long enough for the WEI test to run.

Obviously the WEI score system only exists for Vista/7 - but if Mike's assessment in comment 46 is correct, then this patch doesn't benefit XP anyway - so the lack of WEI wouldn't matter.

Anyway, hope you don't mind the drive-by comments - the above is just something that occurred to me and thought it wouldn't hurt to mention...
Comment 60 (dormant account) 2011-01-31 12:22:26 PST
The windows experience index crossed my mind. Problem is that at least on gandalf's system above the index is out of date and the harddrive performed significantly worse than indicated.

I'll experiment with a few solutions, but given the complexity of heuristics mentioned so far, this will take at least another week to figure out.

I checked on XP. The speed doubles there too(from 10s to 5s on Kurt's system). Chrome's use of pagefaults to preload stuff might be why our approaches differ on XP.
Comment 61 (dormant account) 2011-01-31 14:33:24 PST
Another look at xperf traces (and gandalf's startup measurements above) suggest a bad interaction with windows prefetch. In Csaba's profile a full second is spent on IO before my preload code even runs.

So I might just be able to disable preload if prefetch took forever.
Comment 62 Jake 2011-01-31 23:28:50 PST
Why don't add an option at about:config? this way everybody can switch it on or off at will, no way you can satisfy anybody due to the complexity, even eventually you find a way you think is perfect, you might still need add this option, somenoe at some times just want to turn it off, you never know.
Comment 63 Jake 2011-01-31 23:50:51 PST
It may happens before FF read prefs, but you can mark it at startupCache or somewhere when people change it at about:config. it's better than -preload because everybody knows how to customize at about:config, it's the first thing you learn as a FF user.
Comment 64 sashafan2 2011-02-01 01:23:43 PST
What does the "main" property mean?
Comment 65 (dormant account) 2011-02-02 14:24:42 PST
.main marks when the platform-independent bits of firefox start running. It's useful for developers to gauge library/other-platform-overhead.

Would be useful to have people compare startup of
http://people.mozilla.com/~tglek/startup/firefox-4.0b10pre.en-US.win32.zip
and
http://people.mozilla.com/~tglek/startup/vanilla_ff.zip

Also tell me what the messagebox in the first build says. I added a heuristic to disable preload if we seem to be executing with hot cache or if Windows prefetch has spent a long time caching before the preload code runs.
Comment 66 Eddward 2011-02-03 05:48:31 PST
latest nightly:
main	680
sessionRestored	8461
firstPaint	8681

latest nightly 2.test:
main	480
sessionRestored	8731
firstPaint	9001

build with patch:
main	680
sessionRestored	4431
firstPaint	4571

build with patch 2.test:
main	990
sessionRestored	4521
firstPaint	4681

D2D/DW/D3D10 - enabled
looks good :)
Comment 67 Eddward 2011-02-03 06:21:15 PST
Netbook - Win 7 Pro
nightly:
main	1482
firstPaint	22107
sessionRestored	25508

build with patch:
main	5007
sessionRestored	13822
firstPaint	14165

D2D/DW/D3D - disabled
Comment 68 (dormant account) 2011-02-03 11:55:37 PST
Thanks for the tests, keep it up please. I need more stats.
It's not clear what people are comparing. I specifically need 

http://people.mozilla.com/~tglek/startup/vanilla_ff.zip

compared to

http://people.mozilla.com/~tglek/startup/firefox-4.0b10pre.en-US.win32.zip

I would also like to knows what the messagebox on startup of firefox-4.0b10pre.en-US.win32.zip says.
Comment 69 Eddward 2011-02-03 13:04:03 PST
Notebook, Win 7 Pro 32bit, D2D/DW/D3D - disabled, clean new profile
with patch(firefox-4.0b10pre.en-US.win32.zip):
main	2855
sessionRestored	5289
firstPaint	5508

with patch2:
main	3651
firstPaint	6538
sessionRestored	6647

with patch3:
main	3994
firstPaint	6007
sessionRestored	6085
-------
vanilla:
main	1669
firstPaint	7692
sessionRestored	7785

vanilla2:
main	2637
firstPaint	7989
sessionRestored	8082

vanilla2:
main	3651
firstPaint	6818
sessionRestored	6912

With patch it's better but not significantly...

err, I see no messagebox on startup of firefox-4.0b10pre.en-US.win32.zip
Comment 70 (dormant account) 2011-02-03 13:33:45 PST
Eddward, looks like firefox-4.0b10pre.en-US.win32.zip was a slightly stale version. Fixed that now, sorry & thanks for catching the problem. Can you retest it?
Comment 71 Eddward 2011-02-03 14:35:24 PST
Okay, so this is now correct link http://people.mozilla.com/~tglek/startup/firefox-4.0b11pre.en-US.win32.zip yes?

results:
1. cold startup
2900ms since start. Things seems to be cached.
main	3151
sessionRestored	5991
firstPaint	6428

2. cold startup
2900ms since start. Things seems to be cached.
main	3260
firstPaint	6958
sessionRestored	7083
Comment 72 (dormant account) 2011-02-03 14:43:30 PST
Ok thanks Eddward, looks like on your system windows prefetch does a lot of io similarly to gandalf's.


Anyone else?
Comment 73 Eddward 2011-02-03 15:25:50 PST
Results from another system (desktop) Win 7 Ultimate 32bit D2D/DW/D3D10 - enabled.

firefox-4.0b11pre.en-US.win32.zip (with patch)
1. cold startup
800ms since start. Optimizing
main	1280
sessionRestored	2551
firstPaint	2741

2. cold startup
1400ms since start. Things seems to be cached.
main	1540
sessionRestored	2491
firstPaint	2611

3. cold startup
1300ms since start. Things seems to be cached.
main	1430
sessionRestored	2811
firstPaint	2961

vanilla:
results very similar; first paint ~2600-2700ms
Comment 74 Loic 2011-02-04 15:46:22 PST
System: laptop Vista Home 32bit - D2D/DW blocked/disabled

Test: 3 runs / only tab about:home / working offline

Vanilla version:

main	46
sessionRestored	2060
firstPaint	2153

main	62
sessionRestored	2060
firstPaint	2154

main	156
sessionRestored	2201
firstPaint	2294

Patched version firefox-4.0b11pre.en-US.win32:


0 ms since start. Things seem to be cached
main	94
sessionRestored	2136
firstPaint	2370


0 ms since start. Things seem to be cached
main	63
sessionRestored	2123
firstPaint	2388


0 ms since start. Things seem to be cached
main	125
sessionRestored	2190
firstPaint	2439
Comment 75 (dormant account) 2011-02-09 12:01:30 PST
Sorry for delays. I was triplechecking that prefetch was screwing me. Turns out windows prefetch spends a lot of time prefetching before executable code is even run making any optimization within firefox impossible. Time to give up on approach in the patches.

Instead I moved prefetching into a separate executable. Please copy http://people.mozilla.com/~tglek/startup/wrapper.exe into your firefox directory and try running firefox by launching wrapper.exe. Use any nightly build to test.

i need about:startup times after reboot(as before) + the message that is shown after firefox exits. I also need about:startup numbers when running firefox.exe directly.


Here I see 1.5-2x speeds ups without any/much prefetch slowdown. Would like to see that confirmed.
Comment 76 Brian Carpenter [:geeknik] 2011-02-09 12:17:41 PST
I was hoping to help test this, but about:startup doesn't exist in this nightly build. When I run your wrapper and let Minefield startup and then exit it, I get 100ms to preload. This is w/o rebooting the computer, which I can't do right now.
Comment 77 (dormant account) 2011-02-09 12:19:17 PST
(In reply to comment #76)
> I was hoping to help test this, but about:startup doesn't exist in this nightly
> build. When I run your wrapper and let Minefield startup and then exit it, I
> get 100ms to preload. This is w/o rebooting the computer, which I can't do
> right now.

http://glandium.org/blog/?p=1575 for about:startup extension.

I need people to reboot the computer for this.
Comment 78 Brian Carpenter [:geeknik] 2011-02-09 12:45:21 PST
Okay, here we go...

*rebooted* 
Ran w/ the wrapper:

main	3109
sessionRestored	14469
firstPaint	15360

700ms to preload

*rebooted again*

Ran w/o the wrapper:

main	3203
sessionRestored	14704
firstPaint	17063

System specs: Intel i7-920 @ 4.2ghz, 6GB DDR3 @ 1600mhz, Gigabyte X58A-UD7 motherboard, 4x250GB Western Digital Enterprise drives in RAID0+1, etc etc.

Build I'm using is built from http://hg.mozilla.org/mozilla-central/rev/fd0817e454fe
Comment 79 (dormant account) 2011-02-09 12:49:38 PST
Brian, that's horrible given your specs. Can you try deleting files(either all of them or just wrapper* and firefox*) from c:\windows\prefetch  and rebooting again?
Comment 80 Brian Carpenter [:geeknik] 2011-02-09 13:18:24 PST
Ok, deleted the contents of the prefetch directory. Also, those numbers in Comment 78 reflect Minefield loading 4 app tabs and another 3 or 4 regular tabs. 

These numbers were run after I rebooted, with the wrapper, and only opening about:home.

main    523
sessionRestored    15173
firstPaint    16360

500ms to preload
Comment 81 (dormant account) 2011-02-09 13:41:19 PST
(In reply to comment #80)
Brian, your startup is the worst I've ever seen on a high power system. Would you mind rebooting and then recording an xperf trace? See http://blog.mozilla.com/tglek/2010/10/04/diagnosing-slow-startup/ email me a a link to it.

Looks to me like wrapper.exe is working great on your system(ie main time is decreased correctly), but the rest of your disk IO is killing startup.
Comment 82 Brian Carpenter [:geeknik] 2011-02-09 14:51:11 PST
Okay, I disabled direct*, rebooted windows, disabled the realtime scanner on my AV and here is what I get:

main	2031
sessionRestored	6016
firstPaint	6298

800ms to preload
Comment 83 Brian Carpenter [:geeknik] 2011-02-09 15:04:26 PST
Direct* still disabled, av scanner is enabled this time, no wrapper:

main	6515
sessionRestored	9970
firstPaint	10673
Comment 84 (dormant account) 2011-02-09 15:07:08 PST
(In reply to comment #83)
> Direct* still disabled, av scanner is enabled this time, no wrapper:
> 
> main    6515
> sessionRestored    9970
> firstPaint    10673

cool so looks like wrapper shaved 4seconds here. Not sure things are still slow with directwrite off. Perhaps a defrag will take care of that.
Comment 85 (dormant account) 2011-02-09 16:15:52 PST
Created attachment 511223 [details]
wrapper.exe source

source for wrapper.exe
Comment 86 Brian Carpenter [:geeknik] 2011-02-09 17:06:27 PST
Ok, just to make sure there were no filesystem errors, I rebooted, forced a chkdsk on C:, ran a boot time defrag and then boot into windows and ran the wrapper again and got this:

main	4593
sessionRestored	8469
firstPaint	8579

900 ms to preload

Direct* is still disabled, AV was enabled.
Comment 87 John Daggett (:jtd) 2011-02-09 18:14:31 PST
(In reply to comment #80)
> Ok, deleted the contents of the prefetch directory. Also, those numbers in
> Comment 78 reflect Minefield loading 4 app tabs and another 3 or 4 regular
> tabs. 
> 
> These numbers were run after I rebooted, with the wrapper, and only opening
> about:home.
> 
> main    523
> sessionRestored    15173
> firstPaint    16360
> 
> 500ms to preload

Brian, with Direct* enabled, could you attach the info below for
your cold startup times?

1. DirectWrite version and font cache size (see about:support)
2. Output of enabling the startup logging described in bug 602792, comment 110

I'm guessing your font cache file is fairly large, I would also
suggest testing again after deleting that file.  See instructions
on bug 602792, comment 103.
Comment 88 Brian Carpenter [:geeknik] 2011-02-09 18:22:20 PST
I haven't had a chance to get to the logging portion yet, I apologize, however, here is the other info from about:support

DirectWrite Enabled true (6.1.7601.17514, font cache 8.05 MB)
Comment 89 Brian Carpenter [:geeknik] 2011-02-10 01:30:50 PST
Here is the fontlog.txt:

0[82d140]: InitFontList
0[82d140]: Start: 02/10/2011 03:28:53
0[82d140]: Uptime: 30882.671 s
0[82d140]: dwrite version: 6.1.7601.17514
0[82d140]: Total time in InitFontList:        0.018 ms
0[82d140]:  --- gfxPlatformFontList init:     0.006 ms
0[82d140]:  --- GdiInterop object:            0.012 ms
0[82d140]: DelayedInitFontList
0[82d140]: Start: 02/10/2011 03:28:54
0[82d140]: Uptime: 30882.906 s
0[82d140]: dwrite version: 6.1.7601.17514
0[82d140]: Total time in DelayedInitFontList:        0.960 ms (families: 196, gdi table access)
0[82d140]:  --- GetSystemFontCollection:      0.036 ms
0[82d140]:  --- iterate over families:        0.923 ms

I didn't reboot before running this. If you want me to do that, I will, and then I will also delete the font cache and retest the other scores as well if need be.
Comment 90 Brian Carpenter [:geeknik] 2011-02-10 01:40:17 PST
And here is the fontlog.txt after removing the fontcache files: 

0[72d140]: InitFontList
0[72d140]: Start: 02/10/2011 03:37:54
0[72d140]: Uptime: 31423.343 s
0[72d140]: dwrite version: 6.1.7601.17514
0[72d140]: Total time in InitFontList:        0.019 ms
0[72d140]:  --- gfxPlatformFontList init:     0.006 ms
0[72d140]:  --- GdiInterop object:            0.013 ms
0[72d140]: DelayedInitFontList
0[72d140]: Start: 02/10/2011 03:37:54
0[72d140]: Uptime: 31423.796 s
0[72d140]: dwrite version: 6.1.7601.17514
0[72d140]: Total time in DelayedInitFontList:        0.956 ms (families: 196, gdi table access)
0[72d140]:  --- GetSystemFontCollection:      0.015 ms
0[72d140]:  --- iterate over families:        0.941 ms

And

DirectWrite Enabledtrue (6.1.7601.17514, font cache n/a)
Comment 91 Csaba Kozák [:WonderCsabo] 2011-02-11 09:04:53 PST
Unfortunately i can no longer test with my laptop, but here are the numbers with my PC (Win7 x64, AMD Phenom X3 8450, 4GB DDR2 RAM, 5.7 WXPI for HDD):

Cold startup times with 2011-02-11 build, D2D DW off, 2 tabs restored

w/out wrapper

main	1232
firstPaint	13027
sessionRestored	14509

w/ wrapper

main	1107
firstPaint	6756
sessionRestored	7910

500 ms to preload
Comment 92 Eddward 2011-02-11 09:51:53 PST
Intel Atom N450, 2GB RAM, Win 7 Pro 32bit, D2D,D3D,DW off, clean new profile.

Nightly (2011-02-11) without wrapper:

main	2043
firstPaint	8050
sessionRestored	8487

main	2637
firstPaint	7989
sessionRestored	8082

Nightly (2011-02-11) with wrapper:

main	952
firstPaint	3823
sessionRestored	4073
2200ms to preload

main	889
firstPaint	3792
sessionRestored	4088
1900ms to preload
Comment 93 Justin Lebar (not reading bugmail) 2011-02-11 10:38:49 PST
Windows XP laptop, Core2 Duo T7200 @ 2GHz, 2GB RAM, 7200RPM, no AV.
D2D, DW off.

== No wrapper ==
main	1093
firstPaint	6062

main	1297
firstPaint	7031
sessionRestored	7281

== Wrapper ==
900ms to preload
main	2672
firstPaint	7079
sessionRestored	7297

500ms to preload
main	1047
firstPaint	5844
sessionRestored	6094
Comment 94 (dormant account) 2011-02-11 10:47:12 PST
Ok, thanks for the testing guys. Sounds like this approach doesn't get annihilated by windows prefetch as badly. There is a small issue in windows throwing away the cache too soon(due to me indicating sequential IO). Thus there is potential for better perf.
Comment 95 Justin Lebar (not reading bugmail) 2011-02-11 11:31:03 PST
Here are my numbers after a defrag.

== No wrapper ==
main	922
firstPaint	5125
sessionRestored	5672

== Wrapper ==
300ms to preload
main	953
firstPaint	4750
sessionRestored	4984
Comment 96 (dormant account) 2011-02-11 15:55:06 PST
I think I fixed the cache issue I was seeing before by mmap()ing the area

Here are new versions of wrapper. The first one is threaded, to see if keeping cpu into lower power state helps(ie lower frequency resulting in lower IRQ handling), .
http://people.mozilla.com/~tglek/startup/wrapper_threaded.exe

This one is the same as before but with mmap. 
http://people.mozilla.com/~tglek/startup/wrapper.exe

Justin could you try these to see if your numbers get better?
Comment 97 christian 2011-02-11 16:00:50 PST
Is there any way we can get this in nightlies and/or run A/B testing for the various options (using Test Pilot)? I'm worried that ad-hoc testing may not give us the best outcome across millions of devices. Even running this on all the machines in the QA labs would give us additional clarity via more data points...
Comment 98 (dormant account) 2011-02-11 16:05:38 PST
(In reply to comment #97)
> Is there any way we can get this in nightlies and/or run A/B testing for the
> various options (using Test Pilot)? I'm worried that ad-hoc testing may not
> give us the best outcome across millions of devices. Even running this on all
> the machines in the QA labs would give us additional clarity via more data
> points...

I'm not too worried about testing this. I'm just making sure that preloading doesn't make things slower for people. As far as testing goes we can turn it on in a nightly(assuming the wrapper approach will fly) and then use our startup reporting mechanism(http://blog.mozilla.com/addons/2011/02/10/add-on-metadata-start-up-time/) to see if our startups decrease.
Comment 99 Justin Lebar (not reading bugmail) 2011-02-11 17:55:42 PST
Sorry, Taras; not much movement.

== Wrapper 2 ==
preload 400ms
main	1360
firstPaint	6032
sessionRestored	6704

== wrapper_threaded ==
preload 500ms
main	1625
firstPaint	5594
Comment 100 Csaba Kozák [:WonderCsabo] 2011-02-12 12:39:37 PST
(In reply to comment #91)

With same spec:

wrapper_mmap

main	796
firstPaint	11998
sessionRestored	14322

400ms to preload

wrapper_threaded

main	858
firstPaint	9985
sessionRestored	10952

500ms to preload
Comment 101 Raul 2011-02-12 17:23:58 PST
Using Windows 7 x64 and Firefox latest nightly 20110212(32 bits), the first time I run FF4 once I turn on the laptop, varies from 200ms to 500ms, with no changes to anything, I even have disabled autoupdate of addons, FF and search engines.
Comment 102 marcio pl 2011-02-14 11:54:23 PST
Some numbers on my system, an almost 4 years old Intel Q6600@2.8ghz 4gb on Windows 7 64 bits. Prefetch info was cleared before each reboot. 

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b12pre) Gecko/20110214 Firefox/4.0b12pre with 19 extensions.
D2D and DW enabled
GPU Accelerated Windows: 1/1 Direct3D 10

w/o wrapper before deleting Prefetch:
main	1320
firstPaint	7061
sessionRestored	7251

w/ wrapper before deleting Prefetch (threaded):
300ms preload
main	3910 (weird)
firstPaint	6561
sessionRestored	6661

w/ wrapper, before deleting Prefetch (mmap):
200ms preload
main	710
firstPaint	6371
sessionRestored	6531

---------------------------------

w/ wrapper, after deleting Prefetch (mmap):
300ms preload
main	180
firstPaint	6331
sessionRestored	6531

w/ wrapper, after deleting Prefetch (threaded):
100ms preload
main	140
firstPaint	2921
sessionRestored	3011
Comment 103 Oskar Ivanić (:icecold) 2011-02-16 13:14:20 PST
AMD Athlon 4800+ X2 2.5GHz, 1GB DDR2, Seagate ST3160815A 7200 RPM ATA/100 on Windows 7 Ultimate 32bit.

Build identifier: Mozilla/5.0 (Windows NT 6.1; rv:2.0b12pre) Gecko/20110216 Firefox/4.0b12pre with 2 add-ons (AdblockPlus and about:startup)

Hardware acceleration on. (installed hotfix from MS - KB2467023)

With HWA:
main	109
firstPaint	1655
sessionRestored	1717

With HWA and wrapper_mmap
main	62
firstPaint	1483
sessionRestored	1545

100 ms to preload

With HWA and wrapper_threaded
main	62
firstPaint	1655
sessionRestored	1717

All
Comment 104 Oskar Ivanić (:icecold) 2011-02-16 13:16:49 PST
(In reply to comment #103)
> With HWA and wrapper_threaded
> main    62
> firstPaint    1655
> sessionRestored    1717
200 ms to preload

All with prefetch deleted before.
Font cache 1,71 MB
Comment 105 (dormant account) 2011-02-16 13:40:24 PST
(In reply to comment #104)
> (In reply to comment #103)
> > With HWA and wrapper_threaded
> > main    62
> > firstPaint    1655
> > sessionRestored    1717
> 200 ms to preload
> 
> All with prefetch deleted before.
> Font cache 1,71 MB

ok thanks. How about with prefetch?

For everybody else, thanks for the feedback. Due to having to move to a separate binary to avoid windows prefetch penalty this is too complex to release as part of firefox 4. We are considering options for releasing it this after ff4.
My least bad option at the moment is to install a helper that wipes firefox prefetch entries on windows startup and do prefetch within the firefox executable as before. The other option is to teach firefox that it's being launched from a wrapper and deploy the wrapper(this is tricky to get right for integration into windows, etc).
Comment 106 Oskar Ivanić (:icecold) 2011-02-16 14:41:54 PST
(In reply to comment #105)
> ok thanks. How about with prefetch?

With HWA:
main	109
firstPaint	1685
sessionRestored	1732

With HWA and wrapper_mmap:
main	78
firstPaint	1545
sessionRestored	1607

200 ms to preload

With HWA and wrapper_threaded
main	78
firstPaint	1655
sessionRestored	1733

200 ms to preload

Sometimes numbers are much higher like 1500 (main) .
But after restart it gets lower.

This is without deleting prefetch.
Comment 107 Dave Garrett 2011-02-16 14:59:38 PST
(In reply to comment #105)
> My least bad option at the moment is to install a helper that wipes firefox
> prefetch entries on windows startup and do prefetch within the firefox
> executable as before.

One of the ideas I heard floated a while back was to switch to a new background automatic updates system, possibly run as a Windows service. (I think Google Chrome does something like this) If this becomes a reality for Firefox, it would make sense to merge this sort of preloader concept with the background auto-updater into one Firefox background service. You could probably even go further and have it preload a fair bit during idle time if there's enough memory free.
Comment 108 Robert Strong [:rstrong] (use needinfo to contact me) 2011-02-16 15:06:44 PST
To avoid needing to runas or the UAC prompt the service will be running as the system account (default for services) which is privileged. I would very much prefer not preloading anything since it would add a potential attack vector.
Comment 109 (dormant account) 2011-02-16 15:10:51 PST
(In reply to comment #108)
> To avoid needing to runas or the UAC prompt the service will be running as the
> system account (default for services) which is privileged. I would very much
> prefer not preloading anything since it would add a potential attack vector.

Preloading as in populating the file cache doesn't add any attack vectors.
Comment 110 Robert Strong [:rstrong] (use needinfo to contact me) 2011-02-16 18:04:22 PST
(In reply to comment #109)
> (In reply to comment #108)
> > To avoid needing to runas or the UAC prompt the service will be running as the
> > system account (default for services) which is privileged. I would very much
> > prefer not preloading anything since it would add a potential attack vector.
> 
> Preloading as in populating the file cache doesn't add any attack vectors.
The summary "preload dlls" is what threw me off and it is definitely safer from the service's point of view.
Comment 111 (dormant account) 2011-02-17 12:28:24 PST
(In reply to comment #108)
> To avoid needing to runas or the UAC prompt the service will be running as the
> system account (default for services) which is privileged. I would very much
> prefer not preloading anything since it would add a potential attack vector.

is there a bug for implementing this service?
Comment 112 Robert Strong [:rstrong] (use needinfo to contact me) 2011-02-17 12:31:51 PST
bug 481815
Comment 113 Robert Strong [:rstrong] (use needinfo to contact me) 2011-02-17 12:46:25 PST
Couple of quick notes about the service.
1. it will start during OS startup before any users logon.
2. not all users will be able to install it since it requires admin privs to install.
3. there will likely be an opt-out option in the installer.
Comment 114 CorZ 2011-02-18 01:04:12 PST
(In reply to comment #107)
> (In reply to comment #105)
> > My least bad option at the moment is to install a helper that wipes firefox
> > prefetch entries on windows startup and do prefetch within the firefox
> > executable as before.
> 
> One of the ideas I heard floated a while back was to switch to a new background
> automatic updates system, possibly run as a Windows service. (I think Google
> Chrome does something like this) If this becomes a reality for Firefox, it
> would make sense to merge this sort of preloader concept with the background
> auto-updater into one Firefox background service. You could probably even go
> further and have it preload a fair bit during idle time if there's enough
> memory free.

I don't see any windows services for Chrome. Google does have added some Scheduled Tasks for update, but no service or task for preload.
Comment 115 myahoo 2011-02-18 04:19:50 PST
(In reply to comment #114)
> I don't see any windows services for Chrome. Google does have added some
> Scheduled Tasks for update, but no service or task for preload.

I think you need to watch two things:
* gpedit.msc
* services.msc
Comment 116 Virtual_ManPL [:Virtual] - (ni? me) 2011-02-18 06:31:16 PST
Isn't a better idea to simply add this ddl's to SuperPrefetch windows folders ?
We can add this on installation and next Firefox will be checking on start that this ddls still exist, because can be deleted by some software. And if not Firefox will add it. Checking need to be wise implanted to not burden slow cold start.
Comment 117 Please Ignore This Troll (Account Disabled) 2011-02-20 08:10:11 PST
(In reply to comment #116)
> Isn't a better idea to simply add this ddl's to SuperPrefetch windows folders ?
> We can add this on installation

Will this work for portable Fx too?

> and next Firefox will be checking on start that this ddls
> still exist, because can be deleted by some software. And if not Firefox
> will add it. Checking need to be wise implanted to not burden slow cold start.

E.g. this dll will be copied to SuperPrefetch after 1st Fx run and won't get loaded for the 1st run? Or it is still planned to be done while Windows is loading?
Comment 118 Virtual_ManPL [:Virtual] - (ni? me) 2011-02-20 10:18:25 PST
(In reply to comment #117)
> Will this work for portable Fx too?

No, because portable is like name means portable. It shouldn't add or leave any files & keys in system (except buggy portable "shells") and as far I remember we don't support officially portable Firefox.

(In reply to comment #117)
> E.g. this dll will be copied to SuperPrefetch after 1st Fx run and won't get
> loaded for the 1st run? Or it is still planned to be done while Windows is
> loading?

Files should be added to SuperPrefetch folders after installation/upgrade or when Firefox check they're missing, so Windows will be preload this ddls on next startup.

Regarding first run with cold-start I have idea, that we can called some process that will be preloaded this ddls, before seeing latest installation screen with "Finish / OK" button or if it's possible preload only ddls to memory without it.

So Firefox will be loading much faster with this :)


I'm not a programmer so it's only theoretical discussion.


What devs thing about it ? It's possible ?
Comment 119 Robert Strong [:rstrong] (use needinfo to contact me) 2011-02-20 22:14:38 PST
(In reply to comment #118)
>...
> (In reply to comment #117)
> > E.g. this dll will be copied to SuperPrefetch after 1st Fx run and won't get
> > loaded for the 1st run? Or it is still planned to be done while Windows is
> > loading?
> 
> Files should be added to SuperPrefetch folders after installation/upgrade or
> when Firefox check they're missing, so Windows will be preload this ddls on
> next startup.
>
Prefetch is managed by the OS so we can't add files directly.

> Regarding first run with cold-start I have idea, that we can called some
> process that will be preloaded this ddls, before seeing latest installation
> screen with "Finish / OK" button or if it's possible preload only ddls to
> memory without it.
This would only slightly improve first run after an install since we have to generate our own cache, etc. after installing a different version
Comment 120 Mike Beltzner [:beltzner, not reading bugmail] 2011-02-23 18:27:47 PST
Comment on attachment 505620 [details] [diff] [review]
wip

(bookkeeping)
Comment 121 (dormant account) 2011-03-07 15:51:03 PST
I wrote an extension http://people.mozilla.com/~tglek/startup/startup.service.xpi (source http://hg.mozilla.org/users/tglek_mozilla.com/startup-service) to deal with the "slow windows prefetch" problem.
The extension is a) wrapper around a service .exe that deletes firefox prefetch files and b) wrapper .exe that preloads firefox in a more efficient manner than windows does by default.
The extension installs a "Firefox Service" and removes it on uninstall. It also adds a 'faster firefox' shortcut to the desktop. In theory having this extension installed + launching via the "faster firefox" shortcut should result in the 2x speedup that this bug aims for.
Comment 122 Oskar Ivanić (:icecold) 2011-03-07 17:02:33 PST
AMD Athlon 4800+ X2 2.5GHz, 1GB DDR2, Seagate ST3160815A 7200 RPM ATA/100 on
Windows 7 SP1 Ultimate 32bit. 

5 extensions installed (AdblockPLus, GreaseMonkey, F1, startup, startup service) + 1 userscript and active persona.

HWA on.

Before startup service add-on:
main	1638
firstPaint	13354
sessionRestored	13448

With startup service add-on:
main	858
firstPaint	7848
sessionRestored	7988
Comment 123 Wes Kocher (:KWierso) 2011-03-07 21:28:20 PST
So here are my numbers:

Maybe cold/warm? (This is what about:startup showed initially after I installed it. Don't remember if this was for the first time I started Firefox after rebooting the system, the session was left running over the weekend.)
main	4661
firstPaint	12132
sessionRestored	14184

Warm (with extension) (After installing the startup service addon, I shut down Firefox and started it with the "faster firefox" shortcut.)
main	3631
firstPaint	9751
sessionRestored	13391

Cold (with extension) (I shut down everything and restarted my system completely, then used the "faster firefox" shortcut to launch the browser.)
main	3761
firstPaint	10314
sessionRestored	12202



Installed/enabled extensions:
Add-on Compatibility Reporter
JS Deminifier
RT Site Extender
Stratiform
Feedback
Adblock Plus
Greasemonkey
Status-4-Evar
Bugzilla Tweaks
Add-on Builder Helper
Stylish
Echofon
NoScript
Element Hiding Helper for Adblock Plus
DOM Inspector
startup
startup.service

I have 7 open tabs, including Gmail, about:addons and Facebook.




(Just out of curiosity, do the startup numbers take the time spent in the profile manager's UI into account?)
Comment 124 Oskar Ivanić (:icecold) 2011-03-08 02:52:39 PST
I tested extension on other computer on Windows 7 SP 1.
AMD Athlon 2800+ 2.0 GHz, 1GB DDR RAM, HDD - Western Digital Caviar Blue WD800JD 7200 RPM SATA150
HWA off.
Extensions installed: Adblock Plus, GreaseMonkey (one usersript) , about startup, startup service and active persona theme.

Cold without startup service:
main	2015
firstPaint	4110
sessionRestored	4235

Cold with startup service:
main	688
firstPaint	3251
sessionRestored	3548
Comment 125 Emanuel Hoogeveen [:ehoogeveen] 2011-03-08 03:29:44 PST
I don't know if it's meant to, but the extension doesn't appear to work in Windows XP (at the very least, it complains that CompareStringEx can't be located in kernel32.dll, there may be other issues).
Comment 126 (dormant account) 2011-03-08 10:50:08 PST
(In reply to comment #125)
> I don't know if it's meant to, but the extension doesn't appear to work in
> Windows XP (at the very least, it complains that CompareStringEx can't be
> located in kernel32.dll, there may be other issues).

thanks. I'll fix that in the next rev.
Comment 127 Ed Morley [:emorley] 2011-03-08 19:21:43 PST
Using:
- Win7 x64 ; Minefield x86 opt 2011-03-03
- Core 2 Duo 2.4; WD Raptor 10000 RPM SATA150 WD1500ADFD
- H/W accel disabled (both layers and D2D)
- Session with 5 tabs open; 8 addons + flash

cold: (after waiting 60 secs for win services to finish loading)
main	2652
firstPaint	4977
sessionRestored	5164

warm:
main	62
firstPaint	1779
sessionRestored	1935

with new wrapper from comment 121...

cold: (after waiting 60 secs for win services to finish loading)
main	546
firstPaint	4540
sessionRestored	4837

warm:
main	172
firstPaint	1920
sessionRestored	2092
Comment 128 kolubinowicki 2011-03-09 15:54:54 PST
Gecko/20110308 Firefox/4.0b13pre
Windows 7 sp1 32bit
GPU: 1/1 direct3 d9
DirectWrite: off
HDD: 5400 rpm laptop drive WDC WD3200BEVT
2 Gb RAM
31 extensions
Restored session with: 510 tabs 
(entry in about:config browser.sessionstore.max_concurrent_tabs=0, see also Bug 632012, still have a hope that Bartab extension will be updated to fully unload tabs in Firefox 4.0 like it worked in Firefox 3.6 = session restore should be much more faster)
about one month old profile
+ database vacuum=after use SpeedyFox from http://www.crystalidea.com/speedyfox
+ 1Gb readyboost= using old (slow) 1 gb flash drive as readyboost cache in Windows 7 
_____________________________
Without Installed startup.service.xpi: 
_____________________________
	Cold boot

	main	2730
	firstPaint	5414
	sessionRestored	14946
	
	Hot boot

	main	125
	firstPaint	2076
	sessionRestored	11217

+ 1 Gb readyboost flash drive

	Cold boot

	main	78
	firstPaint	3527
	sessionRestored	12981

	Hot boot

	main	78
	firstPaint	2200
	sessionRestored	11389

+ database vacuum
	
	Cold boot
	
	main	93
	firstPaint	3776
	sessionRestored	13073

	Hot boot

	main	125
	firstPaint	2060
	sessionRestored	10905

+ database vacuum + 1Gb readyboost

	Cold boot

	main	93
	firstPaint	2793
	sessionRestored	12215
	
	Hot boot

	main	63
	firstPaint	1998
	sessionRestored	10953

_____________________________
With Installed startup.service.xpi, start by desktop shortcut: 
_____________________________

	Cold boot

	main	546
	firstPaint	4010
	sessionRestored	13277

	Hot boot

	main	140
	firstPaint	2091
	sessionRestored	11389


+ 1 Gb readyboost flash drive

	Cold boot

	main	484
	firstPaint	3480
	sessionRestored	12715

	Hot boot

	main	140
	firstPaint	2107
	sessionRestored	11404
Comment 129 (dormant account) 2011-03-09 15:59:10 PST
by the way, once http://support.microsoft.com/kb/2505438 is installed, keeping directwrite ON should result in faster startups(ie the opposite of before this microsoft released this patch).

Thanks for the benchmarks. I have enough data now.
Comment 130 (dormant account) 2011-03-09 16:19:27 PST
(In reply to comment #125)
> I don't know if it's meant to, but the extension doesn't appear to work in
> Windows XP (at the very least, it complains that CompareStringEx can't be
> located in kernel32.dll, there may be other issues).

Uploaded a new version to https://addons.mozilla.org/en-US/developers/addon/startupservice/ . Would be nice if someone could test it on xp.
Comment 131 rob64rock 2011-03-09 17:28:05 PST
(In reply to comment #130)
> (In reply to comment #125)
> > I don't know if it's meant to, but the extension doesn't appear to work in
> > Windows XP (at the very least, it complains that CompareStringEx can't be
> > located in kernel32.dll, there may be other issues).
> 
> Uploaded a new version to
> https://addons.mozilla.org/en-US/developers/addon/startupservice/ . Would be
> nice if someone could test it on xp.

I tried to access it using the new link above, but it has me log-in to addons.mozilla.org and then loads a blank page with this Url: https://addons.mozilla.org/en-US/developers/addon/startupservice/edit

No download or install of the extension occurs.
Comment 132 (dormant account) 2011-03-09 17:38:51 PST
(In reply to comment #131)

> I tried to access it using the new link above, but it has me log-in to
> addons.mozilla.org and then loads a blank page with this Url:
> https://addons.mozilla.org/en-US/developers/addon/startupservice/edit
> 
> No download or install of the extension occurs.

sorry, https://addons.mozilla.org/en-US/firefox/addon/startupservice/
Comment 133 MZ (watervole02) 2011-03-09 18:25:32 PST
I'm using the addon on XP and it seems to make startup faster. However, an error shows up on install.

svc.exe - Entry Point Not Found
The procedure entry point GetUserDefaultLocaleName could not be located in the dynamic link library KERNEL32.dll.
Comment 134 Csaba Kozák [:WonderCsabo] 2011-03-10 13:07:51 PST
(In reply to comment #130)
>
> Uploaded a new version to
> https://addons.mozilla.org/en-US/developers/addon/startupservice/ . Would be
> nice if someone could test it on xp.

I think you should write a more detailed description for the extension. Check out review #1 on the addon's page.
Comment 135 Shandy 2011-03-10 20:33:10 PST
- i got this error (picture) http://i56.tinypic.com/2vsp6vo.png and i believe it is related to the followings:

[ExtensionDirs]
Extension0=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{D4DD63FA-01E4-46a7-B6B1-EDAB7D6AD389}.xpi
Extension1=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}.xpi
Extension2=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{b9db16a4-6edc-47ec-a1f4-b86292ed211d}.xpi
Extension3=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{1018e4d6-728f-4b20-ad56-37578a4de76b}
Extension4=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{e4a8a97b-f2ed-450b-b12d-ee082ba24781}.xpi
Extension5=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{46551EC9-40F0-4e47-8E18-8E5CF550CFB8}.xpi
Extension6=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\compatibility@addons.mozilla.org.xpi
Extension7=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{EF522540-89F5-46b9-B6FE-1829E2B572C6}.xpi
Extension8=C:\Program Files (x86)\Mozilla Firefox 4.0 Beta 8\extensions\{CAFEEFAC-0016-0000-0022-ABCDEFFEDCBA}

[ThemeDirs]
Extension0=C:\Program Files (x86)\Mozilla Firefox 4.0 Beta 8\extensions\{972ce4c6-7e08-4474-a285-3208198ce6fd}

FYI: I'm not a programmer but i would like to help out, also i can't install this addon anymore after uninstalling it this morning (now it is at night 11ish pm)

thanks!
Comment 136 Shandy 2011-03-10 20:45:35 PST
(In reply to comment #135)
> - i got this error (picture) http://i56.tinypic.com/2vsp6vo.png and i believe
> it is related to the followings:
> 
> [ExtensionDirs]
> Extension0=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{D4DD63FA-01E4-46a7-B6B1-EDAB7D6AD389}.xpi
> Extension1=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}.xpi
> Extension2=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{b9db16a4-6edc-47ec-a1f4-b86292ed211d}.xpi
> Extension3=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{1018e4d6-728f-4b20-ad56-37578a4de76b}
> Extension4=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{e4a8a97b-f2ed-450b-b12d-ee082ba24781}.xpi
> Extension5=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{46551EC9-40F0-4e47-8E18-8E5CF550CFB8}.xpi
> Extension6=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\compatibility@addons.mozilla.org.xpi
> Extension7=C:\Users\FaTe\AppData\Roaming\Mozilla\Firefox\Profiles\rgn4r093.default\extensions\{EF522540-89F5-46b9-B6FE-1829E2B572C6}.xpi
> Extension8=C:\Program Files (x86)\Mozilla Firefox 4.0 Beta
> 8\extensions\{CAFEEFAC-0016-0000-0022-ABCDEFFEDCBA}
> 
> [ThemeDirs]
> Extension0=C:\Program Files (x86)\Mozilla Firefox 4.0 Beta
> 8\extensions\{972ce4c6-7e08-4474-a285-3208198ce6fd}
> 
> FYI: I'm not a programmer but i would like to help out, also i can't install
> this addon anymore after uninstalling it this morning (now it is at night 11ish
> pm)
> 
> thanks!

Win 7 (no SP1) x64 
- this error occurs when installation was completed and i try faster firefox still slow i believe it is because the installation was not successful
Comment 137 (dormant account) 2011-03-10 20:59:41 PST
Guess it's broken on xp and 64bit windows. Thanks
Comment 138 Ed Morley [:emorley] 2011-03-11 02:10:22 PST
Taras, v0.1 WFM using Win 7 x64 and x86 Minefield; so unless 0.2 regressed x64...
Comment 139 Aaron Kaluszka 2011-03-11 15:15:14 PST
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b13pre) Gecko/20110309 Firefox/4.0b13pre with Start Faster 0.2

I get a dialog titled "mozjs.dll" that says "Could not open" when running Faster Firefox. It loads once I hit OK.
Comment 140 (dormant account) 2011-03-11 15:43:59 PST
(In reply to comment #139)
> 
> I get a dialog titled "mozjs.dll" that says "Could not open" when running
> Faster Firefox. It loads once I hit OK.

Thanks. That's a bug, I'll fix that in next rev.

Shandy, in your screenshot, I can't see the end of the title in the message box. it should end in a number. Would be handy to know what the number is.
Comment 141 *I need to change my userid so it's less racist) 2011-03-11 16:44:10 PST
Extension activated, running Fx after the OS restart:
main	508
sessionRestored	11493
firstPaint	12212

Extension deactivated, running Fx after the OS restart:
main	3375
sessionRestored	10407
firstPaint	11079

P4 3.0Ghz, hwa=on.
Comment 142 (dormant account) 2011-03-11 16:49:26 PST
(In reply to comment #141)
> Extension activated, running Fx after the OS restart:
> main    508
> sessionRestored    11493
> firstPaint    12212
> 
> Extension deactivated, running Fx after the OS restart:
> main    3375
> sessionRestored    10407
> firstPaint    11079
> 
> P4 3.0Ghz, hwa=on.

Is this via faster firefox shortcut?
Comment 143 greybert- 2011-03-11 16:50:04 PST
tried to install ff4 xp sp3 turned off all addons before, on install get the following: svc.exe entry point not found
the proceedure entry point GetUserDefaultLocaleName could not be located in the dynamic link library KERNEL32.dll
addon appears to install but no short cut is created on desktop
Comment 144 Shandy 2011-03-11 18:34:25 PST
(In reply to comment #140)
> (In reply to comment #139)
> > 
> > I get a dialog titled "mozjs.dll" that says "Could not open" when running
> > Faster Firefox. It loads once I hit OK.
> 
> Thanks. That's a bug, I'll fix that in next rev.
> 
> Shandy, in your screenshot, I can't see the end of the title in the message
> box. it should end in a number. Would be handy to know what the number is.

i can't do anything about it =( caz it doesn't let me expand it >.<

sorry & hope you will get this fix soon <3
Comment 145 *I need to change my userid so it's less racist) 2011-03-12 03:06:12 PST
(In reply to comment #142)
> Is this via faster firefox shortcut?

Sorry, didn't notice that shortcut. But I've tried it now and it opens new profile, instead of my. I'm using firefox portable, and probably that's the reason. Not going to test not on my profile.
Comment 146 Shandy 2011-03-12 06:47:56 PST
FYI: the addon was ok until  http://support.microsoft.com/kb/2505438 is installed 

Win 7 (no sp1) x64
Comment 147 Virtual_ManPL [:Virtual] - (ni? me) 2011-03-12 08:29:08 PST
How about installing SP1 with all patches ? Still the same ?
Comment 148 Baraa 2011-03-12 10:26:41 PST
Hi,

My Build identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b13pre) Gecko/20110311 Firefox/4.0b13pre

I have win 7 x64 with sp1 with the mentioned patch installed, and the add-on seems to work fine.
Comment 149 Shandy 2011-03-12 19:43:13 PST
(In reply to comment #148)
> Hi,
> 
> My Build identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b13pre)
> Gecko/20110311 Firefox/4.0b13pre
> 
> I have win 7 x64 with sp1 with the mentioned patch installed, and the add-on
> seems to work fine.

i'm on Firefox RC1 and installed SP1 (took me long time) reinstalled the addon v0.2 still getting same error
Comment 150 (dormant account) 2011-03-14 09:27:16 PDT
there are some problems with reinstalling/upgrading. I'll see about fixing them in next version.
Comment 151 *I need to change my userid so it's less racist) 2011-03-31 17:16:30 PDT
another 2 weeks of ignorance, nice.
Comment 152 Csaba Kozák [:WonderCsabo] 2011-04-19 06:02:17 PDT
Taras, any news on this bug?
Comment 153 (dormant account) 2011-04-19 09:10:51 PDT
(In reply to comment #152)
> Taras, any news on this bug?

Nothing atm. I have a good idea of how to implement it, but we need a windows updating service before I can implement it(which is being worked on)
Comment 154 u88484 2011-04-19 10:11:42 PDT
Do you have a bug number we can follow?
Comment 155 (dormant account) 2011-04-19 10:18:08 PDT
wait on progress in bug 481815
Comment 156 Robert Strong [:rstrong] (use needinfo to contact me) 2011-04-25 09:40:25 PDT
I think with the current method to solve this bug being to just delete the firefox.exe prefetch files it would be simpler to just use a scheduled task. Taras, would this be acceptable?
Comment 157 (dormant account) 2011-04-25 09:49:58 PDT
(In reply to comment #156)
> I think with the current method to solve this bug being to just delete the
> firefox.exe prefetch files it would be simpler to just use a scheduled task.
> Taras, would this be acceptable?

Yes. I think a scheduled task + code in firefox.exe that checks for the firefox.exe* prefetch files before using our own prefetch logic would solve the after-reboot slowness.
Comment 158 Robert Strong [:rstrong] (use needinfo to contact me) 2011-04-25 16:13:05 PDT
Threw together a hacky vbscript as a proof of concept. You can save it to your filesystem, in the task scheduler create a new task, change the user account to run the task to SYSTEM, set the trigger to At system startup, and set the action to Start a program with the vbs file containing the following for the program / script.

On Error Resume Next
Dim wmi, fso, shell, prefetchdir, querydir, query, monevts, evt
Set fso = CreateObject("Scripting.FileSystemObject")
Set shell = WScript.CreateObject("WScript.Shell")
prefetchdir = shell.ExpandEnvironmentStrings("%windir%") & "\Prefetch"
If fso.FolderExists(prefetchdir) Then
  fso.DeleteFile prefetchdir & "\FIREFOX.EXE*", True

  querydir = Replace(prefetchdir, "\", "\\\\")
  Set wmi = GetObject("winmgmts:{impersonationLevel=impersonate}!\\.\root\cimv2")
  query = "SELECT * FROM __InstanceCreationEvent WITHIN 10 WHERE " & _
          "Targetinstance ISA 'CIM_DirectoryContainsFile' and " & _
          "TargetInstance.GroupComponent= " & _
          "'Win32_Directory.Name=""" & querydir & """'"
  Set monevts = wmi.ExecNotificationQuery(query)
  Do
    Set evt = monevts.NextEvent
    If fso.FolderExists(prefetchdir) Then
      fso.DeleteFile prefetchdir & "\FIREFOX.EXE*", True
    End If
  Loop
End If
Comment 159 (dormant account) 2011-04-25 16:21:40 PDT
(In reply to comment #158)
> Threw together a hacky vbscript as a proof of concept. You can save it to your
> filesystem, in the task scheduler create a new task, change the user account to
> run the task to SYSTEM, set the trigger to At system startup, and set the
> action to Start a program with the vbs file containing the following for the
> program / script.
> 

What does the part part "Set wmi" do? Is this to keep deleting firefox.exe* if it appears?

The only hard part left is registering this script with the task scheduler during Firefox update/install. Is that something that can be done for Firefox 6?
Comment 160 Brian Carpenter [:geeknik] 2011-04-25 16:22:36 PDT
I assume this will fail gracefully and not crash if the user is not using prefetch? My boot drive in Windows 7 is a 120GB SSD and I've turned off Superfetch and deleted the prefetch directory.
Comment 161 Robert Strong [:rstrong] (use needinfo to contact me) 2011-04-25 16:59:18 PDT
(In reply to comment #159)
> (In reply to comment #158)
> > Threw together a hacky vbscript as a proof of concept. You can save it to your
> > filesystem, in the task scheduler create a new task, change the user account to
> > run the task to SYSTEM, set the trigger to At system startup, and set the
> > action to Start a program with the vbs file containing the following for the
> > program / script.
> > 
> 
> What does the part part "Set wmi" do? Is this to keep deleting firefox.exe* if
> it appears?
It is a reference to windows management instrumentation. monevts.NextEvent will wait for the next create file event in the prefetch dir and then delete any FIREFOX.EXE* prefetch files.

> The only hard part left is registering this script with the task scheduler
> during Firefox update/install. Is that something that can be done for Firefox
> 6?
Likely

(In reply to comment #160)
> I assume this will fail gracefully and not crash if the user is not using
> prefetch?
yes
Comment 162 (dormant account) 2011-05-04 12:17:25 PDT
Created attachment 530115 [details] [diff] [review]
detect when prefetch is off(for whatever reason) and preload our way

Here is another WiP. Turns out we can detect prefetch.
Comment 163 (dormant account) 2011-05-04 12:35:16 PDT
Created attachment 530119 [details] [diff] [review]
detect when prefetch is off(for whatever reason) and preload our way

Lets land a conservative version of this and track progress via our startup-speed tracking.
This uses GetProcessIOCounters as an indicator whether prefetch is ruining our startup or whether we still have room to optimize it ourselves. This only preloads xul.dll and mozjs.dll(if it exists) because each of these has more overhead than the other files combined.

This will make startup faster for everybody who has broken windows prefetch(which seems relatively common), is running on ssd or uses another method to opt out (ie Rob's script).
Then as step two we can install Rob's script from the installer and enjoy awesomer startup times on every machine.
Comment 164 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-05-04 12:52:43 PDT
Comment on attachment 530119 [details] [diff] [review]
detect when prefetch is off(for whatever reason) and preload our way

Review of attachment 530119 [details] [diff] [review]:

::: toolkit/xre/nsWindowsWMain.cpp
@@ -93,1 +93,13 @@
-#endif  
+#endif
+
+#ifdef XRE_PRELOAD_XUL
+static void
... 9 more ...

pick a style for bracing single-line consequents, and stick to it please.  (I think the rest of this file doesn't brace single lines, but I'm sure you can tell at a glanc

@@ +104,5 @@
+    return;
+  }
+  
+  DWORD dwBytesRead;
+  DWORD total = 0;

total appears to not be used other than to accumulate read sizes and then be discarded.

@@ +108,5 @@
+  DWORD total = 0;
+  // Do dummy reads to trigger kernel-side readhead via FILE_FLAG_SEQUENTIAL_SCAN.
+  // Abort when underfilling because during testing the buffers are read fully
+  // A buffer that's not keeping up would imply that readahead isn't working right
+  while (ReadFile(fd, buf, sizeof(buf), &dwBytesRead, NULL)

this would fit on one line, no?

with the total-addition removed, you'd want something like /*nothing*/; as the one-line body I think.
Comment 165 (dormant account) 2011-05-04 14:52:37 PDT
Created attachment 530154 [details] [diff] [review]
style fixes
Comment 166 (dormant account) 2011-05-06 12:09:15 PDT
Created attachment 530687 [details] [diff] [review]
telemetry probes

These are a bit ugly since we need to capture the stats early during startup(before xul.dll is loaded) and pass em into xul.dll.
Comment 167 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-05-06 12:43:55 PDT
Comment on attachment 530154 [details] [diff] [review]
style fixes

Review of attachment 530154 [details] [diff] [review]:
-----------------------------------------------------------------

I gave you r+ before, didn't need to re-review!  But while I'm here...

::: toolkit/xre/nsWindowsWMain.cpp
@@ +117,5 @@
> +{
> +  wchar_t buf[MAX_PATH];
> +  size_t pathlen = wcslen(appPath);
> +
> +  for(;pathlen && appPath[pathlen] != L'\\'; pathlen--);

this is pretty opaque, can you switch to something that's not pure side-effect, like:

while (pathlen && appPath[pathlen] != L'\\')
  pathlen--;

?

r+ with that, or without if you feel strongly.
Comment 168 (dormant account) 2011-05-06 12:52:20 PDT
Comment on attachment 530687 [details] [diff] [review]
telemetry probes

Benjamin, requesting your review here since you might have a suggestion on how to better pass the startup io data into xul.dll
Comment 169 (dormant account) 2011-05-06 12:59:25 PDT
http://hg.mozilla.org/mozilla-central/rev/c4b4470534b4
Comment 170 Florian J. [:FeuerFliege] 2011-05-07 04:36:56 PDT
hi, my last build takes ages to start, so I investigated the latest changesets and found this one. Is it possible that this is the reason for the huge delay (~7sec)? My OS is Win7 64bit.

I am now building two firefox versions. One short before this patch landed and the other with this patch. How do I compare these versions best? After a reboot with the about-startup extension?
Comment 171 (dormant account) 2011-05-07 19:44:38 PDT
(In reply to comment #170)
> hi, my last build takes ages to start, so I investigated the latest
> changesets and found this one. Is it possible that this is the reason for
> the huge delay (~7sec)? My OS is Win7 64bit.
> 
> I am now building two firefox versions. One short before this patch landed
> and the other with this patch. How do I compare these versions best? After a
> reboot with the about-startup extension?

yes it's possible. That's the way to test.
Comment 172 Florian J. [:FeuerFliege] 2011-05-08 04:04:05 PDT
No, it is not this patch:

before patch:

main    sessionRestored	firstPaint
483     2341            2138
477     2759            2493
452     2684            2435

after:

main    sessionRestored	firstPaint
718     2450            2232
702     2387            2169
736     2569            2351


The reason must be somewhere else. My latest build gave me this results:

main    sessionRestored	firstPaint
3508    10566           10359
7399    17492           17088	
3591    19871           19510
Comment 173 (dormant account) 2011-05-08 14:38:11 PDT
Florian,
Try capturing an xperf profile. See http://blog.mozilla.com/tglek/2010/10/04/diagnosing-slow-startup/ for instructions, then email it to me.

I can take a look at it. I wonder if we hit a font-enumeration regression.
Comment 174 Benjamin Smedberg [:bsmedberg] 2011-05-09 12:27:49 PDT
Comment on attachment 530687 [details] [diff] [review]
telemetry probes

As noted in IRC, I'd prefer a little bit that this all live in libxul: we can avoid passing the static from firefox.exe in by moving recordstartupcounters into XRE_main and calling GetProcessIoCounters again. remark r? if that's not a good idea for some reason.
Comment 175 (dormant account) 2011-05-09 15:21:45 PDT
Comment on attachment 530687 [details] [diff] [review]
telemetry probes

(In reply to comment #174)
> As noted in IRC, I'd prefer a little bit that this all live in libxul: we
> can avoid passing the static from firefox.exe in by moving
> recordstartupcounters into XRE_main and calling GetProcessIoCounters again.
> remark r? if that's not a good idea for some reason.

This patch reports data that we base our decision to preload xul.dll on.

We can't move the GetProcessIoCounters. It has to happen early on in firefox.exe before other .dlls started loading.
Comment 176 (dormant account) 2011-05-11 12:32:24 PDT
Comment on attachment 530687 [details] [diff] [review]
telemetry probes

Looks like the way forward is to incorporate this into Mike Hommey's xpcomglue preload
Comment 177 Mike Hommey [:glandium] 2011-05-16 01:01:44 PDT
Let's say this is closed by the landing. The patch from here will be backed out before landing bug 552864 and bug 632404. The latter will reintroduce the windows dll preloading.

I'll file a new bug for the telemetry probes.
Comment 178 (dormant account) 2011-05-20 11:35:27 PDT
backed out this preload http://hg.mozilla.org/mozilla-central/rev/a5165f98aefb http://hg.mozilla.org/mozilla-central/rev/38aaa0624674

bug 552864 has a better approach
Comment 179 Daniel Gutson 2011-09-29 05:54:59 PDT
>while (ReadFile(fd, buf, sizeof(buf), &dwBytesRead, NULL) && dwBytesRead == sizeof(buf))


shouldn't be dwBytesRead < sizeof

 ?
Comment 180 RFOG 2011-09-29 08:57:26 PDT
Code like this:

+  while (ReadFile(fd, buf, sizeof(buf), &dwBytesRead, NULL) && dwBytesRead == sizeof(buf))
+    /* Nothing */;

Is a very bad practice in Windows development, despite the new way to do threading Windows has.

First of all current thread is locked until while loop ends. I do not know if this is done in a separated thread or in main one. If it is in main, it locks all UI and message loop until it finishes. Being in a separated thread it locks the thread and dramatically slow down program performance because CPU goes to 100%.

Best way to solve this is using Completion Ports (http://msdn.microsoft.com/en-us/library/windows/desktop/aa365198(v=vs.85).aspx) and/or Overlapped CreateFule way to read a file (http://msdn.microsoft.com/en-us/library/windows/desktop/aa365683(v=vs.85).aspx).

Only if code is in a different thread than main one, it is necessary to add a Sleep(0) or Sleep(1) into the loop (or another value, but those have special meaning in Win32). 

If yo specify Sleep(0) you are telling Windows do only the most urgent tasks and return to the loop. Win Sleep(1) you tell Windows: do all the urgent tasks and process some thread message loop messages.

This is a server performance problem. If you intend to preload some DLL into memory, you are locking Firefox until all the files are loaded if this code is in main thread, and dramatically slow down it if it is in a secondary one. This code is inefficient and forces Windows not use its internal DLL and file caching schemes.
Comment 181 Siddharth Agarwal [:sid0] (inactive) 2011-09-29 09:06:56 PDT
(In reply to RFOG from comment #180)
> Code like this:
> 
> +  while (ReadFile(fd, buf, sizeof(buf), &dwBytesRead, NULL) && dwBytesRead
> == sizeof(buf))
> +    /* Nothing */;
> 
> Is a very bad practice in Windows development, despite the new way to do
> threading Windows has.
> 
> First of all current thread is locked until while loop ends. I do not know
> if this is done in a separated thread or in main one. If it is in main, it
> locks all UI and message loop until it finishes. Being in a separated thread
> it locks the thread and dramatically slow down program performance because
> CPU goes to 100%.

Please understand the context of the code before commenting on it. This runs right at startup (notice the wmain?) on the main thread before any UI is displayed.

> 
> This is a server performance problem. If you intend to preload some DLL into
> memory, you are locking Firefox until all the files are loaded if this code
> is in main thread

Yes, that is the point of the code.

> This code is inefficient and forces Windows not use its internal DLL
> and file caching schemes.

We've found that Windows' file caching is broken. Please read <https://blog.mozilla.com/tglek/2011/09/20/firefox-7-cheating-the-operating-system-to-start-faster/> and all of Taras's other posts before commenting again.
Comment 182 (dormant account) 2011-09-29 09:40:42 PDT
(In reply to Daniel Gutson from comment #179)
> >while (ReadFile(fd, buf, sizeof(buf), &dwBytesRead, NULL) && dwBytesRead == sizeof(buf))
> 
> 
> shouldn't be dwBytesRead < sizeof
> 
>  ?

This is intentional. This loop exists solely to populate the windows filesystem cache. Typically this loop succeeds(ie does full reads for the whole file). Assumption is that if there is an under-read, then something weird is going on and we should abort our heuristic to not risk introducing a slowdown.

Note You need to log in before you can comment on or make changes to this bug.