76004 - Check in the preloader

Status: VERIFIED FIXED

(SeaMonkey :: Build Config, defect)

Description

[dprice (gone)]

get the preloader checked in.

Comment 1

[dprice (gone)]

I've got the code from rick, wedged it into my mozilla/extentions directory, and
hooked it into the build system.  Looks like it is ready to be checked in.  I'll
attach everything then get a super reviewer

Comment 2

[dprice (gone)]

Attachment: zip of the new preloader source code

Comment 3

[dprice (gone)]

Attachment: diff of changes to mozilla\extensions\makefile.win

Comment 4

[dprice (gone)]

Attachment: diff -uN at waterson's request

Comment 5

[cls]

I have no idea what this does but if it's going to be on by default, is
extensions/ really the place for it?

Comment 6

[dprice (gone)]

I'm open to suggestions for a different and better home.

Comment 7

[cls]

modules/ ? I don't know.  If we're really going to scrap the idea of making
extensions/ be optional (a la bug 18352), then let me know and I'll stop giving
Morse a hard time about cookie & wallet (well, ok, probably not).  

Comment 8

[dprice (gone)]

I would view the preloader as something that is optional for building mozilla.
It creates a separate executable that doesn't rely on anything else in the tree.
 If someone were to remove their extensions directory and build they simply
wouldn't get the advantage of a preloader.  Sounds to me like it is in line with
18352

The only thing that would really depend on the existance of the preloader would
be actually building an installer package.  But that's something that happens
post build.

-- adding waterson to CC, he's providing the sr=

Comment 9

[Brendan Eich [:brendan]]

modules is deprecated (it originated in the post-Netscape 3.0(!) "modularity"
jihad).  Why not mozilla/tools rather than mozilla/extensions?

/be

Comment 10

[Chris Waterson]

1. Use *m*akefile.win, with a lower-case `m'.

2. Add MPL. Also, are there copyright issues? Ask mitchell@ and/or
   [e-mail] about...

+Portions derived from TrayIcon.exe by Antti Markus ([e-mail]).

   Mail 'em the license from trayicon.cpp

3. This might get optimized away; make it do something silly like pass `*p'
   to itoa() or something.

+      if (p && (*p)){  
+        //don't actually invoke it, just cause it to load into memory...
+
+        //note that modules have different number of entry points.
+        //so not all modules have an entry point at theCurrentProc index.
+      }

Do that, and sr=waterson

Comment 11

[Brendan Eich [:brendan]]

As rjesup suggested, declaring p to be volatile long * should ensure that (*p)
loads from it.

/be

Comment 12

[dprice (gone)]

I'll go ahead and move it into mozilla/tools/preloader and hook it into the
build.  Diffs are forthcoming.

I've changed it to volatile long *p;

mitchell doesn't believe we can take the trayicon.cpp code with its current
license.  I'm working with her to find ways to get it in.

Comment 13

[dprice (gone)]

Attachment: diff to build preloader in mozilla/tools

Comment 14

[rickg]

I can trivailly rewrite the trayicon code. I used this version as a starting 
point for expediency sake. I'll do that with my next update. Don't bother with 
the license.

Comment 15

[chris hofmann]

did this make it in last night?

Comment 16

[dprice (gone)]

unfortunately no. :(  The legal issues stopped it.

Comment 17

[chris hofmann]

moving off the 0.9 milestone since it won't make the release builds there.
dprice can you update the bug with rickg last e-mail comments.

I'm not sure the options for tunning are the highest priority.
we need to just get an agressive fast loader in the hands of
a lot of folks to start testing.   options to tune can come 
later.

Comment 18

[dprice (gone)]

Attachment: new version without gpl code

Comment 19

[dprice (gone)]

I've attached a new version for super review.

All of the code from trayicon.* has been removed and replaced by new code in
preloader.cpp

Comment 20

[chris hofmann]

Attachment: itty-bity binary for testing...

Comment 21

[chris hofmann]

go play!  lets lots of folks starting to run the preloader
figure out anwsers to some of the questions below.

In addtion in some of my first sessions when I ran the preloader
I was scratching my head after some crashes and wondering if
the increase in number of zombie process I needed to kill off
to start correctly might be due to the preloader trying to keep
things in memory...

chris h.

David Price wrote:
> 
>      Attached is a new version of preloader.exe  The arguments are:
> 
> -p path  specify the location of the install (c:\program
> files\netscape\netscape 6 in my case)
> -m       use the mozilla modules list
> -n       use the netscape 6.5 modules list
> -f       use the complete modules list

jrgm wrote:
Just to save some others some time, note that the order of the switches must 
be -n then -p. e.g., 

  C:\preloader\preloader.exe -n  -p E:\mozilla-bin\2001-04-18-09-trunk

The reverse 'exe -p path -n' doesn't work. 

John 
>    Make sure that you specify the path.  The code that pulls the path
> and module list from the registry doesn't work very well.  For example,
> if you specify -m it will only check for .8.  Toss the .exe into your
> startup folder.  Edit the shortcut to include the -p option.  Reboot.
> After you log in, watch the task bar.  Once the mozilla icon comes out
> of its fits, check with taskinfo 2000.  It should list a lot of open
> modules for preloader.exe  We've confirmed that preloader.exe has gotten
> everything into memory.  Go ahead and measure stuff.  If it isn't
> loading modules correctly, just right-click on the icon and exit.  Then
> just run it from the command line.
> 
>    Questions people have asked:
> 
>    Is the overall system slower while the preloader is running?  Not for
> me but I've got a fun computer.
> 
>    Will this have impact on something that's sensitive to system
> performance, like a CD writer?  I dunno.
> 
>    John's also concerned that if we try to start mozilla too soon, we'll
> get slower startup times because there's still noise OS related
> startup.  Perhaps it is a good idea to wait for a minute or two?  My
> feeling is that if the user tries to start too soon after login, they'll
> attribute the general slowness of startup to the OS anyway.
> 
>    Finally, it's very aggressive.  I've set it to ping 64 functions in
> each module every 50 milliseconds. :)
> 
> Enjoy
> Dave

Comment 22

[rickg]

Couple of points for folks:

-m tells the preloader that it's ONLY supposed to assume a mozilla build.
-n tells the preloader that it's a netscape install, and is the default. You 
shouldn't have to specify this flag (but you can).

I can't imagine why order of the args would matter, but I'll take a look.
My next update will include a UI to tune the preloader, and to stop if from 
doing any work once mozilla/netscape are running.

Comment 23

[Alex]

I just want to verfy the behavior on the preloader: the first time Mozilla is
launched with the preloader running, there doesn't seem to be an affect on the
startup time. But afterwards, Mozilla launches much faster.

Comment 24

[Hong Kwon]

Doesn't Mozilla launch faster anyway on the second relaunch (even without the 
preloader)?  Either I'm not configured correctly or the preloader is having no 
effect for me (other than some minor amusement as Moz spasms in the taskbar).  
Just counting the seconds, my startup times are about the same as those 
documented at 
http://www.mozilla.org/quality/perf/browser-archive/launch-results.html with or 
without the preloader installed.  I'm using the following command line:

preloader.exe -n -p c:\program files\netscape\netscape 6

Minor problem: more than one instance of preloader can be launched at once.  
Probably want to prevent that.

The configuration UI doesn't seem to be implemented, as per rickg's comments and 
the fact that none of the values stick in the UI.  

Comment 25

[rickg]

I don't know how your preloader is configured. But it's performance can vary 
(based on how it's tuned) from having little/no effect to having a really big 
effect. I'll be publishing a new version that let's you configure the preload 
via a GUI from the system tray.

Also -- the new version will prevent multiple instances. Thanks for the 
reminder.

Comment 26

[:deanis74]

Minor nits:

1. The tray context menu needs the TPM_RIGHTBUTTON flag.
2. The tray context menu needs access keys, as does the configuration dialog.
3. The tip for the tray icon reads "Click to configure...", but you have to 
right-click.

Comment 27

[dprice (gone)]

**)#)&#%)#&%)*#% tree.  I don't have permission to check into mozilla/tools.
Getting that.

Comment 28

[rickg]

Dean: Thanks for the comments; I've implemented all your suggestions.

Comment 29

[dprice (gone)]

the code was checked in to mozilla/tools/preloader  changes to the global
makefile.win will be checked in when the tree opens for .9.1
anyone should be able to check out that directory and nmake -f makefile.win to
build their own preloader.

hong:  The big win for the preloader come with the first startup

Comment 30

[dprice (gone)]

Attachment: latest and greatest preloader

Comment 31

[dprice (gone)]

latest and greatest version of the preloader is checked in.  attached an
executable to this bug.

Comment 32

[Alec Flett]

what format is the latest version in? I can't open it with zip...

Comment 33

[Alec Flett]

nevermind, I found it in the tree
for future reference, it's in tools/preloader

Comment 34

[David Ascher]

Also FYI, if you save the attachment called 'showattachment.cgi' or some such 
and rename it preloader.exe, it works. =)

Comment 35

[chris hofmann]

hamerly is signed up to help testing on this.. ;-)

lets get some test results posted with the latest experimental builds
and figure out the next set of areas to tune or places to fix.

Comment 36

[hamerly]

config: Win 98 Dell 500 MHz laptop / 128 MB memory / slow disk
Build ID: 2001043504, verified with 2001050204. content is about:blank

        without preloader   with preloader  improvement
cold       16.9 secs          12.5 secs        26%
warm        8.0 secs           6.2 secs        23%                              

cold = startup time immediately following reboot
warm = startup time following multiple launches

Comment 37

[selmer (gone)]

I ran this on my win98, 350 Mhz, 256 Mb machine using two different
configurations of the preloader and didn't find the benefit.  What I noted is
that the little gecko tray icon flashes a little during bootup but that
launching the app causes the disk to grind bringing in the DLLs (or whatever.) 
I used the Max settings on the modules and entry points w/ medium frequency,
then I used medium settings for all configurations with basically the same results.

The fact that the disk had to grind as much as it did when I launched the app
seems to indicate that the DLLs had not yet been brought into memory by the time
I launched.  The only explanation I can think of for that is that the preloader
is not aggressively trying to touch *everything* when it is just started up. 
This may be the bulk of the variance seen between runs and machines.

For this tool to fulfill its purpose, it needs to bring in *everything* during
the bootup.  Only after that has happened is it OK to fall back to maintaining
their presence via some timed algorithm.  

My experience makes it clear that this is not happening.  I waited for the
bootup activities to finish, for the gecko to stop flashing, and for the disk to
stop grinding before launching and I *still* got disk grinding when I launched.
 All that disk grinding needs to happen during the bootup!!

Comment 38

[rickg]

The preloader loads the percentage of modules indicated by the modules config 
setting in the UI (you can prove this by using a tool like taskInfo2000 to see 
the modules are in fact loaded). What the preloader doesn't do is load the data 
pages whereas the browser does. That's why you hear more grinding when the 
browser runs.

We can discuss ways that the preloader might load data pages as well (it's 
harder than it seems). I'm looking into that and other techniques to 
further improve the preloader.

Can you do two things for me? First, get taskinfo2000 and install it. Configure 
the preloader to load all the modules and then restart your 
machine. After the preloader runs, launch taskinfo2000 and look at the list of 
modules under taskbar2.exe. That should be a fairly complete 
set of mozilla modules. Second, please get your stopwatch out and try launching 
with and without the preloader.

Rick

Comment 39

[selmer (gone)]

Rick, I timed my runs this morning and got ~20 seconds both times with the
preloader running on first launch after reboot.  That's just too slow for my
configuration.  If the data is the main part of this, then we're going to need
that in the solution. 

taskinfo2000 looks pretty cool and I'll try that when I get a chance.  I doubt
my results will contradict you.  Why would I spend so much more time bringing in
data than you though?  (I presume I can compare the module list of the preloader
to the netscp6.exe?  BTW, I took my preloader from dprice's email that said it
had all the combined changes and it called itself preloader.exe.  Do I have the
wrong version?)

While using the preloader, I discovered that the configure dialog's cancel
button sets all the values back to "Min".  Are you tracking preloader bugs here
or somewhere else?

Comment 40

[rickg]

You might have the wrong version. Everyone that has reported data in the last 
few days has seen a 20-30% speedup. Since you don't you may have an older 
version. 

As for the name, he checked the code into the mozilla tree, and the output of 
his makefile is preloader.exe. I'm still using my project file which creates an 
app calle taskbar2.exe. Same code though.

Please add bugs here. I found the min problem earlier today and it's fixed in my 
tree. Also, I'm currently looking into other ways that the preloader can improve 
time.

Rick

Comment 41

[rickg]

I just had a really scary idea that I'm going to try out:

I know the VM codepage size (4K or 8k on alpha). If I can get the base address 
of each DLL in memory, I can compute my own pointers -- one for each codepage. 
By derefererencing that ptr (as I do now) I might be able to cause both 
executable code pages and *data* code pages to load (today I only get the exe 
pages).

This will either work beautifully or crash in a spectactular way. 
In either case, it's going to be fun to see. :)

Comment 42

[selmer (gone)]

Rick, could you cross that idea with the current one and only keep an entrypoint
in the list if it's not in the codepage of an entrypoint already in the list?

Also, if your theory is correct, getting the data pages would be the bigger win...

Comment 43

[selmer (gone)]

Preloader bug:  The configure sliders do not have any markings.  This makes it
impossible to know what the settings are or to duplicate old settings after
making changes.

Comment 44

[selmer (gone)]

Attachment: DLLs N6.5 loads that Preloader does not...

Comment 45

[selmer (gone)]

Used taskinfo2000 to collect DLLs loaded by both N6.5 and Preloader on my win98
350 Mhz 256 Mb machine.  Sorted and diffed into previous attachment.  The list
did not change when I bumped all configurations up to max (except the mouse
driver came into the preloader.)

How do I clear my configuration changes?  I tried deleting the program and
downloading a new one, but it remembered my settings.  Are the values in the
registry somewhere?

Comment 46

[selmer (gone)]

Attachment: DLLs N6.5 loads that Preloader w/default settings does not (w95, 350Mhz, 256Mb)

Comment 47

[selmer (gone)]

Ah, found configuration registry info in the dependent bug: 
HKLM/SW/NS/NS6.5/Main/Settings.

Looks like the modules list is not optimized for commercial builds.  Perhaps
this explains all the variance.  (I bet fullsoft is pretty big all by itself :-)

Give me a version that covers all the DLLs shown in my diffs and I'll try this
again.

Comment 48

[selmer (gone)]

Come to think of it, my list is probably incomplete too.  The list of DLLs needs
to be generated from a fully bulked up installation.  My install doesn't have
anything from the second screen of the installer except talkback.  The full list
should be generated by taking EVERYTHING and then using taskinfo2000 to get the
list of modules...

Comment 49

[:deanis74]

Is that registry key customizable?  Obviously, it would make more sense to 
store under Mozilla in the registry for Moz builds.

Comment 50

[selmer (gone)]

I didn't grok all the code.  You should have a mozilla key under software, is it
under there?

Comment 51

[dprice (gone)]

there's a bug with the preloader's module lists.  If you use -p to specify the
path that the modules are loaded from, it uses the 6.5 module list in all cases.
 This could be causing errors like "why is it not loading this module when the
slider is up at full?"

Comment 52

[selmer (gone)]

I just put the preloader in my startup folder and didn't pass any parameters
into it.  How can I tell which module list it tried to use?  (I want it to use
the N6.5 list.)

Comment 53

[rickg]

Attachment: zip file containing preloader and project file...

Comment 54

[rickg]

Attachment: Latest preloader binary...

Comment 55

[Alec Flett]

can you check in the latest changes to the source as well? That way we can all
help work on it.

Comment 56

[rickg]

Alex, I don't have time tonight to merge the changes. But I DID just post the 
source to this bug. 

Here's the latest update of the preloader.exe. The following changes have been 
made:

1. A new commandline argument has been added (-r) which removes the preloader 
registry settings.
2. The configuration UI has new fields that allow you to independently preload 
mail or AIM.

Note: You MUST run preloader.exe -r once so that your old registry settings get 
cleared out.

Next up: I'm working on preloading data pages, which I expect to significantly 
improve performance. Then I'll try preloading JAR files to see 
what effect that has. We should know a lot more by next monday.

Rick

Comment 57

[Keyser Sose]

Not sure if this has been mentioned yet but i wastalking with my friend and he
mentioned he would like the ability to turn off the icon so there was no visible
signs that the preloader was running. Guess the idea is to keep the taskbar
clean. At least an option to do that seems like a good idea to me.

Comment 58

[Matthew Tuck [:CodeMachine]]

If this is done then running the preloader a second time should tell the user
the preloader is loaded and ask whether they want to unload it.

Comment 59

[timeless]

yeah, the spec for w2k says that the tray is for notification.  And WinXP 
starts to enforce that spec.

Comment 60

[hamerly]

preloader3 (20010511) with latreste build:

config: Win 98 Dell 500 MHz laptop / 128 MB memory / slow disk
Build ID: 2001051104

        N4.X    IE5.5  6.5 w/o preloader with preloader  improvement
cold                      15.9 secs      12.0 secs         25%
warm     2.6    4.5        7.5 secs       6.2 secs         17%

cold = startup time immediately following reboot
warm = startup time following multiple launches

Comment 61

[rickg]

Attachment: new update that doesn't require any changes to mozilla

Comment 62

[rickg]

The newest update adds dynamic loading of JAR files, and it can load data pages 
as well as code pages from each module. In tests of my 4 machines, average 
loadtime with the preloader is 36% faster than without it. I've even seen it 
run as fast as 44% faster (a big machine with lots of ram).

I'll land the code over the weekend.

Comment 63

[selmer (gone)]

Could we get a new executable to match the sources posted today?  

Comment 64

[selmer (gone)]

OK, I got the executable via email.  On my win98 350Mhz 256Mb machine I see an
improvement from 20 seconds using old preloader to 15 seconds using new
preloader with all default options except using 8192 for page size.

The preloader seems to ignore the HKLM/SW/NSCP/Netscape 6/CurrentVersion key and
always puts its info in the 6.0 (en) subkey if it exists (or something.)  I had
to delete all the registry info for versions other than 6.5b1 (en) to get
preloader to do the right thing.  (And confirmed it via taskinfo2000.)

There is one very large file still missing from the preloader:  component.reg. 
This is the third largest file used by the app, so it deserves to be preloaded
as well.

Comment 65

[selmer (gone)]

My win98 350Mhz 256Mb non-preloader start time is 22 seconds.  Latest preloader
therefore gives me ~32% speedup on cold start.

Comment 66

[selmer (gone)]

I'm not seeing any warm start improvement - 8 seconds when preloader running or not.

Comment 67

[dprice (gone)]

pdt wants this in .9.2 

Comment 68

[Ben Goodger (use ben at mozilla dot org for email)]

I'm trying to test this, so far without luck, so a few comments on aspects of
the preloader not related to its performance:

UI:
1) In commercial builds, the Preloader should have a Netscape icon, similar to
the AOL tray icon. Currently, the icon in the system tray looks like an
unidentifiable collection of pixels that moves about randomly when there is
activity ;) 

2) The tooltip for the preloader reads 'Right click to configure the preloader'.
Preloader for what application? It should say (in Commercial builds) 'Right
click to configure Netscape Startup Agent' or similar, with "Mozilla" in Mozilla
builds. 

3) (Ideally) clicking the preloader once would launch a browser window. I
believe this is what the AOL startup agent does. 

4) Needless to say, the options in the configuration dialog are suitable for
testing, but not suitable for end users. We need to make sure this is tidied up
before we release an RTM. 

5) The dialog with the text "Are you sure you want to disable the preloader,
after all our hard work?" needs to be removed, or the text revised. 

6) The "Preload" and "Path" fields in the configuration dialog were disabled. I
had to manually edit my registry in order to set the Path from an incorrect path
to a correct path. 

7) For Mozilla builds, AIM is irrelevant and should not list itself as an option.

8) Java should not be enabled as a preload item by default. Work was done to
ensure that Java was only launched when a Java applet was found on a webpage,
and preloading it here defeats that. 

Comment 69

[dprice (gone)]

Attachment: new exe with registry fixed

Comment 70

[dprice (gone)]

I spoke with ssu about the registry keys that locate installation path of
netscape/mozilla.  He told me they were switched from HKLM to HKCU because
'regular' win2k and NT users cannot always modify keys in HKLM.  The latest
execuabtle looks in HKCU

Comment 71

[timeless]

correct protocol is to try HKLM first, if that fails, use HKCU.

Comment 72

[Cathleen]

from today's meeting...

selmer also wanted to get component.reg as a pre-loaded data file.

Comment 73

[:deanis74]

What about the .exe?  I couldn't see anywhere in the source that this is 
preloaded.

Comment 74

[dprice (gone)]

sliding this to the future

Comment 75

[:deanis74]

I'm confused.  I thought this was the -turbo startup parameter, which is already 
functional.

Comment 76

[dprice (gone)]

There were a number of different preloaders.  One from law which is the -turbo
mode, one for rickg which brings popular pages into memory.  This bug refers to
rick's.

Comment 77

[dprice (gone)]

I'm resolving this one as fixed.  The code has been checked in.  The only
remaining task is to hook this into the build system.  But this is not the
preloader we're going with.  Having everyone build it seems like it would waste
a lot of cycles.  It is in the tree for anyone who wants to use/experiment with it.

Comment 78

[Jon Granrose]

verified.