Closed Bug 7249 Opened 21 years ago Closed 20 years ago

Run rebase.exe to enhance performance on debug builds

Categories

(SeaMonkey :: Build Config, defect, P3)

x86
Windows 95
defect

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: leger, Assigned: leaf)

Details

(Keywords: perf)

Can we run rebase.exe, which is a post-processing build utility which "rebases"
a set of dlls so that they can load without having conflicting base addresses.
Troy or Phil knows more about this.  This could be a big win when loading dlls.
Whiteboard: [Perf]
Target Milestone: M7
Assignee: cyeh → leaf
so does rebase.exe come with the standard msft tools?

leaf: check in with phil about what's required.
Yes, it's part of the msdev tools. Phil thinks that you can just point it at a
bunch of dlls and let it work its magic. I'd look at what 4.5 was doing as a
starting point.
I should say that I'm not 100% sure we're not running it already on release
builds. I don't think we are, but we could be.
Status: NEW → ASSIGNED
So, i'm trying out rebase.exe today... it tells me i have to specify either the
Initial base address to each dll, or give it a file which will tell it what base
address to use. I'm not sure i know what values to give...
can someone (bienvenu) lend a hand as to what base address we should be using?
sorry, I have no clue (if I did, I would have volunteered to do it myself).
What did 4.x use? I'll poke around a little bit, but someone with a clue (Phil?
Troy?) should really provide the answer.
Here are the exciting lines from winfe\mkfiles32\mozilla.mak:

REBASE=rebase.exe
!if [for %i in (. %PATH%) do @if exist %i\$(REBASE) echo %i\$(REBASE) >
rebase.yes]
!endif
!if exist(rebase.yes)
!if [for %i in ($(OUTDIR)\*.dll) do @echo %i >> rebase.lst]
!endif
!if [for %i in ($(OUTDIR)\java\bin\*.dll) do @echo %i >> rebase.lst]
!endif
!if [for %i in ($(OUTDIR)\spellchk\*.dll) do @echo %i >> rebase.lst]
!endif
!endif

rebase:
!if exist(rebase.lst)
	$(REBASE) -b 60000000 -R . -G rebase.lst
	del rebase.lst
!endif
!if exist(rebase.yes)
	del rebase.yes
!endif
okay, so i went and looked at mkfiles32/mozilla.mak and boned up on the usage of

rebase. for it to be most effective, you would want to do this with a group of

dll's that load together so that they don't have conflicting base addresses.



it's not clear to me if we should do this with each dll or not, and what kind of

side effects that will result if we do this on components etc.



my suggestion is this: someone should go try this and see if you get a perfomance

win. the way you can do this outside of a makefile is to do the following:



create a file that has a list of dll's that you want to rebase together. (it's

important that the list be complete, so that rebase can juggle all of the dll's

together.) put it in a file named "rebase.lst"



Then do this:



rebase.exe -b 60000000 -R . -G rebase.lst



If this results in a speed improvement, great, let's do it. But I'm leary of just

applying tools to dll's because they worked under the mozilla classic codebase.

for all we know right now, running rebase might slow down the code.
Further questions: in the self-registering component world, is rebase going to
do what we think it will? I can rebase all the .dlls that are parallel to
apprunner.exe (which are not of the self-registering variety, presumably).

I still need someone that can do performance testing.
cyeh wrote:
> for all we know right now, running rebase might slow down the code

I can't think of a single reason why that would be. Why do you say that?

leaf wrote:
> in the self-registering component world, is rebase going to do what we
> think it will?

Yes, I think so. What does self-registration have to do with rebasing? The DLL
still has to get loaded by the OS loader, and the loader still has to decide
whether to leave the DLL at the address is was loaded at, or relocate it (with
performance degradation) someplace else.
Target Milestone: M7 → M8
lets get some development testing on this early in m8.
too late for the m7 train
I hear troy's back, so I thought I'd cc: him on this bug to say "hiya!"

/be
so does it save time?
and would setting the base address at 6MB's require a minimum footprint of 6MB
for the application?
troy: we are waiting to hear whether rebasing is going to really help performance
or not, and whether setting the base address at 6MB forces a minimum 6MB
footprint.
Yes, rebasing is a big performance win, and no it does change the footprint.

Basically, some sort of rebasing is going to happen. Either we do it when we
build (or sometime before we ship), or it happens at runtime when the DLL is
loaded. The OS loader will detect that the image base address conflicts with a
DLL already loaded, and it will relocate it to a different address. That's an
expensive operation that happens each and every time the browser is run.
The other thing that it's critical that we do is to "bind" the .exe and .dll
files. Unline rebasing (which we can do during the build process), binding
happens when the files are installed on the user's machine.

Binding is basically a "fixup" of the external functions and doing it during
install means that when the DLL is loaded less fixups are needed.

It's probably obvious, but worth mentioning anyway, that if a DLL is rebased
when loaded, then all DLLs that reference the rebased DLL must also be fixed up
again. That means if we don't rebase our DLLs before we ship them, then we can't
bind them when installing them

That means we lose twice, and the startup time of our app will suffer.

Sean Su knows about binding of the DLLs when doing an install, because that's
something we do for Communicator
Blocks: 7251
Status: ASSIGNED → RESOLVED
Closed: 21 years ago
Resolution: --- → FIXED
i've added rebase rules to the mozilla win32 build system, so that every build
will get rebased at the end of the install phase.

I've also added this to the rule that splits symbols in ns (since we should
probably strip the symbols before doing the rebase).
Status: RESOLVED → VERIFIED
Marking Verified.
Status: VERIFIED → REOPENED
When I launch the debug version of apprunner on Win32, I see tons of OS loader
warning messages that DLLs have been reloated. So I don't think this is working
right. Reopening.
Status: REOPENED → ASSIGNED
Are you removing your component.reg? If you have an old one, you will get this
kind of error.

If you *are* removing it, note that here and i'll try and figure out what's
going wrong... I think this might be related to the fact that the installer
isn't binding our dlls right now.
I deleted dist\win32_d.obj\bin\component.reg, and deleted all of the DLLs and
EXEs in bin, and then rebuilt from client.mak, and I still get the relocation
warnings.

Looking at the makefile, it appears to only rebase if MOZILLA_OFFICIAL is
defined. Not sure what the purpose of that env var is, but I'll try it out.
Seems like we should rebase all the time, though. Otherwise we're just wasting
developers' time.
As expected, defining MOZILLA_OFFICIAL causes rebasing to happen properly.
Resolution: FIXED → ---
Please accept my apology. I didn't realize this was something for developers as
well as release builds. I'm not sure why errors are occuring when you don't
rebase, since rebasing was added six weeks or so ago. Why didn't these errors
occur then?

I'll remove the MOZILLA_OFFICIAL wrapper tomorrow if there are no more pressing
fires.
Target Milestone: M8 → M10
Moving from M8 to M10.  leaf, if you will be doing this to the M9 branch build,
then please set Target Milestone to M9.
> I didn't realize this was something for developers as
> well as release builds.

Yah, it'll make developer turnaround time a little faster, since relocating DLLs
in memory takes time every time the app is launched.

> I'm not sure why errors are occuring when you don't rebase, since rebasing was
> added six weeks or so ago. Why didn't these errors occur then?

I'm sure they did. I've been on sabbatical for six weeks, and am just getting
back now.
yes, the errors have been there forever in debug builds, and they didn't go away
when re-basing was added. I'm not sure that we've ever rebased debug dlls
consistently, even in 4.x, since one could argue that doing the actual rebasing
takes longer than resolving the conflicts at load time.
> I'm not sure that we've ever rebased debug dlls consistently, even in 4.x

My recollection is that we always rebased, and that seems to be what the old
mozilla.mak does, as I look at it now.

> one could argue that doing the actual rebasing takes longer than resolving
> the conflicts at load time

Maybe if you always compile before running, but I figured people sometimes run
without compiling. If that's true, rebasing seens like a win.
we're doing this even on debug now, right? if yes, then close the bug as fixed.
Nope. Still only when MOZILLA_OFFICIAL is defined.
Sorry, still waiting for a day of 1.) little enough tree fire fighting and 2.)
enough open-green-tree time to check this build change in.
No longer blocks: 7251
Severity: critical → normal
Summary: Run rebase.exe to enhance performance → Run rebase.exe to enhance performance on debug builds
Whiteboard: [Perf]
Target Milestone: M10 → M14
pushing milestone out to M14 (not a beta blocker) since we do this for the
release builds.

setting severity to normal.

yes, it's a nice thing to do for debug builds, but trying to make bugs reflect
priority.
Keywords: perf
Adding perf to keyword field.
The restart time is about 4 seconds with the debug build on my current NT

system.  I rebased the debug build and there wasn't a noticeable difference.  I

don't see a developer win.  Troy mentioned that we should be binding the

executable during install.  Is there a separate bug for that?

Rebasing and binding are two separate issues. The rebasing issue concerns the 
base address of the DLL. We need to make sure that each of our DLLs has a 
different base address; otherwise, when a DLL is loaded the OS loader will 
rebase the DLL (move it to a different place in memory). This can get done as a 
post process step of the build process

The binding step can't get done until install time.

Both steps are important and we need to do them
But do we really need to do them on debug builds? (we already rebase for
optimized builds, not sure if the installer does binding yet)
It doesn't hurt to rebase for debug builds, and it will mean the loader doesn't 
have to do it.

What's the argument for not doing it for debug builds?
I just talked to jevering about this today, and he said that it ended up not
being much of a win for him. (Since this is for -debug- builds only, it's kind
of a "let's make life better for hackers" thing anyway, and has no effect on the
end product. As a hacker, I'd prefer a quicker build, anyway.)

I'm thinking we should close the bug & mark invalid. Troy: you seem to be the
big champion for this on debug builds; does it make things noticibly better for
you?
Whatever. My major concern was that we do it for optimized builds.
Ok, we've been doing it for release builds for... almost five months. Closing
bug.
Status: ASSIGNED → RESOLVED
Closed: 21 years ago20 years ago
Resolution: --- → FIXED
Marking Verified.
Status: RESOLVED → VERIFIED
Product: Browser → Seamonkey
You need to log in before you can comment on or make changes to this bug.