Closed Bug 15115 Opened 21 years ago Closed 19 years ago

[help wanted] Contents of registry file are read multiple times

Categories

(Core :: XPCOM, defect, P1)

All
Mac System 8.5
defect

Tracking

()

RESOLVED FIXED
mozilla0.9

People

(Reporter: sfraser_bugs, Assigned: dveditz)

References

()

Details

(Keywords: helpwanted, perf, Whiteboard: [PDT-])

Attachments

(2 files)

From pierre's post (URL above):

We can improve libreg even more (XP)
-------------------------------
We had tremendous improvements in libreg in the past few weeks
but we can do even better. The total time spent in the file i/o
functions is 1 second, during which the Mozilla Registry is opened
4 times and we read a total of 7Mb. Since the Component Registry
takes about 240 Kb and the other files have an insignificant size,
it means that we read the Component Registry entirely almost 30 times!

   op       # of calls                     approx time
  ----      ----------                     -----------
  read        1735 (for 7,145,105 bytes)     0.87 s
  write         50                           0.00
  seek        1774                           0.08
  open           6                           0.02
  close          5                           0.01

  Total       3570                           0.99

We need to not read 7Mb data.
Assignee: dp → dveditz
The tree-structure of the data in the registry pretty much guarantees that any
client needs to read throughout the file, the beginning to pick up the roots,
and the middle/end to get recent data.

An easy change would be to make the buffer size bigger than the expected
registry, like 250-300K.  Would that cause any problems for the Mac? Remember
that there are likely to be two registries open (at least at startup), the
component registry and the global mozilla registry, so double that buffer size.

Changing the buffer size is trivial. With more work we could add an API to set
the buffersize for individual registries and only take the memory hit for the
component registry. But that would lock us into this particular buffering
mechanism to some extent.

There are also some changes that can be made to the componentManager which will
eliminate many of those reads.
Do we really need to double the buffer size? The Component Registry can certainly

reach 300K but I haven't seen the Mozilla Registry at more then 2K. Can we

allocate the buffers according to the registry size?
We could do something along those lines, but that's more work than changing a
#define. Can we go for the cheap speed gain now and fix the memory issue later?
Or could we put off the speedup.

Actually, could you change the buffer size on your build and re-profile it as
a reality check? It may not even help all that much compared to other start-up
costs. Look for #define BUFIO_BUFSIZE at the top of nr_bufio.c
How hard can it be to use NSPR to get the file size, and allocate the buffer
accordingly?  I don't see why #define is so much easier.  What am I missing?

Could we get this bug target-milestoned and assigned and stuff?  Dogfood,
so I'm cc'ing chofmann.

/be
Whiteboard: [perf]
Well for one thing the component.reg doesn't exist at first startup, so how in
the world would we know we want to use a 250K buffer for it?

We need to change the Open API anyway for other feature requests, the preferred
buffer size could just be another option.
Here are some results for the current buffer size, and a 256k buffer:


64k buffer

NS_InitXPCOM		215 ms
bufio_loadBuf		 91 ms		51 calls

256k buffer

NS_InitXPCOM		208 ms
bufio_loadBuf		 49 ms		1 call
Target Milestone: M15
Whiteboard: [perf]
Keywords: perf
Summary: [Perf] Contents of registry file are read multiple times → Contents of registry file are read multiple times
Not sure what your last entry means Simon, is this still a major bottleneck?  If 
you think it needs to be fixed for beta, please add beta1 keyword. cc'ing 
waterson.
Yes, this is still extrememly bad news on the mac, with a simple fix (bigger 
buffer).
Keywords: beta1
Putting on PDT+ radar for beta1.
Whiteboard: [PDT+]
Making pdt+ bugs P1
Priority: P3 → P1
Pdt+ beta1 bug can't be M15
Target Milestone: M15 → M14
In a recent quantify run, bufio_loadBuf shows up as 6.75% of total runtime (to 
launch to the mozilla.org page) after eliminating time spent in several places:

- idling in the dns service (windows event loop)
- idling in the socket transport (in PR_Poll)
- idling in the file transport (in PR_Wait)
- time spent in LoadLibrary.

The last item is a significant performance impediment, although the first 3 are 
things quantify doesn't understand are ok. Nevertheless, fixing the registry 
file read time could have a significant performance impact.
Here's something I've noticed too: NR_RegOpen gets called 9 times on startup 
from the following callers:

CALLER					PERCENT	CALLS	PROPAGATED-TIME
vr_Init					41.77	1	93564.65
nsRegistry::OpenDefault			30.48	2	68278.75
nsRegistry::Open			15.04	1	33691.43
nsRegistry::OpenWellKnownRegistry	9.74	4	21826.25
PerformScheduledTasks			2.96	1	6626.18

Presumably the registry could be opened once and read into an in-memory data 
structure, and from there the file system shouldn't be hit again for it.
I don't know whether it's relevant but each time I launch Mozilla, the date of 

the Component Registry file changes indicating that the file is modified. However 

when I compare the data of the two copies before and after running the 

application, it doesn't show any difference.

One of the "open registry" calls may be coming from the code that constructs the
charset menu, which is fairly well buried, and rarely used. I could dig into
that case further if you like; it may be that we can defer reading the registry
until somebody opens the menu...
responding to several of warren's comments: 
 - it may be 7% on windows, it's far far worse on Mac.
 - There are at least three different registries getting opened at startup, 
component reg, version reg, profile reg, (and possibly history reg?). Doug's 
"update notification" call to the version reg should be pushed off to some 
later time.
 - NR_RegOpen is a near no-op if the requested registry is already open. Dp and 
I have worked to make sure that's the case in the past but it could stand 
examing again to make sure the usage pattern is the same. A better call to 
check would be bufio_open
 - When I land the autoreg-only-after-install fix the component reg can be 
opened read-only most of the time.
Whiteboard: [PDT+] → [PDT+] 2/15
Blocks: 27510
Whiteboard: [PDT+] 2/15 → [PDT+] 2/18
2/18 is passed.  Need to know when fix can land.  This could be a win for start 
up time.
On windows this suckage only accounts for about 1% of startup time. In order to 
really fix the problem we plan some major re-writes after beta so I'd like to 
avoid putting in a semi-risky (and hugely memory bloating) hack now.

We cannot simply allocate a file-sized buffer for all registries because the 
profile folks open up the Netscape registry to find 4.x profiles, and that file 
can be a couple of megabytes. We'd need to implement a flag that could be 
passed down the call chain, and then rip it out later when we do the real fix.
Whiteboard: [PDT+] 2/18 → [PDT+] I'd like to beg off -- see comment below
Also, not only this accounts only 1% of startup time, the fix will only help the 
startup time when splash screen is displayed up to profile picker comes up.  So, 
by perception, it's not going to be much help since the long pull in startup 
(also by perception) is after profile picker gets dismissed up till a browser 
window comes up.  That's the block of time when nothing seem to happen to users.
Removing PDT+ for reconsideration
Whiteboard: [PDT+] I'd like to beg off -- see comment below
Putting on PDT- radar for beta1.
Whiteboard: [PDT-]
Keywords: helpwanted
Summary: Contents of registry file are read multiple times → [help wanted] Contents of registry file are read multiple times
Target Milestone: M14 → M16
M16 has been out for a while now, these bugs target milestones need to be 
updated.
marking invalid.  HR's are block elements, and in strict mode they get laid out 
just like any other block.

let me know if you think we should do otherwise in strict mode, which would mean 
treating HR's as blocks in most cases, but as inlines if they are impacted by 
floaters.  I don't think the CSS spec wants to get into a situation where it 
forces style changes based on geometry.
Status: ASSIGNED → RESOLVED
Closed: 20 years ago
Resolution: --- → INVALID
Buster, you changed the wrong bug.
Status: RESOLVED → REOPENED
Resolution: INVALID → ---
yep, my bad.  sorry for the spam.
Resetting missed milestones
Keywords: beta1
Target Milestone: M16 → ---
dp is no longer @netscape.com. reassigning qa contact to default for this component
QA Contact: dp → rayw
Dan, what's up with this bug?  Is there still anything useful to do?
kandrot fixed the bulk of this (under another bug, I guess). I do have one 
small tweak I found so I might as well accept this bug, check it in, and close 
this out.
Keywords: nsbeta1+
Target Milestone: --- → mozilla0.9
Note that previous attachment showed attempted, not actual read sizes (fread may 
return less if you hit eof).
Redux of data in the last attachment show that we re-read registry files way more 
than we should:

File                   Total read   File size  Times read
---------------------------------------------------------
Application Registry       8140       4070            2.0
Mozilla Registry            543        543            1.0
Component Registry      3737086     366226           10.2
I think kandrot ``fixed'' this by just jacking the buffer size up large enough
so that the entire file is kept in memory. (Then, after the initialization is
done, we close the file and re-set the buffer size or some such.) So I'm not
sure whether or not this will result in disk hits? Can your PR_Read() inspektor
tell?
fread calls in <http://bugzilla.mozilla.org/showattachment.cgi?attach_id=29755> 
are in temporal order, so inspecting these data should show that. I do note that 
we read the entire components.reg twice, once from NS_InitXPCOM and again from 
the first AutoRegister call in the component loader. All subsequent 10K reads are 
coming out of charset stuff.

Filed bug 74815 on i18n's use of libreg.
That's basically what kandrot did, but he missed an early case (Platform init I 
think) that helps cut the read size down considerably. (everything up to the 
10K reads should all be one big read).

Reducing reads by increasing memory use is a tradeoff. In this case the attempt 
is to use the memory at startup and then reduce it after that, but apparently 
we should have left the buffer large until after i18n is finished using the 
data we've already read once.

Other alternatives are to kill component.reg, or restrict its use to small 
groupings of data read at random times like categories which benefit from the 
hierarchical structure. Take the big flat chunks that are read in toto at 
startup and never again, like lists of CID's or charset converters, into flat 
csv files or the like.
Fixed, but at the cost of memory. Opened bug 76329 on that issue.
Status: REOPENED → RESOLVED
Closed: 20 years ago19 years ago
Resolution: --- → FIXED
Component: XPCOM Registry → XPCOM
QA Contact: rayw → xpcom
You need to log in before you can comment on or make changes to this bug.