Closed Bug 15115 Opened 21 years ago Closed 19 years ago
[help wanted] Contents of registry file are read multiple times
From pierre's post (URL above): We can improve libreg even more (XP) ------------------------------- We had tremendous improvements in libreg in the past few weeks but we can do even better. The total time spent in the file i/o functions is 1 second, during which the Mozilla Registry is opened 4 times and we read a total of 7Mb. Since the Component Registry takes about 240 Kb and the other files have an insignificant size, it means that we read the Component Registry entirely almost 30 times! op # of calls approx time ---- ---------- ----------- read 1735 (for 7,145,105 bytes) 0.87 s write 50 0.00 seek 1774 0.08 open 6 0.02 close 5 0.01 Total 3570 0.99 We need to not read 7Mb data.
The tree-structure of the data in the registry pretty much guarantees that any client needs to read throughout the file, the beginning to pick up the roots, and the middle/end to get recent data. An easy change would be to make the buffer size bigger than the expected registry, like 250-300K. Would that cause any problems for the Mac? Remember that there are likely to be two registries open (at least at startup), the component registry and the global mozilla registry, so double that buffer size. Changing the buffer size is trivial. With more work we could add an API to set the buffersize for individual registries and only take the memory hit for the component registry. But that would lock us into this particular buffering mechanism to some extent. There are also some changes that can be made to the componentManager which will eliminate many of those reads.
Do we really need to double the buffer size? The Component Registry can certainly reach 300K but I haven't seen the Mozilla Registry at more then 2K. Can we allocate the buffers according to the registry size?
We could do something along those lines, but that's more work than changing a #define. Can we go for the cheap speed gain now and fix the memory issue later? Or could we put off the speedup. Actually, could you change the buffer size on your build and re-profile it as a reality check? It may not even help all that much compared to other start-up costs. Look for #define BUFIO_BUFSIZE at the top of nr_bufio.c
How hard can it be to use NSPR to get the file size, and allocate the buffer accordingly? I don't see why #define is so much easier. What am I missing? Could we get this bug target-milestoned and assigned and stuff? Dogfood, so I'm cc'ing chofmann. /be
Well for one thing the component.reg doesn't exist at first startup, so how in the world would we know we want to use a 250K buffer for it? We need to change the Open API anyway for other feature requests, the preferred buffer size could just be another option.
Here are some results for the current buffer size, and a 256k buffer: 64k buffer NS_InitXPCOM 215 ms bufio_loadBuf 91 ms 51 calls 256k buffer NS_InitXPCOM 208 ms bufio_loadBuf 49 ms 1 call
Summary: [Perf] Contents of registry file are read multiple times → Contents of registry file are read multiple times
Not sure what your last entry means Simon, is this still a major bottleneck? If you think it needs to be fixed for beta, please add beta1 keyword. cc'ing waterson.
Yes, this is still extrememly bad news on the mac, with a simple fix (bigger buffer).
Putting on PDT+ radar for beta1.
Making pdt+ bugs P1
Priority: P3 → P1
Pdt+ beta1 bug can't be M15
Target Milestone: M15 → M14
In a recent quantify run, bufio_loadBuf shows up as 6.75% of total runtime (to launch to the mozilla.org page) after eliminating time spent in several places: - idling in the dns service (windows event loop) - idling in the socket transport (in PR_Poll) - idling in the file transport (in PR_Wait) - time spent in LoadLibrary. The last item is a significant performance impediment, although the first 3 are things quantify doesn't understand are ok. Nevertheless, fixing the registry file read time could have a significant performance impact.
Here's something I've noticed too: NR_RegOpen gets called 9 times on startup from the following callers: CALLER PERCENT CALLS PROPAGATED-TIME vr_Init 41.77 1 93564.65 nsRegistry::OpenDefault 30.48 2 68278.75 nsRegistry::Open 15.04 1 33691.43 nsRegistry::OpenWellKnownRegistry 9.74 4 21826.25 PerformScheduledTasks 2.96 1 6626.18 Presumably the registry could be opened once and read into an in-memory data structure, and from there the file system shouldn't be hit again for it.
I don't know whether it's relevant but each time I launch Mozilla, the date of the Component Registry file changes indicating that the file is modified. However when I compare the data of the two copies before and after running the application, it doesn't show any difference.
One of the "open registry" calls may be coming from the code that constructs the charset menu, which is fairly well buried, and rarely used. I could dig into that case further if you like; it may be that we can defer reading the registry until somebody opens the menu...
responding to several of warren's comments: - it may be 7% on windows, it's far far worse on Mac. - There are at least three different registries getting opened at startup, component reg, version reg, profile reg, (and possibly history reg?). Doug's "update notification" call to the version reg should be pushed off to some later time. - NR_RegOpen is a near no-op if the requested registry is already open. Dp and I have worked to make sure that's the case in the past but it could stand examing again to make sure the usage pattern is the same. A better call to check would be bufio_open - When I land the autoreg-only-after-install fix the component reg can be opened read-only most of the time.
2/18 is passed. Need to know when fix can land. This could be a win for start up time.
On windows this suckage only accounts for about 1% of startup time. In order to really fix the problem we plan some major re-writes after beta so I'd like to avoid putting in a semi-risky (and hugely memory bloating) hack now. We cannot simply allocate a file-sized buffer for all registries because the profile folks open up the Netscape registry to find 4.x profiles, and that file can be a couple of megabytes. We'd need to implement a flag that could be passed down the call chain, and then rip it out later when we do the real fix.
Whiteboard: [PDT+] 2/18 → [PDT+] I'd like to beg off -- see comment below
Also, not only this accounts only 1% of startup time, the fix will only help the startup time when splash screen is displayed up to profile picker comes up. So, by perception, it's not going to be much help since the long pull in startup (also by perception) is after profile picker gets dismissed up till a browser window comes up. That's the block of time when nothing seem to happen to users.
Removing PDT+ for reconsideration
Whiteboard: [PDT+] I'd like to beg off -- see comment below
Putting on PDT- radar for beta1.
Summary: Contents of registry file are read multiple times → [help wanted] Contents of registry file are read multiple times
Target Milestone: M14 → M16
M16 has been out for a while now, these bugs target milestones need to be updated.
marking invalid. HR's are block elements, and in strict mode they get laid out just like any other block. let me know if you think we should do otherwise in strict mode, which would mean treating HR's as blocks in most cases, but as inlines if they are impacted by floaters. I don't think the CSS spec wants to get into a situation where it forces style changes based on geometry.
Status: ASSIGNED → RESOLVED
Closed: 20 years ago
Resolution: --- → INVALID
Buster, you changed the wrong bug.
Status: RESOLVED → REOPENED
Resolution: INVALID → ---
yep, my bad. sorry for the spam.
Resetting missed milestones
Target Milestone: M16 → ---
dp is no longer @netscape.com. reassigning qa contact to default for this component
QA Contact: dp → rayw
Dan, what's up with this bug? Is there still anything useful to do?
kandrot fixed the bulk of this (under another bug, I guess). I do have one small tweak I found so I might as well accept this bug, check it in, and close this out.
Target Milestone: --- → mozilla0.9
Note that previous attachment showed attempted, not actual read sizes (fread may return less if you hit eof).
Redux of data in the last attachment show that we re-read registry files way more than we should: File Total read File size Times read --------------------------------------------------------- Application Registry 8140 4070 2.0 Mozilla Registry 543 543 1.0 Component Registry 3737086 366226 10.2
I think kandrot ``fixed'' this by just jacking the buffer size up large enough so that the entire file is kept in memory. (Then, after the initialization is done, we close the file and re-set the buffer size or some such.) So I'm not sure whether or not this will result in disk hits? Can your PR_Read() inspektor tell?
fread calls in <http://bugzilla.mozilla.org/showattachment.cgi?attach_id=29755> are in temporal order, so inspecting these data should show that. I do note that we read the entire components.reg twice, once from NS_InitXPCOM and again from the first AutoRegister call in the component loader. All subsequent 10K reads are coming out of charset stuff. Filed bug 74815 on i18n's use of libreg.
That's basically what kandrot did, but he missed an early case (Platform init I think) that helps cut the read size down considerably. (everything up to the 10K reads should all be one big read). Reducing reads by increasing memory use is a tradeoff. In this case the attempt is to use the memory at startup and then reduce it after that, but apparently we should have left the buffer large until after i18n is finished using the data we've already read once. Other alternatives are to kill component.reg, or restrict its use to small groupings of data read at random times like categories which benefit from the hierarchical structure. Take the big flat chunks that are read in toto at startup and never again, like lists of CID's or charset converters, into flat csv files or the like.
Fixed, but at the cost of memory. Opened bug 76329 on that issue.
Status: REOPENED → RESOLVED
Closed: 20 years ago → 19 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.