87725 - Instrument NSS use of PLArena

Status: RESOLVED FIXED

(NSS :: Libraries, defect, P1)

Description

[larryh (gone)]

NSS uses PLArena in a number of places as a substitute for malloc(). PLArena 
uses malloc() subdividing the malloc'd pieces to its caller. By subdividing a 
malloc'd piece, performance pressure on malloc is reduced. There is speculation 
that the CHUNKSIZEes for the arena pools are not optimal and that the malloc() 
overhead may be high in spite of NSS's intent.

This bugzilla documents a temporary instrumentation to capture data to support 
or refute the above assertion. The instrumentation will capture the "high water 
mark" for the arena pool. That is: the total size of all allocation taken from 
an arena pool. The instrumentation code will not be checked in to the tree. The 
patch will be attached to this report in case it is needed again.

No attempt will be made to account for the effects of calls to mark and release, 
other than to note that they occured. If analysis of the instrumentation data 
show heavy use of mark and release, this issue will be revisited.

The test harness selfserv and strsclnt will be used to gather data. After NSS 
are satisfied that it works, iWS will be asked to use the instrumented parts and 
provide us with the captured data.

If the instrumentation shows that NSS can improve its performance with PLArena, 
then adjustments to NSS will be made on a separate ticket.

Comment 1

[larryh (gone)]

Alas, the use of mark & release in NSS arenas cannot be ignored. In the inital 
instrumentation, I emitted a message for each use of mark & release. There's a 
lot of use for this. Therefore, the original goal of just keeping a 
high water mark (HWM) within the PORTArena structure is not adaquate.

Some post processing of the log data is required to get the true high water 
mark. Write a little perl, keep score of the HWM for each distinct PORTArena 
structure by correlating a release with the HWM at the time of the mark.

Comment 2

[larryh (gone)]

add cc Bob Relyea, Julien Pierre

Comment 3

[larryh (gone)]

Attachment: instrumentation added to secport.c

Comment 4

[larryh (gone)]

Attachment: perl script for analyzing instrumentation data

Comment 5

[larryh (gone)]

Attachment: fixed bug in perl script. was showing more use than actual

Comment 6

[larryh (gone)]

I ran the test programs selfserv, with instrumented secport.c, and strsclnt with 
the following strsclnt commands:
strsclnt -c 1000
strsclnt -c 1000 -N

(2000 transactions)

This yielded a log file containing 110142 lines, where each line represented 
some use of the PLArena functions in secport.c

I fed the logfile to the perl script. The summary section of the perl script 
reported:
0 greater than 8k
0 greater than 6k
6 greater than 4k
6 greater than 2k
13223 less    than 2k

A little hand analysis says that for each transaction, 6 arenapools were used.
Each transaction called an arena function 55 times.

The data on arenapool size says that for almost all transactions, the arenapool 
use was limited to a single arena.

I conclude that there is no merit to trying to optimize the default arenapool 
size to limit the number of calls to malloc(); it is already close to optimal.

Finally, wtc suggests that I hand-build a solaris NSS with the instrumented 
secport.c for use by iWS in their performance testing. We (NSS group, really: 
me) will run the perl script against the logfile generated to see how iWS use of 
NSS fares. ... Julien, gimme a call, please. Tell me which version (3.2.1, 3.3) 
of NSS you want to test with.

Comment 7

[larryh (gone)]

Julien ran iWS using the instrumented NSS 3.3 using 200 transactions; 100 
generating new keys; 100 using cached keys. The follwing are the data, after 
reducing the log file.

4 greater than 8k
28 greater than 6k
249 greater than 4k
152 greater than 2k
7387 less    than 2k

It shows some higher use of arenas greater than 2k, but it still does not appear 
that increasing the default arenasize in NSS would buy much performance.

Comment 8

[Wan-Teh Chang]

This was completed.