196880 - Excessive malloc() calls

Status: RESOLVED INACTIVE

(NSS :: Libraries, defect, P2)

Description

[Kirk Erickson]

Sun BugTraq Bug Id: 4830310.
Filed by [e-mail].

Release summary: 3.3
Synopsis: NSS/HTTPS Performance is significantly poor due to
	excessive malloc() call
Keywords: malloc, nss, performance
Justification:
	NSS performance is very important to stay competitive
	in HTTPS arena use SunONE web-server
Suggested fix: The SSL layer could accept a memory allocator
	hook from the webserver during NSS-Init -- The webserver would
	maintanin per-thread memory-pool, that can be used by NSS

Using NSS 3.3

The number of malloc() call per HTTPS/ops is 298 -- This need
to be reduced, I have analyzed each malloc call and the whole
call stack

A summary of the analysis is attached (malloc.stack) that can be
used to aggresively work on reducing the number of malloc calls.

Comment 1

[Kirk Erickson]

Note, this problem has been addressed since NSS 3.3, see:
http://bugzilla.mozilla.org/show_bug.cgi?id=97485
malloc lock contention

A zone allocator was invented to address this issue.  It was
incorporated into NSPR 4.2 in January, 2002.  This solution was
offered in part to satisfy the heap needs of NSS and NSPR without
interfering with things like Java's run time (ala SmartHeap).

To engage: setenv NSPR_USE_ZONE_ALLOCATOR 1

NSS 3.4 or later uses NSPR 4.2 or later.  I do basically
all selfserv stress with the zone allocator engaged because
performance is dramatically improved, particularly on restarts.

Added Pallab to CC list.

Comment 2

[Kirk Erickson]

[e-mail] 2003-03-11
Used NSS3.7 and NSPR4.3 -- the number of malloc is significantly down to
57 -- in these 2 stacks only --

malloc: size = 256, retval =a2ad08
....../lib/libc.so.1:calloc+0x58
....../lib/libfreebl_hybrid_3.so:mp_init_size+0x58
....../lib/libfreebl_hybrid_3.so:s_mp_div+0x84
....../lib/libfreebl_hybrid_3.so:mp_div+0x17c
....../lib/libfreebl_hybrid_3.so:s_mp_to_mont+0x2c
....../lib/libfreebl_hybrid_3.so:mp_exptmod_f+0x98
....../lib/libfreebl_hybrid_3.so:mp_exptmod+0x1e0
....../lib/libfreebl_hybrid_3.so:rsa_PrivateKeyOpCRTNoCheck+0x19c
....../lib/libfreebl_hybrid_3.so:rsa_PrivateKeyOp+0x218
....../lib/libsoftokn3.so:RSA_DecryptBlock+0x54
....../lib/libsoftokn3.so:NSC_Decrypt+0x104

malloc: size = 9760, retval =a2a9f0
....../lib/libfreebl_hybrid_3.so:mp_exptmod_f+0xf4
....../lib/libfreebl_hybrid_3.so:mp_exptmod+0x1e0
....../lib/libfreebl_hybrid_3.so:rsa_PrivateKeyOpCRTNoCheck+0x19c
....../lib/libfreebl_hybrid_3.so:rsa_PrivateKeyOp+0x218
....../lib/libsoftokn3.so:RSA_DecryptBlock+0x54
....../lib/libsoftokn3.so:NSC_Decrypt+0x104

However, the Zone allocator has moved the problem from one location
to another location (instead of fixing it) -- 
This zone allocator does not take into account the number of CPUs in the
system -- and as such would make 'mz->lock' a hot lock -- a group of threads
would contend on a given "[zone][pool]" on large systems, on small systems it
may not exhibit any difference --

Comment 3

[Roland Mainz]

Slightly offtopic: Would it make sense to switch to an alternate memory
allocator as workaround for this issue (see bug 193622 ("RFE: Use libmtmalloc()
(or libbsdmalloc()) for memory allocations...")) ?

Comment 4

[Nelson Bolyard (seldom reads bugmail)]

P2.  Of business value to an active supporter of NSS.

The present implementation of Zone Allocator has up to 11 different memory 
pools for each "zone" (size of allocation block).  The CPUs/threads should
be reasonably well distributed across those 11 pools.  That suggests that
there should be minimal contention until there are at least 11 CPUs, but 
there have been reports that contention becomes an issue with as few as 4
CPUs.  If that is true, we should find out why.  That may be easy to correct 
once we know why it is not happening.  

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, and this bug has priority 'P2'.
:beurdouche, could you have a look please?

For more information, please visit auto_nag documentation.