547704 - Very low SSL performance on shared databases.

Status: RESOLVED WONTFIX

(NSS :: Libraries, defect)

Description

[Slavomir Katuscak]

For a few weeks I see that memory leak tests on machine nssamdrhel4 are not finished on time, so I performed some analysis.

The biggest lost of time is in SSL tests (script ssl.sh). This script runs 4 times per build in combinations (standard, pkix, upgraded db from old format to shared db, shared db) where in standard run there are more combinations of bypass and fips used so ssl_run_all function runs 4 times, in other three combination there is only bypass/bypass used, so ssl_run_all runs only once.

I compared how long ssl.sh took on Linux and on Solaris builds 20091201.1 and 20091201.1:

machine simplify.1 (solaris):
normal:     65min -> 90min
pkix:       16min -> 28min 
upgradedb:  17min -> 57min
shareddb:   17min -> 56min

machine nssamdrhel4.1 (linux)
normal:     14min -> 18min
pkix:        5min -> 11min
upgradedb:   8min -> 70min
shareddb:    7min -> 68min


You can see that difference is on every test combination, that can be caused by different number of test in ssl.sh and different network behaviour, however for shared databases (upgradedb and shareddb) is difference very significant (from less than 10 minutes to more then hour).

Some chronological data:
On nssamdrhel4 we run every night 6 test combinations - 32bit/dbg, 32bit/opt, 64bit/dbg, 64bit/opt, 32bit/dbg/memleak, 32bit/opt/memleak. Every day, before nightly build is started, testing of old build is killed and report with results (even incomplete) is sent. 

Until 20100129.1 I received complete results from nightly builds. 20100129.1 was the first build when results from 32bit/opt/memleak build was not finished in time, since this I received everyday incomplete results from this machine. Since build 20100209.1 even results from 32bit/dbg/memleak aren't finished and 32bit/opt/memleak isn't tested at all.

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

Are these shared DBs being accessed over NFS?  
If so, change the scripts so they don't do that, but instead access the 
DBs locally.

Comment 2

[Christophe Ravel]

The shared DBs were on NFS then (2009-12-01) and now. So even if we know that the performance of sqlite on NFS are not good, there seems to be a regression.
SQLITE3 version 3.6.22 was integrated in NSS source tree on 2010-01-07.

Comment 3

[Christophe Ravel]

Please note that I wrote "seems" to be a regression. We need to run more tests to affirm that.

Comment 4

[Slavomir Katuscak]

I performed test on local disk drive on the same machine (nssamdrhel4) with shareddb. Test script ssl.sh took less than 4 minutes in both old and new version, therefore problem really is in NFS. It is possible that newer version of SQLITE can require more disk operations that can take more time on NFS, however on local drive there is no effect on performance. 

So the problem is really in NFS performance. One solution is to fix network problems, other is to use local drive and copy results back after testing is finished.

Comment 5

[Wan-Teh Chang]

Based on Slavo's comment 4, I suspect the underlying cause of the
poor shared database performance on NFS is bug 578561, which reports
that the network filesystem test in the softoken is broken with new
versions of sqlite.

Since Slavo is no longer working on NSS, I'm afraid that I have
to mark this bug WONTFIX.  Let's fix the broken network filesystem
test in bug 578561.