Closed Bug 477777 Opened 15 years ago Closed 15 years ago

Selfserv crashed in client/server tests.

Categories

(NSS :: Libraries, defect, P1)

Product:
NSS
Component:
Libraries
Platform:
All
Solaris
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
3.12.3

People

(Reporter: slavomir.katuscak+mozilla, Assigned: alvolkov.bgs)

Details

(Whiteboard: PKIX)

Attachments

(1 file)

hashtable patch v1 - incref counters before unlocking the table.
1.05 KB, patch
nelson
: review+
Details | Diff | Splinter Review 
Selfserv crashed in client/server tests:

ssl.sh: Stress SSL2 RC4 128 with MD5 (no reuse, client auth) ----
selfserv starting at Mon Feb  9 18:36:07 PST 2009
selfserv -D -p 8111 -d ../server -n dositups.red.iplanet.com -B -s \
         -e dositups.red.iplanet.com-ec -w nss -r -r -i ../tests_pid.7305  &
trying to connect to selfserv at Mon Feb  9 18:36:07 PST 2009
tstclnt -p 8111 -h dositups.red.iplanet.com -B -s -q \
        -d ../client < /export/tinderlight/data/dositups_32_DBG/mozilla/security/nss/tests/ssl/sslreq.dat
kill -0 26801 >/dev/null 2>/dev/null
selfserv with PID 26801 found at Mon Feb  9 18:36:07 PST 2009
selfserv with PID 26801 started at Mon Feb  9 18:36:07 PST 2009
strsclnt -q -p 8111 -d ../client -B -s -w nss -c 100 -C A -N -n TestUser \
          dositups.red.iplanet.com
strsclnt started at Mon Feb  9 18:36:07 PST 2009
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: 0 cache hits; 0 cache misses, 0 cache not reusable
          0 stateless resumes
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
8[817e078]: Assertion failure: objType < PKIX_NUMTYPES, at pkix_pl_object.c:136
Assertion failure: objType < PKIX_NUMTYPES, at pkix_pl_object.c:136
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: PR_Send returned error -5938:
Encountered end of file.
strsclnt: PR_Send returned error -5938:
Encountered end of file.
strsclnt: PR_Send returned error -5938:
Encountered end of file.
strsclnt: PR_Send returned error -5938:
Encountered end of file.
strsclnt: PR_Send returned error -5961:
TCP connection reset by peer.
strsclnt: PR_Send returned error -5938:
Encountered end of file.
strsclnt: PR_Send returned error -5938:
Encountered end of file.
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
strsclnt: SSL2 - 78 server certificates tested.
strsclnt: NoReuse - 78 server certificates tested.
./ssl.sh: line 421: 26801 Abort                   (core dumped) ${PROFTOOL} ${BINDIR}/selfserv -D -p ${PORT} -d ${P_R_SERVERDIR} -n ${HOSTADDR} ${SERVER_OPTIONS} ${ECC_OPTIONS} -w nss ${sparam} -i ${R_SERVERPID} $verbose
strsclnt completed at Mon Feb  9 18:36:10 PST 2009
ssl.sh: #4881: Stress SSL2 RC4 128 with MD5 (no reuse, client auth) produced a returncode of 1, expected is 0.  - Core file is detected - FAILED

Happened 2 times on Tinderbox (2 different machines), only in PKIX cycle (NSS_ENABLE_PKIX_VERIFY="1"), on both exactly the same tests and only once (in relatively short time frame), both on SunOS (Sparc + X86).
Core analysis:

t@null (l@1) terminated by signal KILL (Killed)
0xbfb1dc57: __pollsys+0x0007:   jb       __cerror       [ 0xbfa9dad0, .-0x80187 ]
Current function is pt_poll_now
  602                                   rv = poll(&tmp_pfd, 1, msecs);
(dbx) threads                                                                
      t@1  a  l@1   ?()   LWP suspended in  __pollsys() 
      t@2  a  l@2   _pt_root()   LWP suspended in  __lwp_park() 
      t@3  a  l@3   _pt_root()   LWP suspended in  __lwp_park() 
      t@4  a  l@4   _pt_root()   LWP suspended in  __pollsys() 
      t@5  a  l@5   _pt_root()   LWP suspended in  __pollsys() 
      t@6  a  l@6   _pt_root()   LWP suspended in  __pollsys() 
      t@7  a  l@7   _pt_root()   LWP suspended in  s_mpv_mul_d_add_prop() 
o     t@8  a  l@8   _pt_root()   signal SIGABRT in  __lwp_kill() 
      t@9  a  l@9   _pt_root()   sleep on 0x8186ca4  in  __lwp_park() 
(dbx) where t@8
current thread: t@8
=>[1] __lwp_kill(0x8, 0x6), at 0xbfb1df57 
  [2] raise(0x6), at 0xbfacadab 
  [3] abort(0x0, 0x0, 0xbfee8150, 0xbfc73dcf, 0xbeffaad0, 0xbfe83db8), at 0xbfaae8f9 
  [4] PR_Assert(s = 0xbfed796c "objType < PKIX_NUMTYPES", file = 0xbfed7984 "pkix_pl_object.c", ln = 136), line 562 in "prlog.c"
  [5] pkix_pl_Object_GetHeader(object = 0x81e9624, pObjectHeader = 0xbeffab2c, plContext = 0x825ef00), line 136 in "pkix_pl_object.c"
  [6] PKIX_PL_Object_IncRef(object = 0x81e9624, plContext = 0x825ef00), line 831 in "pkix_pl_object.c"
  [7] PKIX_PL_HashTable_Lookup(ht = 0x817991c, key = 0x8297ae4, pResult = 0xbeffac08, plContext = 0x825ef00), line 409 in "pkix_pl_hashtable.c"
  [8] pkix_CacheCertChain_Lookup(targetCert = 0x82d6adc, anchors = 0x82f3874, testDate = 0x819abcc, pFound = 0xbeffac7c, pBuildResult = 0xbeffac90, plContext = 0x825ef00), line 635 in "pkix_tools.c"
  [9] pkix_Build_CheckInCache(state = 0x82979bc, pBuildResult = 0xbeffad14, pNBIOContext = 0xbeffad68, plContext = 0x825ef00), line 2955 in "pkix_build.c"
  [10] pkix_Build_InitiateBuildChain(procParams = 0x82f380c, pNBIOContext = 0xbeffae1c, pState = 0xbeffae24, pBuildResult = 0xbeffae20, pVerifyNode = 0xbeffae9c, plContext = 0x825ef00), line 3441 in "pkix_build.c"
  [11] PKIX_BuildChain(procParams = 0x82f380c, pNBIOContext = 0xbeffae98, pState = 0xbeffae94, pBuildResult = 0xbeffaea0, pVerifyNode = 0xbeffae9c, plContext = 0x825ef00), line 3641 in "pkix_build.c"
  [12] cert_BuildAndValidateChain(procParams = 0x82f380c, pResult = 0xbeffaee0, pVerifyNode = 0xbeffaedc, plContext = 0x825ef00), line 820 in "certvfypkix.c"
  [13] cert_VerifyCertChainPkix(cert = 0x82f9990, checkSig = 1, requiredUsage = certUsageSSLClient, time = 1234233368385632LL, wincx = (nil), log = (nil), pSigerror = (nil), pRevoked = (nil)), line 1259 in "certvfypkix.c"
  [14] cert_VerifyCertChain(handle = 0x81732c0, cert = 0x82f9990, checkSig = 1, sigerror = (nil), certUsage = certUsageSSLClient, t = 1234233368385632LL, wincx = (nil), log = (nil), revoked = (nil)), line 870 in "certvfy.c"
  [15] CERT_VerifyCertChain(handle = 0x81732c0, cert = 0x82f9990, checkSig = 1, certUsage = certUsageSSLClient, t = 1234233368385632LL, wincx = (nil), log = (nil)), line 882 in "certvfy.c"
  [16] CERT_VerifyCert(handle = 0x81732c0, cert = 0x82f9990, checkSig = 1, certUsage = certUsageSSLClient, t = 1234233368385632LL, wincx = (nil), log = (nil)), line 1478 in "certvfy.c"
  [17] CERT_VerifyCertNow(handle = 0x81732c0, cert = 0x82f9990, checkSig = 1, certUsage = certUsageSSLClient, wincx = (nil)), line 1529 in "certvfy.c"
  [18] SSL_AuthCertificate(arg = 0x81732c0, fd = 0x81ab570, checkSig = 1, isServer = 1), line 255 in "sslauth.c"
  [19] mySSLAuthCertificate(arg = 0x81732c0, fd = 0x81ab570, checkSig = 1, isServer = 1), line 339 in "selfserv.c"
  [20] ssl2_HandleClientCertificate(ss = 0x81de8f8, certType = '\001', cd = 0x82554c6 "0\x82^B]0\x82^A\xc6\xa0^C^B^A^B^B^AF0^M^F^I*\x86H\x86\xf7^M^A^A^E^E", cdLen = 609U, response = 0x8255727 "^\xf2\xcd\xa6\x92\xdf^T\xc1V\x9e]\x99\xe6\x87\xd0~\xd6nu~\xd4^Z\xf7mkg\xf8WA9\xad\x95\xce^L^GR\xb7\x85\xdb^CY\x9f\xa1\x91\x9e^WZ\xf4LD\x92\xec^^x\xf9\xdc\x9e\xb2h\x91\x9b{^[r\xee^T^Gs=@g\x9eJ\xf5\xf7\xd93\xfdSn\xf1NN]^GA\xb9\xbe\xb25l\xf8\x94\xc1\xd9\xa0^M^[\xc3$\xdfE\xf6\\xf2\xab\x88`^I\x96\xa9\xa3dl^Xr9\xef]s!\xb1\xc3\xa0\x90\xd2=\xc3", responseLen = 128U), line 2499 in "sslcon.c"
  [21] ssl2_HandleMessage(ss = 0x81de8f8), line 2647 in "sslcon.c"
  [22] ssl_Do1stHandshake(ss = 0x81de8f8), line 151 in "sslsecur.c"
  [23] ssl_SecureRecv(ss = 0x81de8f8, buf = 0xbeffb51c "GET /abc HTTP/1.0^M\n^M\n", len = 10239, flags = 0), line 1113 in "sslsecur.c"
  [24] ssl_SecureRead(ss = 0x81de8f8, buf = 0xbeffb51c "GET /abc HTTP/1.0^M\n^M\n", len = 10239), line 1132 in "sslsecur.c"
  [25] ssl_Read(fd = 0x81ab570, buf = 0xbeffb51c, len = 10239), line 1468 in "sslsock.c"
  [26] PR_Read(fd = 0x81ab570, buf = 0xbeffb51c, amount = 10239), line 141 in "priometh.c"
  [27] handle_connection(tcp_sock = 0x81ab570, model_sock = 0x8087310, requestCert = 2), line 1047 in "selfserv.c"
  [28] jobLoop(a = (nil), b = (nil), c = 2), line 528 in "selfserv.c"
  [29] thread_wrapper(arg = 0x817e9e0), line 496 in "selfserv.c"
  [30] _pt_root(arg = 0x817e078), line 221 in "ptthread.c"
  [31] _thr_setup(0xbf401400), at 0xbfb1cf2f 
  [32] _lwp_start(), at 0xbfb1d220 

For more info I created backup of build, you can find core on machine dositups at:
/export/tinderlight/data/dositups_32_DBG.last.fail/mozilla/tests_results/security/dositups.1/pkix/client/core
Assignee: nobody → alexei.volkov.bugs
Severity: normal → major
Whiteboard: PKIX
Similar problem occured on the same Tinderbox machine (dositups) now:

ssl.sh: Stress SSL3 RC4 128 with MD5 (no reuse, client auth) ----
selfserv starting at Tue Feb 10 06:46:47 PST 2009
selfserv -D -p 8333 -d ../server -n dositups.red.iplanet.com -B -s \
         -e dositups.red.iplanet.com-ec -w nss -r -r -i ../tests_pid.4791  &
trying to connect to selfserv at Tue Feb 10 06:46:47 PST 2009
tstclnt -p 8333 -h dositups.red.iplanet.com -B -s -q \
        -d ../client < /export/tinderlight/data/dositups_64_DBG/mozilla/security/nss/tests/ssl/sslreq.dat
kill -0 28996 >/dev/null 2>/dev/null
selfserv with PID 28996 found at Tue Feb 10 06:46:48 PST 2009
selfserv with PID 28996 started at Tue Feb 10 06:46:48 PST 2009
strsclnt -q -p 8333 -d ../client -B -s -w nss -c 100 -C c -T -N -n TestUser \
          dositups.red.iplanet.com
strsclnt started at Tue Feb 10 06:46:48 PST 2009
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: 0 cache hits; 5 cache misses, 0 cache not reusable
          0 stateless resumes
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Invalid, err -8026.
Libpkix internal error occured during cert validation.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
selfserv: HDX PR_Read returned error -12271:
SSL peer cannot verify your certificate.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: PR_Send returned error -8026:
Libpkix internal error occured during cert validation.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
(pkix_CacheCertChain_Add: PKIX_PL_HashTable_Add for CertChain skipped: entry existed
strsclnt: 0 cache hits; 38 cache misses, 0 cache not reusable
          0 stateless resumes
strsclnt: NoReuse - 38 server certificates tested.
strsclnt completed at Tue Feb 10 06:46:48 PST 2009
ssl.sh: #4882: Stress SSL3 RC4 128 with MD5 (no reuse, client auth) produced a returncode of 1, expected is 0.  - FAILED

Both previous failures with core were on test #4881 (Stress SSL2 RC4 128 with MD5 (no reuse, client auth)), now #4881 passed and #4882 (Stress SSL3 RC4 128 with MD5 (no reuse, client auth)) failed, without core, just error messages.
If increfed after, another operation(hashtable remove object) can potentially set a counter of the references to a object to value of zero, thus forcing object to be destroyed. The thread that just obtained the pointer to an object by calling hashtable lookup will be left with dangling pointer.
Attachment #361796 - Flags: review?
Attachment #361796 - Flags: review? → review?(nelson)
Comment on attachment 361796 [details] [diff] [review]
hashtable patch v1 - incref counters before unlocking the table.

r=nelson
Attachment #361796 - Flags: review?(nelson) → review+
committed.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Priority: -- → P1
You need to log in before you can comment on or make changes to this bug.