1607257 - Enable SSLTokenCache on nightly

Status: RESOLVED FIXED

(Core :: Networking, task, P2)

Description

[Kershaw Chang [:kershaw]]

This is a follow up bug of bug 1597956. To get the telemetry data, we'd like to enable SSLTokenCache on nightly.

Comment 1

[Kershaw Chang [:kershaw]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0153a1508773c84dbbe0abfff1739efac9a260ab&selectedJob=287432584

Comment 2

[Kershaw Chang [:kershaw]]

Hi Dana,

When I tried to turn on SSLTokensCache and pushed to try server, ASAN keeps complaining about the memory leak of the session id created from SSLExp_SetResumptionToken. Please see the stack below.

[task 2020-02-04T11:22:21.401Z] 11:22:21     INFO - GECKO(1128) | Direct leak of 1453 byte(s) in 41 object(s) allocated from:
[task 2020-02-04T11:22:21.402Z] 11:22:21     INFO - GECKO(1128) |     #0 0x5583dc75ab9d in malloc /builds/worker/fetches/llvm-project/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:145:3
[task 2020-02-04T11:22:21.403Z] 11:22:21     INFO - GECKO(1128) |     #1 0x7f69d03b2d61 in PORT_Alloc_Util /builds/worker/workspace/build/src/security/nss/lib/util/secport.c:87:14
[task 2020-02-04T11:22:21.404Z] 11:22:21     INFO - GECKO(1128) |     #2 0x7f69d03b2d61 in PORT_Strdup_Util /builds/worker/workspace/build/src/security/nss/lib/util/secport.c:192:22
[task 2020-02-04T11:22:21.405Z] 11:22:21     INFO - GECKO(1128) |     #3 0x7f69d0149b9e in ssl3_NewSessionID /builds/worker/workspace/build/src/security/nss/lib/ssl/ssl3con.c:8094:49
[task 2020-02-04T11:22:21.406Z] 11:22:21     INFO - GECKO(1128) |     #4 0x7f69d01a0524 in SSLExp_SetResumptionToken /builds/worker/workspace/build/src/security/nss/lib/ssl/sslsock.c:4358:11
[task 2020-02-04T11:22:21.407Z] 11:22:21     INFO - GECKO(1128) |     #5 0x7f69b3c377bb in mozilla::net::nsSocketTransport::InitiateSocket() /builds

However, I think the session id should be released when the socket is destroyed, like the stack below. I did add some logs before calling ssl_DestroySID and I can see the the session id is really released.
I have no idea why ASAN still thinks that session id is leaked. Dana, maybe you have an idea about this?

  * frame #0: 0x000000010123c39d libnss3.dylib`ssl_DestroySID(sid=0x0000000138ca9900, freeIt=1) at sslnonce.c:170:9 [opt]
    frame #1: 0x000000010123c546 libnss3.dylib`ssl_FreeSID [inlined] ssl_FreeLockedSID(sid=0x0000000138ca9900) at sslnonce.c:222:9 [opt]
    frame #2: 0x000000010123c517 libnss3.dylib`ssl_FreeSID(sid=0x0000000138ca9900) at sslnonce.c:237 [opt]
    frame #3: 0x000000010124008c libnss3.dylib`ssl_DestroySecurityInfo [inlined] ssl_ResetSecurityInfo(sec=0x000000012b5f10a8, doMemset=0) at sslsecur.c:602:9 [opt]
    frame #4: 0x0000000101240039 libnss3.dylib`ssl_DestroySecurityInfo(sec=0x000000012b5f10a8) at sslsecur.c:618 [opt]
    frame #5: 0x0000000101243f1a libnss3.dylib`ssl_DestroySocketContents(ss=0x000000012b5f1000) at sslsock.c:441:5 [opt]
    frame #6: 0x0000000101243cd4 libnss3.dylib`ssl_FreeSocket(ss=0x000000012b5f1000) at sslsock.c:492:5 [opt]
    frame #7: 0x000000010123a0f6 libnss3.dylib`ssl_DefClose(ss=0x000000012b5f1000) at ssldef.c:221:5 [opt]
    frame #8: 0x000000011589ffa5 XUL`nsNSSSocketInfo::CloseSocketAndDestroy(this=0x00000001346a6200) at nsNSSIOLayer.cpp:679:21 [opt]
    frame #9: 0x00000001103007d1 XUL`mozilla::net::nsSocketTransport::ReleaseFD_Locked(PRFileDesc*) [inlined] mozilla::net::nsSocketTransport::CloseSocket(aFd=<unavailable>, aTelemetryEnabled=<unavailable>) at nsSocketTransport2.cpp:3589:3 [opt]
    frame #10: 0x00000001103007c2 XUL`mozilla::net::nsSocketTransport::ReleaseFD_Locked(this=0x0000000135041000, fd=<unavailable>) at nsSocketTransport2.cpp:2145 [opt]
    frame #11: 0x0000000110306ef6 XUL`mozilla::net::nsSocketTransport::OnSocketDetached(this=0x0000000135041000, fd=<unavailable>) at nsSocketTransport2.cpp:2528:7 [opt]

Comment 3

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Odd. I guess I would check that ssl_DestroySID gets called as many times as ssl3_NewSessionID if you haven't already. Other than that, maybe an NSS developer like :kjacobs would have a better idea.

Comment 4

[Kershaw Chang [:kershaw]]

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #3)

Odd. I guess I would check that ssl_DestroySID gets called as many times as ssl3_NewSessionID if you haven't already. Other than that, maybe an NSS developer like :kjacobs would have a better idea.

Yes, I already did this check. Please see https://treeherder.mozilla.org/#/jobs?repo=try&revision=c2af0c977732d5051478b36ca85a50ee82319dca&selectedJob=288214229.

:kjacobs, do you probably have an idea about this? Could you point me a direction of how to debug this?
Thanks.

Comment 5

[Kevin Jacobs [:kjacobs]]

I think this is getting leaked on reallocation. First in ssl3_NewSessionID and again (without free) in ssl_DecodeResumptionToken.

Edit: Confirmed and filed bug 1614870 in NSS.

Comment 6

[Kershaw Chang [:kershaw]]

(In reply to Kevin Jacobs [:kjacobs] from comment #5)

I think this is getting leaked on reallocation. First in ssl3_NewSessionID and again (without free) in ssl_DecodeResumptionToken.

Edit: Confirmed and filed bug 1614870 in NSS.

Thanks!

Comment 7

[Kershaw Chang [:kershaw]]

Attachment: Bug 1607257 - Enable ssl tokens cache on nightly

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c5a8b641d905
Enable ssl tokens cache on nightly r=keeler

Comment 9

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/c5a8b641d905