Closed Bug 1239609 Opened 4 years ago Closed 4 years ago

Intermittent test_hmac.js | xpcshell return code: -11 | application crashed [@ nsNSSShutDownObject::shutdown(nsNSSShutDownObject::CalledFromType)]

Categories

(Core :: Security: PSM, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
mozilla47
Tracking Status
firefox46 --- fixed
firefox47 --- fixed

People

(Reporter: cbook, Assigned: keeler)

References

()

Details

(Keywords: crash, intermittent-failure)

Attachments

(1 file)

https://treeherder.mozilla.org/logviewer.html#?job_id=6590212&repo=fx-team 

16:15:22 WARNING - TEST-UNEXPECTED-FAIL | security/manager/ssl/tests/unit/test_hmac.js | xpcshell return code: -11

 16:15:33  WARNING -  PROCESS-CRASH | security/manager/ssl/tests/unit/test_hmac.js | application crashed [@ nsNSSShutDownObject::shutdown(nsNSSShutDownObject::CalledFromType)]
 16:15:33     INFO -  Crash dump filename: /tmp/xpc-other-tEv2FF/33f86c79-d20f-fb55-658ed573-3bcf523e.dmp
 16:15:33     INFO -  Operating system: Linux
 16:15:33     INFO -                    0.0.0 Linux 3.2.0-76-generic #111-Ubuntu SMP Tue Jan 13 22:16:09 UTC 2015 x86_64
 16:15:33     INFO -  CPU: amd64
 16:15:33     INFO -       family 6 model 62 stepping 4
 16:15:33     INFO -       1 CPU
 16:15:33     INFO -  Crash reason:  SIGSEGV
 16:15:33     INFO -  Crash address: 0x28
 16:15:33     INFO -  Process uptime: not available
 16:15:33     INFO -  Thread 0 (crashed)
 16:15:33     INFO -   0  libxul.so!nsNSSShutDownObject::shutdown(nsNSSShutDownObject::CalledFromType) [nsNSSShutDown.h:f71701b82ab0 : 203 + 0x3]
 16:15:33     INFO -      rax = 0x0000000000000018   rdx = 0x00007fde8132d060
 16:15:33     INFO -      rcx = 0x00007fde89d0d410   rbx = 0x00007fde81349078
 16:15:33     INFO -      rsi = 0x0000000000000000   rdi = 0x00007fde81349078
 16:15:33     INFO -      rbp = 0x00007fffb9298920   rsp = 0x00007fffb9298910
 16:15:33     INFO -       r8 = 0x0000000000000001    r9 = 0x0000000000002ad0
 16:15:33     INFO -      r10 = 0x00007fffb92984e0   r11 = 0x0000000000000000
 16:15:33     INFO -      r12 = 0x00007fde89d68320   r13 = 0x00007fde89d683a0
 16:15:33     INFO -      r14 = 0x00007fde813a1280   r15 = 0x00007fde89d0c2a8
 16:15:33     INFO -      rip = 0x00007fde9573d133
 16:15:33     INFO -      Found by: given as instruction pointer in context
 16:15:33     INFO -   1  libxul.so!nsNSSShutDownList::evaporateAllNSSResources() [nsNSSShutDown.cpp:f71701b82ab0 : 136 + 0xb]
 16:15:33     INFO -      rbx = 0x00007fffb9298958   rbp = 0x00007fffb92989b0
 16:15:33     INFO -      rsp = 0x00007fffb9298930   r12 = 0x00007fde89d68320
 16:15:33     INFO -      r13 = 0x00007fde89d683a0   r14 = 0x00007fde813a1280
 16:15:33     INFO -      r15 = 0x00007fde89d0c2a8   rip = 0x00007fde95754ce7
 16:15:33     INFO -      Found by: call frame info
 16:15:33     INFO -   2  libxul.so!nsNSSComponent::ShutdownNSS() [nsNSSComponent.cpp:f71701b82ab0 : 1145 + 0x9]
 16:15:33     INFO -      rbx = 0x00007fde8132a110   rbp = 0x00007fffb92989f0
 16:15:33     INFO -      rsp = 0x00007fffb92989c0   r12 = 0x00007fde815974c0
 16:15:33     INFO -      r13 = 0x0000000000000000   r14 = 0x00007fffb92990c0
 16:15:33     INFO -      r15 = 0x00007fde89d0c2a8   rip = 0x00007fde95757088
 16:15:33     INFO -      Found by: call frame info
 16:15:33     INFO -   3  libxul.so!nsNSSComponent::~nsNSSComponent() [nsNSSComponent.cpp:f71701b82ab0 : 270 + 0x8]
 16:15:33     INFO -      rbx = 0x00007fde8132a110   rbp = 0x00007fffb9298a10
 16:15:33     INFO -      rsp = 0x00007fffb9298a00   r12 = 0x00007fde815974c0
 16:15:33     INFO -      r13 = 0x0000000000000000   r14 = 0x00007fffb92990c0
 16:15:33     INFO -      r15 = 0x00007fde89d0c2a8   rip = 0x00007fde95759350
 16:15:33     INFO -      Found by: call frame info
 16:15:33     INFO -   4  libxul.so!nsNSSComponent::~nsNSSComponent() [nsNSSComponent.cpp:f71701b82ab0 : 281 + 0x5]
 16:15:33     INFO -      rbx = 0x00007fde8132a110   rbp = 0x00007fffb9298a30
 16:15:33     INFO -      rsp = 0x00007fffb9298a20   r12 = 0x00007fde8132a128
 16:15:33     INFO -      r13 = 0x0000000000000000   r14 = 0x00007fffb92990c0
 16:15:33     INFO -      r15 = 0x00007fde89d0c2a8   rip = 0x00007fde95759417
 16:15:33     INFO -      Found by: call frame info
 16:15:33     INFO -   5  libxul.so!nsNSSComponent::Release() [nsNSSComponent.cpp:f71701b82ab0 : 1223 + 0x8]
 16:15:33     INFO -      rbx = 0x00007fde8132a110   rbp = 0x00007fffb9298a60
 16:15:33     INFO -      rsp = 0x00007fffb9298a40   r12 = 0x00007fde8132a128
 16:15:33     INFO -      r13 = 0x0000000000000000   r14 = 0x00007fffb92990c0
 16:15:33     INFO -      r15 = 0x00007fde89d0c2a8   rip = 0x00007fde957594b4
 16:15:33     INFO -      Found by: call frame info
 16:15:33     INFO -   6  libxul.so!nsComponentManagerImpl::FreeServices() [nsCOMPtr.h:f71701b82ab0 : 886 + 0xc]
 16:15:33     INFO -      rbx = 0x00007fffb9298a70   rbp = 0x00007fffb9298ac0
 16:15:33     INFO -      rsp = 0x00007fffb9298a70   r12 = 0x00007fde8613db60
 16:15:33     INFO -      r13 = 0x00007fde8613db70   r14 = 0x00007fffb92990c0
 16:15:33     INFO -      r15 = 0x00007fde89d0c2a8   rip = 0x00007fde9371f16e
 16:15:33     INFO -      Found by: call frame info
 16:15:33     INFO -   7  libxul.so!mozilla::ShutdownXPCOM(nsIServiceManager*) [XPCOMInit.cpp:f71701b82ab0 : 910 + 0x5]
 16:15:33     INFO -      rbx = 0x0000000000000000   rbp = 0x00007fffb9298b10
 16:15:33     INFO -      rsp = 0x00007fffb9298ad0   r12 = 0x00007fde89d6a338
 16:15:33     INFO -      r13 = 0x00007fde84703000   r14 = 0x00007fffb92990c0
 16:15:33     INFO -      r15 = 0x00007fde89d0c2a8   rip = 0x00007fde9375084a
 16:15:33     INFO -      Found by: call frame info
 16:15:33     INFO -   8  libxul.so!XRE_XPCShellMain [XPCShellImpl.cpp:f71701b82ab0 : 1572 + 0x7]
 16:15:33     INFO -      rbx = 0x0000000000000000   rbp = 0x00007fffb9298f70
 16:15:33     INFO -      rsp = 0x00007fffb9298b20   r12 = 0x0000000000000000
 16:15:33     INFO -      r13 = 0x00007fde84703000   r14 = 0x00007fffb92990c0
 16:15:33     INFO -      r15 = 0x00007fde89d0c2a8   rip = 0x00007fde93d52055
 16:15:33     INFO -      Found by: call frame info
 16:15:33     INFO -   9  xpcshell!main [xpcshell.cpp:f71701b82ab0 : 54 + 0x10]
 16:15:33     INFO -      rbx = 0x0000000000000000   rbp = 0x00007fffb9298fa0
 16:15:33     INFO -      rsp = 0x00007fffb9298f80   r12 = 0x0000000000404f80
 16:15:33     INFO -      r13 = 0x00007fffb9299080   r14 = 0x0000000000000000
 16:15:33     INFO -      r15 = 0x0000000000000000   rip = 0x0000000000404ea1
 16:15:33     INFO -      Found by: call frame info
 16:15:33     INFO -  10  libc-2.15.so!__libc_start_main + 0xed
 16:15:33     INFO -      rbx = 0x0000000000000000   rbp = 0x0000000000000000
 16:15:33     INFO -      rsp = 0x00007fffb9298fb0   r12 = 0x0000000000404f80
 16:15:33     INFO -      r13 = 0x00007fffb9299080   r14 = 0x0000000000000000
 16:15:33     INFO -      r15 = 0x0000000000000000   rip = 0x00007fde8fd3d76d
 16:15:33     INFO -      Found by: call frame info
 16:15:33     INFO -  11  xpcshell!_start + 0x29
 16:15:33     INFO -      rbx = 0x0000000000000000   rbp = 0x0000000000000000
 16:15:33     INFO -      rsp = 0x00007fffb9299070   r12 = 0x0000000000404f80
 16:15:33     INFO -      r13 = 0x00007fffb9299080   r14 = 0x0000000000000000
 16:15:33     INFO -      r15 = 0x0000000000000000   rip = 0x0000000000404fa9
 16:15:33     INFO -      Found by: call frame info
 16:15:33     INFO -  Thread 1
 16:15:33     INFO -   0  libc-2.15.so!__poll + 0x53
 16:15:33     INFO -      rax = 0xfffffffffffffdfc   rdx = 0xffffffffffffffff
 16:15:33     INFO -      rcx = 0xffffffffffffffff   rbx = 0x00007fde89d1ebe0
 16:15:33     INFO -      rsi = 0x0000000000000001   rdi = 0x00007fde89d0c2f8
 16:15:33     INFO -      rbp = 0x00007fde89d0c2f8   rsp = 0x00007fde890a8af0
 16:15:33     INFO -       r8 = 0x0000000000000000    r9 = 0x0000000000002ad1
 16:15:33     INFO -      r10 = 0x00007fde890a8780   r11 = 0x0000000000000293
 16:15:33     INFO -      r12 = 0x00007fde90941f00   r13 = 0x00000000ffffffff
 16:15:33     INFO -      r14 = 0x0000000000000001   r15 = 0x0000000000000001
 16:15:33     INFO -      rip = 0x00007fde8fe04933
 16:15:33     INFO -      Found by: given as instruction pointer in context
 16:15:33     INFO -   1  libglib-2.0.so.0.3200.1!g_main_context_iterate [gmain.c : 3417 + 0xc]
 16:15:33     INFO -      rbx = 0x00007fde89d1ebe0   rbp = 0x00007fde89d0c2f8
 16:15:33     INFO -      rsp = 0x00007fde890a8b20   r12 = 0x00007fde90941f00
 16:15:33     INFO -      r13 = 0x00000000ffffffff   r14 = 0x0000000000000001
 16:15:33     INFO -      r15 = 0x0000000000000001   rip = 0x00007fde90934ff6
 16:15:33     INFO -      Found by: call frame info
 16:15:33     INFO -   2  libglib-2.0.so.0.3200.1!g_main_loop_run [gmain.c : 3317 + 0x12]
 16:15:33     INFO -      rbx = 0x00007fde89d2a850   rbp = 0x0000000000000000
 16:15:33     INFO -      rsp = 0x00007fde890a8b70   r12 = 0x00007fffb9298850
 16:15:33     INFO -      r13 = 0x00007fde890a99c0   r14 = 0x0000000000000000
 16:15:33     INFO -      r15 = 0x0000000000000003   rip = 0x00007fde9093545a
 16:15:33     INFO -      Found by: call frame info
 16:15:33     INFO -   3  libdconfsettings.so!dconf_context_thread [dconfcontext.c : 11 + 0x8]
Assignee: nobody → nobody
Component: Test → Security: PSM
Product: NSS → Core
Version: 3.2.1 → unspecified
It turns out I can reliably reproduce the crash on my local setup.

It looks like the Part 2 patch of Bug 1230377 introduced the crash:
 - https://hg.mozilla.org/integration/mozilla-inbound/rev/7e6a05835d04 is fine
 - https://hg.mozilla.org/integration/mozilla-inbound/rev/705dc2c96e94 and after crash
Blocks: 1230377
FWIW, on my setup, reverting the NS_KEYMODULEOBJECTFACTORY_CID value fixes the crash.

(Alternatively, doing |hg update -r 7e6a05835d04| then doing nothing except update the NS_KEYMODULEOBJECTFACTORY_CID value introduces the crash).
Thanks for having a look at this. It looks like I didn't do the nsKeyObjectFactory destructor correctly in bug 1230377. I found a few other nsNSSShutDownObject destructors that were potentially problematic, so I fixed those as well.
Assignee: nobody → dkeeler
Comment on attachment 8711198 [details]
MozReview Request: bug 1239609 - audit nsNSSShutDownObject destructors for correctness r?Cykesiopka r?sworkman

https://reviewboard.mozilla.org/r/32015/#review28887

The security/ parts LGTM.

FWIW, I also did a non-in depth audit and didn't find any other problematic classes.

::: security/manager/ssl/nsNSSShutDown.h:127
(Diff revision 1)
>    must then call destructorSafeDestroyNSSReference() and then

You may want to integrate the new comment below here, or update this comment to say that in some cases it's not necessary and point to the new comment.

The comments kinda contradict each other otherwise.
Attachment #8711198 - Flags: review?(cykesiopka.bmo) → review+
Attachment #8711198 - Flags: review?(sworkman) → review+
Comment on attachment 8711198 [details]
MozReview Request: bug 1239609 - audit nsNSSShutDownObject destructors for correctness r?Cykesiopka r?sworkman

https://reviewboard.mozilla.org/r/32015/#review28967
:vchang may not be available. Cykesiopka, would you mind also reviewing the WifiCertService.cpp change? Thanks.
Flags: needinfo?(cykesiopka.bmo)
(In reply to David Keeler [:keeler] (use needinfo?) from comment #7)
> :vchang may not be available. Cykesiopka, would you mind also reviewing the
> WifiCertService.cpp change? Thanks.

Sure, I'll review that as well.
Flags: needinfo?(cykesiopka.bmo)
https://reviewboard.mozilla.org/r/32015/#review29259

The WifiCertService changes LGTM as well.
Comment on attachment 8711198 [details]
MozReview Request: bug 1239609 - audit nsNSSShutDownObject destructors for correctness r?Cykesiopka r?sworkman

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/32015/diff/1-2/
Attachment #8711198 - Attachment description: MozReview Request: bug 1239609 - audit nsNSSShutDownObject destructors for correctness r?Cykesiopka r?vchang r?sworkman → MozReview Request: bug 1239609 - audit nsNSSShutDownObject destructors for correctness r?Cykesiopka r?sworkman
Attachment #8711198 - Flags: review?(changyihsin)
Great - thanks. I updated the comment to make it more clear what the requirements were.
https://hg.mozilla.org/mozilla-central/rev/9a1977d9b21b
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla47
Comment on attachment 8711198 [details]
MozReview Request: bug 1239609 - audit nsNSSShutDownObject destructors for correctness r?Cykesiopka r?sworkman

Approval Request Comment
[Feature/regressing bug #]: bug 1230377
[User impact if declined]: potential crashes, UAF
[Describe test coverage new/current, TreeHerder]: has tests
[Risks and why]: low - it's pretty clear what the mistake was, and it's pretty clear that this is the correct fix
[String/UUID change made/needed]: none
Attachment #8711198 - Flags: approval-mozilla-aurora?
Comment on attachment 8711198 [details]
MozReview Request: bug 1239609 - audit nsNSSShutDownObject destructors for correctness r?Cykesiopka r?sworkman

Fixes recent regression, approved for uplift to aurora
Attachment #8711198 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
You need to log in before you can comment on or make changes to this bug.