1278460 - Intermittent PROCESS-CRASH | netwerk/test/unit_ipc/test_progress_wrap.js | application crashed [@ mozalloc_abort(char const*)]

Status: RESOLVED INCOMPLETE

(Core :: Networking, defect, P2)

Description

[Iris Hsiao [:ihsiao]]

16:41:15  WARNING -  PROCESS-CRASH | netwerk/test/unit_ipc/test_progress_wrap.js | application crashed [@ mozalloc_abort(char const*)]

https://treeherder.mozilla.org/logviewer.html#?job_id=29607608&repo=mozilla-inbound

Comment 1

[Valentin Gosu [:valentin] (he/him)]

This seems to be OS X 10.10 only.
11 failures in the past 5 days.

https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278460&entireHistory=true&tree=all

 16:41:15  WARNING -  PROCESS-CRASH | netwerk/test/unit_ipc/test_progress_wrap.js | application crashed [@ mozalloc_abort(char const*)]
 16:41:15     INFO -  Crash dump filename: /var/folders/z3/t5k6xjnj1b7c750r159xpw7800000w/T/xpc-other-ZxtiHO/8C66918B-6DA6-474E-A21A-642B4E60E735.dmp
 16:41:15     INFO -  Operating system: Mac OS X
 16:41:15     INFO -                    10.10.5 14F27
 16:41:15     INFO -  CPU: amd64
 16:41:15     INFO -       family 6 model 69 stepping 1
 16:41:15     INFO -       4 CPUs
 16:41:15     INFO -  Crash reason:  EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
 16:41:15     INFO -  Crash address: 0x0
 16:41:15     INFO -  Process uptime: 16 seconds
 16:41:15     INFO -  Thread 2 (crashed)
 16:41:15     INFO -   0  libmozglue.dylib!mozalloc_abort(char const*) [mozalloc_abort.cpp:865d7ba1d841 : 33 + 0x0]
 16:41:15     INFO -      rax = 0x0000000000000000   rdx = 0x00007fff72f4b1f8
 16:41:15     INFO -      rcx = 0x0000000000000000   rbx = 0x00007fff72f4bc50
 16:41:15     INFO -      rsi = 0x0000100000001000   rdi = 0x00000f0000001003
 16:41:15     INFO -      rbp = 0x000000010ad58e20   rsp = 0x000000010ad58e10
 16:41:15     INFO -       r8 = 0x000000010ad58dc0    r9 = 0x000000010ad59000
 16:41:15     INFO -      r10 = 0x0000000000000000   r11 = 0x0000000000000246
 16:41:15     INFO -      r12 = 0x00000001020aebed   r13 = 0x00007fff72f4bc50
 16:41:15     INFO -      r14 = 0x0000000000000366   r15 = 0x00000001020b2d30
 16:41:15     INFO -      rip = 0x00000001087c9d41
 16:41:15     INFO -      Found by: given as instruction pointer in context
 16:41:15     INFO -   1  libmozglue.dylib!abort [mozalloc_abort.cpp:865d7ba1d841 : 80 + 0xc]
 16:41:15     INFO -      rbx = 0x00000001020af330   rbp = 0x000000010ad58e30
 16:41:15     INFO -      rsp = 0x000000010ad58e30   r12 = 0x00000001020aebed
 16:41:15     INFO -      r13 = 0x00007fff72f4bc50   r14 = 0x0000000000000366
 16:41:15     INFO -      r15 = 0x00000001020b2d30   rip = 0x00000001087c9d70
 16:41:15     INFO -      Found by: call frame info
 16:41:15     INFO -   2  libnss3.dylib!PR_Assert [prlog.c:865d7ba1d841 : 553 + 0x5]
 16:41:15     INFO -      rbx = 0x00000001020af330   rbp = 0x000000010ad58e70
 16:41:15     INFO -      rsp = 0x000000010ad58e40   r12 = 0x00000001020aebed
 16:41:15     INFO -      r13 = 0x00007fff72f4bc50   r14 = 0x0000000000000366
 16:41:15     INFO -      r15 = 0x00000001020b2d30   rip = 0x000000010204eaed
 16:41:15     INFO -      Found by: call frame info
 16:41:15     INFO -   3  libsystem_pthread.dylib!_pthread_tsd_cleanup + 0x56
 16:41:15     INFO -      rbx = 0x0000000000000103   rbp = 0x000000010ad58ea0
 16:41:15     INFO -      rsp = 0x000000010ad58e80   r12 = 0x000000010ad59000
 16:41:15     INFO -      r13 = 0x00000000000008ff   r14 = 0x00007fff70f36c40
 16:41:15     INFO -      r15 = 0x0000000000000000   rip = 0x00007fff8370f739
 16:41:15     INFO -      Found by: call frame info
 16:41:15     INFO -   4  libsystem_pthread.dylib!_pthread_exit + 0x75
 16:41:15     INFO -      rbp = 0x000000010ad58ef0   rsp = 0x000000010ad58eb0
 16:41:15     INFO -      rip = 0x00007fff8370f460
 16:41:15     INFO -      Found by: previous frame's frame pointer
 16:41:15     INFO -   5  libsystem_pthread.dylib!_pthread_body + 0x8e
 16:41:15     INFO -      rbp = 0x000000010ad58f10   rsp = 0x000000010ad58f00
 16:41:15     INFO -      rip = 0x00007fff83710065
 16:41:15     INFO -      Found by: previous frame's frame pointer
 16:41:15     INFO -   6  libsystem_pthread.dylib!_pthread_start + 0xb0
 16:41:15     INFO -      rbp = 0x000000010ad58f50   rsp = 0x000000010ad58f20
 16:41:15     INFO -      rip = 0x00007fff8370ffd7
 16:41:15     INFO -      Found by: previous frame's frame pointer
 16:41:15     INFO -   7  libsystem_pthread.dylib!thread_start + 0xd
 16:41:15     INFO -      rbp = 0x000000010ad58f78   rsp = 0x000000010ad58f60
 16:41:15     INFO -      rip = 0x00007fff8370d3ed
 16:41:15     INFO -      Found by: previous frame's frame pointer
 16:41:15     INFO -   8  XUL + 0x684b50
 16:41:15     INFO -      rsp = 0x000000010ad59030   rip = 0x00000001027c8b50
 16:41:15     INFO -      Found by: stack scanning

Comment 2

[Valentin Gosu [:valentin] (he/him)]

Nick, you knew some OS X stuff, right?
Any clue what's going on here?

Comment 3

[Intermittent Failures Robot]

13 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 8
* fx-team: 5

Platform breakdown:
* osx-10-10: 13

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278460&startday=2016-06-06&endday=2016-06-12&tree=all

Comment 4

[u408661]

(In reply to Valentin Gosu [:valentin] from comment #2)
> Nick, you knew some OS X stuff, right?

Not really, I just happen to use it for dev. I don't know anything os x-specific :)

> Any clue what's going on here?

Nope - this stack is not particularly helpful. Looks like we're trying to start a thread, and something goes wrong there, causing an assert failure. Problem is, PR_Assert (which appears in the stack) should fprintf something to stderr, stating what assertion failed, and that appears nowhere. So, either the stack is misleading (most likely, IMHO), or something even worse is going on. Either way, we're in an ugly position for figuring it out. Making things even uglier - this is another one of those bugs that's not particularly common so far, so reproducing and debugging locally could take some time.

Comment 5

[Valentin Gosu [:valentin] (he/him)]

In that case I'm going to put this into the backlog for now.

Comment 6

[Patrick McManus [:mcmanus]]

this isn't that low volume and its new.. maybe 15 crash reports listed in brasstacks. I think we should at least give it a try.

seems to be crashing in pthread_tsd_cleanup, which it seems has something to do with thread local data .. but out of create/exit on the stack - which is certainly odd.

Comment 7

[u408661]

This same stack seems to be appearing in bug 1281307, bug 1281319, bug 1281509, and bug 1281694 - all of which are also e10s tests.

Comment 8

[BMO Automation]

Bulk assigning P3 to all open intermittent bugs without a priority set in Firefox components per bug 1298978.

Comment 9

[Intermittent Failures Robot]

5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 5

Platform breakdown:
* windowsxp: 4
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278460&startday=2016-10-03&endday=2016-10-09&tree=all

Comment 10

[Intermittent Failures Robot]

9 failures in 722 pushes (0.012 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 2
* mozilla-central: 2
* autoland: 2
* try: 1
* mozilla-beta: 1
* graphics: 1

Platform breakdown:
* osx-10-10: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278460&startday=2017-01-09&endday=2017-01-15&tree=all

Comment 11

[Intermittent Failures Robot]

6 failures in 812 pushes (0.007 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* mozilla-inbound: 4
* autoland: 2

Platform breakdown:
* osx-10-10: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278460&startday=2017-02-20&endday=2017-02-26&tree=all

Comment 12

[Intermittent Failures Robot]

2 failures in 814 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 2

Platform breakdown:
* windows8-64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278460&startday=2017-06-12&endday=2017-06-18&tree=all

Comment 13

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

The latest failures are on windows and shows different crash stack, which complains about potential deadlock between Timer and nsSocketTransportService. @dragana might have better insight on this.

>[Parent 12260] ###!!! ASSERTION: Potential deadlock detected:
>Cyclical dependency starts at
>Mutex : TimerThread.mMonitor
>Next dependency:
>Mutex : nsSocketTransportService::mLock (currently acquired)
>Next dependency:
>Mutex : nsTimerImpl::mMutex (currently acquired)
>Cycle completed at
>Mutex : TimerThread.mMonitor
>Deadlock may happen for some other execution


> 0  mozglue.dll!mozalloc_abort(char const * const) [mozalloc_abort.cpp:97877fd398ad : 33 + 0x1b]
> 1  xul.dll!NS_DebugBreak [nsDebugImpl.cpp:97877fd398ad : 438 + 0xd]
> 2  xul.dll!mozilla::BlockingResourceBase::CheckAcquire() [BlockingResourceBase.cpp:97877fd398ad : 308 + 0x24]
> 3  xul.dll!mozilla::OffTheBooksMutex::Lock() [BlockingResourceBase.cpp:97877fd398ad : 382 + 0x9]
> 4  xul.dll!TimerThread::RemoveTimer(nsTimerImpl *) [TimerThread.cpp:97877fd398ad : 575 + 0x9]
> 5  xul.dll!nsTimerImpl::InitCommon(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator> const &,unsigned int,nsTimerImpl::Callback &&) [nsTimerImpl.cpp:97877fd398ad : 223 + 0x8]
> 6  xul.dll!nsTimerImpl::InitCommon(unsigned int,unsigned int,nsTimerImpl::Callback &&) [nsTimerImpl.cpp:97877fd398ad : 203 + 0x2e]
> 7  xul.dll!nsTimerImpl::Init(nsIObserver *,unsigned int,unsigned int) [nsTimerImpl.cpp:97877fd398ad : 328 + 0xe]
> 8  xul.dll!<lambda_5a2b5c6d24e46eeba8c7c381d7cd37fb>::operator() [nsSocketTransportService2.cpp:97877fd398ad : 1626 + 0x3f]
> 9  xul.dll!mozilla::detail::RunnableFunction<<lambda_5a2b5c6d24e46eeba8c7c381d7cd37fb> >::Run() [nsThreadUtils.h:97877fd398ad : 460 + 0x9]
>10  xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:97877fd398ad : 1406 + 0x14]
>11  xul.dll!NS_ProcessNextEvent(nsIThread *,bool) [nsThreadUtils.cpp:97877fd398ad : 472 + 0xd]
>12  xul.dll!mozilla::SpinEventLoopUntil<1,<lambda_56960601ff793302007fcdece32fa3b7> >(<lambda_56960601ff793302007fcdece32fa3b7> &&,nsIThread *) [nsThreadUtils.h:97877fd398ad : 281 + 0xa]
>13  xul.dll!nsThread::Shutdown() [nsThread.cpp:97877fd398ad : 1032 + 0x13]
>14  xul.dll!nsDestroyThreadEvent::Run() [nsCacheUtils.cpp:97877fd398ad : 20 + 0xf]
>15  xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:97877fd398ad : 1406 + 0x14]
>16  xul.dll!NS_ProcessNextEvent(nsIThread *,bool) [nsThreadUtils.cpp:97877fd398ad : 472 + 0xd]
>17  xul.dll!mozilla::SpinEventLoopUntil<1,<lambda_ba23c3a8cdf222e68bd8d404642209b6> >(<lambda_ba23c3a8cdf222e68bd8d404642209b6> &&,nsIThread *) [nsThreadUtils.h:97877fd398ad : 281 + 0xa]
>18  xul.dll!mozilla::net::nsHttpConnectionMgr::Shutdown() [nsHttpConnectionMgr.cpp:97877fd398ad : 245 + 0x1a]
>19  xul.dll!mozilla::net::nsHttpHandler::ShutdownConnectionManager() [nsHttpHandler.cpp:97877fd398ad : 2622 + 0x5]
>20  xul.dll!mozilla::net::nsHttpHandler::Observe(nsISupports *,char const *,char16_t const *) [nsHttpHandler.cpp:97877fd398ad : 2237 + 0x9]
>21  xul.dll!nsObserverList::NotifyObservers(nsISupports *,char const *,char16_t const *) [nsObserverList.cpp:97877fd398ad : 112 + 0x19]
>22  xul.dll!nsObserverService::NotifyObservers(nsISupports *,char const *,char16_t const *) [nsObserverService.cpp:97877fd398ad : 288 + 0x11]
>23  xul.dll!mozilla::ShutdownXPCOM(nsIServiceManager *) [XPCOMInit.cpp:97877fd398ad : 879 + 0x20]
>24  xul.dll!XRE_XPCShellMain(int,char * *,char * *,XREShellData const *) [XPCShellImpl.cpp:97877fd398ad : 1550 + 0x7]
>25  xpcshell.exe!NS_internal_main(int,char * *,char * *) [xpcshell.cpp:97877fd398ad : 68 + 0x18]
>26  xpcshell.exe!wmain [nsWindowsWMain.cpp:97877fd398ad : 115 + 0x14]
>27  xpcshell.exe!__scrt_common_main_seh [exe_common.inl : 253 + 0x22]
>28  kernel32.dll!BaseThreadInitThunk + 0x1a
>29  ntdll.dll!RtlUserThreadStart + 0x21
>30  KERNELBASE.dll!GetLegacyComposition + 0x1180

Comment 14

[Dragana Damjanovic [:dragana]]

Is this(comment 13) the same as bug 1370448?

Comment 15

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

Maybe a different issue since bug 1370448 is landed two weeks ago but still seeing failure last week.

Comment 16

[(no longer active)]

Comment 13 is bug 1370448 but it seems like it was misstarred...

Comment 17

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

no instances in 4 months, lets close this out

Comment 19

[Cosmin Sabou [:CosminS]]

https://treeherder.mozilla.org/logviewer.html#?job_id=188061276&repo=mozilla-inbound&lineNumber=4674

Comment 20

[Intermittent Failures Robot]

1 failures in 623 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1278460&startday=2018-07-09&endday=2018-07-15&tree=trunk

Comment 21

[Selena Deckelmann :selenamarie :selena]

Code referenced seems unrelated. Given that this hasn't recurred, closing.