Closed Bug 1503822 Opened Last year Closed Last year

Intermittent PROCESS-CRASH | damp | application crashed [@ libsystem_kernel.dylib + 0x16166]

Categories

(Core :: JavaScript Engine, defect, P5, critical)

defect

Tracking

()

RESOLVED FIXED
mozilla65
Tracking Status
firefox-esr60 --- unaffected
firefox63 --- unaffected
firefox64 --- fixed
firefox65 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: ochameau)

References

Details

(Keywords: crash, intermittent-failure, Whiteboard: [stockwell disable-recommended])

Crash Data

Attachments

(1 file)

Filed by: rgurzau [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=209105472&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/QTINVQa5SAiz_6WrLxtU9g/runs/0/artifacts/public/logs/live_backing.log

01:38:25     INFO -  TEST-INFO | 2120: exit 0
01:38:25     INFO -  mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/YTiCv8VhRDWMr8-dXaOBIw/artifacts/public/build/target.crashreporter-symbols.zip
01:39:25     INFO -  mozcrash Copy/paste: /Users/cltbld/tasks/task_1541058480/build/macosx64-minidump_stackwalk /var/folders/xf/qrthc7gd4tg0fs0j2pl985wm00000x/T/tmpR0Fajp/profile/minidumps/2116DB2B-F283-420B-9587-F01D9F3E07A7-browser.dmp /var/folders/xf/qrthc7gd4tg0fs0j2pl985wm00000x/T/tmpwnTvVJ
01:39:43     INFO -  mozcrash Saved minidump as /Users/cltbld/tasks/task_1541058480/build/blobber_upload_dir/2116DB2B-F283-420B-9587-F01D9F3E07A7-browser.dmp
01:39:43     INFO -  PROCESS-CRASH | damp | application crashed [@ libsystem_kernel.dylib + 0x16166]
01:39:43     INFO -  Crash dump filename: /var/folders/xf/qrthc7gd4tg0fs0j2pl985wm00000x/T/tmpR0Fajp/profile/minidumps/2116DB2B-F283-420B-9587-F01D9F3E07A7-browser.dmp
01:39:43     INFO -  Operating system: Mac OS X
01:39:43     INFO -                    10.10.5 14F27
01:39:43     INFO -  CPU: amd64
01:39:43     INFO -       family 6 model 69 stepping 1
01:39:43     INFO -       4 CPUs
01:39:43     INFO -  GPU: UNKNOWN
01:39:43     INFO -  Crash reason:  EXC_BREAKPOINT / EXC_I386_BPT
01:39:43     INFO -  Crash address: 0x7fff87ee2166
01:39:43     INFO -  Process uptime: 83 seconds
01:39:43     INFO -  Thread 0 (crashed)
01:39:43     INFO -   0  libsystem_kernel.dylib + 0x16166
01:39:43     INFO -      rax = 0x000000000200012d   rdx = 0x0000000000000000
01:39:43     INFO -      rcx = 0x00007fff58ed1c48   rbx = 0x0000000000000203
01:39:43     INFO -      rsi = 0x0000000000000203   rdi = 0x00007fff58ed2068
01:39:43     INFO -      rbp = 0x00007fff58ed1cd0   rsp = 0x00007fff58ed1c48
01:39:43     INFO -       r8 = 0x0000000000002060    r9 = 0x0000000000000000
01:39:43     INFO -      r10 = 0x000000000000f155   r11 = 0x0000000000000246
01:39:43     INFO -      r12 = 0x00007fff58ed2080   r13 = 0x0000000000000203
01:39:43     INFO -      r14 = 0x0000000000000000   r15 = 0x00007fff58ed2068
01:39:43     INFO -      rip = 0x00007fff87ee2166
01:39:43     INFO -      Found by: given as instruction pointer in context
01:39:43     INFO -   1  XUL!google_breakpad::ExceptionHandler::WriteMinidump(bool) [exception_handler.cc:e9b8b0ba18f4a73caa8425c60ebd5c4df133f58d : 300 + 0x5]
01:39:43     INFO -      rbp = 0x00007fff58ed1f60   rsp = 0x00007fff58ed1ce0
01:39:43     INFO -      rip = 0x000000010c2753a8
01:39:43     INFO -      Found by: previous frame's frame pointer
01:39:43     INFO -   2  XUL!CrashReporter::CreateMinidumpsAndPair(unsigned int, unsigned int, nsTSubstring<char> const&, nsIFile*, nsIFile**) [exception_handler.cc:e9b8b0ba18f4a73caa8425c60ebd5c4df133f58d : 315 + 0xd]
01:39:43     INFO -      rbp = 0x00007fff58ed20f0   rsp = 0x00007fff58ed1f70
01:39:43     INFO -      rip = 0x000000010c27297c
01:39:43     INFO -      Found by: previous frame's frame pointer
01:39:43     INFO -   3  XUL!mozilla::dom::ContentParent::KillHard(char const*) [CrashReporterHost.h:e9b8b0ba18f4a73caa8425c60ebd5c4df133f58d : 77 + 0x7]
01:39:43     INFO -      rbp = 0x00007fff58ed21c0   rsp = 0x00007fff58ed2100
01:39:43     INFO -      rip = 0x000000010a86b644
01:39:43     INFO -      Found by: previous frame's frame pointer
01:39:43     INFO -   4  XUL!nsTimerImpl::Fire(int) [nsTimerImpl.cpp:e9b8b0ba18f4a73caa8425c60ebd5c4df133f58d : 684 + 0x7]
01:39:43     INFO -      rbp = 0x00007fff58ed2680   rsp = 0x00007fff58ed21d0
01:39:43     INFO -      rip = 0x0000000107c4076d
01:39:43     INFO -      Found by: previous frame's frame pointer
01:39:43     INFO -   5  XUL!nsTimerEvent::Run() [TimerThread.cpp:e9b8b0ba18f4a73caa8425c60ebd5c4df133f58d : 297 + 0x5]
01:39:43     INFO -      rbp = 0x00007fff58ed26b0   rsp = 0x00007fff58ed2690
01:39:43     INFO -      rip = 0x0000000107c2bac7
01:39:43     INFO -      Found by: previous frame's frame pointer
01:39:43     INFO -   6  XUL!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:e9b8b0ba18f4a73caa8425c60ebd5c4df133f58d : 1245 + 0x9]
01:39:43     INFO -      rbp = 0x00007fff58ed2bf0   rsp = 0x00007fff58ed26c0
01:39:43     INFO -      rip = 0x0000000107c32d89
01:39:43     INFO -      Found by: previous frame's frame pointer
01:39:43     INFO -   7  XUL!nsThread::Shutdown() [nsThreadUtils.cpp:e9b8b0ba18f4a73caa8425c60ebd5c4df133f58d : 530 + 0x11]
01:39:43     INFO -      rbp = 0x00007fff58ed2c30   rsp = 0x00007fff58ed2c00
01:39:43     INFO -      rip = 0x0000000107c31e8c
01:39:43     INFO -      Found by: previous frame's frame pointer
01:39:43     INFO -   8  XUL!mozilla::dom::workerinternals::RuntimeService::ShutdownIdleThreads(nsITimer*, void*) [RuntimeService.cpp:e9b8b0ba18f4a73caa8425c60ebd5c4df133f58d : 1714 + 0x5]
01:39:43     INFO -      rbp = 0x00007fff58ed2d40   rsp = 0x00007fff58ed2c40
01:39:43     INFO -      rip = 0x000000010a8c4cf3
01:39:43     INFO -      Found by: previous frame's frame pointer
01:39:43     INFO -   9  XUL!nsTimerImpl::Fire(int) [nsTimerImpl.cpp:e9b8b0ba18f4a73caa8425c60ebd5c4df133f58d : 684 + 0x7]
01:39:43     INFO -      rbp = 0x00007fff58ed3200   rsp = 0x00007fff58ed2d50
01:39:43     INFO -      rip = 0x0000000107c4076d
01:39:43     INFO -      Found by: previous frame's frame pointer
01:39:43     INFO -  10  XUL!nsTimerEvent::Run() [TimerThread.cpp:e9b8b0ba18f4a73caa8425c60ebd5c4df133f58d : 297 + 0x5]
01:39:43     INFO -      rbp = 0x00007fff58ed3230   rsp = 0x00007fff58ed3210
01:39:43     INFO -      rip = 0x0000000107c2bac7
01:39:43     INFO -      Found by: previous frame's frame pointer
Crash Signature: [@ libsystem_kernel.dylib + 0x16166] → [@ libsystem_kernel.dylib + 0x16166] [@ SnowWhiteKiller::Trace(JS::Heap<JSObject*>*, char const*, void*) const] [@ js::jit::MacroAssembler::Push(js::jit::TypedOrValueRegister)] [@ mozilla::net::SubstitutingURL::Mutator::QueryInterface(nsID const&, void…
Component: Talos → JavaScript Engine
Product: Testing → Core
Version: Version 3 → unspecified
Crash Signature: , arena_t*)] [@ nsXULElement::~nsXULElement()] [@ Fold(JSContext*, js::frontend::ParseNode**, js::frontend::PerHandlerParser<js::frontend::FullParseHandler>&)] → , arena_t*)] [@ nsXULElement::~nsXULElement()] [@ Fold(JSContext*, js::frontend::ParseNode**, js::frontend::PerHandlerParser<js::frontend::FullParseHandler>&)] [@ BaselineCacheIRCompiler::compile()]
I think you meant to NI Jan here.
Flags: needinfo?(jcoppeard) → needinfo?(jdemooij)
Crash Signature: , arena_t*)] [@ nsXULElement::~nsXULElement()] [@ Fold(JSContext*, js::frontend::ParseNode**, js::frontend::PerHandlerParser<js::frontend::FullParseHandler>&)] [@ BaselineCacheIRCompiler::compile()] → , arena_t*)] [@ nsXULElement::~nsXULElement()] [@ Fold(JSContext*, js::frontend::ParseNode**, js::frontend::PerHandlerParser<js::frontend::FullParseHandler>&)] [@ BaselineCacheIRCompiler::compile()] [@ js::InternalCallOrConstruct(JSContext*, JS::CallA…
Crash Signature: , JS::CallArgs const&, js::MaybeConstruct)] → , JS::CallArgs const&, js::MaybeConstruct)] [@ js::MapObject::set_impl(JSContext*, JS::CallArgs const&)] [@ js::jit::JitcodeGlobalTable::allocateTower(unsigned int)]
Crash Signature: , JS::CallArgs const&, js::MaybeConstruct)] [@ js::MapObject::set_impl(JSContext*, JS::CallArgs const&)] [@ js::jit::JitcodeGlobalTable::allocateTower(unsigned int)] → , JS::CallArgs const&, js::MaybeConstruct)] [@ js::MapObject::set_impl(JSContext*, JS::CallArgs const&)] [@ js::jit::JitcodeGlobalTable::allocateTower(unsigned int)] @ js::NativeObject::create(JSContext*, js::gc::AllocKind, js::gc::InitialHeap, JS::Han…
Crash Signature: , JS::CallArgs const&, js::MaybeConstruct)] [@ js::MapObject::set_impl(JSContext*, JS::CallArgs const&)] [@ js::jit::JitcodeGlobalTable::allocateTower(unsigned int)] @ js::NativeObject::create(JSContext*, js::gc::AllocKind, js::gc::InitialHeap, JS::Han… → , JS::CallArgs const&, js::MaybeConstruct)] [@ js::MapObject::set_impl(JSContext*, JS::CallArgs const&)] [@ js::jit::JitcodeGlobalTable::allocateTower(unsigned int)] [@ js::NativeObject::create(JSContext*, js::gc::AllocKind, js::gc::InitialHeap, JS::Ha…
(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #1)
> Jon, can you take a look at this very frequent failure, please? Bug 1503522
> and bug 1503523 landed before this started.

These two bugs are pretty trivial and it's very unlikely they introduced crashes. What makes you think they're related? :)

Can we do some retriggers on the relevant revisions?
Flags: needinfo?(jdemooij)
Maybe this one?

https://hg.mozilla.org/integration/autoland/rev/496185f049334fd0e6e900676ed4468fdc623930

That made a change to DAMP right before this started so it looks very suspicious.
Flags: needinfo?(poirot.alex)
Retriggers confirm it's from bug 1480953. The JS engine commits had been regarded as suspicious because many of the crash signature were about it.
Blocks: 1480953
I'm looking into this.
Assignee: nobody → poirot.alex
Flags: needinfo?(poirot.alex)
It is not clear to me why a JS stack has been reported in this bug. To me, if I process the crash logs correctly, the crashing stack is the one from comment 0, which is about a Timer, not about JS.
So if that's really about a Timer, I don't have any clue of what is crashing here, we may have timer about anywhere...

But I blindly tried various workarounds, as it looks like it is crashing only when firefox closes, not when DAMP tests are still running. Unfortunately none of them worked:

* Wait 3seconds before printing to stdout test results, which should force the python script to close firefox:
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=ff28e479d7bc79111e01676fc4a503a3a764e893
  https://hg.mozilla.org/try/rev/6485bde694478365b6e57f0f8b52f25d4f3ca826

* Do not reset the default selected tab on test end:
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=8d0dec57bab146256e0033fc387fabbe17335ec5
  https://hg.mozilla.org/try/rev/9a72f610bee58b4d5aa3ff5231077cdd4a920e4d

Then I tried to revert piece by piece what landed in bug 1480953:

* Restore open animations, still fails:
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=50856b3802cc525b1620146e6c140dc46d76db69
  https://hg.mozilla.org/try/rev/fcfb92971e0a7f464a50360066da7692cbffa6ae

* Restore the 1s wait in netmonitor, seems to pass!
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=c66c0b3e2d787d8b499ca47a33729ba8bf4049cb
  https://hg.mozilla.org/try/rev/18aec1fcbcf8157ee552eaff928c6f2eeabab5e9

* Restore close animation, also seems to pass!
  https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=210183432&revision=b5774d6491cf99464892957762b5e44713d11c6f
  https://hg.mozilla.org/try/rev/b6a774845be9057a492a40107f9684d42d621225

So I'm not sure it is about tab animation, but rather about something happening on toolbox destroy that isn't waited correctly. But given the stack trace, there is no way to find the issue source.
Given all that, I think it is best to restore the 1s wait in netmonitor...
Pushed by apoirot@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5c21b021bcf3
Wait on netmonitor test end to prevent a crash during firefox shutdown. r=jdescottes
https://hg.mozilla.org/mozilla-central/rev/5c21b021bcf3
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla65
You need to log in before you can comment on or make changes to this bug.