Intermittent application crashed [@ mozilla::detail::InvalidArrayIndex_CRASH] | single tracking bug
Categories
(Core :: Disability Access APIs, defect)
Tracking
()
People
(Reporter: intermittent-bug-filer, Assigned: eeejay)
References
Details
(Keywords: crash, intermittent-failure, intermittent-testcase)
Crash Data
Attachments
(1 file)
Filed by: ayeddi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=426377360&repo=try
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/F-czDvO5RZ-6G0BtkGdFCg/runs/0/artifacts/public/logs/live_backing.log
Failed twice in a row while running Try with `--enable-a11y-checks` flag
Failed twice while running of the Linux 18.04 x64 WebRender opt Mochitests with accessibility checks enabled test-linux1804-64-qr/opt-mochitest-devtools-chrome-a11y-checks dt1 while testing WIP patch https://phabricator.services.mozilla.com/D118126
PROCESS-CRASH | application crashed [@ mozilla::detail::InvalidArrayIndex_CRASH] | devtools/client/webconsole/test/browser/browser_webconsole_console_logging_workers_api.js
Log
Comment hidden (Intermittent Failures Robot) |
Updated•1 year ago
|
New crashes on Try (devtools a11y-checks job):
PROCESS-CRASH | application crashed [@ mozilla::detail::InvalidArrayIndex_CRASH] | devtools/client/inspector/rules/test/browser_rules_color_scheme_simulation.js
PROCESS-CRASH | application crashed [@ mozilla::detail::InvalidArrayIndex_CRASH] | devtools/client/webconsole/test/browser/browser_webconsole_console_logging_workers_api.js
PROCESS-CRASH | application crashed [@ mozilla::detail::InvalidArrayIndex_CRASH] | devtools/client/debugger/test/mochitest/browser_dbg-features-source-tree.js
PROCESS-CRASH | application crashed [@ mozilla::detail::InvalidArrayIndex_CRASH] | devtools/client/framework/test/browser_toolbox_meatball.js
Comment hidden (Intermittent Failures Robot) |
new crashes (try):
CRASH devtools/client/webconsole/test/browser/browser_webconsole_split_persist.js
application crashed [@ mozilla::detail::InvalidArrayIndex_CRASH]
and from 10/6 run (try):
CRASH devtools/client/webconsole/test/browser/browser_webconsole_split_persist.js
application crashed [@ mozilla::detail::InvalidArrayIndex_CRASH]
Comment 6•1 year ago
|
||
This seems to happen outside of devtools as well, but I see accessibility related objects in the trace
PROCESS-CRASH | application crashed [@ mozilla::detail::InvalidArrayIndex_CRASH] | browser/components/urlbar/tests/browser/browser_searchMode_setURI.js
[task 2023-10-07T14:53:25.086Z] 14:53:25 INFO - 0 firefox-bin!MOZ_Crash(char const*, int, char const*) [Assertions.h:ef73c2e8cd831f2a46ff5704623db63232fd912d : 281]
[task 2023-10-07T14:53:25.086Z] 14:53:25 INFO - Found by: inlining
[task 2023-10-07T14:53:25.087Z] 14:53:25 INFO - 1 firefox-bin!mozilla::detail::InvalidArrayIndex_CRASH(unsigned long, unsigned long) [Assertions.cpp:ef73c2e8cd831f2a46ff5704623db63232fd912d : 50 + 0x0]
[task 2023-10-07T14:53:25.087Z] 14:53:25 INFO - rax = 0x00005637f583d830 rdx = 0x0000000000000001
[task 2023-10-07T14:53:25.088Z] 14:53:25 INFO - rcx = 0x0000000000000022 rbx = 0x00007f7d47f94790
[task 2023-10-07T14:53:25.088Z] 14:53:25 INFO - rsi = 0x00005637f577888d rdi = 0x00007fff86269bf0
[task 2023-10-07T14:53:25.089Z] 14:53:25 INFO - rbp = 0x00007fff8626a040 rsp = 0x00007fff8626a040
[task 2023-10-07T14:53:25.089Z] 14:53:25 INFO - r8 = 0x00000000ffffffff r9 = 0x0000000000000005
[task 2023-10-07T14:53:25.089Z] 14:53:25 INFO - r10 = 0x0000000000000001 r11 = 0x0000000000000000
[task 2023-10-07T14:53:25.090Z] 14:53:25 INFO - r12 = 0x0000000000000003 r13 = 0x0000000000000000
[task 2023-10-07T14:53:25.090Z] 14:53:25 INFO - r14 = 0x00007f7d19518900 r15 = 0x00007f7d47f94ca0
[task 2023-10-07T14:53:25.090Z] 14:53:25 INFO - rip = 0x00005637f5828a3a
[task 2023-10-07T14:53:25.091Z] 14:53:25 INFO - Found by: given as instruction pointer in context
[task 2023-10-07T14:53:25.091Z] 14:53:25 INFO - 2 libxul.so!nsTArray_Impl<mozilla::a11y::LocalAccessible*, nsTArrayInfallibleAllocator>::ElementAt(unsigned long) const [nsTArray.h:ef73c2e8cd831f2a46ff5704623db63232fd912d : 1217]
[task 2023-10-07T14:53:25.092Z] 14:53:25 INFO - Found by: inlining
[task 2023-10-07T14:53:25.092Z] 14:53:25 INFO - 3 libxul.so!mozilla::a11y::LocalAccessible::ContentChildAt(unsigned int) const [LocalAccessible.h:ef73c2e8cd831f2a46ff5704623db63232fd912d : 363]
[task 2023-10-07T14:53:25.092Z] 14:53:25 INFO - Found by: inlining
[task 2023-10-07T14:53:25.093Z] 14:53:25 INFO - 4 libxul.so!mozilla::a11y::DocAccessible::ShutdownChildrenInSubtree(mozilla::a11y::LocalAccessible*) [DocAccessible.cpp:ef73c2e8cd831f2a46ff5704623db63232fd912d : 2633 + 0x4]
[task 2023-10-07T14:53:25.093Z] 14:53:25 INFO - rbx = 0x00007f7d47f94790 rbp = 0x00007fff8626a080
[task 2023-10-07T14:53:25.094Z] 14:53:25 INFO - rsp = 0x00007fff8626a050 r12 = 0x0000000000000003
[task 2023-10-07T14:53:25.094Z] 14:53:25 INFO - r13 = 0x0000000000000000 r14 = 0x00007f7d19518900
[task 2023-10-07T14:53:25.094Z] 14:53:25 INFO - r15 = 0x00007f7d47f94ca0 rip = 0x00007f7d81c7ce91
[task 2023-10-07T14:53:25.095Z] 14:53:25 INFO - Found by: call frame info
[task 2023-10-07T14:53:25.095Z] 14:53:25 INFO - 5 libxul.so!mozilla::a11y::DocAccessible::ShutdownChildrenInSubtree(mozilla::a11y::LocalAccessible*) [DocAccessible.cpp:ef73c2e8cd831f2a46ff5704623db63232fd912d : 2641 + 0xe]
[task 2023-10-07T14:53:25.096Z] 14:53:25 INFO - rbx = 0x00007f7d47e5faf0 rbp = 0x00007fff8626a0c0
[task 2023-10-07T14:53:25.096Z] 14:53:25 INFO - rsp = 0x00007fff8626a090 r12 = 0x0000000000000003
[task 2023-10-07T14:53:25.097Z] 14:53:25 INFO - r13 = 0x0000000000000000 r14 = 0x00007f7d19518900
[task 2023-10-07T14:53:25.097Z] 14:53:25 INFO - r15 = 0x00007f7d47f94790 rip = 0x00007f7d81c7ce71
[task 2023-10-07T14:53:25.097Z] 14:53:25 INFO - Found by: call frame info
[task 2023-10-07T14:53:25.098Z] 14:53:25 INFO - 6 libxul.so!mozilla::a11y::NotificationController::ProcessMutationEvents() [NotificationController.cpp:ef73c2e8cd831f2a46ff5704623db63232fd912d : 593 + 0x4]
[task 2023-10-07T14:53:25.098Z] 14:53:25 INFO - rbx = 0x00007f7d47e5faf0 rbp = 0x00007fff8626a1b0
[task 2023-10-07T14:53:25.098Z] 14:53:25 INFO - rsp = 0x00007fff8626a0d0 r12 = 0x0000000000000000
[task 2023-10-07T14:53:25.099Z] 14:53:25 INFO - r13 = 0x0000000000000000 r14 = 0x00007fff8626a0f0
[task 2023-10-07T14:53:25.099Z] 14:53:25 INFO - r15 = 0x00007f7d20e4b4a0 rip = 0x00007f7d81c48491
[task 2023-10-07T14:53:25.100Z] 14:53:25 INFO - Found by: call frame info
(In reply to Julian Descottes [:jdescottes] from comment #6)
This seems to happen outside of devtools as well, but I see accessibility related objects in the trace
PROCESS-CRASH | application crashed [@ mozilla::detail::InvalidArrayIndex_CRASH] | browser/components/urlbar/tests/browser/browser_searchMode_setURI.js
This crash has happened on this Autoland push while running test-linux1804-64-qr/opt-mochitest-browser-chrome-swr-a11y-checks-3 - log
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
New crash on Try shippable (log)
devtools/client/debugger/test/mochitest/browser_dbg-breakpoints-list.js
application crashed [@ mozilla::detail::InvalidArrayIndex_CRASH]
Comment 11•1 year ago
|
||
We try to get an invalid index here while shutting down a subtree after we handle its hide event. I've been trying to fathom how that's possible.
Most of the time, we (deliberately) repeatedly retrieve index 0; i.e. jdx doesn't change from 0. We rely on the fact that shutting down a child removes that child from its parent. In other words, we remove children from the first to the last. The only exception is that we skip a child if it's not bound to its parent, since the child can't remove itself. That's the only time we increment jdx. This really should never happen.
The only thing i can think is that some child is removing a sibling, but i can't think of any reason that could happen. aria-owns can mess with the sibling tree, but aria-owns gets cleaned up during ContentRemoved. Owned elements should be back in their correct places by the time we get here.
Eitan, can you think of any reason a child would remove a sibling? Or any other reason we might be crashing here that I'm missing?
One way to wallpaper this would be to check ContentChildCount before trying to call ContentChildAt in every iteration of the loop. That should avoid the crash, but it's wallpapering something i really don't understand.
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 13•1 year ago
|
||
I reproduced this locally and i'm looking into it. from a first glance looks like the accessible's doc is shutdown somehow within ShutdownChildrenInSubtree
which causes things to get badly. trying to figure out how that is possible.
Comment 14•1 year ago
|
||
Hmm. This reminds me a bit of bug 1690456. I wonder if something we do during shutdown allows JS to run, which allows a document or even the accessibility service to shut down? See also bug 1849179 comment 16, where the accessibility service has mysteriously shut down with a document still alive.
Assignee | ||
Comment 15•1 year ago
|
||
Yeah, those can all be a similar issue. In this crash what is happening is:
- ShutdownChildrenInSubtree is called on a container, and it loops through the children
- A child of the container is shut down.
- The child is also the last accessible in an xpcom doc's cache.
- Since it is the last accessible,
RemoveFromXPCDocumentCache
is called on the accessible's xpc doc. - It is also the last xpc doc so the accessibility service is shut down.
- the DocAccessibles are shut down, all their referenced accessibles are shutdown.
- We get index 0 in the container's children, but the container is now defunct and mChildren is empty.
- Boom
At least in this case I don't see a need to keep a11y alive after we lose the last xpcom doc, so an IsDefunct check of the container in ShutdownChildrenInSubtree makes sense to me. As for the focus manager deref, I guess I would do the same just have a good comment there.
Assignee | ||
Comment 16•1 year ago
|
||
Looked a bit closer at bug 1849179, and I think it is a similar problem there, only the a11y shutdown happens after the last/only child of a container is shutdown, so then the recursion stack pops to unbind the parent container and since the focus manager is already removed things go bad.
I think I have a fix that is more specific than just a defunct check that will remedy both issues..
Assignee | ||
Comment 17•1 year ago
|
||
Unbinding an accessible from a document can cause accessibility to shut
down if the accessible is the last remaining one with an xpcom wrapper.
In that case we need to return early from ShutdownChildrenInSubtree.
Updated•1 year ago
|
Comment hidden (Intermittent Failures Robot) |
Comment 19•1 year ago
|
||
Comment 20•1 year ago
|
||
bugherder |
Updated•1 year ago
|
Comment hidden (Intermittent Failures Robot) |
Description
•