1711543 - Stack overflow running gfx/layers/apz/test/mochitest/test_group_hittest-2.html

Status: NEW

(Core :: Graphics: WebRender, defect, P3)

Description

[Andrew Osmond [:aosmond] (he/him)]

We appear to be hitting a stack overflow with the following test case:

https://searchfox.org/mozilla-central/source/gfx/layers/apz/test/mochitest/test_group_hittest-2.html

In the sub test:

https://searchfox.org/mozilla-central/source/gfx/layers/apz/test/mochitest/helper_hittest_deep_scene_stack.html

[task 2021-05-16T13:09:55.507Z] 13:09:55     INFO - TEST-OK | gfx/layers/apz/test/mochitest/test_group_hittest-1.html | took 7099ms
[task 2021-05-16T13:09:55.603Z] 13:09:55     INFO - TEST-START | gfx/layers/apz/test/mochitest/test_group_hittest-2.html
[task 2021-05-16T13:09:56.254Z] 13:09:56     INFO - GECKO(6297) | WaitUntilApzStable: flushed APZ repaints in parent proc, waiting for callback...
[task 2021-05-16T13:09:56.257Z] 13:09:56     INFO - GECKO(6297) | WaitUntilApzStable: APZ flush done in parent proc
[task 2021-05-16T13:09:56.328Z] 13:09:56     INFO - GECKO(6297) | AddressSanitizer:DEADLYSIGNAL
[task 2021-05-16T13:09:56.328Z] 13:09:56     INFO - GECKO(6297) | =================================================================
[task 2021-05-16T13:09:56.328Z] 13:09:56    ERROR - GECKO(6297) | ==6366==ERROR: AddressSanitizer: stack-overflow on address 0x7ffc4e43ee58 (pc 0x55d29804de3b bp 0x7ffc4e43f6a0 sp 0x7ffc4e43ee60 T0)
[task 2021-05-16T13:09:56.382Z] 13:09:56     INFO - GECKO(6297) |     #0 0x55d29804de3b in __asan_memset /builds/worker/fetches/llvm-project/llvm/projects/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:26:3
[task 2021-05-16T13:09:56.920Z] 13:09:56     INFO - GECKO(6297) |     #1 0x7efc696a82be in nsCSSPropertyIDSet::nsCSSPropertyIDSet(std::initializer_list<nsCSSPropertyID>) /builds/worker/workspace/obj-build/dist/include/nsCSSPropertyIDSet.h:33:9
[task 2021-05-16T13:09:56.940Z] 13:09:56     INFO - GECKO(6297) |     #2 0x7efc6eb6d8d5 in OpacityProperties /builds/worker/checkouts/gecko/layout/style/nsCSSPropertyIDSet.h:107:12
[task 2021-05-16T13:09:56.940Z] 13:09:56     INFO - GECKO(6297) |     #3 0x7efc6eb6d8d5 in HasMatchingAnimations<(lambda at /builds/worker/checkouts/gecko/layout/base/nsLayoutUtils.cpp:298:7)> /builds/worker/checkouts/gecko/layout/base/nsLayoutUtils.cpp:222:31
[task 2021-05-16T13:09:56.941Z] 13:09:56     INFO - GECKO(6297) |     #4 0x7efc6eb6d8d5 in nsLayoutUtils::HasEffectiveAnimation(nsIFrame const*, nsCSSPropertyIDSet const&) /builds/worker/checkouts/gecko/layout/base/nsLayoutUtils.cpp:296:10
[task 2021-05-16T13:09:56.980Z] 13:09:56     INFO - GECKO(6297) |     #5 0x7efc6f1ca856 in mozilla::ActiveLayerTracker::IsStyleAnimated(nsDisplayListBuilder*, nsIFrame*, nsCSSPropertyIDSet const&) /builds/worker/checkouts/gecko/layout/painting/ActiveLayerTracker.cpp:543:7
[task 2021-05-16T13:09:56.982Z] 13:09:56     INFO - GECKO(6297) |     #6 0x7efc6f2b3d42 in IsTransformMaybeAnimated /builds/worker/checkouts/gecko/layout/painting/ActiveLayerTracker.cpp:483:10
[task 2021-05-16T13:09:56.983Z] 13:09:56     INFO - GECKO(6297) |     #7 0x7efc6f2b3d42 in nsDisplayTransform::ShouldPrerenderTransformedContent(nsDisplayListBuilder*, nsIFrame*, nsRect*) /builds/worker/checkouts/gecko/layout/painting/nsDisplayList.cpp:7769:8
[task 2021-05-16T13:09:57.040Z] 13:09:57     INFO - GECKO(6297) |     #8 0x7efc6edb2528 in nsIFrame::BuildDisplayListForStackingContext(nsDisplayListBuilder*, nsDisplayList*, bool*) /builds/worker/checkouts/gecko/layout/generic/nsIFrame.cpp:3177:21
[task 2021-05-16T13:09:57.041Z] 13:09:57     INFO - GECKO(6297) |     #9 0x7efc6ec98a69 in nsIFrame::BuildDisplayListForChild(nsDisplayListBuilder*, nsIFrame*, nsDisplayListSet const&, mozilla::EnumSet<nsIFrame::DisplayChildFlag, unsigned int>) /builds/worker/checkouts/gecko/layout/generic/nsIFrame.cpp:4224:12
[task 2021-05-16T13:09:57.067Z] 13:09:57     INFO - GECKO(6297) |     #10 0x7efc6ec6a775 in DisplayLine(nsDisplayListBuilder*, nsLineList_iterator&, bool, nsDisplayListSet const&, nsBlockFrame*, mozilla::css::TextOverflow*, unsigned int, int, int&) /builds/worker/checkouts/gecko/layout/generic/nsBlockFrame.cpp:6904:13
[task 2021-05-16T13:09:57.068Z] 13:09:57     INFO - GECKO(6297) |     #11 0x7efc6ec68fab in nsBlockFrame::BuildDisplayList(nsDisplayListBuilder*, nsDisplayListSet const&) /builds/worker/checkouts/gecko/layout/generic/nsBlockFrame.cpp:7062:9
[task 2021-05-16T13:09:57.071Z] 13:09:57     INFO - GECKO(6297) |     #12 0x7efc6edb3c9b in nsIFrame::BuildDisplayListForStackingContext(nsDisplayListBuilder*, nsDisplayList*, bool*) /builds/worker/checkouts/gecko/layout/generic/nsIFrame.cpp:3414:5
[task 2021-05-16T13:09:57.072Z] 13:09:57     INFO - GECKO(6297) |     #13 0x7efc6ec98a69 in nsIFrame::BuildDisplayListForChild(nsDisplayListBuilder*, nsIFrame*, nsDisplayListSet const&, mozilla::EnumSet<nsIFrame::DisplayChildFlag, unsigned int>) /builds/worker/checkouts/gecko/layout/generic/nsIFrame.cpp:4224:12
[task 2021-05-16T13:09:57.073Z] 13:09:57     INFO - GECKO(6297) |     #14 0x7efc6ec6a775 in DisplayLine(nsDisplayListBuilder*, nsLineList_iterator&, bool, nsDisplayListSet const&, nsBlockFrame*, mozilla::css::TextOverflow*, unsigned int, int, int&) /builds/worker/checkouts/gecko/layout/generic/nsBlockFrame.cpp:6904:13
[task 2021-05-16T13:09:57.075Z] 13:09:57     INFO - GECKO(6297) |     #15 0x7efc6ec68fab in nsBlockFrame::BuildDisplayList(nsDisplayListBuilder*, nsDisplayListSet const&) /builds/worker/checkouts/gecko/layout/generic/nsBlockFrame.cpp:7062:9
[task 2021-05-16T13:09:57.075Z] 13:09:57     INFO - GECKO(6297) |     #16 0x7efc6edb3c9b in nsIFrame::BuildDisplayListForStackingContext(nsDisplayListBuilder*, nsDisplayList*, bool*) /builds/worker/checkouts/gecko/layout/generic/nsIFrame.cpp:3414:5
[task 2021-05-16T13:09:57.079Z] 13:09:57     INFO - GECKO(6297) |     #17 0x7efc6ec98a69 in nsIFrame::BuildDisplayListForChild(nsDisplayListBuilder*, nsIFrame*, nsDisplayListSet const&, mozilla::EnumSet<nsIFrame::DisplayChildFlag, unsigned int>) /builds/worker/checkouts/gecko/layout/generic/nsIFrame.cpp:4224:12
[task 2021-05-16T13:09:57.079Z] 13:09:57     INFO - GECKO(6297) |     #18 0x7efc6ec6a775 in DisplayLine(nsDisplayListBuilder*, nsLineList_iterator&, bool, nsDisplayListSet const&, nsBlockFrame*, mozilla::css::TextOverflow*, unsigned int, int, int&) /builds/worker/checkouts/gecko/layout/generic/nsBlockFrame.cpp:6904:13
[task 2021-05-16T13:09:57.080Z] 13:09:57     INFO - GECKO(6297) |     #19 0x7efc6ec68fab in nsBlockFrame::BuildDisplayList(nsDisplayListBuilder*, nsDisplayListSet const&) /builds/worker/checkouts/gecko/layout/generic/nsBlockFrame.cpp:7062:9
[task 2021-05-16T13:09:57.081Z] 13:09:57     INFO - GECKO(6297) |     #20 0x7efc6edb3c9b in nsIFrame::BuildDisplayListForStackingContext(nsDisplayListBuilder*, nsDisplayList*, bool*) /builds/worker/checkouts/gecko/layout/generic/nsIFrame.cpp:3414:5

Further down:

[task 2021-05-16T13:15:04.470Z] 13:15:04     INFO - TEST-PASS | gfx/layers/apz/test/mochitest/test_group_hittest-2.html | Starting subtest helper_hittest_deep_scene_stack.html 

https://treeherder.mozilla.org/logviewer?job_id=339823504&repo=try

Comment 1

[Miko Mynttinen]

I am getting similar stack-overflow error, except in reflow code. I think this is just ASAN builds requiring more stack space than the default 8M on Linux. Allowing a large stack size with ulimit -s 32000 fixes this on my machine.

Comment 2

[Daniel Veditz [:dveditz]]

We don't need to hide resource exhaustion bugs like this (stack buffer overflow is an entirely different matter)

Comment 3

[Andrew Osmond [:aosmond] (he/him)]

(In reply to Daniel Veditz [:dveditz] from comment #2)

We don't need to hide resource exhaustion bugs like this (stack buffer overflow is an entirely different matter)

Good to know, thanks!