1693496 - Intermittent SUMMARY: ThreadSanitizer: SEGV /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:254:3 in MOZ_Crash

Status: RESOLVED FIXED

(Core :: Web Painting, defect, P3)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=330370944&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FYqvOVPvRuayz20cal7b6A/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-02-18T09:59:28.292Z] 09:59:28     INFO - TEST-OK | toolkit/mozapps/extensions/test/browser/browser_page_options_updates.js | took 3686ms
[task 2021-02-18T09:59:28.347Z] 09:59:28     INFO - checking window state
[task 2021-02-18T09:59:28.435Z] 09:59:28     INFO - TEST-START | toolkit/mozapps/extensions/test/browser/browser_panel_item_accesskey.js
[task 2021-02-18T09:59:30.594Z] 09:59:30     INFO - GECKO(1565) | ThreadSanitizer:DEADLYSIGNAL
[task 2021-02-18T09:59:30.594Z] 09:59:30     INFO - GECKO(1565) | ==1565==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fcce7d458d3 bp 0x5630a2ef13a0 sp 0x7fff6bcf2510 T1565)
[task 2021-02-18T09:59:30.594Z] 09:59:30     INFO - GECKO(1565) | ==1565==The signal is caused by a WRITE memory access.
[task 2021-02-18T09:59:30.594Z] 09:59:30     INFO - GECKO(1565) | ==1565==Hint: address points to the zero page.
[task 2021-02-18T09:59:30.675Z] 09:59:30     INFO - GECKO(1565) |     #0 MOZ_Crash /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:254:3 (libxul.so+0x53858d3)
[task 2021-02-18T09:59:30.675Z] 09:59:30     INFO - GECKO(1565) |     #1 GetOldListIndex /builds/worker/checkouts/gecko/layout/painting/nsDisplayList.h:2357:7 (libxul.so+0x53858d3)
[task 2021-02-18T09:59:30.675Z] 09:59:30     INFO - GECKO(1565) |     #2 MergeState::HasMatchingItemInOldList(nsDisplayItem*, Index<OldListUnits>*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:605:16 (libxul.so+0x53858d3)
[task 2021-02-18T09:59:30.675Z] 09:59:30     INFO - GECKO(1565) |     #3 MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:427:9 (libxul.so+0x530e7b2)
[task 2021-02-18T09:59:30.676Z] 09:59:30     INFO - GECKO(1565) |     #4 RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:815:31 (libxul.so+0x530e209)
[task 2021-02-18T09:59:30.676Z] 09:59:30     INFO - GECKO(1565) |     #5 MergeState::MergeChildLists(nsDisplayItem*, nsDisplayItem*, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:492:37 (libxul.so+0x53859e4)
[task 2021-02-18T09:59:30.677Z] 09:59:30     INFO - GECKO(1565) |     #6 MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:461:9 (libxul.so+0x530ece4)
[task 2021-02-18T09:59:30.678Z] 09:59:30     INFO - GECKO(1565) |     #7 RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:815:31 (libxul.so+0x530e209)
[task 2021-02-18T09:59:30.678Z] 09:59:30     INFO - GECKO(1565) |     #8 MergeState::MergeChildLists(nsDisplayItem*, nsDisplayItem*, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:492:37 (libxul.so+0x53859e4)
[task 2021-02-18T09:59:30.678Z] 09:59:30     INFO - GECKO(1565) |     #9 MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:461:9 (libxul.so+0x530ece4)
[task 2021-02-18T09:59:30.679Z] 09:59:30     INFO - GECKO(1565) |     #10 RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:815:31 (libxul.so+0x530e209)
[task 2021-02-18T09:59:30.680Z] 09:59:30     INFO - GECKO(1565) |     #11 MergeState::MergeChildLists(nsDisplayItem*, nsDisplayItem*, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:492:37 (libxul.so+0x53859e4)
[task 2021-02-18T09:59:30.680Z] 09:59:30     INFO - GECKO(1565) |     #12 MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:461:9 (libxul.so+0x530ece4)
[task 2021-02-18T09:59:30.680Z] 09:59:30     INFO - GECKO(1565) |     #13 RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:815:31 (libxul.so+0x530e209)
[task 2021-02-18T09:59:30.681Z] 09:59:30     INFO - GECKO(1565) |     #14 RetainedDisplayListBuilder::AttemptPartialUpdate(unsigned int) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:1471:7 (libxul.so+0x5312314)
[task 2021-02-18T09:59:30.697Z] 09:59:30     INFO - GECKO(1565) |     #15 nsLayoutUtils::PaintFrame(gfxContext*, nsIFrame*, nsRegion const&, unsigned int, nsDisplayListBuilderMode, nsLayoutUtils::PaintFrameFlags) /builds/worker/checkouts/gecko/layout/base/nsLayoutUtils.cpp:3336:40 (libxul.so+0x4f5439b)
[task 2021-02-18T09:59:30.697Z] 09:59:30     INFO - GECKO(1565) |     #16 mozilla::PresShell::Paint(nsView*, nsRegion const&, mozilla::PaintFlags) /builds/worker/checkouts/gecko/layout/base/PresShell.cpp:6396:5 (libxul.so+0x4ecfd3d)
[task 2021-02-18T09:59:30.699Z] 09:59:30     INFO - GECKO(1565) |     #17 nsViewManager::ProcessPendingUpdatesPaint(nsIWidget*) /builds/worker/checkouts/gecko/view/nsViewManager.cpp:459:18 (libxul.so+0x4b6c616)
[task 2021-02-18T09:59:30.699Z] 09:59:30     INFO - GECKO(1565) |     #18 nsViewManager::ProcessPendingUpdatesForView(nsView*, bool) /builds/worker/checkouts/gecko/view/nsViewManager.cpp:394:22 (libxul.so+0x4b6c085)
[task 2021-02-18T09:59:30.699Z] 09:59:30     INFO - GECKO(1565) |     #19 nsViewManager::ProcessPendingUpdates() /builds/worker/checkouts/gecko/view/nsViewManager.cpp:972:5 (libxul.so+0x4b6d99f)
[task 2021-02-18T09:59:30.700Z] 09:59:30     INFO - GECKO(1565) |     #20 nsRefreshDriver::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:2378:11 (libxul.so+0x4e86b39)
[task 2021-02-18T09:59:30.702Z] 09:59:30     INFO - GECKO(1565) |     #21 TickDriver /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:357:13 (libxul.so+0x4e8e207)
[task 2021-02-18T09:59:30.703Z] 09:59:30     INFO - GECKO(1565) |     #22 mozilla::RefreshDriverTimer::TickRefreshDrivers(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp, nsTArray<RefPtr<nsRefreshDriver> >&) /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:336:7 (libxul.so+0x4e8e207)
[task 2021-02-18T09:59:30.704Z] 09:59:30     INFO - GECKO(1565) |     #23 mozilla::RefreshDriverTimer::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:351:5 (libxul.so+0x4e8e007)
[task 2021-02-18T09:59:30.704Z] 09:59:30     INFO - GECKO(1565) |     #24 RunRefreshDrivers /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:799:5 (libxul.so+0x4e8d7e9)
[task 2021-02-18T09:59:30.705Z] 09:59:30     INFO - GECKO(1565) |     #25 mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::TickRefreshDriver(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:722:16 (libxul.so+0x4e8d7e9)
[task 2021-02-18T09:59:30.705Z] 09:59:30     INFO - GECKO(1565) |     #26 mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::NotifyParentProcessVsync() /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:624:7 (libxul.so+0x4e8d266)
[task 2021-02-18T09:59:30.705Z] 09:59:30     INFO - GECKO(1565) |     #27 mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::ParentProcessVsyncNotifier::Run() /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:514:20 (libxul.so+0x4e8bdd8)
[task 2021-02-18T09:59:30.706Z] 09:59:30     INFO - GECKO(1565) |     #28 mozilla::RunnableTask::Run() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:472:16 (libxul.so+0xbde732)
[task 2021-02-18T09:59:30.706Z] 09:59:30     INFO - GECKO(1565) |     #29 mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:760:26 (libxul.so+0xbdc63c)
[task 2021-02-18T09:59:30.706Z] 09:59:30     INFO - GECKO(1565) |     #30 mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:611:15 (libxul.so+0xbdafb6)
[task 2021-02-18T09:59:30.706Z] 09:59:30     INFO - GECKO(1565) |     #31 mozilla::TaskController::ProcessPendingMTTask(bool) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:395:36 (libxul.so+0xbdb254)
[task 2021-02-18T09:59:30.707Z] 09:59:30     INFO - GECKO(1565) |     #32 operator() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:133:37 (libxul.so+0xbe1727)
[task 2021-02-18T09:59:30.707Z] 09:59:30     INFO - GECKO(1565) |     #33 mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_3>::Run() /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.h:534:5 (libxul.so+0xbe1727)
[task 2021-02-18T09:59:30.708Z] 09:59:30     INFO - GECKO(1565) |     #34 nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1158:16 (libxul.so+0xbf25a6)
[task 2021-02-18T09:59:30.709Z] 09:59:30     INFO - GECKO(1565) |     #35 NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:548:10 (libxul.so+0xbf8b32)
[task 2021-02-18T09:59:30.709Z] 09:59:30     INFO - GECKO(1565) |     #36 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:87:21 (libxul.so+0x14ef61d)
[task 2021-02-18T09:59:30.710Z] 09:59:30     INFO - GECKO(1565) |     #37 RunInternal /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:335:10 (libxul.so+0x147a11c)
[task 2021-02-18T09:59:30.710Z] 09:59:30     INFO - GECKO(1565) |     #38 RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:328:3 (libxul.so+0x147a11c)
[task 2021-02-18T09:59:30.711Z] 09:59:30     INFO - GECKO(1565) |     #39 MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:310:3 (libxul.so+0x147a11c)
[task 2021-02-18T09:59:30.711Z] 09:59:30     INFO - GECKO(1565) |     #40 nsBaseAppShell::Run() /builds/worker/checkouts/gecko/widget/nsBaseAppShell.cpp:137:27 (libxul.so+0x4bbffb6)
[task 2021-02-18T09:59:30.711Z] 09:59:30     INFO - GECKO(1565) |     #41 nsAppStartup::Run() /builds/worker/checkouts/gecko/toolkit/components/startup/nsAppStartup.cpp:271:30 (libxul.so+0x6734808)
[task 2021-02-18T09:59:30.712Z] 09:59:30     INFO - GECKO(1565) |     #42 XREMain::XRE_mainRun() /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5251:22 (libxul.so+0x684f78f)
[task 2021-02-18T09:59:30.712Z] 09:59:30     INFO - GECKO(1565) |     #43 XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5443:8 (libxul.so+0x68506b8)
[task 2021-02-18T09:59:30.713Z] 09:59:30     INFO - GECKO(1565) |     #44 XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5506:21 (libxul.so+0x6850b34)
[task 2021-02-18T09:59:30.713Z] 09:59:30     INFO - GECKO(1565) |     #45 mozilla::BootstrapImpl::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/Bootstrap.cpp:45:12 (libxul.so+0x685b1f2)
[task 2021-02-18T09:59:30.714Z] 09:59:30     INFO - GECKO(1565) |     #46 do_main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:220:22 (firefox+0xc7c70)
[task 2021-02-18T09:59:30.715Z] 09:59:30     INFO - GECKO(1565) |     #47 main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:344:16 (firefox+0xc7c70)
[task 2021-02-18T09:59:30.715Z] 09:59:30     INFO - GECKO(1565) |     #48 __libc_start_main /build/glibc-2ORdQG/glibc-2.27/csu/../csu/libc-start.c:310 (libc.so.6+0x21b96)
[task 2021-02-18T09:59:30.716Z] 09:59:30     INFO - GECKO(1565) |     #49 _start <null> (firefox+0x2ea6a)
[task 2021-02-18T09:59:30.716Z] 09:59:30     INFO - GECKO(1565) | ThreadSanitizer can not provide additional info.
[task 2021-02-18T09:59:30.716Z] 09:59:30     INFO - GECKO(1565) | SUMMARY: ThreadSanitizer: SEGV /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:254:3 in MOZ_Crash
[task 2021-02-18T09:59:30.717Z] 09:59:30     INFO - GECKO(1565) | ==1565==ABORTING
[task 2021-02-18T09:59:30.837Z] 09:59:30     INFO - GECKO(1565) | Exiting due to channel error.
[task 2021-02-18T09:59:30.838Z] 09:59:30     INFO - GECKO(1565) | Exiting due to channel error.
[task 2021-02-18T09:59:30.838Z] 09:59:30     INFO - GECKO(1565) | Exiting due to channel error.
[task 2021-02-18T09:59:30.838Z] 09:59:30     INFO - GECKO(1565) | Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=657.443) Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=663.694) [GFX1-]: Receive IPC close with reason=AbnormalShutdown
[task 2021-02-18T09:59:30.839Z] 09:59:30     INFO - GECKO(1565) | Exiting due to channel error.
[task 2021-02-18T09:59:30.839Z] 09:59:30     INFO - GECKO(1565) | Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=661.557) Exiting due to channel error.
[task 2021-02-18T09:59:30.839Z] 09:59:30     INFO - GECKO(1565) | Exiting due to channel error.
[task 2021-02-18T09:59:31.869Z] 09:59:31     INFO - GECKO(1565) | Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=671.595) Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=672.356)
[task 2021-02-18T09:59:31.917Z] 09:59:31     INFO - TEST-INFO | Main app process: killed by SIGIOT```

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

These intermittent failures are observed during the execution of toolkit/mozapps/extensions/test/browser/browser_panel_item_accesskey.js on Linux tsan since yesterday.

Comment 2

[Intermittent Failures Robot]

6 failures in 3737 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6

Platform and build breakdown:

• linux1804-64-tsan: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-02-15&endday=2021-02-21&tree=all

Comment 3

[Intermittent Failures Robot]

22 failures in 4327 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 20
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-tsan: 22
  • opt: 22

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-02-22&endday=2021-02-28&tree=all

Comment 4

[Intermittent Failures Robot]

10 failures in 3683 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10

Platform and build breakdown:

• linux1804-64-tsan: 9
  • opt: 9
• linux1804-64-asan: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-03-01&endday=2021-03-07&tree=all

Comment 5

[Intermittent Failures Robot]

18 failures in 4622 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 15
• try: 2
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-tsan: 17
  • opt: 17
• linux1804-64-asan: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-03-08&endday=2021-03-14&tree=all

Comment 6

[Intermittent Failures Robot]

19 failures in 4257 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 18
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-tsan: 19
  • opt: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-03-15&endday=2021-03-21&tree=all

Comment 7

[Intermittent Failures Robot]

28 failures in 3967 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 26
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-tsan: 28
  • opt: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-03-22&endday=2021-03-28&tree=all

Comment 8

[Alexandru Michis [:malexandru]]

There have been 43 total failures in the last 7 days.

Matt, could you please redirect this to someone who can take a look?

Comment 9

[Intermittent Failures Robot]

42 failures in 3125 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #35 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 33
• mozilla-central: 9

Platform and build breakdown:

• linux1804-64-tsan: 42
  • opt: 42

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-03-29&endday=2021-04-04&tree=all

Comment 10

[Matt Woodrow (:mattwoodrow)]

Miko, any idea what might have changed to cause this to spike?

Comment 11

[Miko Mynttinen]

(In reply to Matt Woodrow (:mattwoodrow) from comment #10)

Miko, any idea what might have changed to cause this to spike?

No idea. I tried reproducing this locally without luck.

The pushlog around the date this intermittent appeared does not have anything obvious: https://hg.mozilla.org/mozilla-unified/pushloghtml?startdate=2021-02-16&enddate=2021-02-18

Comment 12

[Intermittent Failures Robot]

15 failures in 800 pushes (0.019 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-tsan: 15
  • opt: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-04-07&endday=2021-04-07&tree=all

Comment 13

[Intermittent Failures Robot]

17 failures in 707 pushes (0.024 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-tsan: 17
  • opt: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-04-08&endday=2021-04-08&tree=all

Comment 14

[Atila Butkovits]

Update:

There have been 66 failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2021-04-03&endday=2021-04-10&tree=trunk&bug=1693496

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=335534259&repo=autoland&lineNumber=3101

Comment 15

[Intermittent Failures Robot]

69 failures in 3854 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #18 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 62
• mozilla-central: 6
• try: 1

Platform and build breakdown:

• linux1804-64-tsan: 69
  • opt: 69

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-04-05&endday=2021-04-11&tree=all

Comment 16

[Intermittent Failures Robot]

17 failures in 779 pushes (0.022 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 16
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-tsan: 17
  • opt: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-04-12&endday=2021-04-12&tree=all

Comment 17

[Intermittent Failures Robot]

16 failures in 649 pushes (0.025 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-tsan: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-04-13&endday=2021-04-13&tree=all

Comment 18

[Intermittent Failures Robot]

21 failures in 1076 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21

Platform and build breakdown:

• linux1804-64-tsan: 21
  • opt: 21

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-04-14&endday=2021-04-14&tree=all

Comment 19

[Intermittent Failures Robot]

88 failures in 4255 pushes (0.021 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 84
• mozilla-central: 4

Platform and build breakdown:

• linux1804-64-tsan: 88
  • opt: 88

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-04-12&endday=2021-04-18&tree=all

Comment 20

[Intermittent Failures Robot]

18 failures in 663 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-tsan: 18
  • opt: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-04-19&endday=2021-04-19&tree=all

Comment 21

[Intermittent Failures Robot]

16 failures in 768 pushes (0.021 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 3

Platform and build breakdown:

• linux1804-64-tsan: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-04-21&endday=2021-04-21&tree=all

Comment 22

[Alexandru Michis [:malexandru]]

There have been 63 total failures in the last 7 days, recent failure log.
Only linux1804-64-tsan is affected.

Comment 23

[Andreea Pavel [:apavel]]

Hi Matt, any other person in mind who can take a look at this?

Comment 24

[Intermittent Failures Robot]

59 failures in 3224 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #12 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 51
• mozilla-central: 8

Platform and build breakdown:

• linux1804-64-tsan: 59
  • opt: 59

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-04-19&endday=2021-04-25&tree=all

Comment 25

[Cosmin Sabou [:CosminS]]

Christian, would you like to have a go at this frequent fail on linux1804-64-tsan only?
Recent log: https://treeherder.mozilla.org/logviewer?job_id=337786813&repo=autoland&lineNumber=3102
Thanks in advance.

[task 2021-04-26T21:51:47.268Z] 21:51:47     INFO - TEST-OK | toolkit/mozapps/extensions/test/browser/browser_page_options_updates.js | took 4967ms
[task 2021-04-26T21:51:47.329Z] 21:51:47     INFO - checking window state
[task 2021-04-26T21:51:47.471Z] 21:51:47     INFO - TEST-START | toolkit/mozapps/extensions/test/browser/browser_panel_item_accesskey.js
[task 2021-04-26T21:51:50.549Z] 21:51:50     INFO - GECKO(1568) | ThreadSanitizer:DEADLYSIGNAL
[task 2021-04-26T21:51:50.549Z] 21:51:50     INFO - GECKO(1568) | ==1568==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f3b035800df bp 0x55c7a9e993b0 sp 0x7ffd94c96470 T1568)
[task 2021-04-26T21:51:50.549Z] 21:51:50     INFO - GECKO(1568) | ==1568==The signal is caused by a WRITE memory access.
[task 2021-04-26T21:51:50.549Z] 21:51:50     INFO - GECKO(1568) | ==1568==Hint: address points to the zero page.
[task 2021-04-26T21:51:50.612Z] 21:51:50     INFO - GECKO(1568) |     #0 MOZ_Crash /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:256:3 (libxul.so+0x532f0df)
[task 2021-04-26T21:51:50.612Z] 21:51:50     INFO - GECKO(1568) |     #1 GetOldListIndex /builds/worker/checkouts/gecko/layout/painting/nsDisplayList.h:2350:7 (libxul.so+0x532f0df)
[task 2021-04-26T21:51:50.612Z] 21:51:50     INFO - GECKO(1568) |     #2 MergeState::HasMatchingItemInOldList(nsDisplayItem*, Index<OldListUnits>*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:604:16 (libxul.so+0x532f0df)
[task 2021-04-26T21:51:50.612Z] 21:51:50     INFO - GECKO(1568) |     #3 MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:427:9 (libxul.so+0x52b8852)
[task 2021-04-26T21:51:50.613Z] 21:51:50     INFO - GECKO(1568) |     #4 RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:812:31 (libxul.so+0x52b82a9)
[task 2021-04-26T21:51:50.613Z] 21:51:50     INFO - GECKO(1568) |     #5 MergeState::MergeChildLists(nsDisplayItem*, nsDisplayItem*, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:492:37 (libxul.so+0x532f1e4)
[task 2021-04-26T21:51:50.613Z] 21:51:50     INFO - GECKO(1568) |     #6 MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:461:9 (libxul.so+0x52b8d84)
[task 2021-04-26T21:51:50.613Z] 21:51:50     INFO - GECKO(1568) |     #7 RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:812:31 (libxul.so+0x52b82a9)
[task 2021-04-26T21:51:50.613Z] 21:51:50     INFO - GECKO(1568) |     #8 MergeState::MergeChildLists(nsDisplayItem*, nsDisplayItem*, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:492:37 (libxul.so+0x532f1e4)
[task 2021-04-26T21:51:50.614Z] 21:51:50     INFO - GECKO(1568) |     #9 MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:461:9 (libxul.so+0x52b8d84)
[task 2021-04-26T21:51:50.614Z] 21:51:50     INFO - GECKO(1568) |     #10 RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:812:31 (libxul.so+0x52b82a9)
[task 2021-04-26T21:51:50.614Z] 21:51:50     INFO - GECKO(1568) |     #11 MergeState::MergeChildLists(nsDisplayItem*, nsDisplayItem*, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:492:37 (libxul.so+0x532f1e4)
[task 2021-04-26T21:51:50.616Z] 21:51:50     INFO - GECKO(1568) |     #12 MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:461:9 (libxul.so+0x52b8d84)
[task 2021-04-26T21:51:50.617Z] 21:51:50     INFO - GECKO(1568) |     #13 RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:812:31 (libxul.so+0x52b82a9)
[task 2021-04-26T21:51:50.618Z] 21:51:50     INFO - GECKO(1568) |     #14 RetainedDisplayListBuilder::AttemptPartialUpdate(unsigned int) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:1462:7 (libxul.so+0x52bc364)
[task 2021-04-26T21:51:50.618Z] 21:51:50     INFO - GECKO(1568) |     #15 nsLayoutUtils::PaintFrame(gfxContext*, nsIFrame*, nsRegion const&, unsigned int, nsDisplayListBuilderMode, nsLayoutUtils::PaintFrameFlags) /builds/worker/checkouts/gecko/layout/base/nsLayoutUtils.cpp:3376:40 (libxul.so+0x4efb17e)
[task 2021-04-26T21:51:50.619Z] 21:51:50     INFO - GECKO(1568) |     #16 mozilla::PresShell::Paint(nsView*, nsRegion const&, mozilla::PaintFlags) /builds/worker/checkouts/gecko/layout/base/PresShell.cpp:6392:5 (libxul.so+0x4e768c0)
[task 2021-04-26T21:51:50.620Z] 21:51:50     INFO - GECKO(1568) |     #17 nsViewManager::ProcessPendingUpdatesPaint(nsIWidget*) /builds/worker/checkouts/gecko/view/nsViewManager.cpp:459:18 (libxul.so+0x4af6076)
[task 2021-04-26T21:51:50.621Z] 21:51:50     INFO - GECKO(1568) |     #18 nsViewManager::ProcessPendingUpdatesForView(nsView*, bool) /builds/worker/checkouts/gecko/view/nsViewManager.cpp:394:22 (libxul.so+0x4af5ae5)
[task 2021-04-26T21:51:50.622Z] 21:51:50     INFO - GECKO(1568) |     #19 nsViewManager::ProcessPendingUpdates() /builds/worker/checkouts/gecko/view/nsViewManager.cpp:972:5 (libxul.so+0x4af73ff)
[task 2021-04-26T21:51:50.623Z] 21:51:50     INFO - GECKO(1568) |     #20 nsRefreshDriver::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:2364:11 (libxul.so+0x4e2db7e)
[task 2021-04-26T21:51:50.623Z] 21:51:50     INFO - GECKO(1568) |     #21 operator() /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:1460:25 (libxul.so+0x4e3c5ab)
[task 2021-04-26T21:51:50.624Z] 21:51:50     INFO - GECKO(1568) |     #22 mozilla::detail::RunnableFunction<nsRefreshDriver::EnsureTimerStarted(nsRefreshDriver::EnsureTimerStartedFlags)::$_0>::Run() /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:534:5 (libxul.so+0x4e3c5ab)
[task 2021-04-26T21:51:50.624Z] 21:51:50     INFO - GECKO(1568) |     #23 mozilla::RunnableTask::Run() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:473:16 (libxul.so+0xbecfc7)
[task 2021-04-26T21:51:50.624Z] 21:51:50     INFO - GECKO(1568) |     #24 mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:757:26 (libxul.so+0xbca208)
[task 2021-04-26T21:51:50.625Z] 21:51:50     INFO - GECKO(1568) |     #25 mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:612:15 (libxul.so+0xbc8972)
[task 2021-04-26T21:51:50.625Z] 21:51:50     INFO - GECKO(1568) |     #26 mozilla::TaskController::ProcessPendingMTTask(bool) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:396:36 (libxul.so+0xbc8b14)
[task 2021-04-26T21:51:50.625Z] 21:51:50     INFO - GECKO(1568) |     #27 operator() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:135:37 (libxul.so+0xbf07c7)
[task 2021-04-26T21:51:50.625Z] 21:51:50     INFO - GECKO(1568) |     #28 mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_0>::Run() /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.h:534:5 (libxul.so+0xbf07c7)
[task 2021-04-26T21:51:50.625Z] 21:51:50     INFO - GECKO(1568) |     #29 nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1159:16 (libxul.so+0xbdb191)
[task 2021-04-26T21:51:50.625Z] 21:51:50     INFO - GECKO(1568) |     #30 NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:548:10 (libxul.so+0xbe2142)
[task 2021-04-26T21:51:50.625Z] 21:51:50     INFO - GECKO(1568) |     #31 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:85:21 (libxul.so+0x14eeecd)
[task 2021-04-26T21:51:50.626Z] 21:51:50     INFO - GECKO(1568) |     #32 RunInternal /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:335:10 (libxul.so+0x1479c9c)
[task 2021-04-26T21:51:50.626Z] 21:51:50     INFO - GECKO(1568) |     #33 RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:328:3 (libxul.so+0x1479c9c)
[task 2021-04-26T21:51:50.626Z] 21:51:50     INFO - GECKO(1568) |     #34 MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:310:3 (libxul.so+0x1479c9c)
[task 2021-04-26T21:51:50.626Z] 21:51:50     INFO - GECKO(1568) |     #35 nsBaseAppShell::Run() /builds/worker/checkouts/gecko/widget/nsBaseAppShell.cpp:137:27 (libxul.so+0x4b4ca56)
[task 2021-04-26T21:51:50.626Z] 21:51:50     INFO - GECKO(1568) |     #36 nsAppStartup::Run() /builds/worker/checkouts/gecko/toolkit/components/startup/nsAppStartup.cpp:273:30 (libxul.so+0x6710258)
[task 2021-04-26T21:51:50.627Z] 21:51:50     INFO - GECKO(1568) |     #37 XREMain::XRE_mainRun() /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5348:22 (libxul.so+0x682bbff)
[task 2021-04-26T21:51:50.627Z] 21:51:50     INFO - GECKO(1568) |     #38 XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5539:8 (libxul.so+0x682ca57)
[task 2021-04-26T21:51:50.627Z] 21:51:50     INFO - GECKO(1568) |     #39 XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5598:21 (libxul.so+0x682cea4)
[task 2021-04-26T21:51:50.627Z] 21:51:50     INFO - GECKO(1568) |     #40 mozilla::BootstrapImpl::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/Bootstrap.cpp:45:12 (libxul.so+0x6837852)
[task 2021-04-26T21:51:50.627Z] 21:51:50     INFO - GECKO(1568) |     #41 do_main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:220:22 (firefox+0xc7c50)
[task 2021-04-26T21:51:50.627Z] 21:51:50     INFO - GECKO(1568) |     #42 main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:347:16 (firefox+0xc7c50)
[task 2021-04-26T21:51:50.627Z] 21:51:50     INFO - GECKO(1568) |     #43 __libc_start_main /build/glibc-2ORdQG/glibc-2.27/csu/../csu/libc-start.c:310 (libc.so.6+0x21b96)
[task 2021-04-26T21:51:50.627Z] 21:51:50     INFO - GECKO(1568) |     #44 _start <null> (firefox+0x2ea5a)
[task 2021-04-26T21:51:50.627Z] 21:51:50     INFO - GECKO(1568) | ThreadSanitizer can not provide additional info.
[task 2021-04-26T21:51:50.628Z] 21:51:50     INFO - GECKO(1568) | SUMMARY: ThreadSanitizer: SEGV /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:256:3 in MOZ_Crash
[task 2021-04-26T21:51:50.628Z] 21:51:50     INFO - GECKO(1568) | ==1568==ABORTING
[task 2021-04-26T21:51:50.784Z] 21:51:50     INFO - GECKO(1568) | Exiting due to channel error.
[task 2021-04-26T21:51:50.785Z] 21:51:50     INFO - GECKO(1568) | Exiting due to channel error.
[task 2021-04-26T21:51:50.785Z] 21:51:50     INFO - GECKO(1568) | Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=736.594) Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=734.332) Exiting due to channel error.
[task 2021-04-26T21:51:50.785Z] 21:51:50     INFO - GECKO(1568) | Exiting due to channel error.

Comment 26

[Intermittent Failures Robot]

20 failures in 551 pushes (0.036 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 20

Platform and build breakdown:

• linux1804-64-tsan: 20
  • opt: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-04-26&endday=2021-04-26&tree=all

Comment 27

[Christian Holler (:decoder)]

This failure is not TSan-specific. We are hitting this assertion

https://searchfox.org/mozilla-central/rev/6cbe34b441f7c7c29cd1e5f0e19c7000142f1423/layout/painting/nsDisplayList.h#2350-2353

and just crash. If this only happens in TSan builds, then it is likely due to perf or thread scheduling.

I can try to see if this reproduces locally, but ultimately, this is a regular bug in our code, nothing TSan related.

Comment 28

[Christian Holler (:decoder)]

Fwiw, this is locally reproducible in a TSan build. I had to run the test in a loop like

while MOZCONFIG=.mozconfig-tsan ./mach test toolkit/mozapps/extensions/test/browser/browser_panel_item_accesskey.js; do :; done

and it reproduced for me in about 30 minutes.

Comment 29

[Cosmin Sabou [:CosminS]]

Matt, could you redirect this to someone who can have a look? Thank you.

Comment 30

[Intermittent Failures Robot]

69 failures in 3706 pushes (0.019 failures/push) were associated with this bug in the last 7 days.

This is the #11 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 61
• mozilla-central: 8

Platform and build breakdown:

• linux1804-64-tsan: 69
  • opt: 69

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-04-26&endday=2021-05-02&tree=all

Comment 31

[Timothy Nikkel (:tnikkel)]

This is a regression from

294868c1cf03701f1d1d6cce6187579da5d1c6a9 Luca Greco — Bug 1525179 - Use aboutaddons.html at the root of about:addons. r=robwu

Note that the same changeset also caused bug 1705978 where we have two, nested, root content documents, which breaks assumptions, and could also be involved here.

Comment 32

[Andreea Pavel [:apavel]]

Luca can you please take a look at this?

Comment 33

[Luca Greco [:rpl] [:luca] [:lgreco]]

Sure thing!

First of all:
I doubt that bug 1705978 is related to the same thing, I believe that bug 1705978 is triggered by the browser elements that embeds the Extensions options ui if they do have one, and the test on which bug 1705978 fails is one of those that verify exactly that scenario (and so there is a "about:addons tab browser" which embeds a "extension options ui page" inside another browser element).

In this test file instead there shouldn't be any of that browser elements nesting, none of the extension installed while this test runs have an options ui page to embed in about:addons.

As Christian already pointed out this isn't really a TSAN crash:
it is a crash triggered on purpose by a diagnostic assertion, and as an additional confirmation by trying hard enough (using --verify and running it a bunch of times if it didn't fail in a run) I was able to get the same diagnostic assert to be triggered in a linux64 debug build (less frequently, but often enough).

And so on some non release builds (nightly and debug build for sure, not 100% sure but I think also early beta?) the diagnostic crash would be triggered if those conditions are met, but if I'm reading it correctly on a non-debug release build there shouldn't be any crash, the method would continue and return successfully (I guess that there may be some side-effect like a rendering issue, but I don't know enough about this layer to be able to tell for sure what the side-effect would be on release without studying that much more closely).

And so to get to the bottom of this, we do still likely require help from someone that has more direct knowledge about that layer and in particular what is exactly the unexpected conditions that the diagnostic assert is warning us about.

Looking to the hg logs, I think that maybe Miko Mynttinen can help us with that, if Matt is too busy at the moment.

Comment 34

[Luca Greco [:rpl] [:luca] [:lgreco]]

In the meantime I tried to at least verify which part of the about:addons page was making this particular test able to trigger this assertion:

• the diagnostic crash seems to be triggered exactly when the test is calling the open method on the panel-list custom element
  (that panel-list custom element is defined here in aboutaddons.js), which at least may let us restrict the area we have to look more deeply to figure out what is exactly going on
• the panel-list custom element:
  • does use an open shadow dom root (not sure if it is related or not)
  • the panel-list element scoped stylesheet does set position: absolute on the :host
  • the panel-list element is being hidden by default until open it as a panel popup menu (but not technically a real popup panel, as it is actually part of the page), the setAlign method here should be the one that is technically aligning the element position and make it visible

I'm not sure if it is a coincidence or a side-effect, but it is a detail that may help us to better identify the root cause:

• If I remove or comment our the position: absolute here, the test doesn't seem to be able to trigger that diagnostic assertion anymore

I'll see if I can get some more useful detail, but in the meantime I wanted to update the issue with this one, in case we manage to get some more ideas also from other engineers with more direct knowledge in the "Core :: Web Painting" area.

Comment 35

[Miko Mynttinen]

Thank you for additional information Luca.

As I mentioned in the comment 11, I had previously tried to reproduce this both in debug and TSAN builds but did not succeed even with thousands of test runs. Based on Christian's comment though, it seems that I did not try hard enough.

I will take another look at this and leave the test running longer on my secondary computer in hopes of catching the assertion in a debugger.

Comment 36

[Luca Greco [:rpl] [:luca] [:lgreco]]

Thank you for additional information Luca.

My apologies for having missed to notice you were already in this issue, I was too focused on figuring out the riddle that this issue is hiding behind :-)

I took another look today, I wanted to double-check what other special things the panel-list custom element is doing to be able to trigger this assertion while position is set to absolute, because I was pretty sure that position: absolute alone is not what triggers the unexpected scenario, but more likely it has to be combined with something else.

One other thing that looked suspicious to me was that requestAnimationFrame + setTimeout being used here in setAlign:

• https://searchfox.org/mozilla-central/rev/3151f97de27730793c2e298716df760999423f26/toolkit/mozapps/extensions/content/aboutaddons.js#755-756

And so I tried to repeat the experiment (running the test using --verify on a debug and a tsan builds) with the setTimeout part omitted (running that code directly in the requestAnimationFrame callback) while keeping position: absolute set in the stylesheet as is and that also did not trigger the assertion anymore (in debug builds, in tsan build I was triggering an actual TSAN failure but completely unrelated [1]).

Looking back into why we have that requestAnimationFrame + setTimeout, it seems that we are doing it to try to reduce the chances to trigger an uninterruptible reflow (mentioned in phabricator here) as suggested in this MDN doc page:

• https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Performance_best_practices_for_Firefox_fe_engineers#how_do_i_avoid_triggering_uninterruptible_reflow

The interesting part is that, when I tried the other approach suggested on MDN (requestAnimationFrame + Services.tm.dispatchToMainThread) I did actually trigger another assertion (I think for the same reasons we are triggering the other one, but in a different place likely because of the different timing of setTimeout vs. Services.tm.dispatchToMainThread):

(rr) bt
#0  MergeState::AddNewNode(nsDisplayItem*, mozilla::Maybe<Index<OldListUnits> > const&, mozilla::Span<Index<MergedListUnits> const, 18446744073709551615ul>, mozilla::Maybe<Index<MergedListUnits> > const&)
    (this=this@entry=0x7ffcd2626920, aItem=aItem@entry=0x7f8ea9d42990, aOldIndex=..., aDirectPredecessors=..., aExtraDirectPredecessor=...)
    at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:635
#1  0x00007f8ec62747d6 in MergeState::ProcessOldNode(Index<OldListUnits>, nsTArray<Index<MergedListUnits> >&&) (this=this@entry=0x7ffcd2626920, aNode=..., aDirectPredecessors=...)
    at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:670
#2  0x00007f8ec622aac1 in MergeState::Finalize() (this=this@entry=0x7ffcd2626920) at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:586
#3  0x00007f8ec622a286 in RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*)
    (this=this@entry=0x7f8eb2e3a000, aNewList=<optimized out>, aNewList@entry=0x7ffcd26269f0, aOldList=<optimized out>,
    aOldList@entry=0x7f8ea9d84d10, aOutList=0x7f8ea9d84d10, aOutContainerASR=..., aOuterItem=aOuterItem@entry=0x7f8ea9d84c28) at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:815
#4  0x00007f8ec6273d75 in MergeState::MergeChildLists(nsDisplayItem*, nsDisplayItem*, nsDisplayItem*)
    (this=this@entry=0x7ffcd2628c60, aNewItem=aNewItem@entry=0x0, aOldItem=aOldItem@entry=0x7f8ea9d84c28, aOutItem=aOutItem@entry=0x7f8ea9d84c28)
    at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:492
#5  0x00007f8ec6274750 in MergeState::ProcessOldNode(Index<OldListUnits>, nsTArray<Index<MergedListUnits> >&&) (this=this@entry=0x7ffcd2628c60, aNode=..., aDirectPredecessors=...)
    at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:662
#6  0x00007f8ec6273fed in MergeState::ProcessPredecessorsOfOldNode(Index<OldListUnits>) (this=<optimized out>, this@entry=0x7ffcd2628c60, aNode=...)
    at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:711
#7  0x00007f8ec622a836 in MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) (this=this@entry=0x7ffcd2628c60, aNewItem=0x7f8ea9d85ea0, aPreviousItem=...)
    at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:464
#8  0x00007f8ec622a24c in RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*)
    (this=this@entry=0x7f8eb2e3a000, aNewList=aNewList@entry=0x7f8ea6f88f88, aOldList=<optimized out>, aOldList@entry=0x7f8ea9d43cb0, aOutList=0x7f8ea6f88f88, aOutContainerASR=..., aOuterItem=0x7f8ea6f88f90,
    aOuterItem@entry=0x7f8ea6f88ea0) at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:812
#9  0x00007f8ec6273d75 in MergeState::MergeChildLists(nsDisplayItem*, nsDisplayItem*, nsDisplayItem*)
    (this=this@entry=0x7ffcd2628ea0, aNewItem=aNewItem@entry=0x7f8ea6f88ea0, aOldItem=aOldItem@entry=0x7f8ea9d43bc8, aOutItem=aOutItem@entry=0x7f8ea6f88ea0)
    at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:492
#10 0x00007f8ec622a80f in MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) (this=this@entry=0x7ffcd2628ea0, aNewItem=0x7f8ea6f88ea0, aPreviousItem=...)
    at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:461
#11 0x00007f8ec622a24c in RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*)
    (this=this@entry=0x7f8eb2e3a000, aNewList=aNewList@entry=0x7f8ea6f89bb0, aOldList=<optimized out>, aOldList@entry=0x7f8ea6f86fe8, aOutList=0x7f8ea6f86fe8, aOutContainerASR=..., aOuterItem=0x7f8ea6f89bb8,
    aOuterItem@entry=0x7f8ea6f86f38) at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:812
#12 0x00007f8ec6273d75 in MergeState::MergeChildLists(nsDisplayItem*, nsDisplayItem*, nsDisplayItem*)
    (this=this@entry=0x7ffcd26290e0, aNewItem=aNewItem@entry=0x7f8ea6f89b00, aOldItem=aOldItem@entry=0x7f8ea6f86f38, aOutItem=aOutItem@entry=0x7f8ea6f86f38)
    at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:492
#13 0x00007f8ec622a80f in MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) (this=this@entry=0x7ffcd26290e0, aNewItem=0x7f8ea6f89b00, aPreviousItem=...)
    at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:461
#14 0x00007f8ec622a24c in RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*)
    (this=this@entry=0x7f8eb2e3a000, aNewList=aNewList@entry=0x7ffcd2629220, aOldList=<optimized out>,
    aOldList@entry=0x7f8eb2e3bfb0, aOutList=aOutList@entry=0x7f8eb2e3bfb0, aOutContainerASR=..., aOuterItem=0x7ffcd2629228, aOuterItem@entry=0x0)
    at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:812
#15 0x00007f8ec622c4cb in RetainedDisplayListBuilder::AttemptPartialUpdate(unsigned int) (this=0x7f8eb2e3a000, aBackstop=4294440951) at /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:1462
#16 0x00007f8ec5f8a145 in nsLayoutUtils::PaintFrame(gfxContext*, nsIFrame*, nsRegion const&, unsigned int, nsDisplayListBuilderMode, nsLayoutUtils::PaintFrameFlags)
    (aRenderingContext=<optimized out>, aFrame=<optimized out>, aDirtyRegion=..., aBackstop=4294440951, aBuilderMode=<optimized out>, aFlags=(nsLayoutUtils::PaintFrameFlags::WidgetLayers | nsLayoutUtils::PaintFrameFlags::ExistingTransaction | nsLayoutUtils::PaintFrameFlags::NoComposite)) at /builds/worker/checkouts/gecko/layout/base/nsLayoutUtils.cpp:3376
#17 0x00007f8ec5f21910 in mozilla::PresShell::Paint(nsView*, nsRegion const&, mozilla::PaintFlags) (this=0x7f8eaea1b000, aViewToPaint=<optimized out>, aDirtyRegion=..., aFlags=<optimized out>)
    at /builds/worker/checkouts/gecko/layout/base/PresShell.cpp:6398
#18 0x00007f8ec5c829d2 in nsViewManager::ProcessPendingUpdatesPaint(nsIWidget*) (this=0x7f8eaf446e00, aWidget=aWidget@entry=0x7f8eb6a98000) at /builds/worker/checkouts/gecko/view/nsViewManager.cpp:459
#19 0x00007f8ec5c82717 in nsViewManager::ProcessPendingUpdatesForView(nsView*, bool) (this=this@entry=0x7f8eaf446e00, aView=<optimized out>, aFlushDirtyRegion=<optimized out>)
    at /builds/worker/checkouts/gecko/view/nsViewManager.cpp:394
#20 0x00007f8ec5c834ab in nsViewManager::ProcessPendingUpdates() (this=0x7f8eaf446e00) at /builds/worker/checkouts/gecko/view/nsViewManager.cpp:972

I though that this may be something interesting to mention, in case may help you to get some other ideas and because this looked to be triggered a bit more often and consistently and so it may be something that may also allow you to reproduce the issue locally.

This are the changes I applied locally (in case you want to try them quickly to see if they help you to reproduce it locally as well):

diff --git a/toolkit/mozapps/extensions/content/aboutaddons.js b/toolkit/mozapps/extensions/content/aboutaddons.js
--- a/toolkit/mozapps/extensions/content/aboutaddons.js
+++ b/toolkit/mozapps/extensions/content/aboutaddons.js
@@ -752,8 +752,8 @@ class PanelList extends HTMLElement {
       this.style.left = 0;
       this.style.top = 0;
 
-      requestAnimationFrame(() =>
-        setTimeout(() => {
+      requestAnimationFrame(() => {
+        Services.tm.dispatchToMainThread(() => {
           let anchorNode =
             (this.triggeringEvent && this.triggeringEvent.target) ||
             this.parentNode;
@@ -775,7 +775,7 @@ class PanelList extends HTMLElement {
             winScrollY: scrollY,
           });
         }, 0)
-      );
+      });
     });
 
     // Calculate the left/right alignment.
diff --git a/toolkit/mozapps/extensions/test/browser/browser_panel_item_accesskey.js b/toolkit/mozapps/extensions/test/browser/browser_panel_item_accesskey.js
--- a/toolkit/mozapps/extensions/test/browser/browser_panel_item_accesskey.js
+++ b/toolkit/mozapps/extensions/test/browser/browser_panel_item_accesskey.js
@@ -7,7 +7,7 @@ add_task(async function setup() {
   // Disable Proton for this test due to crashes seen when
   // attempting to enable Proton on autoland
   await SpecialPowers.pushPrefEnv({
-    set: [["browser.proton.enabled", false]],
+    set: [["browser.proton.enabled", true]],
   });
 });

I flipped also the proton pref here because I think it is likely that they were hitting exactly this assertion error, and so I flipped it back to trye in case it was going to help me to trigger it more often (not sure if it did in practice).

I never tried pernosco-submit yet, but I was thinking that it may be another option to let me share with you a pernosco session with the rr recordings I've been able to capture locally.

Let me know if we want to give that a try (or if there is any other detail that I may collect to make it easier for you to investigate this issue).

[1]: I'll look into reporting that in bugzilla separately, in the appropriate component (which wasn't this one for sure).

Comment 37

[Intermittent Failures Robot]

38 failures in 3279 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

This is the #33 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 34
• mozilla-central: 3
• try: 1

Platform and build breakdown:

• linux1804-64-tsan: 38
  • opt: 38

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-05-03&endday=2021-05-09&tree=all

Comment 38

[Miko Mynttinen]

(In reply to Luca Greco [:rpl] [:luca] [:lgreco] from comment #36)

This are the changes I applied locally (in case you want to try them quickly to see if they help you to reproduce it locally as well):

Good stuff Luca. This definitely helped, with these changes I am now able to reliably reproduce the assertion in a normal debugging build.

Comment 39

[Miko Mynttinen]

The issue here is that we build nsDisplayCompositor item for nsHTMLScrollFrame and wrap it in nsDisplayContainer. We later rebuild the same item but this time wrap it in nsDisplayStickyPosition, and display list merging gets confused.

The decision to build nsDisplayStickyPosition is made here1. For both of the items, the frame has sticky positioning set. Matt, do we still need the latter IsScrollFrameActive() check here, and if so, how can we get rid of it?

Comment 40

[Timothy Nikkel (:tnikkel)]

So IsScrollFrameActive is changing without an invalidate? What makes it change, mHasBeenScrolledRecently ?

Comment 41

[Noemi Erli[:noemi_erli]]

We had 33 failures in the past week, all on linux1804-64-tsan opt.
Recent failure: https://treeherder.mozilla.org/logviewer?job_id=339728159&repo=autoland&lineNumber=3209

[task 2021-05-14T13:42:27.224Z] 13:42:27 INFO - TEST-START | toolkit/mozapps/extensions/test/browser/browser_panel_item_accesskey.js
[task 2021-05-14T13:42:29.733Z] 13:42:29 INFO - GECKO(1579) | ThreadSanitizer:DEADLYSIGNAL
[task 2021-05-14T13:42:29.734Z] 13:42:29 INFO - GECKO(1579) | ==1579==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fcb7950146f bp 0x55f999017430 sp 0x7ffdf8148f00 T1579)
[task 2021-05-14T13:42:29.735Z] 13:42:29 INFO - GECKO(1579) | ==1579==The signal is caused by a WRITE memory access.
[task 2021-05-14T13:42:29.735Z] 13:42:29 INFO - GECKO(1579) | ==1579==Hint: address points to the zero page.
[task 2021-05-14T13:42:29.846Z] 13:42:29 INFO - GECKO(1579) | #0 MOZ_Crash /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:256:3 (libxul.so+0x52f246f)
[task 2021-05-14T13:42:29.847Z] 13:42:29 INFO - GECKO(1579) | #1 GetOldListIndex /builds/worker/checkouts/gecko/layout/painting/nsDisplayList.h:2381:7 (libxul.so+0x52f246f)
[task 2021-05-14T13:42:29.850Z] 13:42:29 INFO - GECKO(1579) | #2 MergeState::HasMatchingItemInOldList(nsDisplayItem*, Index<OldListUnits>) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:604:16 (libxul.so+0x52f246f)
[task 2021-05-14T13:42:29.850Z] 13:42:29 INFO - GECKO(1579) | #3 MergeState::ProcessItemFromNewList(nsDisplayItem, mozilla::Maybe<Index<MergedListUnits> > const&) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:427:9 (libxul.so+0x527c872)
[task 2021-05-14T13:42:29.850Z] 13:42:29 INFO - GECKO(1579) | #4 RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:812:31 (libxul.so+0x527c288)
[task 2021-05-14T13:42:29.850Z] 13:42:29 INFO - GECKO(1579) | #5 MergeState::MergeChildLists(nsDisplayItem*, nsDisplayItem*, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:492:37 (libxul.so+0x52f2574)
[task 2021-05-14T13:42:29.850Z] 13:42:29 INFO - GECKO(1579) | #6 MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:461:9 (libxul.so+0x527cda7)
[task 2021-05-14T13:42:29.850Z] 13:42:29 INFO - GECKO(1579) | #7 RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:812:31 (libxul.so+0x527c288)
[task 2021-05-14T13:42:29.851Z] 13:42:29 INFO - GECKO(1579) | #8 MergeState::MergeChildLists(nsDisplayItem*, nsDisplayItem*, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:492:37 (libxul.so+0x52f2574)
[task 2021-05-14T13:42:29.851Z] 13:42:29 INFO - GECKO(1579) | #9 MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:461:9 (libxul.so+0x527cda7)
[task 2021-05-14T13:42:29.853Z] 13:42:29 INFO - GECKO(1579) | #10 RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:812:31 (libxul.so+0x527c288)
[task 2021-05-14T13:42:29.853Z] 13:42:29 INFO - GECKO(1579) | #11 MergeState::MergeChildLists(nsDisplayItem*, nsDisplayItem*, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:492:37 (libxul.so+0x52f2574)
[task 2021-05-14T13:42:29.853Z] 13:42:29 INFO - GECKO(1579) | #12 MergeState::ProcessItemFromNewList(nsDisplayItem*, mozilla::Maybe<Index<MergedListUnits> > const&) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:461:9 (libxul.so+0x527cda7)
[task 2021-05-14T13:42:29.854Z] 13:42:29 INFO - GECKO(1579) | #13 RetainedDisplayListBuilder::MergeDisplayLists(nsDisplayList*, RetainedDisplayList*, RetainedDisplayList*, mozilla::Maybe<mozilla::ActiveScrolledRoot const*>&, nsDisplayItem*) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:812:31 (libxul.so+0x527c288)
[task 2021-05-14T13:42:29.854Z] 13:42:29 INFO - GECKO(1579) | #14 RetainedDisplayListBuilder::AttemptPartialUpdate(unsigned int) /builds/worker/checkouts/gecko/layout/painting/RetainedDisplayListBuilder.cpp:1473:7 (libxul.so+0x5280374)
[task 2021-05-14T13:42:29.864Z] 13:42:29 INFO - GECKO(1579) | #15 nsLayoutUtils::PaintFrame(gfxContext*, nsIFrame*, nsRegion const&, unsigned int, nsDisplayListBuilderMode, nsLayoutUtils::PaintFrameFlags) /builds/worker/checkouts/gecko/layout/base/nsLayoutUtils.cpp:3380:40 (libxul.so+0x4ec1de8)
[task 2021-05-14T13:42:29.864Z] 13:42:29 INFO - GECKO(1579) | #16 mozilla::PresShell::Paint(nsView*, nsRegion const&, mozilla::PaintFlags) /builds/worker/checkouts/gecko/layout/base/PresShell.cpp:6399:5 (libxul.so+0x4e3e667)
[task 2021-05-14T13:42:29.865Z] 13:42:29 INFO - GECKO(1579) | #17 nsViewManager::ProcessPendingUpdatesPaint(nsIWidget*) /builds/worker/checkouts/gecko/view/nsViewManager.cpp:459:18 (libxul.so+0x4ab9e36)
[task 2021-05-14T13:42:29.866Z] 13:42:29 INFO - GECKO(1579) | #18 nsViewManager::ProcessPendingUpdatesForView(nsView*, bool) /builds/worker/checkouts/gecko/view/nsViewManager.cpp:394:22 (libxul.so+0x4ab98ba)
[task 2021-05-14T13:42:29.867Z] 13:42:29 INFO - GECKO(1579) | #19 nsViewManager::ProcessPendingUpdates() /builds/worker/checkouts/gecko/view/nsViewManager.cpp:972:5 (libxul.so+0x4abb16d)
[task 2021-05-14T13:42:29.874Z] 13:42:29 INFO - GECKO(1579) | #20 nsRefreshDriver::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp, nsRefreshDriver::IsExtraTick) /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:2431:11 (libxul.so+0x4df4d71)
[task 2021-05-14T13:42:29.874Z] 13:42:29 INFO - GECKO(1579) | #21 TickDriver /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:346:13 (libxul.so+0x4dfcc16)
[task 2021-05-14T13:42:29.875Z] 13:42:29 INFO - GECKO(1579) | #22 mozilla::RefreshDriverTimer::TickRefreshDrivers(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp, nsTArray<RefPtr<nsRefreshDriver> >&) /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:324:7 (libxul.so+0x4dfcc16)
[task 2021-05-14T13:42:29.876Z] 13:42:29 INFO - GECKO(1579) | #23 mozilla::RefreshDriverTimer::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:340:5 (libxul.so+0x4dfca36)
[task 2021-05-14T13:42:29.876Z] 13:42:29 INFO - GECKO(1579) | #24 mozilla::VsyncRefreshDriverTimer::RunRefreshDrivers(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:773:5 (libxul.so+0x4dfc81e)
[task 2021-05-14T13:42:29.877Z] 13:42:29 INFO - GECKO(1579) | #25 mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::TickRefreshDriver(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:702:16 (libxul.so+0x4dfc125)
[task 2021-05-14T13:42:29.877Z] 13:42:29 INFO - GECKO(1579) | #26 mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::NotifyParentProcessVsync() /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:615:7 (libxul.so+0x4dfbbae)
[task 2021-05-14T13:42:29.878Z] 13:42:29 INFO - GECKO(1579) | #27 mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::ParentProcessVsyncNotifier::Run() /builds/worker/checkouts/gecko/layout/base/nsRefreshDriver.cpp:505:20 (libxul.so+0x4dfa738)
[task 2021-05-14T13:42:29.878Z] 13:42:29 INFO - GECKO(1579) | #28 mozilla::RunnableTask::Run() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:482:16 (libxul.so+0xbecf07)
[task 2021-05-14T13:42:29.879Z] 13:42:29 INFO - GECKO(1579) | #29 mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:766:26 (libxul.so+0xbc9e7a)
[task 2021-05-14T13:42:29.880Z] 13:42:29 INFO - GECKO(1579) | #30 mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:621:15 (libxul.so+0xbc8602)
[task 2021-05-14T13:42:29.880Z] 13:42:29 INFO - GECKO(1579) | #31 mozilla::TaskController::ProcessPendingMTTask(bool) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:405:36 (libxul.so+0xbc87a4)
[task 2021-05-14T13:42:29.881Z] 13:42:29 INFO - GECKO(1579) | #32 operator() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:138:37 (libxul.so+0xbf0b67)
[task 2021-05-14T13:42:29.881Z] 13:42:29 INFO - GECKO(1579) | #33 mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_0>::Run() /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.h:534:5 (libxul.so+0xbf0b67)
[task 2021-05-14T13:42:29.882Z] 13:42:29 INFO - GECKO(1579) | #34 nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1159:16 (libxul.so+0xbdac70)
[task 2021-05-14T13:42:29.883Z] 13:42:29 INFO - GECKO(1579) | #35 NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:548:10 (libxul.so+0xbe1bf2)
[task 2021-05-14T13:42:29.883Z] 13:42:29 INFO - GECKO(1579) | #36 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:85:21 (libxul.so+0x14a970d)
[task 2021-05-14T13:42:29.884Z] 13:42:29 INFO - GECKO(1579) | #37 RunInternal /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:335:10 (libxul.so+0x143391c)
[task 2021-05-14T13:42:29.885Z] 13:42:29 INFO - GECKO(1579) | #38 RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:328:3 (libxul.so+0x143391c)
[task 2021-05-14T13:42:29.885Z] 13:42:29 INFO - GECKO(1579) | #39 MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:310:3 (libxul.so+0x143391c)
[task 2021-05-14T13:42:29.886Z] 13:42:29 INFO - GECKO(1579) | #40 nsBaseAppShell::Run() /builds/worker/checkouts/gecko/widget/nsBaseAppShell.cpp:137:27 (libxul.so+0x4b0a0f6)
[task 2021-05-14T13:42:29.888Z] 13:42:29 INFO - GECKO(1579) | #41 nsAppStartup::Run() /builds/worker/checkouts/gecko/toolkit/components/startup/nsAppStartup.cpp:273:30 (libxul.so+0x6867748)
[task 2021-05-14T13:42:29.888Z] 13:42:29 INFO - GECKO(1579) | #42 XREMain::XRE_mainRun() /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5367:22 (libxul.so+0x6982d4a)
[task 2021-05-14T13:42:29.888Z] 13:42:29 INFO - GECKO(1579) | #43 XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5565:8 (libxul.so+0x6983a97)
[task 2021-05-14T13:42:29.888Z] 13:42:29 INFO - GECKO(1579) | #44 XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5624:21 (libxul.so+0x6984034)
[task 2021-05-14T13:42:29.888Z] 13:42:29 INFO - GECKO(1579) | #45 mozilla::BootstrapImpl::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/Bootstrap.cpp:45:12 (libxul.so+0x698e5f2)
[task 2021-05-14T13:42:29.888Z] 13:42:29 INFO - GECKO(1579) | #46 do_main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:224:22 (firefox+0xca6e0)
[task 2021-05-14T13:42:29.889Z] 13:42:29 INFO - GECKO(1579) | #47 main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:351:16 (firefox+0xca6e0)
[task 2021-05-14T13:42:29.889Z] 13:42:29 INFO - GECKO(1579) | #48 __libc_start_main /build/glibc-2ORdQG/glibc-2.27/csu/../csu/libc-start.c:310 (libc.so.6+0x21b96)
[task 2021-05-14T13:42:29.889Z] 13:42:29 INFO - GECKO(1579) | #49 _start <null> (firefox+0x2ecba)
[task 2021-05-14T13:42:29.889Z] 13:42:29 INFO - GECKO(1579) | ThreadSanitizer can not provide additional info.
[task 2021-05-14T13:42:29.889Z] 13:42:29 INFO - GECKO(1579) | SUMMARY: ThreadSanitizer: SEGV /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:256:3 in MOZ_Crash
[task 2021-05-14T13:42:29.889Z] 13:42:29 INFO - GECKO(1579) | ==1579==ABORTING
[task 2021-05-14T13:42:30.102Z] 13:42:30 INFO - GECKO(1579) | Exiting due to channel error.
[task 2021-05-14T13:42:30.102Z] 13:42:30 INFO - GECKO(1579) | Exiting due to channel error.
[task 2021-05-14T13:42:30.103Z] 13:42:30 INFO - GECKO(1579) | Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=681.508) Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=682.844) Exiting due to channel error.
[task 2021-05-14T13:42:30.104Z] 13:42:30 INFO - GECKO(1579) | Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=686.896) Exiting due to channel error.
[task 2021-05-14T13:42:30.104Z] 13:42:30 INFO - GECKO(1579) | Exiting due to channel error.
[task 2021-05-14T13:42:30.104Z] 13:42:30 INFO - GECKO(1579) | Exiting due to channel error.
[task 2021-05-14T13:42:31.144Z] 13:42:31 INFO - GECKO(1579) | Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=693.679) Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=694.532) Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=671.767)

Comment 42

[Miko Mynttinen]

(In reply to Timothy Nikkel (:tnikkel) from comment #40)

So IsScrollFrameActive is changing without an invalidate? What makes it change, mHasBeenScrolledRecently ?

I looked into this a bit more and yes, IsScrollFrameActive() changes without invalidation. The change is coming from ScrollFrameHelper::IsScrollingActive() -> ScrollFrameHelper::IsAlwaysActive()1, where we check that scroll styles are not hidden.
The scroll style value nsPresContext::mViewportScrollStyles is set by nsPresContext::UpdateViewportScrollStylesOverride()2.

This seems a bit scary, we use the same possibly-changed-without-invalidation value elsewhere too3.

As for the actual intermittent, I wonder if IsScrollFrameActive()4 is a layerization optimization (or based on older comments APZ limitation), that we could now remove.

Comment 43

[Timothy Nikkel (:tnikkel)]

(In reply to Miko Mynttinen [:miko] from comment #42)

This seems a bit scary, we use the same possibly-changed-without-invalidation value elsewhere too[3].

Seems like we are going to have to invalidate then when nsPresContext::mViewportScrollStyles changes, WantAsyncScroll indirectly uses the value and that can change what's in the display list. So even if we can remove that IsScrollFrameActive call for sticky position it won't be enough to fix the underlying issue.

Comment 44

[Timothy Nikkel (:tnikkel)]

Attachment: Bug 1693496. Invalidate when nsPresContext::mViewportScrollStyles changes because it can change whats in our display list. r?miko

Comment 45

[Timothy Nikkel (:tnikkel)]

I put up a patch for that, checked that it fixes the problem on try server But we'll also need to do an audit to make sure we invalidate when anything changes that can change ScrollFrameHelper::WantAsyncScroll or ScrollFrameHelper::IsScrollingActive. (I feel like I've done that before, but I probably skipped GetScrollStylesFromFrame because I thought we surely must invalidate for style changes, but I guess it changes in a way that style changes don't catch).

Comment 46

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f080bff1a09b
Invalidate when nsPresContext::mViewportScrollStyles changes because it can change whats in our display list. r=miko

Comment 47

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/f080bff1a09b

Comment 48

[Intermittent Failures Robot]

20 failures in 3061 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 15
• mozilla-central: 4
• try: 1

Platform and build breakdown:

• linux1804-64-tsan: 20
  • opt: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1693496&startday=2021-05-17&endday=2021-05-23&tree=all