1667202 - Intermittent Assertion failure: false (MOZ_ASSERT_UNREACHABLE: Expected scroll metadata to be available!), at /builds/worker/checkouts/gecko/gfx/layers/wr/ClipManager.cpp:284

Status: RESOLVED FIXED

(Core :: Graphics: Layers, defect)

Description

[Treeherder Bug Filer]

Filed by: rmaries [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=316624347&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/IPkceL2pRHeCBS1rKOHOeg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-09-24T19:40:43.527Z] 19:40:43     INFO - TEST-START | dom/media/mediacontrol/tests/browser_media_control_non_eligible_media.js
[task 2020-09-24T19:41:28.103Z] 19:41:28     INFO - GECKO(2664) | [Child 10108, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp, line 8513
[task 2020-09-24T19:41:28.112Z] 19:41:28     INFO - GECKO(2664) | Assertion failure: false (MOZ_ASSERT_UNREACHABLE: Expected scroll metadata to be available!), at /builds/worker/checkouts/gecko/gfx/layers/wr/ClipManager.cpp:284
[task 2020-09-24T19:41:28.131Z] 19:41:28     INFO - GECKO(2664) | [GPU 4020, Renderer] WARNING: Failed to duplicate file handle for current process!: file /builds/worker/checkouts/gecko/ipc/glue/FileDescriptor.cpp, line 124```

Comment 1

[Intermittent Failures Robot]

7 failures in 4019 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 4

Platform breakdown:

• windows10-64-qr: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1667202&startday=2020-09-21&endday=2020-09-27&tree=all

Comment 2

[Intermittent Failures Robot]

10 failures in 4195 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-central: 1

Platform breakdown:

• windows10-64-qr: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1667202&startday=2020-09-28&endday=2020-10-04&tree=all

Comment 3

[Intermittent Failures Robot]

33 failures in 4130 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #41 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 6
• autoland: 26
• try: 1

Platform breakdown:

• windows10-64-qr: 33

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1667202&startday=2020-10-05&endday=2020-10-11&tree=all

Comment 5

[Intermittent Failures Robot]

27 failures in 3694 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 21
• mozilla-central: 5
• try: 1

Platform breakdown:

• windows10-64-qr: 27

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1667202&startday=2020-10-12&endday=2020-10-18&tree=all

Comment 6

[Intermittent Failures Robot]

25 failures in 4230 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 22
• try: 1
• mozilla-central: 2

Platform breakdown:

• windows10-64-qr: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1667202&startday=2020-10-19&endday=2020-10-25&tree=all

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Medium volume of intermittent failures. Can you investigate?

Comment 8

[Intermittent Failures Robot]

30 failures in 4275 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

This is the #44 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 25
• mozilla-central: 4
• try: 1

Platform breakdown:

• windows10-64-qr: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1667202&startday=2020-10-26&endday=2020-11-01&tree=all

Comment 9

[Natalia Csoregi [:nataliaCs]]

Update:

There have been 32 failures within the last 7 days:
*8 failures on OS X 10.14 WebRender debug
*24 failures on Windows 10 x64 WebRender debug

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=320991302&repo=autoland&lineNumber=29683

Comment 10

[Intermittent Failures Robot]

32 failures in 4128 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #49 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 28
• try: 2
• mozilla-central: 2

Platform breakdown:

• windows10-64-qr: 23
• macosx1014-64-qr: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1667202&startday=2020-11-02&endday=2020-11-08&tree=all

Comment 11

[Jim Mathies [:jimm]]

Hey Jim, can you take a look at this?

Comment 12

[Intermittent Failures Robot]

15 failures in 674 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-central: 2
• try: 1

Platform breakdown:

• windows10-64-qr: 12
• macosx1014-64-qr: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1667202&startday=2020-11-13&endday=2020-11-13&tree=all

Comment 13

[Intermittent Failures Robot]

53 failures in 4040 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #18 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 41
• mozilla-central: 10
• try: 2

Platform breakdown:

• windows10-64-qr: 37
• macosx1014-64-qr: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1667202&startday=2020-11-09&endday=2020-11-15&tree=all

Comment 14

[Jim Blandy :jimb]

Sure!

Comment 15

[Intermittent Failures Robot]

15 failures in 1013 pushes (0.015 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 11
• try: 2
• mozilla-central: 2

Platform breakdown:

• windows10-64-qr: 12
• macosx1014-64-qr: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1667202&startday=2020-11-20&endday=2020-11-20&tree=all

Comment 16

[Andreea Pavel [:apavel]]

There are 75 total failures in the last 7 days on

• windows10-64-qr debug
• macosx1014-64-qr debug

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=322513419&repo=autoland&lineNumber=17112

[task 2020-11-21T06:48:02.996Z] 06:48:02 INFO - TEST-START | toolkit/components/pictureinpicture/tests/browser_saveLastPiPLoc.js
[task 2020-11-21T06:48:03.015Z] 06:48:03 INFO - GECKO(6248) | [Child 8584: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 000002190702E400 == 4 [pid = 8584] [id = 60]
[task 2020-11-21T06:48:03.015Z] 06:48:03 INFO - GECKO(6248) | [Child 8584: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 32 (00000219055C03C0) [pid = 8584] [serial = 141] [outer = 0000000000000000]
[task 2020-11-21T06:48:03.015Z] 06:48:03 INFO - GECKO(6248) | [Child 8584: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 33 (0000021907063800) [pid = 8584] [serial = 142] [outer = 00000219055C03C0]
[task 2020-11-21T06:48:03.156Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOCSHELL 0000015419BAC000 == 1 [pid = 10172] [id = 38] [url = about:blank]
[task 2020-11-21T06:48:03.156Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOCSHELL 0000015413542000 == 0 [pid = 10172] [id = 39] [url = about:blank]
[task 2020-11-21T06:48:03.156Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 20 (000001541D97A800) [pid = 10172] [serial = 83] [outer = 0000000000000000] [url = about:blank]
[task 2020-11-21T06:48:03.156Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 19 (0000015419F77C00) [pid = 10172] [serial = 70] [outer = 0000000000000000] [url = http://example.com/browser/toolkit/components/pictureinpicture/tests/test-page.html]
[task 2020-11-21T06:48:03.156Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 18 (000001541D96C800) [pid = 10172] [serial = 72] [outer = 0000000000000000] [url = about:blank]
[task 2020-11-21T06:48:03.156Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 17 (0000015419BA7C00) [pid = 10172] [serial = 67] [outer = 0000000000000000] [url = about:blank]
[task 2020-11-21T06:48:03.157Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 16 (0000015419F6A400) [pid = 10172] [serial = 78] [outer = 0000000000000000] [url = about:blank]
[task 2020-11-21T06:48:03.157Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 15 (0000015419BAB400) [pid = 10172] [serial = 65] [outer = 0000000000000000] [url = about:blank]
[task 2020-11-21T06:48:03.157Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 14 (000001541B859400) [pid = 10172] [serial = 74] [outer = 0000000000000000] [url = about:blank]
[task 2020-11-21T06:48:03.182Z] 06:48:03 INFO - GECKO(6248) | [Child 8584: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 34 (000002190A4DC800) [pid = 8584] [serial = 143] [outer = 00000219055C03C0]
[task 2020-11-21T06:48:03.281Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 13 (000001541CAAB040) [pid = 10172] [serial = 75] [outer = 0000000000000000] [url = about:blank]
[task 2020-11-21T06:48:03.281Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 12 (000001541CAABE40) [pid = 10172] [serial = 80] [outer = 0000000000000000] [url = about:blank]
[task 2020-11-21T06:48:03.281Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 11 (000001541D960580) [pid = 10172] [serial = 85] [outer = 0000000000000000] [url = about:blank]
[task 2020-11-21T06:48:03.281Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 10 (000001541CAABC80) [pid = 10172] [serial = 82] [outer = 0000000000000000] [url = http://example.com/browser/toolkit/components/pictureinpicture/tests/test-page.html]
[task 2020-11-21T06:48:03.282Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 9 (000001541CAAB740) [pid = 10172] [serial = 77] [outer = 0000000000000000] [url = http://example.com/browser/toolkit/components/pictureinpicture/tests/test-page.html]
[task 2020-11-21T06:48:03.282Z] 06:48:03 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 8 (0000015419BD5C80) [pid = 10172] [serial = 87] [outer = 0000000000000000] [url = about:blank]
[task 2020-11-21T06:48:03.321Z] 06:48:03 INFO - GECKO(6248) | [Parent 6100: Main Thread]: I/DocShellAndDOMWindowLeak --DOCSHELL 000001F271E2A000 == 15 [pid = 6100] [id = 43] [url = chrome://global/content/pictureinpicture/player.xhtml]
[task 2020-11-21T06:48:03.382Z] 06:48:03 INFO - GECKO(6248) | [Parent 6100: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 000001F25B51D800 == 16 [pid = 6100] [id = 49]
[task 2020-11-21T06:48:03.382Z] 06:48:03 INFO - GECKO(6248) | [Parent 6100: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 57 (000001F26C112740) [pid = 6100] [serial = 100] [outer = 0000000000000000]
[task 2020-11-21T06:48:03.382Z] 06:48:03 INFO - GECKO(6248) | [Parent 6100: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 58 (000001F2676CEC00) [pid = 6100] [serial = 101] [outer = 000001F26C112740]
[task 2020-11-21T06:48:03.392Z] 06:48:03 INFO - GECKO(6248) | 1605941283389 Marionette TRACE Received observer notification toplevel-window-ready

task 2020-11-21T06:48:05.727Z] 06:48:05 INFO - GECKO(6248) | ain Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8457
[task 2020-11-21T06:48:05.727Z] 06:48:05 INFO - GECKO(6248) | [Child 8584, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8457
[task 2020-11-21T06:48:05.728Z] 06:48:05 INFO - GECKO(6248) | [Child 8584, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8457
[task 2020-11-21T06:48:05.750Z] 06:48:05 INFO - GECKO(6248) | Assertion failure: false (MOZ_ASSERT_UNREACHABLE: Expected scroll metadata to be available!), at /builds/worker/checkouts/gecko/gfx/layers/wr/ClipManager.cpp:283
[task 2020-11-21T06:48:05.832Z] 06:48:05 INFO - GECKO(6248) | #01: mozilla::layers::WebRenderCommandBuilder::CreateWebRenderCommandsFromDisplayList(nsDisplayList*, nsDisplayItem*, nsDisplayListBuilder*, mozilla::layers::StackingContextHelper const&, mozilla::wr::DisplayListBuilder&, mozilla::wr::IpcResourceUpdateQueue&) [gfx/layers/wr/WebRenderCommandBuilder.cpp:1737]
[task 2020-11-21T06:48:05.834Z] 06:48:05 INFO - GECKO(6248) | #02: nsDisplayOwnLayer::CreateWebRenderCommands(mozilla::wr::DisplayListBuilder&, mozilla::wr::IpcResourceUpdateQueue&, mozilla::layers::StackingContextHelper const&, mozilla::layers::RenderRootStateManager*, nsDisplayListBuilder*) [layout/painting/nsDisplayList.cpp:6419]
[task 2020-11-21T06:48:05.835Z] 06:48:05 INFO - GECKO(6248) | #03: mozilla::layers::WebRenderCommandBuilder::CreateWebRenderCommands(nsDisplayItem*, mozilla::wr::DisplayListBuilder&, mozilla::wr::IpcResourceUpdateQueue&, mozilla::layers::StackingContextHelper const&, nsDisplayListBuilder*) [gfx/layers/wr/WebRenderCommandBuilder.cpp:1655]
[task 2020-11-21T06:48:05.835Z] 06:48:05 INFO - GECKO(6248) | #04: mozilla::layers::WebRenderCommandBuilder::CreateWebRenderCommandsFromDisplayList(nsDisplayList*, nsDisplayItem*, nsDisplayListBuilder*, mozilla::layers::StackingContextHelper const&, mozilla::wr::DisplayListBuilder&, mozilla::wr::IpcResourceUpdateQueue&) [gfx/layers/wr/WebRenderCommandBuilder.cpp:1764]
[task 2020-11-21T06:48:05.837Z] 06:48:05 INFO - GECKO(6248) | #05: mozilla::layers::WebRenderCommandBuilder::BuildWebRenderCommands(mozilla::wr::DisplayListBuilder&, mozilla::wr::IpcResourceUpdateQueue&, nsDisplayList*, nsDisplayListBuilder*, mozilla::layers::WebRenderScrollData&, WrFiltersHolder&&) [gfx/layers/wr/WebRenderCommandBuilder.cpp:1581]
[task 2020-11-21T06:48:05.838Z] 06:48:05 INFO - GECKO(6248) | #06: mozilla::layers::WebRenderLayerManager::EndTransactionWithoutLayer(nsDisplayList*, nsDisplayListBuilder*, WrFiltersHolder&&, mozilla::layers::WebRenderBackgroundData*) [gfx/layers/wr/WebRenderLayerManager.cpp:361]
[task 2020-11-21T06:48:05.838Z] 06:48:05 INFO - GECKO(6248) | #07: nsDisplayList::PaintRoot(nsDisplayListBuilder*, gfxContext*, unsigned int) [layout/painting/nsDisplayList.cpp:2493]
[task 2020-11-21T06:48:05.838Z] 06:48:05 INFO - GECKO(6248) | #08: static nsLayoutUtils::PaintFrame(gfxContext*, nsIFrame*, nsRegion const&, unsigned int, nsDisplayListBuilderMode, nsLayoutUtils::PaintFrameFlags) [layout/base/nsLayoutUtils.cpp:3423]
[task 2020-11-21T06:48:05.839Z] 06:48:05 INFO - GECKO(6248) | #09: mozilla::PresShell::Paint(nsView*, nsRegion const&, mozilla::PaintFlags) [layout/base/PresShell.cpp:6390]
[task 2020-11-21T06:48:05.839Z] 06:48:05 INFO - GECKO(6248) | #10: nsViewManager::ProcessPendingUpdatesPaint(nsIWidget*) [view/nsViewManager.cpp:460]
[task 2020-11-21T06:48:05.840Z] 06:48:05 INFO - GECKO(6248) | #11: nsViewManager::ProcessPendingUpdatesForView(nsView*, bool) [view/nsViewManager.cpp:396]
[task 2020-11-21T06:48:05.840Z] 06:48:05 INFO - GECKO(6248) | #12: nsViewManager::ProcessPendingUpdates() [view/nsViewManager.cpp:1019]
[task 2020-11-21T06:48:05.841Z] 06:48:05 INFO - GECKO(6248) | #13: nsRefreshDriver::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) [layout/base/nsRefreshDriver.cpp:2400]
[task 2020-11-21T06:48:05.841Z] 06:48:05 INFO - GECKO(6248) | #14: mozilla::RefreshDriverTimer::TickRefreshDrivers(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp, nsTArray<RefPtr<nsRefreshDriver> >&) [layout/base/nsRefreshDriver.cpp:353]
[task 2020-11-21T06:48:05.842Z] 06:48:05 INFO - GECKO(6248) | #15: mozilla::RefreshDriverTimer::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) [layout/base/nsRefreshDriver.cpp:370]
[task 2020-11-21T06:48:05.842Z] 06:48:05 INFO - GECKO(6248) | #16: mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::TickRefreshDriver(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) [layout/base/nsRefreshDriver.cpp:747]
[task 2020-11-21T06:48:05.843Z] 06:48:05 INFO - GECKO(6248) | #17: mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::NotifyParentProcessVsync() [layout/base/nsRefreshDriver.cpp:649]
[task 2020-11-21T06:48:05.843Z] 06:48:05 INFO - GECKO(6248) | #18: mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::NotifyVsync(mozilla::VsyncEvent const&) [layout/base/nsRefreshDriver.cpp:570]
[task 2020-11-21T06:48:05.843Z] 06:48:05 INFO - GECKO(6248) | #19: mozilla::layout::VsyncChild::RecvNotify(mozilla::VsyncEvent const&) [layout/ipc/VsyncChild.cpp:57]
[task 2020-11-21T06:48:05.844Z] 06:48:05 INFO - GECKO(6248) | #20: mozilla::layout::PVsyncChild::OnMessageReceived(IPC::Message const&) [s3:gecko-generated-sources:dcfb0c5c8cc64a4b8592c33a9bb2c1a430c98ebd009674a59f93629a7c17a7323253973267a429964267d5e17f515a9e0c9f5ab4ceb87eda82e2dd117c395600/ipc/ipdl/PVsyncChild.cpp::187]
[task 2020-11-21T06:48:05.845Z] 06:48:05 INFO - GECKO(6248) | #21: mozilla::ipc::PBackgroundChild::OnMessageReceived(IPC::Message const&) [s3:gecko-generated-sources:0b7e12b5e7db02df3b66de7381ce5932fbdc83b9fe6a51c4c09d9c09ac6efe21f3876199a54f506814a9570ee68e5a965987da8aa151dbb9504226f43da76ab4/ipc/ipdl/PBackgroundChild.cpp::6247]
[task 2020-11-21T06:48:05.845Z] 06:48:05 INFO - GECKO(6248) | #22: mozilla::ipc::MessageChannel::DispatchAsyncMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message const&) [ipc/glue/MessageChannel.cpp:2147]
[task 2020-11-21T06:48:05.846Z] 06:48:05 INFO - GECKO(6248) | #23: mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) [ipc/glue/MessageChannel.cpp:2070]
[task 2020-11-21T06:48:05.846Z] 06:48:05 INFO - GECKO(6248) | #24: mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::MessageChannel::MessageTask&) [ipc/glue/MessageChannel.cpp:1919]
[task 2020-11-21T06:48:05.847Z] 06:48:05 INFO - GECKO(6248) | #25: mozilla::ipc::MessageChannel::MessageTask::Run() [ipc/glue/MessageChannel.cpp:1951]
[task 2020-11-21T06:48:05.847Z] 06:48:05 INFO - GECKO(6248) | #26: mozilla::RunnableTask::Run() [xpcom/threads/TaskController.cpp:451]
[task 2020-11-21T06:48:05.847Z] 06:48:05 INFO - GECKO(6248) | #27: mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex &> const&) [xpcom/threads/TaskController.cpp:720]
[task 2020-11-21T06:48:05.847Z] 06:48:05 INFO - GECKO(6248) | #28: mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex &> const&) [xpcom/threads/TaskController.cpp:579]
[task 2020-11-21T06:48:05.848Z] 06:48:05 INFO - GECKO(6248) | #29: mozilla::TaskController::ProcessPendingMTTask(bool) [xpcom/threads/TaskController.cpp:373]
[task 2020-11-21T06:48:05.848Z] 06:48:05 INFO - GECKO(6248) | #30: mozilla::detail::RunnableFunction<`lambda at /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:120:7'>::Run() [xpcom/threads/nsThreadUtils.h:578]
[task 2020-11-21T06:48:05.848Z] 06:48:05 INFO - GECKO(6248) | #31: nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1197]
[task 2020-11-21T06:48:05.848Z] 06:48:05 INFO - GECKO(6248) | #32: NS_ProcessNextEvent(nsIThread*, bool) [xpcom/threads/nsThreadUtils.cpp:513]
[task 2020-11-21T06:48:05.848Z] 06:48:05 INFO - GECKO(6248) | #33: mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:87]
[task 2020-11-21T06:48:05.848Z] 06:48:05 INFO - GECKO(6248) | #34: MessageLoop::RunHandler() [ipc/chromium/src/base/message_loop.cc:328]
[task 2020-11-21T06:48:05.848Z] 06:48:05 INFO - GECKO(6248) | #35: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:310]
[task 2020-11-21T06:48:05.848Z] 06:48:05 INFO - GECKO(6248) | #36: nsBaseAppShell::Run() [widget/nsBaseAppShell.cpp:139]
[task 2020-11-21T06:48:05.848Z] 06:48:05 INFO - GECKO(6248) | #37: nsAppShell::Run() [widget/windows/nsAppShell.cpp:602]
[task 2020-11-21T06:48:05.848Z] 06:48:05 INFO - GECKO(6248) | #38: XRE_RunAppShell() [toolkit/xre/nsEmbedFunctions.cpp:913]
[task 2020-11-21T06:48:05.849Z] 06:48:05 INFO - GECKO(6248) | #39: mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:237]
[task 2020-11-21T06:48:05.849Z] 06:48:05 INFO - GECKO(6248) | #40: MessageLoop::RunHandler() [ipc/chromium/src/base/message_loop.cc:328]
[task 2020-11-21T06:48:05.849Z] 06:48:05 INFO - GECKO(6248) | #41: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:310]
[task 2020-11-21T06:48:05.849Z] 06:48:05 INFO - GECKO(6248) | #42: XRE_InitChildProcess(int, char**, XREChildData const*) [toolkit/xre/nsEmbedFunctions.cpp:748]
[task 2020-11-21T06:48:05.849Z] 06:48:05 INFO - GECKO(6248) | #43: NS_internal_main(int, char**, char**) [browser/app/nsBrowserApp.cpp:304]
[task 2020-11-21T06:48:05.849Z] 06:48:05 INFO - GECKO(6248) | #44: wmain(int, wchar_t**) [toolkit/xre/nsWindowsWMain.cpp:131]
[task 2020-11-21T06:48:05.850Z] 06:48:05 INFO - GECKO(6248) | #45: __scrt_common_main_seh() [/builds/worker/workspace/obj-build/browser/app/f:/dd/vctools/crt/vcstartup/src/startup/exe_common.inl:288]
[task 2020-11-21T06:48:05.850Z] 06:48:05 INFO - GECKO(6248) | #46: BaseThreadInitThunk [C:\Windows\System32\KERNEL32.DLL + 0x13034]
[task 2020-11-21T06:48:05.850Z] 06:48:05 INFO - GECKO(6248) | #47: RtlUserThreadStart [C:\Windows\SYSTEM32\ntdll.dll + 0x71461]
[task 2020-11-21T06:48:06.173Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100, Unnamed thread 1f2676c8000] WARNING: Resource acquired is being released in non-LIFO order; why?
[task 2020-11-21T06:48:06.173Z] 06:48:06 INFO - GECKO(6248) | : file /builds/worker/checkouts/gecko/xpcom/threads/BlockingResourceBase.cpp:292
[task 2020-11-21T06:48:06.173Z] 06:48:06 INFO - GECKO(6248) | --- Mutex : dumpSafetyLock (currently acquired)
[task 2020-11-21T06:48:06.173Z] 06:48:06 INFO - GECKO(6248) | calling context
[task 2020-11-21T06:48:06.174Z] 06:48:06 INFO - GECKO(6248) | [stack trace unavailable]
[task 2020-11-21T06:48:06.174Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-11-21T06:48:06.174Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-11-21T06:48:06.176Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-11-21T06:48:06.176Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-11-21T06:48:06.247Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-11-21T06:48:06.247Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checko[Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 0000015419F6E800 == 6 [pid = 10172] [id = 45]
[task 2020-11-21T06:48:06.248Z] 06:48:06 INFO - GECKO(6248) | uts/gecko/ipc/glue[Child 101/ProtocolUtils.cpp:511
[task 2020-11-21T06:48:06.248Z] 06:48:06 INFO - GECKO(6248) | 72, Main Thread] WARNING: NS_ENSURE_TRUE(browserChrome) failed: file /[Parent 6100, Mbuilds/worker/checkouts/gecko/docain Thread] Wshell/base/nsDocShARNING: IPC message discaell.cpp:12061
[task 2020-11-21T06:48:06.249Z] 06:48:06 INFO - GECKO(6248) | rded: actor cannot send: file /builds/wo[Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 19 (000001541CAAB3C0) [pid = 10172] [serial = 101] [outer = 0000000000000000]
[task 2020-11-21T06:48:06.249Z] 06:48:06 INFO - GECKO(6248) | rker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-11-21T06:48:06.249Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100, Main Thread] WARNING: IPC messa[Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 20 (0000015419F77C00) [pid = 10172] [serial = 102] [outer = 000001541CAAB3C0]
[task 2020-11-21T06:48:06.249Z] 06:48:06 INFO - GECKO(6248) | ge discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-11-21T06:48:06.249Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/workspace/obj-build/dist/include/mozilla/ipc/ProtocolUtils.h:295
[task 2020-11-21T06:48:06.249Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100, Main Thread] WARNING: 'aOwner->IsDiscarded()', file /builds/worker/workspace/obj-build/dist/include/mozilla/dom/SyncedContextInlines.h:36
[task 2020-11-21T06:48:06.250Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 000001F26D074800 == 15 [pid = 6100] [id = 54]
[task 2020-11-21T06:48:06.250Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 54 (000001F26783C3C0) [pid = 6100] [serial = 110] [outer = 0000000000000000]
[task 2020-11-21T06:48:06.260Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 55 (000001F26D075400) [pid = 6100] [serial = 111] [outer = 000001F26783C3C0]
[task 2020-11-21T06:48:06.322Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100, Gecko_IOThread] WARNING: file /builds/worker/checkouts/gecko/ipc/chromium/src/base/process_util_win.cc:166
[task 2020-11-21T06:48:06.381Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 56 (000001F26D784000) [pid = 6100] [serial = 112] [outer = 000001F26783C3C0]
[task 2020-11-21T06:48:06.424Z] 06:48:06 INFO - TEST-INFO | started process screenshot
[task 2020-11-21T06:48:06.505Z] 06:48:06 INFO - TEST-INFO | screenshot: exit 0
[task 2020-11-21T06:48:06.506Z] 06:48:06 INFO - Buffered messages logged at 06:48:02

[task 2020-11-21T06:48:06.521Z] 06:48:06 INFO - TEST-PASS | toolkit/components/pictureinpicture/tests/browser_saveLastPiPLoc.js | Width is changed to adjust for aspect ration -
[task 2020-11-21T06:48:06.521Z] 06:48:06 INFO - Buffered messages logged at 06:48:06
[task 2020-11-21T06:48:06.521Z] 06:48:06 INFO - Console message: [JavaScript Error: "remote browser crashed while on http://example.com/browser/toolkit/components/pictureinpicture/tests/test-page.html
[task 2020-11-21T06:48:06.521Z] 06:48:06 INFO - " {file: "chrome://mochikit/content/mochitest-e10s-utils.js" line: 10}]
[task 2020-11-21T06:48:06.521Z] 06:48:06 INFO - e10s_init/<@chrome://mochikit/content/mochitest-e10s-utils.js:10:10
[task 2020-11-21T06:48:06.521Z] 06:48:06 INFO -
[task 2020-11-21T06:48:06.521Z] 06:48:06 INFO - Buffered messages finished
[task 2020-11-21T06:48:06.522Z] 06:48:06 INFO - TEST-UNEXPECTED-FAIL | toolkit/components/pictureinpicture/tests/browser_saveLastPiPLoc.js | Uncaught exception - AbortError: Actor 'SpecialPowers' destroyed before query 'Spawn' was resolved
[task 2020-11-21T06:48:06.522Z] 06:48:06 INFO - Leaving test bound
[task 2020-11-21T06:48:06.522Z] 06:48:06 INFO - GECKO(6248) | MEMORY STAT | vsize 2104465MB | vsizeMaxContiguous 65367447MB | residentFast 329MB | heapAllocated 121MB
[task 2020-11-21T06:48:06.522Z] 06:48:06 INFO - TEST-OK | toolkit/components/pictureinpicture/tests/browser_saveLastPiPLoc.js | took 3487ms
[task 2020-11-21T06:48:06.522Z] 06:48:06 INFO - Not taking screenshot here: see the one that was previously logged
[task 2020-11-21T06:48:06.523Z] 06:48:06 INFO - TEST-UNEXPECTED-FAIL | toolkit/components/pictureinpicture/tests/browser_saveLastPiPLoc.js | Found an unexpected tab at the end of test run: http://example.com/browser/toolkit/components/pictureinpicture/tests/test-page.html -
[task 2020-11-21T06:48:06.523Z] 06:48:06 INFO - GECKO(6248) | [Parent 6100, Main Thread] WARNING: '!inner', file /builds/worker/checkouts/gecko/dom/ipc/jsactor/JSWindowActorProtocol.cpp:181
[task 2020-11-21T06:48:06.523Z] 06:48:06 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 0000015419F78000 == 7 [pid = 10172] [id = 46]
[task 2020-11-21T06:48:06.523Z] 06:48:06 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 21 (000001541CAAB580) [pid = 10172] [serial = 103] [outer = 0000000000000000]
[task 2020-11-21T06:48:06.523Z] 06:48:06 INFO - GECKO(6248) | [Child 10172: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 22 (0000015419F79400) [pid = 10172] [serial = 104] [outer = 000001541CAAB580]
[task 2020-11-21T06:48:06.541Z] 06:48:06 INFO - checking window state

Jim are there any updates here?

Comment 17

[Intermittent Failures Robot]

79 failures in 4810 pushes (0.016 failures/push) were associated with this bug in the last 7 days.

This is the #7 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 58
• mozilla-central: 15
• try: 6

Platform breakdown:

• macosx1014-64-qr: 28
• windows10-64-qr: 51

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1667202&startday=2020-11-16&endday=2020-11-22&tree=all

Comment 20

[Intermittent Failures Robot]

37 failures in 1479 pushes (0.025 failures/push) were associated with this bug in the last 7 days.

This is the #10 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 27
• mozilla-central: 8
• try: 2

Platform breakdown:

• windows10-64-qr: 26
• macosx1014-64-qr: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1667202&startday=2020-11-23&endday=2020-11-29&tree=all

Comment 21

[Intermittent Failures Robot]

16 failures in 601 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 14
• mozilla-central: 2

Platform breakdown:

• windows10-64-qr: 11
• macosx1014-64-qr: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1667202&startday=2020-11-30&endday=2020-11-30&tree=all

Comment 22

[Intermittent Failures Robot]

27 failures in 787 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 7
• autoland: 19
• try: 1

Platform breakdown:

• windows10-64-qr: 12
• macosx1014-64-qr: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1667202&startday=2020-12-02&endday=2020-12-02&tree=all

Comment 23

[Intermittent Failures Robot]

17 failures in 651 pushes (0.026 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 3
• try: 1

Platform breakdown:

• windows10-64-qr: 7
• macosx1014-64-qr: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1667202&startday=2020-12-03&endday=2020-12-03&tree=all

Comment 24

[Razvan Maries]

There have been 103 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2020-11-29&endday=2020-12-06&tree=trunk&bug=1667202

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=323676278&repo=mozilla-central&lineNumber=124209

Comment 25

[Razvan Maries]

Jim, could you take a look?

Comment 26

[Intermittent Failures Robot]

107 failures in 3712 pushes (0.029 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 76
• mozilla-central: 28
• try: 3

Platform breakdown:

• windows10-64-qr: 59
• macosx1014-64-qr: 48

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1667202&startday=2020-11-30&endday=2020-12-06&tree=all

Comment 27

[Jim Blandy :jimb]

The issue here is that we have built ActiveScrolledRoots for the display list, which should imply that scroll metadata is available. Somehow the conditions for building ActiveScrolledRoots and being able to get scroll metadata are getting out of sync.

I wonder whether, when the assertion fails, aASR->mScrollableFrame is an nsHTMLScrollFrame or an nsXULScrollFrame, or both.

Comment 28

[Jim Blandy :jimb]

I took a look through the failures included in comment 23:

• Windows 10 x64 WebRender debug Mochitests bc1, bc2, bc4, bc5
  • toolkit/components/pictureinpicture/tests/browser_saveLastPiPLoc.js
  • toolkit/components/pictureinpicture/tests/browser_shortcutsAfterFocus.js
  • toolkit/components/pictureinpicture/tests/browser_showMessage.js
  • toolkit/components/extensions/test/browser/browser_ext_windows_popup_title.js
  • toolkit/components/pictureinpicture/tests/browser_smallVideoLayout.js
• OS X 10.14 WebRender debug Mochitests without e10s c2, c3
  • dom/base/test/chrome/test_chromeOuterWindowID.xhtml

It seems like the OS X failures are more predictable.

Comment 29

[Jim Blandy :jimb]

Seems to be reproducible like this:
https://treeherder.mozilla.org/jobs?repo=try&revision=a25a75d1f4a50c94693a62d3aef28522de1b1482&selectedTaskRun=HHL-ifxBSX6mY86AWS340g.0

try_task_config.json:

{
    "env": {
        "TRY_SELECTOR": "fuzzy"
    },
    "rebuild": 10,
    "tasks": [
        "test-macosx1014-64-qr/debug-mochitest-chrome-1proc-1",
        "test-macosx1014-64-qr/debug-mochitest-chrome-1proc-2",
        "test-macosx1014-64-qr/debug-mochitest-chrome-1proc-3"
    ],
    "version": 1
}

Comment 30

[Jim Blandy :jimb]

(in four out of ten runs)

Comment 31

[Kartikaya Gupta (email:[e-mail])]

I expect the patch I linked in this comment to fix this. I can't really claim to be actively working on it though.

Comment 32

[Intermittent Failures Robot]

16 failures in 691 pushes (0.023 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• try: 3

Platform breakdown:

• macosx1014-64-qr: 4
• windows10-64-qr: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1667202&startday=2020-12-07&endday=2020-12-07&tree=all

Comment 33

[Narcis Beleuzu [:NarcisB]]

There have been 72 failures in the last 7 days

• 36 failures on macosx1014-64-qr
• 36 failures on windows10-64-qr

Comment 34

[Andreea Pavel [:apavel]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #31)

I expect the patch I linked in this comment to fix this. I can't really claim to be actively working on it though.

I checked that try push and the assertion doesn't show up anymore, but there are other failures there i'm not sure you got a chance to take a look at.
Do you have an estimation on when that patch can be landed?

Comment 35

[Kartikaya Gupta (email:[e-mail])]

The other failures on the try push are actually unrelated to the patch, they were pre-existing. But I can probably extract the part of that patch that fixed the assertion and post it here. It should be relatively safe to land.

Comment 36

[Intermittent Failures Robot]

68 failures in 4060 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #5 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 52
• try: 7
• mozilla-central: 9

Platform breakdown:

• macosx1014-64-qr: 35
• windows10-64-qr: 33

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1667202&startday=2020-12-07&endday=2020-12-13&tree=all

Comment 37

[Kartikaya Gupta (email:[e-mail])]

https://treeherder.mozilla.org/jobs?repo=try&group_state=expanded&revision=aec073ca23f652b30f3e75c6d91dc53f7f0d140c

https://treeherder.mozilla.org/jobs?repo=try&group_state=expanded&revision=b2e9ed1929c16181d9e5da086231b2eed81a362d

Looking good

Comment 38

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1667202 - Allow scroll metadata creation even for APZ-disabled scrollframes. r?jimb

APZ is not always enabled; for example, on the 1proc
chrome mochitests. However, WR still relies on getting a scroll metadata
for ASRs in this situation, so that it can inform WR of the scroll range
of scrollframes. This combination results in an assertion failure where
the scroll metadata is not generated but the caller expects it to be. This
patch fixes the problem by generating the scroll metadata even if APZ is
disabled. This should be harmless, and fixes the assertion
failure.

Comment 39

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/52a4a14e2d32
Allow scroll metadata creation even for APZ-disabled scrollframes. r=botond

Comment 40

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/52a4a14e2d32

Comment 41

[Intermittent Failures Robot]

11 failures in 3421 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-central: 5

Platform breakdown:

• windows10-64-qr: 2
• macosx1014-64-qr: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1667202&startday=2020-12-14&endday=2020-12-20&tree=all