1683002 - Intermittent Assertion failure: zoomedMetrics->GetApzc(), at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:733

Status: RESOLVED FIXED

(Core :: Graphics: Layers, defect)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=324753186&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/MvyCmJoCQoGQVcsG3WvJ9w/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-12-16T22:43:54.725Z] 22:43:54     INFO - GECKO(7144) | Assertion failure: zoomedMetrics->GetApzc(), at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:733
[task 2020-12-16T22:43:54.725Z] 22:43:54     INFO - GECKO(7144) | ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to c:\users\task_1608156496\appdata\local\temp\tmptwlvhs.mozrunner\runtests_leaks_tab_pid3984.log
[task 2020-12-16T22:43:54.725Z] 22:43:54     INFO - GECKO(7144) | [3984, Main Thread] WARNING: XPCOM_MEM_BLOAT_LOG is set, disabling native allocations.: file /builds/worker/checkouts/gecko/tools/pr
[task 2020-12-16T22:43:54.725Z] 22:43:54     INFO - GECKO(7144) | ofiler/core/platform.cpp:251
[task 2020-12-16T22:43:54.744Z] 22:43:54     INFO - GECKO(7144) | [Child 3788, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8459
[task 2020-12-16T22:43:54.744Z] 22:43:54     INFO - GECKO(7144) | [Child 3788, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8459
[task 2020-12-16T22:43:54.744Z] 22:43:54     INFO - GECKO(7144) | [Child 3788, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8459
[task 2020-12-16T22:43:54.745Z] 22:43:54     INFO - GECKO(7144) | [Child 3788, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8459
[task 2020-12-16T22:43:54.747Z] 22:43:54     INFO - GECKO(7144) | [Child 3788, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8459
[task 2020-12-16T22:43:54.752Z] 22:43:54     INFO - GECKO(7144) | [Child 3788, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8459
[task 2020-12-16T22:43:54.836Z] 22:43:54     INFO - GECKO(7144) | #01: mozilla::layers::ForEachNode<mozilla::layers::ForwardIterator,mozilla::layers::Layer *,`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:701:7',`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:741:7'>(mozilla::layers::Layer*, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&) [gfx/layers/TreeTraversal.h:141]
[task 2020-12-16T22:43:54.836Z] 22:43:54     INFO - GECKO(7144) | #02: mozilla::layers::ForEachNode<mozilla::layers::ForwardIterator,mozilla::layers::Layer *,`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:701:7',`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:741:7'>(mozilla::layers::Layer*, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&) [gfx/layers/TreeTraversal.h:142]
[task 2020-12-16T22:43:54.837Z] 22:43:54     INFO - GECKO(7144) | #03: mozilla::layers::ForEachNode<mozilla::layers::ForwardIterator,mozilla::layers::Layer *,`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:701:7',`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:741:7'>(mozilla::layers::Layer*, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&) [gfx/layers/TreeTraversal.h:142]
[task 2020-12-16T22:43:54.837Z] 22:43:54     INFO - GECKO(7144) | #04: mozilla::layers::ForEachNode<mozilla::layers::ForwardIterator,mozilla::layers::Layer *,`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:701:7',`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:741:7'>(mozilla::layers::Layer*, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&) [gfx/layers/TreeTraversal.h:142]
[task 2020-12-16T22:43:54.837Z] 22:43:54     INFO - GECKO(7144) | #05: mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree(mozilla::layers::Layer*, bool*) [gfx/layers/composite/AsyncCompositionManager.cpp:1122]
[task 2020-12-16T22:43:54.838Z] 22:43:54     INFO - GECKO(7144) | #06: mozilla::layers::AsyncCompositionManager::TransformShadowTree(mozilla::layers::SampleTime const&, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>, mozilla::layers::CompositorBridgeParentBase::TransformsToSkip) [gfx/layers/composite/AsyncCompositionManager.cpp:1286]
[task 2020-12-16T22:43:54.838Z] 22:43:54     INFO - GECKO(7144) | #07: mozilla::layers::CompositorBridgeParent::CompositeToTarget(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::gfx::DrawTarget*, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const*) [gfx/layers/ipc/CompositorBridgeParent.cpp:966]
[task 2020-12-16T22:43:54.839Z] 22:43:54     INFO - GECKO(7144) | #08: mozilla::layers::CompositorVsyncScheduler::Composite(mozilla::VsyncEvent const&) [gfx/layers/ipc/CompositorVsyncScheduler.cpp:258]
[task 2020-12-16T22:43:54.839Z] 22:43:54     INFO - GECKO(7144) | #09: mozilla::detail::RunnableMethodImpl<mozilla::layers::CompositorVsyncScheduler *,void (mozilla::layers::CompositorVsyncScheduler::*)(const mozilla::VsyncEvent &),1,mozilla::RunnableKind::Cancelable,mozilla::VsyncEvent>::Run() [xpcom/threads/nsThreadUtils.h:1204]
[task 2020-12-16T22:43:54.839Z] 22:43:54     INFO - GECKO(7144) | #10: nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1200]
[task 2020-12-16T22:43:54.839Z] 22:43:54     INFO - GECKO(7144) | #11: NS_ProcessNextEvent(nsIThread*, bool) [xpcom/threads/nsThreadUtils.cpp:548]
[task 2020-12-16T22:43:54.840Z] 22:43:54     INFO - GECKO(7144) | #12: mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:303]
[task 2020-12-16T22:43:54.840Z] 22:43:54     INFO - GECKO(7144) | #13: MessageLoop::RunHandler() [ipc/chromium/src/base/message_loop.cc:328]
[task 2020-12-16T22:43:54.840Z] 22:43:54     INFO - GECKO(7144) | #14: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:310]
[task 2020-12-16T22:43:54.840Z] 22:43:54     INFO - GECKO(7144) | #15: static nsThread::ThreadFunc(void*) [xpcom/threads/nsThread.cpp:443]
[task 2020-12-16T22:43:55.113Z] 22:43:55     INFO - GECKO(7144) | #16: PR_NativeRunThread(void*) [nsprpub/pr/src/threads/combined/pruthr.c:408]
[task 2020-12-16T22:43:55.113Z] 22:43:55     INFO - GECKO(7144) | #17: pr_root(void*) [nsprpub/pr/src/md/windows/w95thred.c:140]
[task 2020-12-16T22:43:55.118Z] 22:43:55     INFO -  fix-stacks: error: failed to read symbols file `Z:\task_1608156496\build\symbols\ucrtbase.pdb\08A2EF04ED46F71846F69333018A073A1\ucrtbase.sym` for `C:\Windows\System32\ucrtbase.dll`
[task 2020-12-16T22:43:55.118Z] 22:43:55     INFO -  fix-stacks: note: this is expected and harmless for system libraries on debug automation runs
[task 2020-12-16T22:43:55.118Z] 22:43:55     INFO -  fix-stacks: The system cannot find the path specified. (os error 3)
[task 2020-12-16T22:43:55.119Z] 22:43:55     INFO - GECKO(7144) | #18: o_ceil [C:\Windows\System32\ucrtbase.dll + 0x1c4be]
[task 2020-12-16T22:43:55.124Z] 22:43:55     INFO -  fix-stacks: error: failed to read symbols file `Z:\task_1608156496\build\symbols\KERNEL32.DLL\CC28E59125064F9516DBA8AD3D6420A51\KERNEL32.DLL.sym` for `C:\Windows\System32\KERNEL32.DLL`
[task 2020-12-16T22:43:55.124Z] 22:43:55     INFO -  fix-stacks: note: this is expected and harmless for system libraries on debug automation runs
[task 2020-12-16T22:43:55.124Z] 22:43:55     INFO -  fix-stacks: The system cannot find the path specified. (os error 3)
[task 2020-12-16T22:43:55.124Z] 22:43:55     INFO - GECKO(7144) | #19: BaseThreadInitThunk [C:\Windows\System32\KERNEL32.DLL + 0x13034]
[task 2020-12-16T22:43:55.190Z] 22:43:55     INFO - GECKO(7144) | #20: patched_BaseThreadInitThunk(int, void*, void*) [mozglue/dllservices/WindowsDllBlocklist.cpp:587]
[task 2020-12-16T22:43:55.199Z] 22:43:55     INFO -  fix-stacks: error: failed to read symbols file `Z:\task_1608156496\build\symbols\ntdll.pdb\D3AE91CEDD9309EF777F2FD5120010BE1\ntdll.sym` for `C:\Windows\SYSTEM32\ntdll.dll`
[task 2020-12-16T22:43:55.199Z] 22:43:55     INFO -  fix-stacks: note: this is expected and harmless for system libraries on debug automation runs
[task 2020-12-16T22:43:55.199Z] 22:43:55     INFO -  fix-stacks: The system cannot find the path specified. (os error 3)
[task 2020-12-16T22:43:55.200Z] 22:43:55     INFO - GECKO(7144) | #21: RtlUserThreadStart [C:\Windows\SYSTEM32\ntdll.dll + 0x71461]
[task 2020-12-16T22:43:55.200Z] 22:43:55     INFO - GECKO(7144) | [Parent 7552, Unnamed thread 1c13e8f4400] WARNING: Resource acquired is being released in non-LIFO order; why?
[task 2020-12-16T22:43:55.201Z] 22:43:55     INFO - GECKO(7144) | : file /builds/worker/checkouts/gecko/xpcom/threads/BlockingResourceBase.cpp:292
[task 2020-12-16T22:43:55.201Z] 22:43:55     INFO - GECKO(7144) | --- Mutex : dumpSafetyLock (currently acquired)
[task 2020-12-16T22:43:55.202Z] 22:43:55     INFO - GECKO(7144) |  calling context
[task 2020-12-16T22:43:55.203Z] 22:43:55     INFO - GECKO(7144) |   [stack trace unavailable]```

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This hit during the execution of dom/media/mediacontrol/tests/browser/browser_media_control_main_controller.js

Comment 2

[Intermittent Failures Robot]

1 failures in 3421 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2020-12-14&endday=2020-12-20&tree=all

Comment 3

[Intermittent Failures Robot]

3 failures in 1450 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 1

Platform breakdown:

• windows10-64: 1
• windows7-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2020-12-21&endday=2020-12-27&tree=all

Comment 4

[Intermittent Failures Robot]

23 failures in 525 pushes (0.044 failures/push) were associated with this bug in the last 7 days.

This is the #15 most frequent failure this week.

Repository breakdown:

• autoland: 15
• mozilla-central: 7
• try: 1

Platform breakdown:

• windows10-64: 7
• linux1804-64: 15
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2020-12-28&endday=2021-01-03&tree=all

Comment 5

[Razvan Maries]

There have been 46 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2021-01-01&endday=2021-01-08&tree=trunk&bug=1683002

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=326152401&repo=autoland&lineNumber=124357

Jim, could you assign this to someone?

Comment 6

[Intermittent Failures Robot]

48 failures in 2956 pushes (0.016 failures/push) were associated with this bug in the last 7 days.

This is the #11 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 29
• mozilla-central: 14
• try: 5

Platform breakdown:

• linux1804-64: 37
• windows10-64: 8
• windows7-32: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-01-04&endday=2021-01-10&tree=all

Comment 7

[Intermittent Failures Robot]

18 failures in 795 pushes (0.023 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 4
• autoland: 9
• try: 5

Platform breakdown:

• linux1804-64: 15
• windows10-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-01-13&endday=2021-01-13&tree=all

Comment 9

[Botond Ballo [:botond]]

Pushed to Try to get a Pernosco recording: https://treeherder.mozilla.org/#/jobs?repo=try&revision=286d7647e322f907ce4e0751d8fa74458b319672

(I don't suppose there's a way to get a Pernosco recording for an existing autoland failure?)

Comment 10

[Timothy Nikkel (:tnikkel)]

Does
https://github.com/Pernosco/pernosco/wiki/Mozilla-Tips#reproducing-arbitrary-taskcluster-job-failures
do what you want?

Comment 11

[Botond Ballo [:botond]]

(In reply to Botond Ballo [:botond] from comment #9)

Pushed to Try to get a Pernosco recording: https://treeherder.mozilla.org/#/jobs?repo=try&revision=286d7647e322f907ce4e0751d8fa74458b319672

I got a few repros in this Try push, but did not get any emails about Pernosco recordings for them.

Does https://github.com/Pernosco/pernosco/wiki/Mozilla-Tips#reproducing-arbitrary-taskcluster-job-failures do what you want?

I tried this as well, but it seems to be malfunctioning for me. I reached out to :roc about it.

Comment 12

[Atila Butkovits]

There have been 56 failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2021-01-10&endday=2021-01-17&tree=trunk&bug=1683002

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=326957422&repo=autoland&lineNumber=128365

Comment 13

[Intermittent Failures Robot]

68 failures in 3810 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #5 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 35
• try: 16
• mozilla-central: 17

Platform breakdown:

• linux1804-64: 54
• windows10-64: 11
• windows7-32: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-01-11&endday=2021-01-17&tree=all

Comment 14

[Intermittent Failures Robot]

19 failures in 569 pushes (0.033 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 10
• try: 6
• mozilla-central: 3

Platform breakdown:

• linux1804-64: 13
• windows10-64: 5
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-01-18&endday=2021-01-18&tree=all

Comment 15

[Intermittent Failures Robot]

16 failures in 587 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 8
• try: 3
• mozilla-central: 5

Platform breakdown:

• linux1804-64: 7
• windows10-64: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-01-19&endday=2021-01-19&tree=all

Comment 16

[Andreea Pavel [:apavel]]

Attachment: Bug 1683002 - disable some tests to reduce failure rate r?#intermittent-reviewers

Comment 17

[Andreea Pavel [:apavel]]

Checked around 20 logs and most of the failures are in

• windows10-64 debug: toolkit/components/pictureinpicture/[browser_shortcutsAfterFocus.js, browser_saveLastPiPLoc.js, browser_smallVideoLayout.js]
• linux1804-64 debug: toolkit/components/aboutmemory/tests/[test_aboutmemory4.xhtml, test_aboutmemory5.xhtml]

Made disabling patch for them.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f6b5fb268305
disable some tests to reduce failure rate r=intermittent-reviewers,jmaher DONTBUILD

Comment 19

[Botond Ballo [:botond]]

As I've not had success getting a Pernosco recording for this, I tried reproing the mochitest-chrome-1proc-1 failure of toolkit/components/aboutmemory/tests/test_aboutmemory5.xhtml (which seems to be the highest frequency and thus have the best chances of a local repro) locally.

I tried running that locally with:

./mach test --disable-e10s toolkit/components/aboutmemory/tests/test_aboutmemory5.xhtml

(with the --disable-e10s being there as it's a 1proc test job).

However, that fails with FAIL three content processes present in loaded data.

Running without --disable-e10s produces the same error.

Am I doing it wrong?

Comment 20

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/f6b5fb268305

Comment 21

[Intermittent Failures Robot]

69 failures in 3488 pushes (0.02 failures/push) were associated with this bug in the last 7 days.

This is the #13 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 39
• try: 12
• mozilla-central: 16
• comm-central: 2

Platform and build breakdown:

• linux1804-64: 28
  • debug: 28
• windows10-64: 34
  • debug: 34
• windows7-32: 7
  • debug: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-01-18&endday=2021-01-24&tree=all

Comment 22

[Botond Ballo [:botond]]

:mccr8, heads up that test_aboutmemory[4,5].xhtml got disabled on Linux x64 debug in this bug.

Not sure how important these tests are / how high a priority re-enabling them on this platform should be.

Comment 23

[Andrew McCreight [:mccr8]]

Thanks for letting me know. That is a bit odd. I wonder if there's memory reporting for APZ that does something bad. I didn't think we had any but who knows.

Comment 24

[Botond Ballo [:botond]]

(In reply to Andrew McCreight [:mccr8] from comment #23)

Thanks for letting me know. That is a bit odd. I wonder if there's memory reporting for APZ that does something bad. I didn't think we had any but who knows.

I don't think the assertion is actually related to memory reporting. I think it's just an APZ bug that happens to be tickled by this testcase.

I am in principle happy to investigate further, but that's a lot more practical to do so if I can repro and investigate locally, and I haven't been able to get this test to work locally (as per comment 19).

Comment 25

[Intermittent Failures Robot]

18 failures in 3891 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-beta: 1
• mozilla-central: 5
• comm-central: 2

Platform and build breakdown:

• windows10-64: 12
  • debug: 12
• linux1804-64: 1
  • debug: 1
• windows7-32: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-01-25&endday=2021-01-31&tree=all

Comment 26

[Intermittent Failures Robot]

16 failures in 3966 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• comm-central: 2
• mozilla-central: 6
• mozilla-beta: 1

Platform and build breakdown:

• windows10-64: 10
  • debug: 10
• windows7-32: 5
  • debug: 5
• linux1804-64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-02-01&endday=2021-02-07&tree=all

Comment 27

[Intermittent Failures Robot]

23 failures in 4322 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 21
• mozilla-central: 2

Platform and build breakdown:

• windows10-64: 21
  • debug: 21
• windows7-32: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-02-08&endday=2021-02-14&tree=all

Comment 28

[Intermittent Failures Robot]

14 failures in 3737 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• comm-central: 2
• try: 3
• mozilla-central: 3

Platform and build breakdown:

• windows7-32: 3
  • debug: 3
• windows10-64: 11
  • debug: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-02-15&endday=2021-02-21&tree=all

Comment 29

[Intermittent Failures Robot]

24 failures in 4327 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6
• autoland: 16
• try: 2

Platform and build breakdown:

• windows10-64: 21
  • debug: 21
• windows10-64-qr: 1
  • debug: 1
• windows7-32: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-02-22&endday=2021-02-28&tree=all

Comment 30

[Intermittent Failures Robot]

20 failures in 3683 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 13
• mozilla-central: 7

Platform and build breakdown:

• windows7-32: 2
  • debug: 2
• windows10-64: 18
  • debug: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-03-01&endday=2021-03-07&tree=all

Comment 31

[Natalia Csoregi [:nataliaCs]]

Update:

There have been 46 failures within the last 7 days:

• 31 failures on Windows 10 x64 debug
• 15 failures on Windows 7 debug

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=333163230&repo=autoland&lineNumber=48836

/pictureinpicture/player.xhtml]
[task 2021-03-14T09:05:30.000Z] 09:05:30     INFO - GECKO(3648) | Assertion failure: zoomedMetrics->GetApzc(), at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:710
[task 2021-03-14T09:05:30.007Z] 09:05:30     INFO - GECKO(3648) | [Child 6092, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8404
[task 2021-03-14T09:05:30.007Z] 09:05:30     INFO - GECKO(3648) | [Child 6092, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8404
[task 2021-03-14T09:05:30.007Z] 09:05:30     INFO - GECKO(3648) | [Child 6092, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8404
[task 2021-03-14T09:05:30.008Z] 09:05:30     INFO - GECKO(3648) | [Child 6092, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8404
[task 2021-03-14T09:05:30.015Z] 09:05:30     INFO - GECKO(3648) | [Child 6092, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8404
[task 2021-03-14T09:05:30.015Z] 09:05:30     INFO - GECKO(3648) | [Child 6092, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8404
[task 2021-03-14T09:05:30.022Z] 09:05:30     INFO - GECKO(3648) | [Child 1760, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8404
[task 2021-03-14T09:05:30.029Z] 09:05:30     INFO - GECKO(3648) | [Child 1760, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8404
[task 2021-03-14T09:05:30.029Z] 09:05:30     INFO - GECKO(3648) | [Child 1760, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8404
[task 2021-03-14T09:05:30.030Z] 09:05:30     INFO - GECKO(3648) | [Child 1760, Main Thread] WARNING: '!aContent->GetComposedDoc()', file /builds/worker/checkouts/gecko/layout/base/nsCSSFrameConstructor.cpp:8404
[task 2021-03-14T09:05:30.138Z] 09:05:30     INFO - GECKO(3648) | #01: mozilla::layers::ForEachNode<mozilla::layers::ForwardIterator,mozilla::layers::LayerMetricsWrapper,`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:701:15',`lambda at /builds/worker/checkouts/gecko/gfx/layers/TreeTraversal.h:156:26'>(mozilla::layers::LayerMetricsWrapper, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag>::operator()::<unnamed-tag> const&, mozilla::layers::ForEachNode<mozilla::layers::ForwardIterator,mozilla::layers::LayerMetricsWrapper,`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:701:15'>::<unnamed-tag> const&) [gfx/layers/TreeTraversal.h:108]
[task 2021-03-14T09:05:30.139Z] 09:05:30     INFO - GECKO(3648) | #02: mozilla::layers::ForEachNode<mozilla::layers::ForwardIterator,mozilla::layers::Layer *,`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:678:7',`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:718:7'>(mozilla::layers::Layer*, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&) [gfx/layers/TreeTraversal.h:139]
[task 2021-03-14T09:05:30.142Z] 09:05:30     INFO - GECKO(3648) | #03: mozilla::layers::ForEachNode<mozilla::layers::ForwardIterator,mozilla::layers::Layer *,`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:678:7',`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:718:7'>(mozilla::layers::Layer*, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&) [gfx/layers/TreeTraversal.h:143]
[task 2021-03-14T09:05:30.142Z] 09:05:30     INFO - GECKO(3648) | #04: mozilla::layers::ForEachNode<mozilla::layers::ForwardIterator,mozilla::layers::Layer *,`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:678:7',`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:718:7'>(mozilla::layers::Layer*, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&) [gfx/layers/TreeTraversal.h:143]
[task 2021-03-14T09:05:30.142Z] 09:05:30     INFO - GECKO(3648) | #05: mozilla::layers::ForEachNode<mozilla::layers::ForwardIterator,mozilla::layers::Layer *,`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:678:7',`lambda at /builds/worker/checkouts/gecko/gfx/layers/composite/AsyncCompositionManager.cpp:718:7'>(mozilla::layers::Layer*, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&, mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree::<unnamed-tag> const&) [gfx/layers/TreeTraversal.h:143]
[task 2021-03-14T09:05:30.143Z] 09:05:30     INFO - GECKO(3648) | #06: mozilla::layers::AsyncCompositionManager::ApplyAsyncContentTransformToTree(mozilla::layers::Layer*, bool*) [gfx/layers/composite/AsyncCompositionManager.cpp:676]
[task 2021-03-14T09:05:30.143Z] 09:05:30     INFO - GECKO(3648) | #07: mozilla::layers::AsyncCompositionManager::TransformShadowTree(mozilla::layers::SampleTime const&, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>, mozilla::layers::CompositorBridgeParentBase::TransformsToSkip) [gfx/layers/composite/AsyncCompositionManager.cpp:1263]
[task 2021-03-14T09:05:30.143Z] 09:05:30     INFO - GECKO(3648) | #08: mozilla::layers::CompositorBridgeParent::CompositeToTarget(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::gfx::DrawTarget*, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const*) [gfx/layers/ipc/CompositorBridgeParent.cpp:921]
[task 2021-03-14T09:05:30.143Z] 09:05:30     INFO - GECKO(3648) | #09: mozilla::layers::CompositorVsyncScheduler::Composite(mozilla::VsyncEvent const&) [gfx/layers/ipc/CompositorVsyncScheduler.cpp:258]
[task 2021-03-14T09:05:30.144Z] 09:05:30     INFO - GECKO(3648) | #10: mozilla::detail::RunnableMethodImpl<mozilla::layers::CompositorVsyncScheduler *,void (mozilla::layers::CompositorVsyncScheduler::*)(const mozilla::VsyncEvent &) __attribute__((thiscall)),1,mozilla::RunnableKind::Cancelable,mozilla::VsyncEvent>::Run() [xpcom/threads/nsThreadUtils.h:1204]
[task 2021-03-14T09:05:30.144Z] 09:05:30     INFO - GECKO(3648) | #11: nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1152]
[task 2021-03-14T09:05:30.144Z] 09:05:30     INFO - GECKO(3648) | #12: NS_ProcessNextEvent(nsIThread*, bool) [xpcom/threads/nsThreadUtils.cpp:548]
[task 2021-03-14T09:05:30.147Z] 09:05:30     INFO - GECKO(3648) | #13: mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:302]
[task 2021-03-14T09:05:30.147Z] 09:05:30     INFO - GECKO(3648) | #14: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:335]
[task 2021-03-14T09:05:30.148Z] 09:05:30     INFO - GECKO(3648) | #15: MessageLoop::RunHandler() [ipc/chromium/src/base/message_loop.cc:329]
[task 2021-03-14T09:05:30.148Z] 09:05:30     INFO - GECKO(3648) | #16: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:311]
[task 2021-03-14T09:05:30.148Z] 09:05:30     INFO - GECKO(3648) | #17: static nsThread::ThreadFunc(void*) [xpcom/threads/nsThread.cpp:393]
[task 2021-03-14T09:05:30.148Z] 09:05:30     INFO - GECKO(3648) | #18: _PR_NativeRunThread(void*) [nsprpub/pr/src/threads/combined/pruthr.c:399]
[task 2021-03-14T09:05:30.149Z] 09:05:30     INFO - GECKO(3648) | #19: pr_root(void*) [nsprpub/pr/src/md/windows/w95thred.c:139]
[task 2021-03-14T09:05:30.164Z] 09:05:30     INFO -  fix-stacks: error: failed to read debug info file `ucrtbase.pdb` for `Z:\task_1615710058\build\application\firefox\ucrtbase.DLL`
[task 2021-03-14T09:05:30.164Z] 09:05:30     INFO -  fix-stacks: note: this is expected and harmless for all PDB files on opt automation runs
[task 2021-03-14T09:05:30.164Z] 09:05:30     INFO -  fix-stacks: The system cannot find the file specified. (os error 2)
[task 2021-03-14T09:05:30.164Z] 09:05:30     INFO - GECKO(3648) | #20: o____lc_collate_cp_func [Z:\task_1615710058\build\application\firefox\ucrtbase.DLL + 0x3e16f]
[task 2021-03-14T09:05:30.164Z] 09:05:30     INFO - GECKO(3648) | #21: BaseThreadInitThunk [C:\windows\system32\kernel32.dll + 0x4ef3c]
[task 2021-03-14T09:05:30.165Z] 09:05:30     INFO - GECKO(3648) | #22: RtlInitializeExceptionChain [C:\windows\SYSTEM32\ntdll.dll + 0x63618]
[task 2021-03-14T09:05:30.165Z] 09:05:30     INFO - GECKO(3648) | #23: RtlInitializeExceptionChain [C:\windows\SYSTEM32\ntdll.dll + 0x635eb]
[task 2021-03-14T09:05:30.285Z] 09:05:30     INFO - GECKO(3648) | Exiting due to channel error.
[task 2021-03-14T09:05:30.285Z] 09:05:30     INFO - GECKO(3648) | Exiting due to channel error.
[task 2021-03-14T09:05:30.286Z] 09:05:30     INFO - GECKO(3648) | [RDD 5020Exiting due to channel error.

Comment 32

[Intermittent Failures Robot]

48 failures in 4622 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

This is the #23 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• try: 3
• autoland: 41
• mozilla-central: 4

Platform and build breakdown:

• windows10-64: 33
  • debug: 33
• windows7-32: 15
  • debug: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-03-08&endday=2021-03-14&tree=all

Comment 33

[Botond Ballo [:botond]]

I can try to make some more progress on this.

Comment 34

[Botond Ballo [:botond]]

Based on some debugging via Try, this is what I've found:

• The code in question, which is called by AsyncCompositionManager::TransformShadowTree(), is traversing the compositor layer tree, and if it contains an async zoom container, then expecting layers with the container's target scroll id to also have an APZC.
• The async zoom container and scroll id annotations come from the content process via a layers transaction.
• The APZC objects are attached to the layer tree during UpdateHitTestingTree.
• The invariant can therefore fail to hold if, for a given composite, we don't call UpdateHitTestingTree before calling TransformShadowTree.
• We can skip calling UpdateHitTestingTree if mApzUpdater is null, which can happen if the window doesn't use APZ.
• In principle, if the window doesn't use APZ, the content process should know that and not create APZ-related annotations. However, for the async zoom container in particular, whether we build it is gated on AllowZoomingForDocument, which checks apz.allow_zooming, but does not check APZ itself being enabled for the window.

Checking the window's APZ enablement in AllowZoomingForDocument() gives me a Try push that's pretty green.

Comment 35

[Botond Ballo [:botond]]

Attachment: Bug 1683002 - Do not build an async zoom container in APZ-disabled windows. r=tnikkel

Comment 36

[Intermittent Failures Robot]

16 failures in 761 pushes (0.021 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15
• mozilla-central: 1

Platform and build breakdown:

• windows10-64: 10
  • debug: 10
• windows7-32: 6
  • debug: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-03-15&endday=2021-03-15&tree=all

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d7379f2e509c
Do not build an async zoom container in APZ-disabled windows. r=tnikkel

Comment 38

[Intermittent Failures Robot]

18 failures in 943 pushes (0.019 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 1
• autoland: 11
• mozilla-central: 6

Platform and build breakdown:

• windows10-64: 11
  • debug: 11
• windows7-32: 7
  • debug: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-03-16&endday=2021-03-16&tree=all

Comment 39

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/d7379f2e509c

Comment 40

[Intermittent Failures Robot]

34 failures in 4257 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #29 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 26
• mozilla-central: 7
• try: 1

Platform and build breakdown:

• windows10-64: 21
  • debug: 21
• windows7-32: 13
  • debug: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1683002&startday=2021-03-15&endday=2021-03-21&tree=all

Comment 41

[Botond Ballo [:botond]]

Looks like the fix was effective.