1651038 - Intermittent Assertion failure: aElement->HasServoData() (Element without Servo data on a post-traversal? How?), at /builds/worker/checkouts/gecko/layout/base/RestyleManager.cpp:2584

Status: RESOLVED FIXED

(Core :: Layout, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=308810138&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/MmYQAP5VSV6lU_1r8JYo-Q/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/MmYQAP5VSV6lU_1r8JYo-Q/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

[task 2020-07-07T10:54:12.447Z] 10:54:12 INFO - REFTEST TEST-PASS | docshell/base/crashtests/1257730-1.html | (LOAD ONLY)
[task 2020-07-07T10:54:12.448Z] 10:54:12 INFO - REFTEST TEST-END | docshell/base/crashtests/1257730-1.html
[task 2020-07-07T10:54:12.449Z] 10:54:12 INFO - Assertion failure: aElement->HasServoData() (Element without Servo data on a post-traversal? How?), at /builds/worker/checkouts/gecko/layout/base/RestyleManager.cpp:2584
[task 2020-07-07T10:54:12.450Z] 10:54:12 INFO - #01: mozilla::RestyleManager::ProcessPostTraversal(mozilla::dom::Element*, mozilla::ServoRestyleState&, mozilla::ServoPostTraversalFlags) [layout/base/RestyleManager.cpp:2801]
[task 2020-07-07T10:54:12.450Z] 10:54:12 INFO - #02: mozilla::RestyleManager::DoProcessPendingRestyles(mozilla::ServoTraversalFlags) [layout/base/RestyleManager.cpp:3007]
[task 2020-07-07T10:54:12.451Z] 10:54:12 INFO - #03: mozilla::PresShell::DoFlushPendingNotifications(mozilla::ChangesToFlush) [layout/base/PresShell.cpp:4198]
[task 2020-07-07T10:54:12.451Z] 10:54:12 INFO - #04: nsRefreshDriver::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) [layout/base/PresShell.h:0]
[task 2020-07-07T10:54:12.451Z] 10:54:12 INFO - #05: mozilla::RefreshDriverTimer::TickRefreshDrivers(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp, nsTArray<RefPtr<nsRefreshDriver> >&) [layout/base/nsRefreshDriver.cpp:344]
[task 2020-07-07T10:54:12.452Z] 10:54:12 INFO - #06: mozilla::RefreshDriverTimer::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) [layout/base/nsRefreshDriver.cpp:369]
[task 2020-07-07T10:54:12.453Z] 10:54:12 INFO - #07: mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::TickRefreshDriver(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) [layout/base/nsRefreshDriver.cpp:739]
[task 2020-07-07T10:54:12.453Z] 10:54:12 INFO - #08: mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::NotifyParentProcessVsync() [layout/base/nsRefreshDriver.cpp:640]
[task 2020-07-07T10:54:12.454Z] 10:54:12 INFO - #09: mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::ParentProcessVsyncNotifier::Run() [layout/base/nsRefreshDriver.cpp:539]
[task 2020-07-07T10:54:12.455Z] 10:54:12 INFO - #10: mozilla::RunnableTask::Run() [xpcom/threads/TaskController.cpp:210]
[task 2020-07-07T10:54:12.455Z] 10:54:12 INFO - #11: mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:459]
[task 2020-07-07T10:54:12.456Z] 10:54:12 INFO - #12: mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:338]
[task 2020-07-07T10:54:12.456Z] 10:54:12 INFO - #13: mozilla::TaskController::ProcessPendingMTTask() [xpcom/threads/TaskController.cpp:154]
[task 2020-07-07T10:54:12.457Z] 10:54:12 INFO - #14: mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_4>::Run() [xpcom/threads/nsThreadUtils.h:578]
[task 2020-07-07T10:54:12.457Z] 10:54:12 INFO - #15: nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1236]
[task 2020-07-07T10:54:12.457Z] 10:54:12 INFO - #16: NS_ProcessNextEvent(nsIThread*, bool) [xpcom/threads/nsThreadUtils.cpp:513]
[task 2020-07-07T10:54:12.458Z] 10:54:12 INFO - #17: mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:87]
[task 2020-07-07T10:54:12.458Z] 10:54:12 INFO - #18: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:334]
[task 2020-07-07T10:54:12.460Z] 10:54:12 INFO - #19: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:310]
[task 2020-07-07T10:54:12.460Z] 10:54:12 INFO - #20: nsBaseAppShell::Run() [widget/nsBaseAppShell.cpp:139]
[task 2020-07-07T10:54:12.460Z] 10:54:12 INFO - #21: XRE_RunAppShell() [toolkit/xre/nsEmbedFunctions.cpp:913]
[task 2020-07-07T10:54:12.461Z] 10:54:12 INFO - #22: mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:237]
[task 2020-07-07T10:54:12.461Z] 10:54:12 INFO - #23: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:334]
[task 2020-07-07T10:54:12.462Z] 10:54:12 INFO - #24: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:310]
[task 2020-07-07T10:54:12.463Z] 10:54:12 INFO - #25: XRE_InitChildProcess(int, char**, XREChildData const*) [toolkit/xre/nsEmbedFunctions.cpp:744]
[task 2020-07-07T10:54:12.464Z] 10:54:12 INFO - #26: content_process_main(mozilla::Bootstrap*, int, char**) [ipc/contentproc/plugin-container.cpp:57]
[task 2020-07-07T10:54:12.464Z] 10:54:12 INFO - #27: main [browser/app/nsBrowserApp.cpp:303]
[task 2020-07-07T10:54:12.464Z] 10:54:12 INFO - #28: __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6 + 0x21b97]
[task 2020-07-07T10:54:12.465Z] 10:54:12 INFO - #29: ??? [/builds/worker/workspace/build/application/firefox/firefox-bin + 0x10ac9]
[task 2020-07-07T10:54:12.465Z] 10:54:12 INFO - #30: ??? (???:???)
[task 2020-07-07T10:54:12.466Z] 10:54:12 INFO - [Parent 1169, Breakpad Server] WARNING: Resource acquired is being released in non-LIFO order; why?
[task 2020-07-07T10:54:12.466Z] 10:54:12 INFO - : file /builds/worker/checkouts/gecko/xpcom/threads/BlockingResourceBase.cpp, line 292
[task 2020-07-07T10:54:12.466Z] 10:54:12 INFO - --- Mutex : dumpSafetyLock (currently acquired)
[task 2020-07-07T10:54:12.467Z] 10:54:12 INFO - calling context
[task 2020-07-07T10:54:12.467Z] 10:54:12 INFO - [stack trace unavailable]

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This is a new frequent failure and needs investigation what started it.

Comment 2

[Intermittent Failures Robot]

39 failures in 5265 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

This is the #43 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 32
• try: 3
• mozilla-central: 4

Platform breakdown:

• linux1804-64-qr: 6
• linux1804-64-asan-qr: 14
• linux1804-64: 10
• linux1804-64-asan: 6
• macosx1014-64: 2
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-07-06&endday=2020-07-12&tree=all

Comment 3

[Narcis Beleuzu [:NarcisB]]

Retriggers are pointing to Bug 1650510
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=Linux%2C18.04%2Cx64%2CWebRender%2Cdebug%2CReftests%2Cwith%2Cfission%2Cenabled%2Ctest-linux1804-64-qr%2Fdebug-crashtest-fis-e10s%2CC&tochange=537d4a5a3bfd8dfa45416d3edcc88ba337c84c24&fromchange=3b27137fc6639bb91695f04ff63523f6dcce6c10&selectedTaskRun=ZjyqBw9rSKWXqNr6og-iJA.0

Julian, can you please take a look?

Comment 4

[Julian Descottes [:jdescottes]]

Bug 1650510 is a test-only change, and those tests don't run in the suite that fails here (they are devtools mochitests).
It's very unlikely to be related, I think we should do more retriggers here.

I also see failures on earlier pushes, eg https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=RhkJX3rEQeS71Q9qfGVrWA.0&searchStr=linux%2C18.04%2Cx64%2Cwebrender%2Cdebug%2Creftests%2Cwith%2Cfission%2Cenabled%2Ctest-linux1804-64-qr%2Fdebug-crashtest-fis-e10s%2Cc&revision=e60508a2639b6256c0d97c8b2a5a89eb68f848f3

Might be related to Bug 1630704 which modified RestyleManager.cpp ?

Comment 5

[Intermittent Failures Robot]

32 failures in 4014 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #46 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 25
• mozilla-central: 7

Platform breakdown:

• macosx1014-64: 3
• linux1804-64-asan-qr: 13
• windows10-64-qr: 1
• linux1804-64-qr: 7
• linux1804-64: 7
• linux1804-64-asan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-07-13&endday=2020-07-19&tree=all

Comment 6

[Sean Voisen (:svoisen)]

Emilio: Thoughts on this? Bug 1630704?

Comment 7

[Emilio Cobos Álvarez (:emilio)]

Doubtful, that shouldn't have any behavior change. The initial stack includes the new scheduler so there may be the case that that caused it. Given this repros across platforms, is there any chance we can see a pernosco recording of this? I'd be interested innpoking at this.

Alternatively can we do a couple more retriggers?

Comment 8

[Intermittent Failures Robot]

30 failures in 4310 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

This is the #42 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 24
• try: 2
• mozilla-central: 4

Platform breakdown:

• linux1804-64-qr: 10
• macosx1014-64: 4
• windows10-64-qr: 1
• linux1804-64-asan: 2
• linux1804-64-asan-qr: 8
• linux1804-64: 4
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-07-20&endday=2020-07-26&tree=all

Comment 9

[Bogdan Tara[:bogdan_tara | bogdant]]

In the last 7 days there have been 24 occurrences, most on linux1804-64 debug and opt.

Recent failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=311726261&repo=mozilla-central&lineNumber=1836

Comment 10

[Cosmin Sabou [:CosminS]]

Re-triggers are inconclusive so far, ended up till 6th of July: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&classifiedState=unclassified&fromchange=e2920747d7df2de2756296a9bfdfd115d66bcd6c&searchStr=linux%2C18.04%2Cx64%2Cwebrender%2Cdebug%2Creftests%2Cwith%2Cfission%2Cenabled%2Ctest-linux1804-64-qr%2Fdebug-crashtest-fis-e10s%2Cc&tochange=888968797a46fe61a6b543fa81a10db8edf9c359&selectedTaskRun=anKxX_tJS8O0i8Sc7crFiA.0

Comment 11

[Intermittent Failures Robot]

31 failures in 4289 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

This is the #46 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• try: 7
• autoland: 18
• mozilla-central: 5
• mozilla-beta: 1

Platform breakdown:

• linux1804-64-qr: 9
• linux1804-64: 7
• linux1804-64-asan-qr: 11
• macosx1014-64: 1
• linux1804-64-asan: 2
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-07-27&endday=2020-08-02&tree=all

Comment 12

[Intermittent Failures Robot]

18 failures in 4618 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 15
• try: 1
• mozilla-central: 2

Platform breakdown:

• macosx1014-64: 2
• linux1804-64: 6
• linux1804-64-qr: 5
• windows7-32: 1
• linux1804-64-asan-qr: 2
• macosx1014-64-qr: 1
• linux1804-64-asan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-08-03&endday=2020-08-09&tree=all

Comment 13

[Bogdan Tara[:bogdan_tara | bogdant]]

In the last 7 days there have been 23 occurrences on linux1804-64 and macosx1014-64 build types debug and opt.

Recent failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=313053320&repo=autoland&lineNumber=1627

Sean, could you, please, take a look?

Comment 14

[Intermittent Failures Robot]

21 failures in 3601 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

This is the #45 most frequent failure this week.

Repository breakdown:

• mozilla-beta: 1
• autoland: 17
• try: 2
• mozilla-central: 1

Platform breakdown:

• linux1804-64: 4
• macosx1014-64: 4
• macosx1014-64-qr: 2
• linux1804-64-qr: 7
• linux1804-64-asan: 2
• linux1804-64-asan-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-08-10&endday=2020-08-16&tree=all

Comment 15

[Intermittent Failures Robot]

19 failures in 4100 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• try: 4
• mozilla-central: 4
• mozilla-release: 1

Platform breakdown:

• linux1804-64-qr: 6
• linux1804-64: 6
• macosx1014-64: 3
• linux1804-64-asan-qr: 2
• linux1804-64-asan: 1
• macosx1014-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-08-17&endday=2020-08-23&tree=all

Comment 16

[Intermittent Failures Robot]

14 failures in 4046 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 12
• try: 1

Platform breakdown:

• linux1804-64-qr: 11
• linux1804-64: 1
• macosx1014-64: 1
• linux1804-64-asan-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-08-24&endday=2020-08-30&tree=all

Comment 17

[Intermittent Failures Robot]

13 failures in 2984 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• autoland: 7
• try: 2

Platform breakdown:

• linux1804-64: 5
• linux1804-64-qr: 3
• linux1804-64-asan: 2
• linux1804-64-asan-qr: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-08-31&endday=2020-09-06&tree=all

Comment 18

[Intermittent Failures Robot]

17 failures in 3410 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 12
• try: 3
• mozilla-central: 2

Platform breakdown:

• linux1804-64-asan: 3
• linux1804-64-qr: 4
• linux1804-64-asan-qr: 5
• linux1804-64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-09-07&endday=2020-09-13&tree=all

Comment 19

[Intermittent Failures Robot]

16 failures in 3760 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• autoland: 10
• mozilla-central: 4

Platform breakdown:

• linux1804-64: 3
• linux1804-64-asan-qr: 3
• windows7-32: 1
• linux1804-64-asan: 3
• linux1804-64-qr: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-09-14&endday=2020-09-20&tree=all

Comment 20

[Intermittent Failures Robot]

10 failures in 4019 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 2

Platform breakdown:

• linux1804-64: 2
• linux1804-64-asan-qr: 3
• macosx1014-64: 1
• windows7-32: 1
• linux1804-64-qr: 2
• linux1804-64-asan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-09-21&endday=2020-09-27&tree=all

Comment 21

[Intermittent Failures Robot]

12 failures in 4195 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 2
• try: 1
• mozilla-release: 1

Platform breakdown:

• linux1804-64-asan-qr: 4
• linux1804-64-qr: 3
• linux1804-64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-09-28&endday=2020-10-04&tree=all

Comment 22

[Intermittent Failures Robot]

8 failures in 4130 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 2
• try: 1

Platform breakdown:

• windows7-32: 2
• macosx1014-64: 2
• linux1804-64-asan-qr: 1
• linux1804-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-10-05&endday=2020-10-11&tree=all

Comment 23

[Intermittent Failures Robot]

8 failures in 3694 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• try: 1

Platform breakdown:

• macosx1014-64: 1
• linux1804-64-qr: 3
• windows7-32: 2
• linux1804-64-asan-qr: 1
• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-10-12&endday=2020-10-18&tree=all

Comment 24

[Andreea Pavel [:apavel]]

I've checked the last 10 logs and this occurs after:
[task 2020-10-23T08:25:01.101Z] 08:25:01 INFO - REFTEST TEST-START | docshell/base/crashtests/1257730-1.html

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=319217000&repo=mozilla-central&lineNumber=2452

[task 2020-10-21T04:21:50.127Z] 04:21:50 INFO - REFTEST TEST-START | docshell/base/crashtests/1257730-1.html
[task 2020-10-21T04:21:50.127Z] 04:21:50 INFO - REFTEST INFO | RESTORE PREFERENCE pref(dom.disable_open_during_load,true)
[task 2020-10-21T04:21:50.127Z] 04:21:50 INFO - REFTEST INFO | SET PREFERENCE pref(browser.send_pings,true)
[task 2020-10-21T04:21:50.127Z] 04:21:50 INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/docshell/base/crashtests/1257730-1.html | 34 / 3856 (0%)
[task 2020-10-21T04:21:50.132Z] 04:21:50 INFO - REFTEST TEST-PASS | docshell/base/crashtests/1257730-1.html | (LOAD ONLY)
[task 2020-10-21T04:21:50.132Z] 04:21:50 INFO - REFTEST TEST-END | docshell/base/crashtests/1257730-1.html
[task 2020-10-21T04:21:50.134Z] 04:21:50 INFO - Assertion failure: aElement->HasServoData() (Element without Servo data on a post-traversal? How?), at /builds/worker/checkouts/gecko/layout/base/RestyleManager.cpp:2583
[task 2020-10-21T04:21:50.134Z] 04:21:50 INFO - #01: mozilla::RestyleManager::ProcessPostTraversal(mozilla::dom::Element*, mozilla::ServoRestyleState&, mozilla::ServoPostTraversalFlags) [layout/base/RestyleManager.cpp:2800]
[task 2020-10-21T04:21:50.134Z] 04:21:50 INFO - #02: mozilla::RestyleManager::DoProcessPendingRestyles(mozilla::ServoTraversalFlags) [layout/base/RestyleManager.cpp:3006]
[task 2020-10-21T04:21:50.134Z] 04:21:50 INFO - #03: mozilla::PresShell::DoFlushPendingNotifications(mozilla::ChangesToFlush) [layout/base/PresShell.cpp:4184]
[task 2020-10-21T04:21:50.135Z] 04:21:50 INFO - #04: nsRefreshDriver::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) [layout/base/nsRefreshDriver.cpp:2197]
[task 2020-10-21T04:21:50.135Z] 04:21:50 INFO - #05: mozilla::RefreshDriverTimer::TickRefreshDrivers(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp, nsTArray<RefPtr<nsRefreshDriver> >&) [layout/base/nsRefreshDriver.cpp:347]
[task 2020-10-21T04:21:50.135Z] 04:21:50 INFO - #06: mozilla::RefreshDriverTimer::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) [layout/base/nsRefreshDriver.cpp:370]
[task 2020-10-21T04:21:50.137Z] 04:21:50 INFO - #07: mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::TickRefreshDriver(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) [layout/base/nsRefreshDriver.cpp:749]
[task 2020-10-21T04:21:50.137Z] 04:21:50 INFO - #08: mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::NotifyParentProcessVsync() [layout/base/nsRefreshDriver.cpp:650]
[task 2020-10-21T04:21:50.137Z] 04:21:50 INFO - #09: mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::NotifyVsync(mozilla::VsyncEvent const&) [layout/base/nsRefreshDriver.cpp:570]
[task 2020-10-21T04:21:50.137Z] 04:21:50 INFO - #10: mozilla::layout::VsyncChild::RecvNotify(mozilla::VsyncEvent const&) [layout/ipc/VsyncChild.cpp:57]
[task 2020-10-21T04:21:50.138Z] 04:21:50 INFO - #11: mozilla::layout::PVsyncChild::OnMessageReceived(IPC::Message const&) [s3:gecko-generated-sources:dcfb0c5c8cc64a4b8592c33a9bb2c1a430c98ebd009674a59f93629a7c17a7323253973267a429964267d5e17f515a9e0c9f5ab4ceb87eda82e2dd117c395600/ipc/ipdl/PVsyncChild.cpp::187]
[task 2020-10-21T04:21:50.138Z] 04:21:50 INFO - #12: mozilla::ipc::PBackgroundChild::OnMessageReceived(IPC::Message const&) [s3:gecko-generated-sources:6d2e160d458a876379d79ad4e64e8a33e9758fbaa3dc353140531b3d7521e7ee52f6b8637c2f4e39bd439aa125d19781c1c523a6a2f78b752637dfb8b696677e/ipc/ipdl/PBackgroundChild.cpp::6231]
[task 2020-10-21T04:21:50.138Z] 04:21:50 INFO - #13: mozilla::ipc::MessageChannel::DispatchAsyncMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message const&) [ipc/glue/MessageChannel.cpp:2151]
[task 2020-10-21T04:21:50.138Z] 04:21:50 INFO - #14: mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) [ipc/glue/MessageChannel.cpp:2077]
[task 2020-10-21T04:21:50.140Z] 04:21:50 INFO - #15: mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::MessageChannel::MessageTask&) [ipc/glue/MessageChannel.cpp:1922]
[task 2020-10-21T04:21:50.140Z] 04:21:50 INFO - #16: mozilla::ipc::MessageChannel::MessageTask::Run() [ipc/glue/MessageChannel.cpp:1955]
[task 2020-10-21T04:21:50.140Z] 04:21:50 INFO - #17: mozilla::RunnableTask::Run() [xpcom/threads/TaskController.cpp:246]
[task 2020-10-21T04:21:50.140Z] 04:21:50 INFO - #18: mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:515]
[task 2020-10-21T04:21:50.141Z] 04:21:50 INFO - #19: mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:374]
[task 2020-10-21T04:21:50.141Z] 04:21:50 INFO - #20: mozilla::TaskController::ProcessPendingMTTask(bool) [xpcom/threads/TaskController.cpp:171]
[task 2020-10-21T04:21:50.141Z] 04:21:50 INFO - #21: mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_3>::Run() [xpcom/threads/nsThreadUtils.h:578]
[task 2020-10-21T04:21:50.141Z] 04:21:50 INFO - #22: nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1200]
[task 2020-10-21T04:21:50.141Z] 04:21:50 INFO - #23: NS_ProcessNextEvent(nsIThread*, bool) [xpcom/threads/nsThreadUtils.cpp:513]
[task 2020-10-21T04:21:50.143Z] 04:21:50 INFO - #24: mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:87]
[task 2020-10-21T04:21:50.143Z] 04:21:50 INFO - #25: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:334]
[task 2020-10-21T04:21:50.143Z] 04:21:50 INFO - #26: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:310]
[task 2020-10-21T04:21:50.143Z] 04:21:50 INFO - #27: nsBaseAppShell::Run() [widget/nsBaseAppShell.cpp:139]
[task 2020-10-21T04:21:50.143Z] 04:21:50 INFO - #28: XRE_RunAppShell() [toolkit/xre/nsEmbedFunctions.cpp:913]
[task 2020-10-21T04:21:50.143Z] 04:21:50 INFO - #29: mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:237]
[task 2020-10-21T04:21:50.145Z] 04:21:50 INFO - #30: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:334]
[task 2020-10-21T04:21:50.145Z] 04:21:50 INFO - #31: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:310]
[task 2020-10-21T04:21:50.145Z] 04:21:50 INFO - #32: XRE_InitChildProcess(int, char**, XREChildData const*) [toolkit/xre/nsEmbedFunctions.cpp:744]
[task 2020-10-21T04:21:50.145Z] 04:21:50 INFO - #33: content_process_main(mozilla::Bootstrap*, int, char**) [ipc/contentproc/plugin-container.cpp:57]
[task 2020-10-21T04:21:50.146Z] 04:21:50 INFO - #34: main [browser/app/nsBrowserApp.cpp:304]
[task 2020-10-21T04:21:50.146Z] 04:21:50 INFO - #35: __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6 + 0x21b97]
[task 2020-10-21T04:21:50.146Z] 04:21:50 INFO - #36: ??? [/builds/worker/workspace/build/application/firefox/firefox-bin + 0x3f7c9]
[task 2020-10-21T04:21:50.146Z] 04:21:50 INFO - #37: ??? (???:???)
[task 2020-10-21T04:21:50.146Z] 04:21:50 INFO - [Parent 1406, Breakpad Server] WARNING: Resource acquired is being released in non-LIFO order; why?
[task 2020-10-21T04:21:50.146Z] 04:21:50 INFO - : file /builds/worker/checkouts/gecko/xpcom/threads/BlockingResourceBase.cpp:292
[task 2020-10-21T04:21:50.148Z] 04:21:50 INFO - --- Mutex : dumpSafetyLock (currently acquired)
[task 2020-10-21T04:21:50.148Z] 04:21:50 INFO - calling context
[task 2020-10-21T04:21:50.148Z] 04:21:50 INFO - [stack trace unavailable]
[task 2020-10-21T04:21:50.148Z] 04:21:50 INFO - [Parent 1406, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-10-21T04:21:50.148Z] 04:21:50 INFO - [Parent 1406, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-10-21T04:21:50.150Z] 04:21:50 INFO - [Parent 1406, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-10-21T04:21:50.150Z] 04:21:50 INFO - [Parent 1406, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-10-21T04:21:50.150Z] 04:21:50 INFO - [Parent 1406, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-10-21T04:21:50.150Z] 04:21:50 INFO - [Parent 1406, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-10-21T04:21:50.152Z] 04:21:50 INFO - [Parent 1406, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-10-21T04:21:50.152Z] 04:21:50 INFO - [Parent 1406, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-10-21T04:21:50.152Z] 04:21:50 INFO - [Parent 1406, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-10-21T04:21:50.152Z] 04:21:50 INFO - [Parent 1406, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:511
[task 2020-10-21T04:21:50.152Z] 04:21:50 INFO - [Parent 1406, Compositor] WARNING: bad Shmem: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:470
[task 2020-10-21T04:21:50.153Z] 04:21:50 INFO - [Parent 1406, Compositor] WARNING: bad Shmem: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:470
[task 2020-10-21T04:21:50.153Z] 04:21:50 INFO - [Parent 1406, Compositor] WARNING: bad Shmem: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:470
[task 2020-10-21T04:21:50.153Z] 04:21:50 ERROR - A content process crashed and MOZ_CRASHREPORTER_SHUTDOWN is set, shutting down
[task 2020-10-21T04:21:50.153Z] 04:21:50 INFO - [Parent 1406, Compositor] WARNING: Possibly dropping task posted to updater thread: file /builds/worker/checkouts/gecko/gfx/layers/apz/src/APZUpdater.cpp:370
[task 2020-10-21T04:21:50.155Z] 04:21:50 INFO - [Parent 1406, Compositor] WARNING: Possibly dropping task posted to updater thread: file /builds/worker/checkouts/gecko/gfx/layers/apz/src/APZUpdater.cpp:370
[task 2020-10-21T04:21:50.155Z] 04:21:50 INFO - [Parent 1406, Main Thread] WARNING: '!inner', file /builds/worker/checkouts/gecko/dom/ipc/jsactor/JSWindowActorProtocol.cpp:172
[task 2020-10-21T04:21:50.155Z] 04:21:50 INFO - [Parent 1406, Main Thread] WARNING: '!inner', file /builds/worker/checkouts/gecko/dom/ipc/jsactor/JSWindowActorProtocol.cpp:172
[task 2020-10-21T04:21:50.155Z] 04:21:50 INFO - [Parent 1406, Compositor] WARNING: Possibly dropping task posted to updater thread: file /builds/worker/checkouts/gecko/gfx/layers/apz/src/APZUpdater.cpp:370
[task 2020-10-21T04:21:50.155Z] 04:21:50 INFO - ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-10-21T04:21:50.156Z] 04:21:50 INFO - [2020-10-21T04:21:46Z WARN xulstore::persist] tried to remove key that isn't in the store
[task 2020-10-21T04:21:50.156Z] 04:21:50 INFO - [2020-10-21T04:21:46Z WARN xulstore::persist] tried to remove key that isn't in the store
[task 2020-10-21T04:21:50.156Z] 04:21:50 INFO - [Parent 1406, Main Thread] WARNING: 'aOwner->IsDiscarded()', file /builds/worker/workspace/obj-build/dist/include/mozilla/dom/SyncedContextInlines.h:36
[task 2020-10-21T04:21:50.156Z] 04:21:50 INFO - !!! error running onStopped callback: TypeError: callback is not a function

Sean are you working on this?

Comment 25

[Intermittent Failures Robot]

15 failures in 4230 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 6
• try: 1

Platform breakdown:

• macosx1014-64: 1
• linux1804-64-qr: 5
• linux1804-64-asan-qr: 2
• linux1804-64: 3
• windows7-32: 3
• linux1804-64-asan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-10-19&endday=2020-10-25&tree=all

Comment 26

[Intermittent Failures Robot]

6 failures in 4275 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• try: 1

Platform breakdown:

• linux1804-64: 4
• windows7-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-10-26&endday=2020-11-01&tree=all

Comment 27

[Intermittent Failures Robot]

11 failures in 4128 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• try: 3
• mozilla-beta: 1

Platform breakdown:

• windows7-32: 1
• linux1804-64-qr: 3
• linux1804-64: 6
• linux1804-64-asan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651038&startday=2020-11-02&endday=2020-11-08&tree=all

Comment 28

[Intermittent Failures Robot]

2 failures in 4040 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• linux1804-64-qr: 1
• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2020-11-09&endday=2020-11-15&tree=all

Comment 29

[Intermittent Failures Robot]

6 failures in 4810 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3
• mozilla-central: 1
• autoland: 1
• mozilla-beta: 1

Platform breakdown:

• linux1804-64: 3
• linux1804-64-asan-qr: 2
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2020-11-16&endday=2020-11-22&tree=all

Comment 30

[Intermittent Failures Robot]

5 failures in 1479 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 2
• mozilla-central: 2

Platform breakdown:

• linux1804-64: 2
• linux1804-64-asan: 1
• linux1804-64-asan-qr: 1
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2020-11-23&endday=2020-11-29&tree=all

Comment 31

[Intermittent Failures Robot]

4 failures in 3712 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• autoland: 1
• mozilla-central: 1

Platform breakdown:

• linux1804-64: 3
• linux1804-64-asan-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2020-11-30&endday=2020-12-06&tree=all

Comment 32

[Intermittent Failures Robot]

7 failures in 4060 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• try: 2
• autoland: 2

Platform breakdown:

• linux1804-64-qr: 4
• linux1804-64: 2
• linux1804-64-asan-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2020-12-07&endday=2020-12-13&tree=all

Comment 33

[Intermittent Failures Robot]

5 failures in 3421 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 3
• mozilla-central: 1

Platform breakdown:

• linux1804-64: 3
• linux1804-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2020-12-14&endday=2020-12-20&tree=all

Comment 34

[Intermittent Failures Robot]

3 failures in 1450 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• autoland: 1

Platform breakdown:

• linux1804-64: 1
• linux1804-64-qr: 1
• linux1804-64-asan-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2020-12-21&endday=2020-12-27&tree=all

Comment 35

[Intermittent Failures Robot]

4 failures in 525 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-beta: 1
• mozilla-central: 1

Platform breakdown:

• linux1804-64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2020-12-28&endday=2021-01-03&tree=all

Comment 36

[Intermittent Failures Robot]

5 failures in 2956 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-beta: 1

Platform breakdown:

• linux1804-64: 3
• linux1804-64-asan-qr: 1
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2021-01-04&endday=2021-01-10&tree=all

Comment 37

[Intermittent Failures Robot]

18 failures in 3810 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 16
• mozilla-central: 1
• try: 1

Platform breakdown:

• linux1804-64-qr: 9
• linux1804-64: 5
• windows7-32: 2
• macosx1014-64: 1
• linux1804-64-asan-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2021-01-11&endday=2021-01-17&tree=all

Comment 38

[Intermittent Failures Robot]

15 failures in 3488 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 13
• try: 1

Platform and build breakdown:

• linux1804-64-asan: 2
  • opt: 2
• linux1804-64-asan-qr: 1
  • opt: 1
• windows7-32: 5
  • debug: 5
• linux1804-64-qr: 4
  • debug: 4
• linux1804-64: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2021-01-18&endday=2021-01-24&tree=all

Comment 39

[Intermittent Failures Robot]

12 failures in 3891 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 7
• try: 1
• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 2
  • opt: 2
• linux1804-64: 5
  • debug: 5
• linux1804-64-qr: 3
  • debug: 3
• windows7-32: 1
  • debug: 1
• linux1804-64-asan: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2021-01-25&endday=2021-01-31&tree=all

Comment 40

[Intermittent Failures Robot]

10 failures in 3966 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• try: 1

Platform and build breakdown:

• linux1804-64: 4
  • debug: 3
  • opt: 1
• linux1804-64-qr: 1
  • debug: 1
• linux1804-64-asan-qr: 2
  • opt: 2
• windows7-32: 1
  • debug: 1
• macosx1014-64: 1
  • debug: 1
• android-em-7-0-x86_64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2021-02-01&endday=2021-02-07&tree=all

Comment 41

[Intermittent Failures Robot]

20 failures in 931 pushes (0.021 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17
• try: 1
• mozilla-central: 2

Platform and build breakdown:

• windows7-32: 18
  • debug: 18
• linux1804-64-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2021-02-09&endday=2021-02-09&tree=all

Comment 42

[Alexandru Michis [:malexandru]]

There were 57 total failures in the last 7 days:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2021-02-06&endday=2021-02-13&tree=trunk&bug=1651038
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=329865406&repo=autoland&lineNumber=2481

Affected platforms are:

• windows7-32 debug
• windows10-64 asan
• macosx1014-64 debug
• linux1804-64-qr debug
• linux1804-64-asan-qr and linux1804-64-asan
• linux1804-64 debug

Comment 43

[Intermittent Failures Robot]

70 failures in 4322 pushes (0.016 failures/push) were associated with this bug in the last 7 days.

This is the #10 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 10
• autoland: 58
• try: 1
• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 3
  • opt: 3
• linux1804-64: 3
  • debug: 3
• windows7-32: 53
  • debug: 53
• linux1804-64-qr: 3
  • debug: 3
• windows10-64: 6
  • asan: 6
• macosx1014-64: 1
  • debug: 1
• linux1804-64-asan: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2021-02-08&endday=2021-02-14&tree=all

Comment 44

[Andreea Pavel [:apavel]]

Hi Daniel, can you take a look or assign someone to? Thank you in advance.

Comment 45

[Daniel Holbert [:dholbert]]

Emilio, maybe you could take a look? Looks like you'd expressed some interest in comment 7. Thanks!

Comment 46

[Emilio Cobos Álvarez (:emilio)]

Yes, I'm in fact interested, but it doesn't seem there is any sort of reproducer that I could look at? Andreea, do you know which tests usually fail like this? Maybe I could leave some of them running and I could get a pernosco / rr recording. Otherwise this is not particularly actionable. We could downgrade the assert but that'd make the bug even harder to catch...

Jason, do you know if any of the fuzzers may have a repro for this?

Comment 47

[Andreea Pavel [:apavel]]

Hi!
Last 7 days we have like this:

• linux1804-64-asan-qr: 92788-; 1.html; 1511130.html; 1257730-1.html;
• windows10-64 asan & debug: 1257730-1.html; 1003441.xhtml;
• windows7-32 debug: 1257730-1.html; 1003441.xhtml;

Majority of failures on each platform is on docshell/base/crashtests/1257730-1.html, same as the first failure reported.
Let me know if you need me to look on a larger period of time.

Comment 48

[Intermittent Failures Robot]

19 failures in 3737 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 17
• mozilla-central: 2

Platform and build breakdown:

• windows7-32: 6
  • debug: 6
• windows10-64: 6
  • debug: 1
  • asan: 5
• linux1804-64-asan-qr: 5
  • opt: 5
• linux1804-64-qr: 1
  • debug: 1
• macosx1014-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2021-02-15&endday=2021-02-21&tree=all

Comment 49

[Jason Kratzer [:jkratzer]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #46)

Yes, I'm in fact interested, but it doesn't seem there is any sort of reproducer that I could look at? Andreea, do you know which tests usually fail like this? Maybe I could leave some of them running and I could get a pernosco / rr recording. Otherwise this is not particularly actionable. We could downgrade the assert but that'd make the bug even harder to catch...

Jason, do you know if any of the fuzzers may have a repro for this?

Unfortunately not. The closest crash we have to that signature was filed in bug 1462401. However, that testcase no longer reproduces and the last crash for that signature was seen on 2019-01-20.

Comment 50

[Jason Kratzer [:jkratzer]]

Looks like I spoke too soon. I managed to get a pernosco session for this issue. You can find it here:
https://pernos.co/debug/BUvPrCCUEf7vdiAvYrhJzg/index.html

Comment 51

[Jason Kratzer [:jkratzer]]

Attachment: testcase.zip

The attached testcase cane be reproduced using grizzly[1]:

1. pip install grizzly-framework
2. unzip testcase.zip -d testcase-dir
3. python3 -m grizzly.replay --xvfb --repeat 10 ~/builds/mc-debug/firefox ./testcase-dir

[1] https://github.com/MozillaSecurity/grizzly

Comment 52

[Emilio Cobos Álvarez (:emilio)]

Attachment: Test-case from Jason

Beautiful. Here's the test-case slightly simplified. That's more than enough to crash Nightly for me. Digging now. Off hand it seems like an XML parser bug/issue, because .innerHTML triggers an unexpected flush from the parser itself...

Comment 53

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1651038 - nsXMLContentSink::DidBuildModel should flush pending tags. r=smaug

As it won't after that if something flushes (because
FlushPendingNotifications will just ignore the flush if the sink is no
longer a document observer).

Otherwise we might not notify of some document insertions, and that
might reasonably confuse layout if it has started before.

The reason we don't have more similar crashes seems to be two different
things:

• That in the crashtest, the custom element triggers a partial parser
  flush via the innerHTML setter while the parser is creating the
  element. That causes the parser to flush the previous text node
  (which tells layout to go look under the <body> for children) but
  unless we FlushTags() once we're done we won't know about the
  custom-element child.

• That there's no end tag for the <body> or <html> nodes, so we don't
  reach this code when we end them:

  https://searchfox.org/mozilla-central/rev/df23c6e58c6be1eb8399e475878f73d4867bef87/dom/xml/nsXMLContentSink.cpp#1041-1047

Comment 54

[Emilio Cobos Álvarez (:emilio)]

Jason, thanks so so much for the test-case!

Comment 55

[Emilio Cobos Álvarez (:emilio)]

James, wpt lint complains it can't parse the file, but that's kind of the point (this bug happens partially because the test-case doesn't contain trailing closing tags). Do you know where can I annotate it?

Comment 56

[James Graham [:jgraham]]

https://searchfox.org/mozilla-central/source/testing/web-platform/tests/lint.ignore#124 is the file/pattern to follow.

Comment 57

[Intermittent Failures Robot]

12 failures in 4327 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-central: 2

Platform and build breakdown:

• windows10-64: 7
  • debug: 2
  • asan: 5
• windows7-32: 1
  • debug: 1
• linux1804-64-qr: 1
  • debug: 1
• macosx1014-64-qr: 1
  • debug: 1
• macosx1014-64: 1
  • debug: 1
• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2021-02-22&endday=2021-02-28&tree=all

Comment 58

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b6cdff635c85
nsXMLContentSink::DidBuildModel should flush pending tags. r=hsivonen

Comment 59

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/b6cdff635c85

Comment 60

[Intermittent Failures Robot]

14 failures in 3683 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-central: 2
• try: 2

Platform and build breakdown:

• windows10-64: 8
  • asan: 6
  • debug: 2
• macosx1014-64-qr: 3
  • debug: 3
• linux1804-64-asan-qr: 1
  • opt: 1
• linux1804-64-qr: 1
  • debug: 1
• linux1804-64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2021-03-01&endday=2021-03-07&tree=all

Comment 61

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/27993 for changes under testing/web-platform/tests

Comment 62

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 63

[Intermittent Failures Robot]

1 failures in 4622 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1

Platform and build breakdown:

• linux1804-64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1651038&startday=2021-03-08&endday=2021-03-14&tree=all