1427172 - Intermittent table-anonymous-boxes/162063-1.xhtml | application timed out after 370 seconds with no output

Status: RESOLVED INCOMPLETE

(Core :: Layout: Tables, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: aciure [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=153418942&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/Jbqu9xmRTWmlgy0WkQMObA/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/Jbqu9xmRTWmlgy0WkQMObA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

Comment 1

[Intermittent Failures Robot]

13 failures in 147 pushes (0.088 failures/push) were associated with this bug in the last 7 days. 

This is the #15 most frequent failure this week.   

Repository breakdown:
* mozilla-central: 6
* mozilla-inbound: 4
* autoland: 3

Platform breakdown:
* linux32-stylo-disabled: 11
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1427172&startday=2017-12-25&endday=2017-12-31&tree=all

Comment 2

[Tiberius Oros[:tiberius_oros]]

There have been 31 failures in the last week.

Most of the failures occur on linux32-stylo-disabled (19 occurrences)and other occur on windows10-64, Windows 7, linux64-stylo-disabled, Linux x64, Linux and linux64-qr.

The most affected build is debug (21 occurrences)followed by opt and pgo.

Here is a recent relevant log file: https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=154348956
And a link to the reftest analyzer file: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/e6xWyQImRLagy7Drb1GE1Q/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

:jet, would you please take a look? Thank you.

Comment 3

[Intermittent Failures Robot]

35 failures in 462 pushes (0.076 failures/push) were associated with this bug in the last 7 days. 

This is the #22 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 15
* mozilla-inbound: 12
* mozilla-central: 5
* try: 3

Platform breakdown:
* linux32-stylo-disabled: 19
* windows10-64: 6
* windows7-32: 4
* linux64: 2
* linux32: 2
* linux64-stylo-disabled: 1
* linux64-qr: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1427172&startday=2018-01-01&endday=2018-01-07&tree=all

Comment 4

[Jet Villegas (inactive)]

The Linux crashes appear to be some variant of this one:
[task 2018-01-05T11:33:10.981Z] 11:33:10     INFO - [Parent 966, Gecko_IOThread] WARNING: pipe error (79): Connection reset by peer: file /builds/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353
[task 2018-01-05T11:33:10.987Z] 11:33:10     INFO - ###!!! [Parent][MessageChannel] Error: (msgtype=0x150084,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv
[task 2018-01-05T11:39:30.558Z] 11:39:30     INFO - Crash reason:  SIGABRT
[task 2018-01-05T11:39:30.559Z] 11:39:30     INFO - Crash address: 0x3bc
[task 2018-01-05T11:39:30.559Z] 11:39:30     INFO - Process uptime: not available
[task 2018-01-05T11:39:30.560Z] 11:39:30     INFO - 
[task 2018-01-05T11:39:30.560Z] 11:39:30     INFO - Thread 0 (crashed)
[task 2018-01-05T11:39:30.561Z] 11:39:30     INFO -  0  linux-gate.so + 0xcd9
[task 2018-01-05T11:39:30.562Z] 11:39:30     INFO -     eip = 0xf77b6cd9   esp = 0xffe6a974   ebp = 0xffe6a9e8   ebx = 0xe449a480
[task 2018-01-05T11:39:30.563Z] 11:39:30     INFO -     esi = 0xf71447b0   edi = 0xe449a480   eax = 0xfffffffc   ecx = 0x00000004
[task 2018-01-05T11:39:30.564Z] 11:39:30     INFO -     edx = 0xffffffff   efl = 0x00000292
[task 2018-01-05T11:39:30.565Z] 11:39:30     INFO -     Found by: given as instruction pointer in context
[task 2018-01-05T11:39:30.565Z] 11:39:30     INFO -  1  libglib-2.0.so.0.4800.2 + 0x4711c
[task 2018-01-05T11:39:30.566Z] 11:39:30     INFO -     eip = 0xf5e5d11c   esp = 0xffe6a9f0   ebp = 0x00000004
[task 2018-01-05T11:39:30.567Z] 11:39:30     INFO -     Found by: previous frame's frame pointer
[task 2018-01-05T11:39:30.568Z] 11:39:30     INFO -  2  libglib-2.0.so.0.4800.2 + 0x8cfec
[task 2018-01-05T11:39:30.569Z] 11:39:30     INFO -     eip = 0xf5ea2fec   esp = 0xffe6aa00   ebp = 0x00000004
[task 2018-01-05T11:39:30.570Z] 11:39:30     INFO -     Found by: stack scanning
[task 2018-01-05T11:39:30.571Z] 11:39:30     INFO -  3  libxul.so!IsFrameContentNodeInNamespace [nsNativeThemeGTK.cpp:de5351e9d43f : 113 + 0x4]
[task 2018-01-05T11:39:30.571Z] 11:39:30     INFO -     eip = 0xf13d2333   esp = 0xffe6aa04   ebp = 0x00000004
[task 2018-01-05T11:39:30.571Z] 11:39:30     INFO -     Found by: stack scanning
[task 2018-01-05T11:39:30.573Z] 11:39:30     INFO -  4  libglib-2.0.so.0.4800.2 + 0x1285b0
[task 2018-01-05T11:39:30.573Z] 11:39:30     INFO -     eip = 0xf5f3e5b0   esp = 0xffe6aa14   ebp = 0x00000004
[task 2018-01-05T11:39:30.574Z] 11:39:30     INFO -     Found by: stack scanning

...and I don't know if I trust that stack ^ at all. I'm adding a dependency on bug 1428509 so at least we get better logs for IPC pipe failures on Linux.

However, I did find a Windows crash with what looks like a clearer stack, here:
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=154056717&lineNumber=25378

Matt: can you have a look and see if this is a reproducible DL crash? Thx!

Comment 5

[Matt Woodrow (:mattwoodrow)]

Not sure why the assertion isn't firing, but this is almost certainly the same issue as bug 1427221 (same failing test).

Comment 6

[Jed Davis [:jld] ⟨⏰|UTC-8⟩ ⟦he/him⟧]

(In reply to Jet Villegas (:jet) from comment #4)
> [task 2018-01-05T11:39:30.565Z] 11:39:30     INFO -  1  libglib-2.0.so.0.4800.2 + 0x4711c
> [task 2018-01-05T11:39:30.567Z] 11:39:30     INFO -     Found by: previous frame's frame pointer

$ addr2line -Cfie usr/lib/debug/.build-id/fe/a2996b7f031dd2269e17664f32e93ccc99bd5b.debug 0x4711c
g_main_context_poll
/build/glib2.0-RnwmWL/glib2.0-2.48.2/./glib/gmain.c:4135
g_main_context_iterate
/build/glib2.0-RnwmWL/glib2.0-2.48.2/./glib/gmain.c:3835

So we SIGABRTed an event loop that was waiting for events; I think that's what normally happens when the test timeout expires.  But there's another crash report in that log:

[task 2018-01-05T11:39:40.131Z] 11:39:40     INFO - Crash reason:  SIGSEGV
[task 2018-01-05T11:39:40.131Z] 11:39:40     INFO - Crash address: 0xf724781b
[task 2018-01-05T11:39:40.133Z] 11:39:40     INFO - Process uptime: not available
[task 2018-01-05T11:39:40.133Z] 11:39:40     INFO - 
[task 2018-01-05T11:39:40.134Z] 11:39:40     INFO - Thread 0 (crashed)
[task 2018-01-05T11:39:40.134Z] 11:39:40     INFO -  0  libxul.so!nsLayoutUtils::GetCrossDocParentFrame [nsLayoutUtils.cpp:de5351e9d43f : 1717 + 0x11]
[task 2018-01-05T11:39:40.135Z] 11:39:40     INFO -     eip = 0xf152b810   esp = 0xff8dad40   ebp = 0xff8dad58   ebx = 0xdf335e70
[task 2018-01-05T11:39:40.137Z] 11:39:40     INFO -     esi = 0x00000000   edi = 0xf53cd000   eax = 0xf72477ff   ecx = 0x0000001d
[task 2018-01-05T11:39:40.137Z] 11:39:40     INFO -     edx = 0xf72477ff   efl = 0x00010296
[task 2018-01-05T11:39:40.138Z] 11:39:40     INFO -     Found by: given as instruction pointer in context
[task 2018-01-05T11:39:40.139Z] 11:39:40     INFO -  1  libxul.so!nsLayoutUtils::IsAncestorFrameCrossDoc [nsLayoutUtils.cpp:de5351e9d43f : 1750 + 0xa]
[task 2018-01-05T11:39:40.140Z] 11:39:40     INFO -     eip = 0xf152b8b5   esp = 0xff8dad60   ebp = 0xff8dad78   ebx = 0xdf335e70
[task 2018-01-05T11:39:40.141Z] 11:39:40     INFO -     esi = 0xd5283000   edi = 0xf53cd000
[task 2018-01-05T11:39:40.141Z] 11:39:40     INFO -     Found by: call frame info
[task 2018-01-05T11:39:40.143Z] 11:39:40     INFO -  2  libxul.so!nsDisplayListBuilder::FindAnimatedGeometryRootFrameFor [nsDisplayList.cpp:de5351e9d43f : 1828 + 0x16]
[task 2018-01-05T11:39:40.143Z] 11:39:40     INFO -     eip = 0xf1757321   esp = 0xff8dad80   ebp = 0xff8dadb8   ebx = 0xdf335e70
[task 2018-01-05T11:39:40.143Z] 11:39:40     INFO -     esi = 0xd5283000   edi = 0xf53cd000
[task 2018-01-05T11:39:40.144Z] 11:39:40     INFO -     Found by: call frame info
[task 2018-01-05T11:39:40.145Z] 11:39:40     INFO -  3  libxul.so!nsDisplayListBuilder::FindAnimatedGeometryRootFor [nsDisplayList.cpp:de5351e9d43f : 1149 + 0xc]
[task 2018-01-05T11:39:40.146Z] 11:39:40     INFO -     eip = 0xf17675cc   esp = 0xff8dadc0   ebp = 0xff8dadf8   ebx = 0xd5283000
[task 2018-01-05T11:39:40.146Z] 11:39:40     INFO -     esi = 0xff8daddc   edi = 0xdf335e70
[task 2018-01-05T11:39:40.147Z] 11:39:40     INFO -     Found by: call frame info
[task 2018-01-05T11:39:40.149Z] 11:39:40     INFO -  4  libxul.so!RetainedDisplayListBuilder::ComputeRebuildRegion [RetainedDisplayListBuilder.cpp:de5351e9d43f : 608 + 0xd]
[task 2018-01-05T11:39:40.149Z] 11:39:40     INFO -     eip = 0xf17676b2   esp = 0xff8dae00   ebp = 0xff8daec8   ebx = 0xd5283000
[task 2018-01-05T11:39:40.150Z] 11:39:40     INFO -     esi = 0xff8daf2c   edi = 0xff8daf10
[task 2018-01-05T11:39:40.150Z] 11:39:40     INFO -     Found by: call frame info
[task 2018-01-05T11:39:40.151Z] 11:39:40     INFO -  5  libxul.so!RetainedDisplayListBuilder::AttemptPartialUpdate [RetainedDisplayListBuilder.cpp:de5351e9d43f : 828 + 0x19]
[task 2018-01-05T11:39:40.152Z] 11:39:40     INFO -     eip = 0xf17980de   esp = 0xff8daed0   ebp = 0xff8daf98   ebx = 0xd5283000
[task 2018-01-05T11:39:40.153Z] 11:39:40     INFO -     esi = 0xff8daf2c   edi = 0xff8daf10
[task 2018-01-05T11:39:40.153Z] 11:39:40     INFO -     Found by: call frame info
[task 2018-01-05T11:39:40.155Z] 11:39:40     INFO -  6  libxul.so!nsLayoutUtils::PaintFrame [nsLayoutUtils.cpp:de5351e9d43f : 3808 + 0x10]
[task 2018-01-05T11:39:40.156Z] 11:39:40     INFO -     eip = 0xf154e927   esp = 0xff8dafa0   ebp = 0xff8dc848   ebx = 0xf53cd000
[task 2018-01-05T11:39:40.156Z] 11:39:40     INFO -     esi = 0xff8db014   edi = 0xd528457c
[task 2018-01-05T11:39:40.157Z] 11:39:40     INFO -     Found by: call frame info
[task 2018-01-05T11:39:40.157Z] 11:39:40     INFO -  7  libxul.so!mozilla::PresShell::Paint [PresShell.cpp:de5351e9d43f : 6486 + 0x1b]
[task 2018-01-05T11:39:40.158Z] 11:39:40     INFO -     eip = 0xf1516dad   esp = 0xff8dc850   ebp = 0xff8dc988   ebx = 0xf7044350
[task 2018-01-05T11:39:40.159Z] 11:39:40     INFO -     esi = 0xd5cc7000   edi = 0xff8dc900
[task 2018-01-05T11:39:40.159Z] 11:39:40     INFO -     Found by: call frame info

…

[task 2018-01-05T11:39:40.222Z] 11:39:40     INFO - 33  firefox!content_process_main [plugin-container.cpp:de5351e9d43f : 63 + 0xc]
[task 2018-01-05T11:39:40.222Z] 11:39:40     INFO -     eip = 0x0804d9ef   esp = 0xff8dd8f0   ebp = 0xff8dd938   ebx = 0x00000015
[task 2018-01-05T11:39:40.223Z] 11:39:40     INFO -     esi = 0xff8dda34   edi = 0xf700e380
[task 2018-01-05T11:39:40.224Z] 11:39:40     INFO -     Found by: call frame info


So the content process crashed in layout, but the main process continued running until the test harness killed it, and then the main process's “crash” was logged first.  Bug 1428509 probably wouldn't change this much besides adding “child process NNNNN killed by SIGSEGV” or similar; that information *is* already in the log, but it's not as obvious.

Also, I notice that some of the crashes from that orangefactor link have the same assertion failure as the bug that comment #5 mentions:  https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=153947769&lineNumber=25318

Comment 7

[Intermittent Failures Robot]

49 failures in 788 pushes (0.062 failures/push) were associated with this bug in the last 7 days. 

This is the #28 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-inbound: 23
* autoland: 12
* mozilla-central: 9
* try: 5

Platform breakdown:
* linux32-stylo-disabled: 23
* linux64-stylo-disabled: 11
* linux64: 6
* windows7-32: 4
* osx-10-10: 3
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1427172&startday=2018-01-08&endday=2018-01-14&tree=all