Closed
Bug 1403220
Opened 7 years ago
Closed 6 years ago
Windows 7 pgo: all tests using application either failing with connection errors, or normal behavior
Categories
(Core :: Layout: Text and Fonts, defect)
Core
Layout: Text and Fonts
Tracking
()
RESOLVED
FIXED
| Tracking | Status | |
|---|---|---|
| firefox57 | --- | fixed |
People
(Reporter: aryx, Assigned: away)
Details
(Whiteboard: [stockwell fixed:product])
Attachments
(1 file)
|
3.23 KB,
patch
|
Details | Diff | Splinter Review |
Ryan, have you seen anything like this in your Try pushes and can you CC the relevant people? Thank you. On Windows 7 pgo, today either all tests using the application are either failing with connection errors, or it's normal behavior (few, other failures). It only affects inbound as far as I noticed. First push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=590f81062c5f6b6a5df5a0d992fde2d633527257&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-searchStr=windows+pgo See pushes with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=491e7fdae961010548bc12900fccee33c1b3fdf7&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-searchStr=windows%20pgo&tochange=3053445906837db96cce7fc059576c9b5e821a1b Almost all of them have this in the log: 10:33:46 INFO - Application command: Z:\task_1506421162\build\application\firefox\firefox.exe -marionette -foreground -profile c:\users\genericworker\appdata\local\temp\tmpkgc_vv.mozrunner 10:33:46 INFO - runtests.py | Application pid: 1520 10:33:46 INFO - TEST-INFO | started process GECKO(1520) 10:33:47 INFO - GECKO(1520) | 1506422027510 Marionette INFO Enabled via --marionette 10:36:47 INFO - Traceback (most recent call last): 10:36:47 INFO - File "Z:\task_1506421162\build\tests\mochitest\runtests.py", line 2631, in doTests 10:36:47 INFO - marionette_args=marionette_args, 10:36:47 INFO - File "Z:\task_1506421162\build\tests\mochitest\runtests.py", line 2156, in runApp 10:36:47 INFO - self.marionette.start_session() 10:36:47 INFO - File "Z:\task_1506421162\build\venv\lib\site-packages\marionette_driver\decorators.py", line 28, in _ 10:36:47 INFO - m._handle_socket_failure() 10:36:47 INFO - File "Z:\task_1506421162\build\venv\lib\site-packages\marionette_driver\decorators.py", line 23, in _ 10:36:47 INFO - return func(*args, **kwargs) 10:36:47 INFO - File "Z:\task_1506421162\build\venv\lib\site-packages\marionette_driver\marionette.py", line 1228, in start_session 10:36:47 INFO - self.protocol, _ = self.client.connect() 10:36:47 INFO - File "Z:\task_1506421162\build\venv\lib\site-packages\marionette_driver\transport.py", line 223, in connect 10:36:47 INFO - self.sock.connect((self.addr, self.port)) 10:36:47 INFO - File "c:\mozilla-build\python\Lib\socket.py", line 228, in meth 10:36:47 INFO - return getattr(self._sock,name)(*args) 10:36:47 INFO - error: [Errno 10061] No connection could be made because the target machine actively refused it Talos g4 is different: https://treeherder.mozilla.org/logviewer.html#?job_id=133306285&repo=mozilla-inbound 03:43:52 INFO - PID 3208 | [#16] 1080p.60fps.mp4_scale_2_startup Cycles:12 Average:8.00 Median:8.02 stddev:0.09 (1.1%) stddev-sans-first:0.08 03:43:52 INFO - PID 3208 | Values: 8.1 8.1 8.0 8.0 8.0 7.9 8.1 8.0 8.0 8.1 7.9 7.9 03:43:52 INFO - PID 3208 | 03:43:52 INFO - PID 3208 | [#17] 1080p.60fps.mp4_scale_2_inclip Cycles:12 Average:7.58 Median:7.65 stddev:0.10 (1.3%) stddev-sans-first:0.10 03:43:52 INFO - PID 3208 | Values: 7.6 7.7 7.7 7.5 7.4 7.5 7.7 7.5 7.7 7.7 7.5 7.5 03:43:52 INFO - PID 3208 | -------- Summary: end -------- 03:43:52 INFO - PID 3208 | 03:43:52 INFO - PID 3208 | [Parent 3208, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346 03:43:53 INFO - PID 3208 | [GPU 3380, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346 03:43:54 INFO - TEST-INFO | 3208: exit 0 03:43:54 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/U3yEwwBwQR-k-glN7QeyvQ/artifacts/public/build/target.crashreporter-symbols.zip 03:44:01 INFO - mozcrash Copy/paste: C:\slave\test\build\win32-minidump_stackwalk.exe c:\users\cltbld\appdata\local\temp\tmpygo7g2\profile\minidumps\3167a0c6-3353-4ae1-90a9-1e906b754b02.dmp c:\users\cltbld\appdata\local\temp\tmpvjovs4 03:44:07 INFO - mozcrash Saved minidump as C:\slave\test\build\blobber_upload_dir\3167a0c6-3353-4ae1-90a9-1e906b754b02.dmp 03:44:07 INFO - mozcrash Saved app info as C:\slave\test\build\blobber_upload_dir\3167a0c6-3353-4ae1-90a9-1e906b754b02.extra 03:44:07 INFO - PROCESS-CRASH | basic_compositor_video | application crashed [@ PropertyProvider::InitFontGroupAndFontMetrics()] 03:44:07 INFO - Crash dump filename: c:\users\cltbld\appdata\local\temp\tmpygo7g2\profile\minidumps\3167a0c6-3353-4ae1-90a9-1e906b754b02.dmp 03:44:07 INFO - Operating system: Windows NT 03:44:07 INFO - 6.1.7601 Service Pack 1 03:44:07 INFO - CPU: x86 03:44:07 INFO - GenuineIntel family 6 model 30 stepping 5 03:44:07 INFO - 8 CPUs 03:44:07 INFO - GPU: UNKNOWN 03:44:07 INFO - Crash reason: EXCEPTION_ACCESS_VIOLATION_READ 03:44:07 INFO - Crash address: 0x3c 03:44:07 INFO - Process uptime: 5 seconds 03:44:07 INFO - Thread 0 (crashed) 03:44:07 INFO - 0 xul.dll!PropertyProvider::InitFontGroupAndFontMetrics() [nsTextFrame.cpp:590f81062c5f : 3232 + 0x1e] 03:44:07 INFO - eip = 0x638219d4 esp = 0x0012c370 ebp = 0x0012c380 ebx = 0x00000000 03:44:07 INFO - esi = 0x0012c4b0 edi = 0x0012c4b0 eax = 0x0012c37c ecx = 0x00000000 03:44:07 INFO - edx = 0x20afd9a0 efl = 0x00010202 03:44:07 INFO - Found by: given as instruction pointer in context 03:44:07 INFO - 1 xul.dll!PropertyProvider::GetFontMetrics() [nsTextFrame.cpp:590f81062c5f : 3217 + 0x5] 03:44:07 INFO - eip = 0x6381fd43 esp = 0x0012c388 ebp = 0x0012c53c 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 2 xul.dll!nsTextFrame::ReflowText(nsLineLayout &,int,mozilla::gfx::DrawTarget *,mozilla::ReflowOutput &,nsReflowStatus &) [nsTextFrame.cpp:590f81062c5f : 9719 + 0xc] 03:44:07 INFO - eip = 0x63912eb6 esp = 0x0012c390 ebp = 0x0012c53c 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 3 xul.dll!nsLineLayout::ReflowFrame(nsIFrame *,nsReflowStatus &,mozilla::ReflowOutput *,bool &) [nsLineLayout.cpp:590f81062c5f : 923 + 0x1f] 03:44:07 INFO - eip = 0x63905be2 esp = 0x0012c544 ebp = 0x0012c6e0 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 4 xul.dll!nsLineLayout::RelativePositionFrames(nsLineLayout::PerSpanData *,nsOverflowAreas &) [nsLineLayout.cpp:590f81062c5f : 3372 + 0x2b] 03:44:07 INFO - eip = 0x639105d4 esp = 0x0012c594 ebp = 0x0012c7b8 03:44:07 INFO - Found by: stack scanning 03:44:07 INFO - 5 xul.dll!nsIFrame::HasPerspective(mozilla::EffectSet *) [nsIFrame.h:590f81062c5f : 1844 + 0x12] 03:44:07 INFO - eip = 0x63910000 esp = 0x0012c80c ebp = 0x0012c62c 03:44:07 INFO - Found by: call frame info with scanning 03:44:07 INFO - 6 xul.dll!nsBlockFrame::ReflowLine(mozilla::BlockReflowInput &,nsLineList_iterator,bool *) [nsBlockFrame.cpp:590f81062c5f : 2873 + 0xb] 03:44:07 INFO - eip = 0x639135d2 esp = 0x0012c87c ebp = 0x0012c8b4 03:44:07 INFO - Found by: call frame info with scanning 03:44:07 INFO - 7 xul.dll!nsBlockFrame::ReflowDirtyLines(mozilla::BlockReflowInput &) [nsBlockFrame.cpp:590f81062c5f : 2409 + 0xb] 03:44:07 INFO - eip = 0x6391210c esp = 0x0012c8bc ebp = 0x0012c984 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 8 xul.dll!nsBlockFrame::Reflow(nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,nsReflowStatus &) [nsBlockFrame.cpp:590f81062c5f : 1235 + 0xd] 03:44:07 INFO - eip = 0x63910987 esp = 0x0012c98c ebp = 0x0012cc48 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 9 xul.dll!nsContainerFrame::ReflowChild(nsIFrame *,nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,mozilla::WritingMode const &,mozilla::LogicalPoint const &,nsSize const &,unsigned int,nsReflowStatus &,nsOverflowContinuationTracker *) [nsContainerFrame.cpp:590f81062c5f : 932 + 0x17] 03:44:07 INFO - eip = 0x638fd217 esp = 0x0012cc50 ebp = 0x0012cc70 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 10 xul.dll!nsHTMLScrollFrame::ReflowScrolledFrame(mozilla::ScrollReflowInput *,bool,bool,mozilla::ReflowOutput *,bool) [nsGfxScrollFrame.cpp:590f81062c5f : 548 + 0x2e] 03:44:07 INFO - eip = 0x638faff2 esp = 0x0012cc78 ebp = 0x0012ce04 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 11 xul.dll!nsHTMLScrollFrame::ReflowContents(mozilla::ScrollReflowInput *,mozilla::ReflowOutput const &) [nsGfxScrollFrame.cpp:590f81062c5f : 660 + 0x29] 03:44:07 INFO - eip = 0x638fac80 esp = 0x0012ce0c ebp = 0x0012ceb0 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 12 xul.dll!nsHTMLScrollFrame::Reflow(nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,nsReflowStatus &) [nsGfxScrollFrame.cpp:590f81062c5f : 1037 + 0x13] 03:44:07 INFO - eip = 0x638fa018 esp = 0x0012ceb8 ebp = 0x0012cfd0 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 13 xul.dll!nsContainerFrame::ReflowChild(nsIFrame *,nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,int,int,unsigned int,nsReflowStatus &,nsOverflowContinuationTracker *) [nsContainerFrame.cpp:590f81062c5f : 976 + 0x17] 03:44:07 INFO - eip = 0x639fa86d esp = 0x0012cfd8 ebp = 0x0012cffc 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 14 xul.dll!nsTextControlFrame::ReflowTextControlChild(nsIFrame *,nsPresContext *,mozilla::ReflowInput const &,nsReflowStatus &,mozilla::ReflowOutput &) [nsTextControlFrame.cpp:590f81062c5f : 690 + 0x1e] 03:44:07 INFO - eip = 0x639fa7e0 esp = 0x0012d004 ebp = 0x0012d174 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 15 xul.dll!nsTextControlFrame::Reflow(nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,nsReflowStatus &) [nsTextControlFrame.cpp:590f81062c5f : 649 + 0xe] 03:44:07 INFO - eip = 0x639fa6a7 esp = 0x0012d17c ebp = 0x0012d1d0 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 16 xul.dll!nsFrame::BoxReflow(nsBoxLayoutState &,nsPresContext *,mozilla::ReflowOutput &,gfxContext *,int,int,int,int,bool) [nsFrame.cpp:590f81062c5f : 10160 + 0x19] 03:44:07 INFO - eip = 0x638fde5b esp = 0x0012d1d8 ebp = 0x0012d3d4 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 17 xul.dll!nsFrame::RefreshSizeCache(nsBoxLayoutState &) [nsFrame.cpp:590f81062c5f : 9707 + 0x27] 03:44:07 INFO - eip = 0x63a87830 esp = 0x0012d3dc ebp = 0x0012d4a0 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 18 xul.dll!nsFrame::GetXULPrefSize(nsBoxLayoutState &) [nsFrame.cpp:590f81062c5f : 9785 + 0xa] 03:44:07 INFO - eip = 0x63a879d8 esp = 0x0012d4a8 ebp = 0x0012d4c0 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 19 xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12] 03:44:07 INFO - eip = 0x6390a1c8 esp = 0x0012d4c8 ebp = 0x0012d514 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 20 xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb] 03:44:07 INFO - eip = 0x6390c7e2 esp = 0x0012d51c ebp = 0x0012d558 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 21 xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12] 03:44:07 INFO - eip = 0x6390a1c8 esp = 0x0012d560 ebp = 0x0012d5ac 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 22 xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb] 03:44:07 INFO - eip = 0x6390c7e2 esp = 0x0012d5b4 ebp = 0x0012d5f0 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 23 xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12] 03:44:07 INFO - eip = 0x6390a1c8 esp = 0x0012d5f8 ebp = 0x0012d644 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 24 xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb] 03:44:07 INFO - eip = 0x6390c7e2 esp = 0x0012d64c ebp = 0x0012d688 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 25 xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12] 03:44:07 INFO - eip = 0x6390a1c8 esp = 0x0012d690 ebp = 0x0012d6dc 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 26 xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb] 03:44:07 INFO - eip = 0x6390c7e2 esp = 0x0012d6e4 ebp = 0x0012d720 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 27 xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12] 03:44:07 INFO - eip = 0x6390a1c8 esp = 0x0012d728 ebp = 0x0012d774 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 28 xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb] 03:44:07 INFO - eip = 0x6390c7e2 esp = 0x0012d77c ebp = 0x0012d7b8 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 29 xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12] 03:44:07 INFO - eip = 0x6390a1c8 esp = 0x0012d7c0 ebp = 0x0012d80c 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 30 xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb] 03:44:07 INFO - eip = 0x6390c7e2 esp = 0x0012d814 ebp = 0x0012d850 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 31 xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12] 03:44:07 INFO - eip = 0x6390a1c8 esp = 0x0012d858 ebp = 0x0012d8a4 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 32 xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb] 03:44:07 INFO - eip = 0x6390c7e2 esp = 0x0012d8ac ebp = 0x0012d8e8 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 33 xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12] 03:44:07 INFO - eip = 0x6390a1c8 esp = 0x0012d8f0 ebp = 0x0012d93c 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 34 xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb] 03:44:07 INFO - eip = 0x6390c7e2 esp = 0x0012d944 ebp = 0x0012d980 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 35 xul.dll!nsStackLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsStackLayout.cpp:590f81062c5f : 70 + 0x12] 03:44:07 INFO - eip = 0x638fc81c esp = 0x0012d988 ebp = 0x0012d9bc 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 36 xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb] 03:44:07 INFO - eip = 0x6390c7e2 esp = 0x0012d9c4 ebp = 0x0012da00 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 37 xul.dll!nsSprocketLayout::PopulateBoxSizes(nsIFrame *,nsBoxLayoutState &,nsBoxSize * &,int &,int &,int &) [nsSprocketLayout.cpp:590f81062c5f : 746 + 0x12] 03:44:07 INFO - eip = 0x6390aec8 esp = 0x0012da08 ebp = 0x0012da98 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 38 xul.dll!nsSprocketLayout::XULLayout(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 216 + 0x51] 03:44:07 INFO - eip = 0x6390a450 esp = 0x0012daa0 ebp = 0x0012dbdc 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 39 xul.dll!nsBoxFrame::DoXULLayout(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 917 + 0x8] 03:44:07 INFO - eip = 0x63908484 esp = 0x0012dbe4 ebp = 0x0012dd50 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 40 xul.dll!nsIFrame::XULLayout(nsBoxLayoutState &) [nsBox.cpp:590f81062c5f : 501 + 0xc] 03:44:07 INFO - eip = 0x638fcd16 esp = 0x0012dd58 ebp = 0x0012dd64 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 41 xul.dll!nsStackLayout::XULLayout(nsIFrame *,nsBoxLayoutState &) [nsStackLayout.cpp:590f81062c5f : 369 + 0x8] 03:44:07 INFO - eip = 0x638fcbde esp = 0x0012dd6c ebp = 0x0012de44 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 42 xul.dll!nsBoxFrame::DoXULLayout(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 917 + 0x8] 03:44:07 INFO - eip = 0x63908484 esp = 0x0012de4c ebp = 0x0012dfb8 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 43 xul.dll!nsIFrame::XULLayout(nsBoxLayoutState &) [nsBox.cpp:590f81062c5f : 501 + 0xc] 03:44:07 INFO - eip = 0x638fcd16 esp = 0x0012dfc0 ebp = 0x0012dfcc 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 44 xul.dll!nsBoxFrame::Reflow(nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,nsReflowStatus &) [nsBoxFrame.cpp:590f81062c5f : 713 + 0xc] 03:44:07 INFO - eip = 0x638f8cfa esp = 0x0012dfd4 ebp = 0x0012e070 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 45 xul.dll!nsContainerFrame::ReflowChild(nsIFrame *,nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,int,int,unsigned int,nsReflowStatus &,nsOverflowContinuationTracker *) [nsContainerFrame.cpp:590f81062c5f : 976 + 0x17] 03:44:07 INFO - eip = 0x639fa86d esp = 0x0012e078 ebp = 0x0012e09c 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 46 xul.dll!mozilla::ViewportFrame::Reflow(nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,nsReflowStatus &) [ViewportFrame.cpp:590f81062c5f : 330 + 0x24] 03:44:07 INFO - eip = 0x638fc558 esp = 0x0012e0a4 ebp = 0x0012e218 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 47 xul.dll!mozilla::PresShell::DoReflow(nsIFrame *,bool) [PresShell.cpp:590f81062c5f : 9380 + 0x1c] 03:44:07 INFO - eip = 0x638269bd esp = 0x0012e220 ebp = 0x0012e3f0 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 48 xul.dll!mozilla::PresShell::ProcessReflowCommands(bool) [PresShell.cpp:590f81062c5f : 9553 + 0xb] 03:44:07 INFO - eip = 0x638effde esp = 0x0012e3f8 ebp = 0x0012e45c 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 49 xul.dll!mozilla::PresShell::DoFlushPendingNotifications(mozilla::ChangesToFlush) [PresShell.cpp:590f81062c5f : 4207 + 0xe] 03:44:07 INFO - eip = 0x638275e8 esp = 0x0012e464 ebp = 0x0012e488 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 50 xul.dll!nsIPresShell::FlushPendingNotifications(mozilla::ChangesToFlush) [nsIPresShell.h:590f81062c5f : 566 + 0x8] 03:44:07 INFO - eip = 0x637221c1 esp = 0x0012e490 ebp = 0x0012e494 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 51 xul.dll!nsRefreshDriver::Tick(__int64,mozilla::TimeStamp) [nsRefreshDriver.cpp:590f81062c5f : 1956 + 0x16] 03:44:07 INFO - eip = 0x63721e77 esp = 0x0012e49c ebp = 0x0012e700 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 52 xul.dll!mozilla::RefreshDriverTimer::TickDriver(nsRefreshDriver *,__int64,mozilla::TimeStamp) [nsRefreshDriver.cpp:590f81062c5f : 337 + 0x1a] 03:44:07 INFO - eip = 0x63721947 esp = 0x0012e708 ebp = 0x0012e738 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 53 xul.dll!mozilla::RefreshDriverTimer::TickRefreshDrivers(__int64,mozilla::TimeStamp,nsTArray<RefPtr<nsRefreshDriver> > &) [nsRefreshDriver.cpp:590f81062c5f : 307 + 0x1a] 03:44:07 INFO - eip = 0x637218d1 esp = 0x0012e740 ebp = 0x0012e77c 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 54 xul.dll!mozilla::RefreshDriverTimer::Tick(__int64,mozilla::TimeStamp) [nsRefreshDriver.cpp:590f81062c5f : 329 + 0x1e] 03:44:07 INFO - eip = 0x63721813 esp = 0x0012e784 ebp = 0x0012e7c0 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 55 xul.dll!mozilla::RefreshDriverTimer::Tick() [nsRefreshDriver.cpp:590f81062c5f : 291 + 0x1c] 03:44:07 INFO - eip = 0x64db8295 esp = 0x0012e7c8 ebp = 0x0012e818 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 56 xul.dll!nsTimerImpl::Fire(int) [nsTimerImpl.cpp:590f81062c5f : 512 + 0x9] 03:44:07 INFO - eip = 0x6394ce52 esp = 0x0012e820 ebp = 0x0012e8fc 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 57 xul.dll!nsTimerEvent::Run() [TimerThread.cpp:590f81062c5f : 286 + 0xb] 03:44:07 INFO - eip = 0x6394cfde esp = 0x0012e904 ebp = 0x0012e934 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 58 xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:590f81062c5f : 1039 + 0xa] 03:44:07 INFO - eip = 0x6376d99a esp = 0x0012e93c ebp = 0x0012eeac 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 59 xul.dll!mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) [MessagePump.cpp:590f81062c5f : 97 + 0x16] 03:44:07 INFO - eip = 0x636ca823 esp = 0x0012eeb4 ebp = 0x0012eee0 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 60 xul.dll!MessageLoop::RunHandler() [message_loop.cc:590f81062c5f : 319 + 0x8] 03:44:07 INFO - eip = 0x637554f0 esp = 0x0012eee8 ebp = 0x0012ef18 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 61 xul.dll!MessageLoop::Run() [message_loop.cc:590f81062c5f : 299 + 0x7] 03:44:07 INFO - eip = 0x637554ae esp = 0x0012ef20 ebp = 0x0012ef38 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 62 xul.dll!nsBaseAppShell::Run() [nsBaseAppShell.cpp:590f81062c5f : 158 + 0x14] 03:44:07 INFO - eip = 0x63bc7c61 esp = 0x0012ef40 ebp = 0x0012ef48 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 63 xul.dll!nsAppShell::Run() [nsAppShell.cpp:590f81062c5f : 230 + 0x8] 03:44:07 INFO - eip = 0x63bc7a06 esp = 0x0012ef50 ebp = 0x0012ef58 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 64 xul.dll!nsAppStartup::Run() [nsAppStartup.cpp:590f81062c5f : 288 + 0x6] 03:44:07 INFO - eip = 0x63bc79bb esp = 0x0012ef60 ebp = 0x0012ef6c 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 65 xul.dll!XREMain::XRE_mainRun() [nsAppRunner.cpp:590f81062c5f : 4701 + 0xc] 03:44:07 INFO - eip = 0x63bc85cc esp = 0x0012ef74 ebp = 0x0012f160 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 66 xul.dll!XREMain::XRE_main(int,char * * const,mozilla::BootstrapConfig const &) [nsAppRunner.cpp:590f81062c5f : 4865 + 0x7] 03:44:07 INFO - eip = 0x63bc65f6 esp = 0x0012f168 ebp = 0x0012f1ac 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 67 xul.dll!XRE_main(int,char * * const,mozilla::BootstrapConfig const &) [nsAppRunner.cpp:590f81062c5f : 4960 + 0xc] 03:44:07 INFO - eip = 0x63cf638b esp = 0x0012f1b4 ebp = 0x0012f308 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 68 xul.dll!mozilla::BootstrapImpl::XRE_main(int,char * * const,mozilla::BootstrapConfig const &) [Bootstrap.cpp:590f81062c5f : 45 + 0xe] 03:44:07 INFO - eip = 0x63cf634d esp = 0x0012f310 ebp = 0x0012f314 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 69 firefox.exe!do_main [nsBrowserApp.cpp:590f81062c5f : 236 + 0x32] 03:44:07 INFO - eip = 0x00aa1c4b esp = 0x0012f31c ebp = 0x0012f460 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 70 firefox.exe!wmain [nsWindowsWMain.cpp:590f81062c5f : 115 + 0x16e] 03:44:07 INFO - eip = 0x00aa133d esp = 0x0012f468 ebp = 0x0012f7b8 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 71 firefox.exe!__scrt_common_main_seh [exe_common.inl : 253 + 0x1d] 03:44:07 INFO - eip = 0x00aa6289 esp = 0x0012f7c0 ebp = 0x0012f800 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 72 kernel32.dll!BaseThreadInitThunk + 0x12 03:44:07 INFO - eip = 0x75c83c45 esp = 0x0012f808 ebp = 0x0012f80c 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 73 ntdll.dll!__RtlUserThreadStart + 0x27 03:44:07 INFO - eip = 0x771837f5 esp = 0x0012f814 ebp = 0x0012f84c 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 74 ntdll.dll!_RtlUserThreadStart + 0x1b 03:44:07 INFO - eip = 0x771837c8 esp = 0x0012f854 ebp = 0x0012f864 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - Thread 1 03:44:07 INFO - 0 ntdll.dll!KiFastSystemCallRet + 0x0 03:44:07 INFO - eip = 0x771670b4 esp = 0x0185f3a8 ebp = 0x0185f3d4 ebx = 0x000000f4 03:44:07 INFO - esi = 0x000000f0 edi = 0x75c7bcb4 eax = 0x0185f410 ecx = 0x0185f3d8 03:44:07 INFO - edx = 0x771670b4 efl = 0x00000246 03:44:07 INFO - Found by: given as instruction pointer in context 03:44:07 INFO - 1 ntdll.dll!NtRemoveIoCompletion + 0xc 03:44:07 INFO - eip = 0x771663a4 esp = 0x0185f3ac ebp = 0x0185f3d4 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 2 KERNELBASE.dll!GetQueuedCompletionStatus + 0x29 03:44:07 INFO - eip = 0x75460647 esp = 0x0185f3b0 ebp = 0x0185f3d4 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 3 firefox.exe!sandbox::BrokerServicesBase::TargetEventsThread(void *) [broker_services.cc:590f81062c5f : 200 + 0x30] 03:44:07 INFO - eip = 0x00aa9548 esp = 0x0185f3dc ebp = 0x0185f800 03:44:07 INFO - Found by: previous frame's frame pointer 03:44:07 INFO - 4 kernel32.dll!BaseThreadInitThunk + 0x12 03:44:07 INFO - eip = 0x75c83c45 esp = 0x0185f808 ebp = 0x0185f80c 03:44:07 INFO - Found by: previous frame's frame pointer 03:44:07 INFO - 5 mozglue.dll!patched_BaseThreadInitThunk [WindowsDllBlocklist.cpp:590f81062c5f : 824 + 0x17] 03:44:07 INFO - eip = 0x7080bae5 esp = 0x0185f814 ebp = 0x0185f820 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 6 ntdll.dll!__RtlUserThreadStart + 0x27 03:44:07 INFO - eip = 0x771837f5 esp = 0x0185f828 ebp = 0x0185f860 03:44:07 INFO - Found by: call frame info 03:44:07 INFO - 7 ntdll.dll!_RtlUserThreadStart + 0x1b 03:44:07 INFO - eip = 0x771837c8 esp = 0x0185f868 ebp = 0x0185f878 03:44:07 INFO - Found by: call frame info Gankro's bug 1400411 landed yesterday and he says a missed nullptr would hit more often. Maybe pgo + bug 1391196?
Flags: needinfo?(ryanvm)
|
Reporter | |
Comment 1•7 years ago
|
||
Backouts of bug 1391196 and bug 1400411 each passed tests (used a11y because it has a short run time of 6 minutes compared to >1 hour for many others) but this doesn't mean anything because the issue wasn't a permfail. bug 1391196 (didn't update the commit message of the try push): https://treeherder.mozilla.org/#/jobs?repo=try&revision=05a2b804e4cd04f75dc6298de661edf86f7d3f4c bug 1400411: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9ac89832124aeb6f2bc551abc698a685f176659f
| Comment hidden (Intermittent Failures Robot) |
|
||
Comment 3•7 years ago
|
||
That "error: [Errno 10061] No connection could be made because the target machine actively refused it" is likely masking a startup crash. If so, I consider it a bug that the underlying crash isn't being reported in the error. I'm assuming the stack trace reported by Talos G4 is this startup crash.
|
|
Reporter | |
Comment 4•7 years ago
|
||
Philor pointed out that this already hit on Saturday and likely before, but might have done so with a lower frequency: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=0b8b08421289dd2df01352b7594102acab8e3ddc RyanVM suggested the crash signature might be related to bug 1400382.
|
||
Comment 5•7 years ago
|
||
(In reply to Sebastian Hengst [:aryx][:archaeopteryx] (needinfo on intermittent or backout) from comment #4) > RyanVM suggested the crash signature might be related to bug 1400382. To be clear, that was before the information came to light about this starting last week already. That said, this has all the hallmarks of MSVC non-deterministic PGO miscompilation. We can maybe try to bisect this with Try pushes and a lot of build retriggers, but realistically, we probably need to have one of our stability/debugging gurus to jump in and help here. Per the IRC discussion, I believe that guru is dmajor? :)
Flags: needinfo?(ryanvm)
I managed to capture this in a Time Travel Debugging trace. \o/ I believe this code captures the problem: http://searchfox.org/mozilla-central/source/layout/generic/nsTextFrame.cpp#2013 xul!GetFontGroupForFrame+0xb [[inlined in xul!PropertyProvider::InitFontGroupAndFontMetrics+0x39 [z:\build\build\src\layout\generic\nstextframe.cpp @ 3232]]: 66125547 e812151a00 call xul!nsLayoutUtils::GetFontMetricsForFrame (662c6a5e) 6612554c 895dfc mov dword ptr [ebp-4],ebx 6612554f 8b08 mov ecx,dword ptr [eax] 66125551 8918 mov dword ptr [eax],ebx 66125553 8b713c mov esi,dword ptr [ecx+3Ch] We call GetFontMetricsForFrame, and the return value in eax is a stack location containing the address of the nsFontMetrics that we returned. (This indirection is probably due to the already_AddRefed/nsRefPtr.) Looks fine; we'll deref it two lines later to get the raw nsFontMetrics: 6612554f 8b08 mov ecx,dword ptr [eax] But that pointer is now null! The intervening instruction had clobbered it: 6612554c 895dfc mov dword ptr [ebp-4],ebx <-- ebp-4 is the same as the return value eax and ebx is null I suspect that the guilty function here is GetFontGroupForFrame for doing this clobber, but to increase our odds of fixing this on the first attempt, I would also long-shot suspect GetFontMetricsForFrame in case it's maybe signaling "I'm returning something interesting in ebx".
|
|
||
Comment 8•7 years ago
|
||
I'm going to land this on m-c with rs=me to stop the bleeding for now, but I think Jonathan should probably take a look too based on the investigation in comment 6.
Pushed by ryanvm@gmail.com: https://hg.mozilla.org/mozilla-central/rev/6100472d3aa8 De-optimize some font functions to work around an MSVC PGO crash. r+a=RyanVM
|
|
Assignee | |
Comment 10•7 years ago
|
||
(In reply to David Major [:dmajor] from comment #6) > 6612554f 8b08 mov ecx,dword ptr [eax] > But that pointer is now null! The intervening instruction had clobbered it: I wasn't clear about this: eax is still the same value as before, but the memory at that address (in other words, dword ptr [eax], aka dword ptr [ebp-4]) is what is now null.
|
|
Assignee | |
Comment 11•7 years ago
|
||
I spot-checked some other call sites that do GetFontMetricsForFrame()->GetThebesFontGroup(), and they don't have this clobber business going on. xul!mozilla::a11y::TextAttrsMgr::FontWeightTextAttr::GetFontWeight+0x14 [z:\build\build\src\accessible\base\textattrs.cpp @ 638]: 6806e74e e80b8325fe call xul!nsLayoutUtils::GetFontMetricsForFrame (662c6a5e) 6806e753 6a20 push 20h 6806e755 8b38 mov edi,dword ptr [eax] 6806e757 832000 and dword ptr [eax],0 6806e75a 8b4f3c mov ecx,dword ptr [edi+3Ch] Also I checked that GetFontMetricsForFrame pops the correct amount of words to return esp to the right place. So it's looking more and more like GetFontGroupForFrame is the culprit.
|
||
Comment 12•7 years ago
|
||
(In reply to Gregory Szorc [:gps] from comment #3) > That "error: [Errno 10061] No connection could be made because the target > machine actively refused it" is likely masking a startup crash. If so, I That's correct. I will fix the broken reporting by mochitests and reftests on bug 1403616. With the patch landed we will see the PROCESS_CRASH log output again.
|
|
Assignee | |
Comment 13•7 years ago
|
||
If enough time passes and nothing catches fire, I'd like to remove the pragma from GetFontMetrics, and limit the pragma for GetFontGroup to defined(_M_IX86).
| Comment hidden (Intermittent Failures Robot) |
|
||
Comment 15•7 years ago
|
||
this is looking to be fixed right now.
Whiteboard: [stockwell disable-recommended] → [stockwell fixed:product]
|
||
Comment 16•7 years ago
|
||
Pushed by dmajor@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/c6edffdecb57 followup: Limit the PGO workaround to just where it's needed.
|
||
Comment 17•7 years ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/c6edffdecb57
| Comment hidden (Intermittent Failures Robot) |
|
|
||
Comment 19•7 years ago
|
||
Interestingly, this is hitting Beta now too. https://treeherder.mozilla.org/logviewer.html#?job_id=134393345&repo=mozilla-beta
|
|
||
Comment 20•7 years ago
|
||
| bugherder uplift | ||
https://hg.mozilla.org/releases/mozilla-beta/rev/bd13ea56e690
status-firefox57:
--- → fixed
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
|
||
Comment 23•6 years ago
|
||
https://wiki.mozilla.org/Bug_Triage/Projects/Bug_Handling/Bug_Husbandry#Update_affecteds
status-firefox59:
--- → ?
|
|
||
Updated•6 years ago
|
You need to log in
before you can comment on or make changes to this bug.
Description
•