Windows 7 pgo: all tests using application either failing with connection errors, or normal behavior

RESOLVED FIXED

Status

()

defect
--
major
RESOLVED FIXED
2 years ago
8 months ago

People

(Reporter: aryx, Assigned: dmajor)

Tracking

Trunk
Points:
---

Firefox Tracking Flags

(firefox57 fixed)

Details

(Whiteboard: [stockwell fixed:product])

Attachments

(1 attachment)

Ryan, have you seen anything like this in your Try pushes and can you CC the relevant people? Thank you.

On Windows 7 pgo, today either all tests using the application are either failing with connection errors, or it's normal behavior (few, other failures). It only affects inbound as far as I noticed.

First push with failures:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=590f81062c5f6b6a5df5a0d992fde2d633527257&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-searchStr=windows+pgo

See pushes with failures:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=491e7fdae961010548bc12900fccee33c1b3fdf7&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-searchStr=windows%20pgo&tochange=3053445906837db96cce7fc059576c9b5e821a1b

Almost all of them have this in the log:

10:33:46     INFO -  Application command: Z:\task_1506421162\build\application\firefox\firefox.exe -marionette -foreground -profile c:\users\genericworker\appdata\local\temp\tmpkgc_vv.mozrunner
10:33:46     INFO -  runtests.py | Application pid: 1520
10:33:46     INFO -  TEST-INFO | started process GECKO(1520)
10:33:47     INFO -  GECKO(1520) | 1506422027510	Marionette	INFO	Enabled via --marionette
10:36:47     INFO -  Traceback (most recent call last):
10:36:47     INFO -    File "Z:\task_1506421162\build\tests\mochitest\runtests.py", line 2631, in doTests
10:36:47     INFO -      marionette_args=marionette_args,
10:36:47     INFO -    File "Z:\task_1506421162\build\tests\mochitest\runtests.py", line 2156, in runApp
10:36:47     INFO -      self.marionette.start_session()
10:36:47     INFO -    File "Z:\task_1506421162\build\venv\lib\site-packages\marionette_driver\decorators.py", line 28, in _
10:36:47     INFO -      m._handle_socket_failure()
10:36:47     INFO -    File "Z:\task_1506421162\build\venv\lib\site-packages\marionette_driver\decorators.py", line 23, in _
10:36:47     INFO -      return func(*args, **kwargs)
10:36:47     INFO -    File "Z:\task_1506421162\build\venv\lib\site-packages\marionette_driver\marionette.py", line 1228, in start_session
10:36:47     INFO -      self.protocol, _ = self.client.connect()
10:36:47     INFO -    File "Z:\task_1506421162\build\venv\lib\site-packages\marionette_driver\transport.py", line 223, in connect
10:36:47     INFO -      self.sock.connect((self.addr, self.port))
10:36:47     INFO -    File "c:\mozilla-build\python\Lib\socket.py", line 228, in meth
10:36:47     INFO -      return getattr(self._sock,name)(*args)
10:36:47     INFO -  error: [Errno 10061] No connection could be made because the target machine actively refused it


Talos g4 is different: https://treeherder.mozilla.org/logviewer.html#?job_id=133306285&repo=mozilla-inbound
03:43:52     INFO -  PID 3208 | [#16] 1080p.60fps.mp4_scale_2_startup  Cycles:12  Average:8.00  Median:8.02  stddev:0.09 (1.1%)  stddev-sans-first:0.08
03:43:52     INFO -  PID 3208 | Values: 8.1  8.1  8.0  8.0  8.0  7.9  8.1  8.0  8.0  8.1  7.9  7.9
03:43:52     INFO -  PID 3208 |
03:43:52     INFO -  PID 3208 | [#17] 1080p.60fps.mp4_scale_2_inclip  Cycles:12  Average:7.58  Median:7.65  stddev:0.10 (1.3%)  stddev-sans-first:0.10
03:43:52     INFO -  PID 3208 | Values: 7.6  7.7  7.7  7.5  7.4  7.5  7.7  7.5  7.7  7.7  7.5  7.5
03:43:52     INFO -  PID 3208 | -------- Summary: end --------
03:43:52     INFO -  PID 3208 |
03:43:52     INFO -  PID 3208 | [Parent 3208, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
03:43:53     INFO -  PID 3208 | [GPU 3380, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
03:43:54     INFO -  TEST-INFO | 3208: exit 0
03:43:54     INFO -  mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/U3yEwwBwQR-k-glN7QeyvQ/artifacts/public/build/target.crashreporter-symbols.zip
03:44:01     INFO -  mozcrash Copy/paste: C:\slave\test\build\win32-minidump_stackwalk.exe c:\users\cltbld\appdata\local\temp\tmpygo7g2\profile\minidumps\3167a0c6-3353-4ae1-90a9-1e906b754b02.dmp c:\users\cltbld\appdata\local\temp\tmpvjovs4
03:44:07     INFO -  mozcrash Saved minidump as C:\slave\test\build\blobber_upload_dir\3167a0c6-3353-4ae1-90a9-1e906b754b02.dmp
03:44:07     INFO -  mozcrash Saved app info as C:\slave\test\build\blobber_upload_dir\3167a0c6-3353-4ae1-90a9-1e906b754b02.extra
03:44:07     INFO -  PROCESS-CRASH | basic_compositor_video | application crashed [@ PropertyProvider::InitFontGroupAndFontMetrics()]
03:44:07     INFO -  Crash dump filename: c:\users\cltbld\appdata\local\temp\tmpygo7g2\profile\minidumps\3167a0c6-3353-4ae1-90a9-1e906b754b02.dmp
03:44:07     INFO -  Operating system: Windows NT
03:44:07     INFO -                    6.1.7601 Service Pack 1
03:44:07     INFO -  CPU: x86
03:44:07     INFO -       GenuineIntel family 6 model 30 stepping 5
03:44:07     INFO -       8 CPUs
03:44:07     INFO -  GPU: UNKNOWN
03:44:07     INFO -  Crash reason:  EXCEPTION_ACCESS_VIOLATION_READ
03:44:07     INFO -  Crash address: 0x3c
03:44:07     INFO -  Process uptime: 5 seconds
03:44:07     INFO -  Thread 0 (crashed)
03:44:07     INFO -   0  xul.dll!PropertyProvider::InitFontGroupAndFontMetrics() [nsTextFrame.cpp:590f81062c5f : 3232 + 0x1e]
03:44:07     INFO -      eip = 0x638219d4   esp = 0x0012c370   ebp = 0x0012c380   ebx = 0x00000000
03:44:07     INFO -      esi = 0x0012c4b0   edi = 0x0012c4b0   eax = 0x0012c37c   ecx = 0x00000000
03:44:07     INFO -      edx = 0x20afd9a0   efl = 0x00010202
03:44:07     INFO -      Found by: given as instruction pointer in context
03:44:07     INFO -   1  xul.dll!PropertyProvider::GetFontMetrics() [nsTextFrame.cpp:590f81062c5f : 3217 + 0x5]
03:44:07     INFO -      eip = 0x6381fd43   esp = 0x0012c388   ebp = 0x0012c53c
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -   2  xul.dll!nsTextFrame::ReflowText(nsLineLayout &,int,mozilla::gfx::DrawTarget *,mozilla::ReflowOutput &,nsReflowStatus &) [nsTextFrame.cpp:590f81062c5f : 9719 + 0xc]
03:44:07     INFO -      eip = 0x63912eb6   esp = 0x0012c390   ebp = 0x0012c53c
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -   3  xul.dll!nsLineLayout::ReflowFrame(nsIFrame *,nsReflowStatus &,mozilla::ReflowOutput *,bool &) [nsLineLayout.cpp:590f81062c5f : 923 + 0x1f]
03:44:07     INFO -      eip = 0x63905be2   esp = 0x0012c544   ebp = 0x0012c6e0
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -   4  xul.dll!nsLineLayout::RelativePositionFrames(nsLineLayout::PerSpanData *,nsOverflowAreas &) [nsLineLayout.cpp:590f81062c5f : 3372 + 0x2b]
03:44:07     INFO -      eip = 0x639105d4   esp = 0x0012c594   ebp = 0x0012c7b8
03:44:07     INFO -      Found by: stack scanning
03:44:07     INFO -   5  xul.dll!nsIFrame::HasPerspective(mozilla::EffectSet *) [nsIFrame.h:590f81062c5f : 1844 + 0x12]
03:44:07     INFO -      eip = 0x63910000   esp = 0x0012c80c   ebp = 0x0012c62c
03:44:07     INFO -      Found by: call frame info with scanning
03:44:07     INFO -   6  xul.dll!nsBlockFrame::ReflowLine(mozilla::BlockReflowInput &,nsLineList_iterator,bool *) [nsBlockFrame.cpp:590f81062c5f : 2873 + 0xb]
03:44:07     INFO -      eip = 0x639135d2   esp = 0x0012c87c   ebp = 0x0012c8b4
03:44:07     INFO -      Found by: call frame info with scanning
03:44:07     INFO -   7  xul.dll!nsBlockFrame::ReflowDirtyLines(mozilla::BlockReflowInput &) [nsBlockFrame.cpp:590f81062c5f : 2409 + 0xb]
03:44:07     INFO -      eip = 0x6391210c   esp = 0x0012c8bc   ebp = 0x0012c984
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -   8  xul.dll!nsBlockFrame::Reflow(nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,nsReflowStatus &) [nsBlockFrame.cpp:590f81062c5f : 1235 + 0xd]
03:44:07     INFO -      eip = 0x63910987   esp = 0x0012c98c   ebp = 0x0012cc48
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -   9  xul.dll!nsContainerFrame::ReflowChild(nsIFrame *,nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,mozilla::WritingMode const &,mozilla::LogicalPoint const &,nsSize const &,unsigned int,nsReflowStatus &,nsOverflowContinuationTracker *) [nsContainerFrame.cpp:590f81062c5f : 932 + 0x17]
03:44:07     INFO -      eip = 0x638fd217   esp = 0x0012cc50   ebp = 0x0012cc70
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  10  xul.dll!nsHTMLScrollFrame::ReflowScrolledFrame(mozilla::ScrollReflowInput *,bool,bool,mozilla::ReflowOutput *,bool) [nsGfxScrollFrame.cpp:590f81062c5f : 548 + 0x2e]
03:44:07     INFO -      eip = 0x638faff2   esp = 0x0012cc78   ebp = 0x0012ce04
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  11  xul.dll!nsHTMLScrollFrame::ReflowContents(mozilla::ScrollReflowInput *,mozilla::ReflowOutput const &) [nsGfxScrollFrame.cpp:590f81062c5f : 660 + 0x29]
03:44:07     INFO -      eip = 0x638fac80   esp = 0x0012ce0c   ebp = 0x0012ceb0
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  12  xul.dll!nsHTMLScrollFrame::Reflow(nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,nsReflowStatus &) [nsGfxScrollFrame.cpp:590f81062c5f : 1037 + 0x13]
03:44:07     INFO -      eip = 0x638fa018   esp = 0x0012ceb8   ebp = 0x0012cfd0
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  13  xul.dll!nsContainerFrame::ReflowChild(nsIFrame *,nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,int,int,unsigned int,nsReflowStatus &,nsOverflowContinuationTracker *) [nsContainerFrame.cpp:590f81062c5f : 976 + 0x17]
03:44:07     INFO -      eip = 0x639fa86d   esp = 0x0012cfd8   ebp = 0x0012cffc
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  14  xul.dll!nsTextControlFrame::ReflowTextControlChild(nsIFrame *,nsPresContext *,mozilla::ReflowInput const &,nsReflowStatus &,mozilla::ReflowOutput &) [nsTextControlFrame.cpp:590f81062c5f : 690 + 0x1e]
03:44:07     INFO -      eip = 0x639fa7e0   esp = 0x0012d004   ebp = 0x0012d174
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  15  xul.dll!nsTextControlFrame::Reflow(nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,nsReflowStatus &) [nsTextControlFrame.cpp:590f81062c5f : 649 + 0xe]
03:44:07     INFO -      eip = 0x639fa6a7   esp = 0x0012d17c   ebp = 0x0012d1d0
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  16  xul.dll!nsFrame::BoxReflow(nsBoxLayoutState &,nsPresContext *,mozilla::ReflowOutput &,gfxContext *,int,int,int,int,bool) [nsFrame.cpp:590f81062c5f : 10160 + 0x19]
03:44:07     INFO -      eip = 0x638fde5b   esp = 0x0012d1d8   ebp = 0x0012d3d4
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  17  xul.dll!nsFrame::RefreshSizeCache(nsBoxLayoutState &) [nsFrame.cpp:590f81062c5f : 9707 + 0x27]
03:44:07     INFO -      eip = 0x63a87830   esp = 0x0012d3dc   ebp = 0x0012d4a0
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  18  xul.dll!nsFrame::GetXULPrefSize(nsBoxLayoutState &) [nsFrame.cpp:590f81062c5f : 9785 + 0xa]
03:44:07     INFO -      eip = 0x63a879d8   esp = 0x0012d4a8   ebp = 0x0012d4c0
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  19  xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12]
03:44:07     INFO -      eip = 0x6390a1c8   esp = 0x0012d4c8   ebp = 0x0012d514
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  20  xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb]
03:44:07     INFO -      eip = 0x6390c7e2   esp = 0x0012d51c   ebp = 0x0012d558
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  21  xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12]
03:44:07     INFO -      eip = 0x6390a1c8   esp = 0x0012d560   ebp = 0x0012d5ac
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  22  xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb]
03:44:07     INFO -      eip = 0x6390c7e2   esp = 0x0012d5b4   ebp = 0x0012d5f0
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  23  xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12]
03:44:07     INFO -      eip = 0x6390a1c8   esp = 0x0012d5f8   ebp = 0x0012d644
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  24  xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb]
03:44:07     INFO -      eip = 0x6390c7e2   esp = 0x0012d64c   ebp = 0x0012d688
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  25  xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12]
03:44:07     INFO -      eip = 0x6390a1c8   esp = 0x0012d690   ebp = 0x0012d6dc
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  26  xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb]
03:44:07     INFO -      eip = 0x6390c7e2   esp = 0x0012d6e4   ebp = 0x0012d720
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  27  xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12]
03:44:07     INFO -      eip = 0x6390a1c8   esp = 0x0012d728   ebp = 0x0012d774
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  28  xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb]
03:44:07     INFO -      eip = 0x6390c7e2   esp = 0x0012d77c   ebp = 0x0012d7b8
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  29  xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12]
03:44:07     INFO -      eip = 0x6390a1c8   esp = 0x0012d7c0   ebp = 0x0012d80c
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  30  xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb]
03:44:07     INFO -      eip = 0x6390c7e2   esp = 0x0012d814   ebp = 0x0012d850
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  31  xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12]
03:44:07     INFO -      eip = 0x6390a1c8   esp = 0x0012d858   ebp = 0x0012d8a4
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  32  xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb]
03:44:07     INFO -      eip = 0x6390c7e2   esp = 0x0012d8ac   ebp = 0x0012d8e8
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  33  xul.dll!nsSprocketLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 1324 + 0x12]
03:44:07     INFO -      eip = 0x6390a1c8   esp = 0x0012d8f0   ebp = 0x0012d93c
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  34  xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb]
03:44:07     INFO -      eip = 0x6390c7e2   esp = 0x0012d944   ebp = 0x0012d980
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  35  xul.dll!nsStackLayout::GetXULPrefSize(nsIFrame *,nsBoxLayoutState &) [nsStackLayout.cpp:590f81062c5f : 70 + 0x12]
03:44:07     INFO -      eip = 0x638fc81c   esp = 0x0012d988   ebp = 0x0012d9bc
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  36  xul.dll!nsBoxFrame::GetXULPrefSize(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 774 + 0xb]
03:44:07     INFO -      eip = 0x6390c7e2   esp = 0x0012d9c4   ebp = 0x0012da00
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  37  xul.dll!nsSprocketLayout::PopulateBoxSizes(nsIFrame *,nsBoxLayoutState &,nsBoxSize * &,int &,int &,int &) [nsSprocketLayout.cpp:590f81062c5f : 746 + 0x12]
03:44:07     INFO -      eip = 0x6390aec8   esp = 0x0012da08   ebp = 0x0012da98
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  38  xul.dll!nsSprocketLayout::XULLayout(nsIFrame *,nsBoxLayoutState &) [nsSprocketLayout.cpp:590f81062c5f : 216 + 0x51]
03:44:07     INFO -      eip = 0x6390a450   esp = 0x0012daa0   ebp = 0x0012dbdc
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  39  xul.dll!nsBoxFrame::DoXULLayout(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 917 + 0x8]
03:44:07     INFO -      eip = 0x63908484   esp = 0x0012dbe4   ebp = 0x0012dd50
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  40  xul.dll!nsIFrame::XULLayout(nsBoxLayoutState &) [nsBox.cpp:590f81062c5f : 501 + 0xc]
03:44:07     INFO -      eip = 0x638fcd16   esp = 0x0012dd58   ebp = 0x0012dd64
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  41  xul.dll!nsStackLayout::XULLayout(nsIFrame *,nsBoxLayoutState &) [nsStackLayout.cpp:590f81062c5f : 369 + 0x8]
03:44:07     INFO -      eip = 0x638fcbde   esp = 0x0012dd6c   ebp = 0x0012de44
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  42  xul.dll!nsBoxFrame::DoXULLayout(nsBoxLayoutState &) [nsBoxFrame.cpp:590f81062c5f : 917 + 0x8]
03:44:07     INFO -      eip = 0x63908484   esp = 0x0012de4c   ebp = 0x0012dfb8
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  43  xul.dll!nsIFrame::XULLayout(nsBoxLayoutState &) [nsBox.cpp:590f81062c5f : 501 + 0xc]
03:44:07     INFO -      eip = 0x638fcd16   esp = 0x0012dfc0   ebp = 0x0012dfcc
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  44  xul.dll!nsBoxFrame::Reflow(nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,nsReflowStatus &) [nsBoxFrame.cpp:590f81062c5f : 713 + 0xc]
03:44:07     INFO -      eip = 0x638f8cfa   esp = 0x0012dfd4   ebp = 0x0012e070
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  45  xul.dll!nsContainerFrame::ReflowChild(nsIFrame *,nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,int,int,unsigned int,nsReflowStatus &,nsOverflowContinuationTracker *) [nsContainerFrame.cpp:590f81062c5f : 976 + 0x17]
03:44:07     INFO -      eip = 0x639fa86d   esp = 0x0012e078   ebp = 0x0012e09c
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  46  xul.dll!mozilla::ViewportFrame::Reflow(nsPresContext *,mozilla::ReflowOutput &,mozilla::ReflowInput const &,nsReflowStatus &) [ViewportFrame.cpp:590f81062c5f : 330 + 0x24]
03:44:07     INFO -      eip = 0x638fc558   esp = 0x0012e0a4   ebp = 0x0012e218
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  47  xul.dll!mozilla::PresShell::DoReflow(nsIFrame *,bool) [PresShell.cpp:590f81062c5f : 9380 + 0x1c]
03:44:07     INFO -      eip = 0x638269bd   esp = 0x0012e220   ebp = 0x0012e3f0
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  48  xul.dll!mozilla::PresShell::ProcessReflowCommands(bool) [PresShell.cpp:590f81062c5f : 9553 + 0xb]
03:44:07     INFO -      eip = 0x638effde   esp = 0x0012e3f8   ebp = 0x0012e45c
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  49  xul.dll!mozilla::PresShell::DoFlushPendingNotifications(mozilla::ChangesToFlush) [PresShell.cpp:590f81062c5f : 4207 + 0xe]
03:44:07     INFO -      eip = 0x638275e8   esp = 0x0012e464   ebp = 0x0012e488
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  50  xul.dll!nsIPresShell::FlushPendingNotifications(mozilla::ChangesToFlush) [nsIPresShell.h:590f81062c5f : 566 + 0x8]
03:44:07     INFO -      eip = 0x637221c1   esp = 0x0012e490   ebp = 0x0012e494
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  51  xul.dll!nsRefreshDriver::Tick(__int64,mozilla::TimeStamp) [nsRefreshDriver.cpp:590f81062c5f : 1956 + 0x16]
03:44:07     INFO -      eip = 0x63721e77   esp = 0x0012e49c   ebp = 0x0012e700
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  52  xul.dll!mozilla::RefreshDriverTimer::TickDriver(nsRefreshDriver *,__int64,mozilla::TimeStamp) [nsRefreshDriver.cpp:590f81062c5f : 337 + 0x1a]
03:44:07     INFO -      eip = 0x63721947   esp = 0x0012e708   ebp = 0x0012e738
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  53  xul.dll!mozilla::RefreshDriverTimer::TickRefreshDrivers(__int64,mozilla::TimeStamp,nsTArray<RefPtr<nsRefreshDriver> > &) [nsRefreshDriver.cpp:590f81062c5f : 307 + 0x1a]
03:44:07     INFO -      eip = 0x637218d1   esp = 0x0012e740   ebp = 0x0012e77c
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  54  xul.dll!mozilla::RefreshDriverTimer::Tick(__int64,mozilla::TimeStamp) [nsRefreshDriver.cpp:590f81062c5f : 329 + 0x1e]
03:44:07     INFO -      eip = 0x63721813   esp = 0x0012e784   ebp = 0x0012e7c0
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  55  xul.dll!mozilla::RefreshDriverTimer::Tick() [nsRefreshDriver.cpp:590f81062c5f : 291 + 0x1c]
03:44:07     INFO -      eip = 0x64db8295   esp = 0x0012e7c8   ebp = 0x0012e818
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  56  xul.dll!nsTimerImpl::Fire(int) [nsTimerImpl.cpp:590f81062c5f : 512 + 0x9]
03:44:07     INFO -      eip = 0x6394ce52   esp = 0x0012e820   ebp = 0x0012e8fc
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  57  xul.dll!nsTimerEvent::Run() [TimerThread.cpp:590f81062c5f : 286 + 0xb]
03:44:07     INFO -      eip = 0x6394cfde   esp = 0x0012e904   ebp = 0x0012e934
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  58  xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:590f81062c5f : 1039 + 0xa]
03:44:07     INFO -      eip = 0x6376d99a   esp = 0x0012e93c   ebp = 0x0012eeac
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  59  xul.dll!mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) [MessagePump.cpp:590f81062c5f : 97 + 0x16]
03:44:07     INFO -      eip = 0x636ca823   esp = 0x0012eeb4   ebp = 0x0012eee0
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  60  xul.dll!MessageLoop::RunHandler() [message_loop.cc:590f81062c5f : 319 + 0x8]
03:44:07     INFO -      eip = 0x637554f0   esp = 0x0012eee8   ebp = 0x0012ef18
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  61  xul.dll!MessageLoop::Run() [message_loop.cc:590f81062c5f : 299 + 0x7]
03:44:07     INFO -      eip = 0x637554ae   esp = 0x0012ef20   ebp = 0x0012ef38
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  62  xul.dll!nsBaseAppShell::Run() [nsBaseAppShell.cpp:590f81062c5f : 158 + 0x14]
03:44:07     INFO -      eip = 0x63bc7c61   esp = 0x0012ef40   ebp = 0x0012ef48
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  63  xul.dll!nsAppShell::Run() [nsAppShell.cpp:590f81062c5f : 230 + 0x8]
03:44:07     INFO -      eip = 0x63bc7a06   esp = 0x0012ef50   ebp = 0x0012ef58
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  64  xul.dll!nsAppStartup::Run() [nsAppStartup.cpp:590f81062c5f : 288 + 0x6]
03:44:07     INFO -      eip = 0x63bc79bb   esp = 0x0012ef60   ebp = 0x0012ef6c
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  65  xul.dll!XREMain::XRE_mainRun() [nsAppRunner.cpp:590f81062c5f : 4701 + 0xc]
03:44:07     INFO -      eip = 0x63bc85cc   esp = 0x0012ef74   ebp = 0x0012f160
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  66  xul.dll!XREMain::XRE_main(int,char * * const,mozilla::BootstrapConfig const &) [nsAppRunner.cpp:590f81062c5f : 4865 + 0x7]
03:44:07     INFO -      eip = 0x63bc65f6   esp = 0x0012f168   ebp = 0x0012f1ac
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  67  xul.dll!XRE_main(int,char * * const,mozilla::BootstrapConfig const &) [nsAppRunner.cpp:590f81062c5f : 4960 + 0xc]
03:44:07     INFO -      eip = 0x63cf638b   esp = 0x0012f1b4   ebp = 0x0012f308
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  68  xul.dll!mozilla::BootstrapImpl::XRE_main(int,char * * const,mozilla::BootstrapConfig const &) [Bootstrap.cpp:590f81062c5f : 45 + 0xe]
03:44:07     INFO -      eip = 0x63cf634d   esp = 0x0012f310   ebp = 0x0012f314
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  69  firefox.exe!do_main [nsBrowserApp.cpp:590f81062c5f : 236 + 0x32]
03:44:07     INFO -      eip = 0x00aa1c4b   esp = 0x0012f31c   ebp = 0x0012f460
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  70  firefox.exe!wmain [nsWindowsWMain.cpp:590f81062c5f : 115 + 0x16e]
03:44:07     INFO -      eip = 0x00aa133d   esp = 0x0012f468   ebp = 0x0012f7b8
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  71  firefox.exe!__scrt_common_main_seh [exe_common.inl : 253 + 0x1d]
03:44:07     INFO -      eip = 0x00aa6289   esp = 0x0012f7c0   ebp = 0x0012f800
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  72  kernel32.dll!BaseThreadInitThunk + 0x12
03:44:07     INFO -      eip = 0x75c83c45   esp = 0x0012f808   ebp = 0x0012f80c
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  73  ntdll.dll!__RtlUserThreadStart + 0x27
03:44:07     INFO -      eip = 0x771837f5   esp = 0x0012f814   ebp = 0x0012f84c
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  74  ntdll.dll!_RtlUserThreadStart + 0x1b
03:44:07     INFO -      eip = 0x771837c8   esp = 0x0012f854   ebp = 0x0012f864
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -  Thread 1
03:44:07     INFO -   0  ntdll.dll!KiFastSystemCallRet + 0x0
03:44:07     INFO -      eip = 0x771670b4   esp = 0x0185f3a8   ebp = 0x0185f3d4   ebx = 0x000000f4
03:44:07     INFO -      esi = 0x000000f0   edi = 0x75c7bcb4   eax = 0x0185f410   ecx = 0x0185f3d8
03:44:07     INFO -      edx = 0x771670b4   efl = 0x00000246
03:44:07     INFO -      Found by: given as instruction pointer in context
03:44:07     INFO -   1  ntdll.dll!NtRemoveIoCompletion + 0xc
03:44:07     INFO -      eip = 0x771663a4   esp = 0x0185f3ac   ebp = 0x0185f3d4
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -   2  KERNELBASE.dll!GetQueuedCompletionStatus + 0x29
03:44:07     INFO -      eip = 0x75460647   esp = 0x0185f3b0   ebp = 0x0185f3d4
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -   3  firefox.exe!sandbox::BrokerServicesBase::TargetEventsThread(void *) [broker_services.cc:590f81062c5f : 200 + 0x30]
03:44:07     INFO -      eip = 0x00aa9548   esp = 0x0185f3dc   ebp = 0x0185f800
03:44:07     INFO -      Found by: previous frame's frame pointer
03:44:07     INFO -   4  kernel32.dll!BaseThreadInitThunk + 0x12
03:44:07     INFO -      eip = 0x75c83c45   esp = 0x0185f808   ebp = 0x0185f80c
03:44:07     INFO -      Found by: previous frame's frame pointer
03:44:07     INFO -   5  mozglue.dll!patched_BaseThreadInitThunk [WindowsDllBlocklist.cpp:590f81062c5f : 824 + 0x17]
03:44:07     INFO -      eip = 0x7080bae5   esp = 0x0185f814   ebp = 0x0185f820
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -   6  ntdll.dll!__RtlUserThreadStart + 0x27
03:44:07     INFO -      eip = 0x771837f5   esp = 0x0185f828   ebp = 0x0185f860
03:44:07     INFO -      Found by: call frame info
03:44:07     INFO -   7  ntdll.dll!_RtlUserThreadStart + 0x1b
03:44:07     INFO -      eip = 0x771837c8   esp = 0x0185f868   ebp = 0x0185f878
03:44:07     INFO -      Found by: call frame info

Gankro's bug 1400411 landed yesterday and he says a missed nullptr would hit more often.

Maybe pgo + bug 1391196?
Flags: needinfo?(ryanvm)
Backouts of bug 1391196 and bug 1400411 each passed tests (used a11y because it has a short run time of 6 minutes compared to >1 hour for many others) but this doesn't mean anything because the issue wasn't a permfail.

bug 1391196 (didn't update the commit message of the try push): https://treeherder.mozilla.org/#/jobs?repo=try&revision=05a2b804e4cd04f75dc6298de661edf86f7d3f4c

bug 1400411: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9ac89832124aeb6f2bc551abc698a685f176659f
Comment hidden (Intermittent Failures Robot)
That "error: [Errno 10061] No connection could be made because the target machine actively refused it" is likely masking a startup crash. If so, I consider it a bug that the underlying crash isn't being reported in the error. I'm assuming the stack trace reported by Talos G4 is this startup crash.
Blocks: 1352671
Philor pointed out that this already hit on Saturday and likely before, but might have done so with a lower frequency: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=0b8b08421289dd2df01352b7594102acab8e3ddc

RyanVM suggested the crash signature might be related to bug 1400382.
(In reply to Sebastian Hengst [:aryx][:archaeopteryx] (needinfo on intermittent or backout) from comment #4)
> RyanVM suggested the crash signature might be related to bug 1400382.

To be clear, that was before the information came to light about this starting last week already. That said, this has all the hallmarks of MSVC non-deterministic PGO miscompilation. We can maybe try to bisect this with Try pushes and a lot of build retriggers, but realistically, we probably need to have one of our stability/debugging gurus to jump in and help here.

Per the IRC discussion, I believe that guru is dmajor? :)
Flags: needinfo?(ryanvm)
Assignee

Comment 6

2 years ago
I managed to capture this in a Time Travel Debugging trace. \o/

I believe this code captures the problem:
http://searchfox.org/mozilla-central/source/layout/generic/nsTextFrame.cpp#2013

xul!GetFontGroupForFrame+0xb [[inlined in xul!PropertyProvider::InitFontGroupAndFontMetrics+0x39 [z:\build\build\src\layout\generic\nstextframe.cpp @ 3232]]:
66125547 e812151a00      call    xul!nsLayoutUtils::GetFontMetricsForFrame (662c6a5e)
6612554c 895dfc          mov     dword ptr [ebp-4],ebx
6612554f 8b08            mov     ecx,dword ptr [eax]
66125551 8918            mov     dword ptr [eax],ebx
66125553 8b713c          mov     esi,dword ptr [ecx+3Ch]

We call GetFontMetricsForFrame, and the return value in eax is a stack location containing the address of the nsFontMetrics that we returned. (This indirection is probably due to the already_AddRefed/nsRefPtr.) Looks fine; we'll deref it two lines later to get the raw nsFontMetrics:

6612554f 8b08            mov     ecx,dword ptr [eax]

But that pointer is now null! The intervening instruction had clobbered it:

6612554c 895dfc          mov     dword ptr [ebp-4],ebx <-- ebp-4 is the same as the return value eax
                                                           and ebx is null

I suspect that the guilty function here is GetFontGroupForFrame for doing this clobber, but to increase our odds of fixing this on the first attempt, I would also long-shot suspect GetFontMetricsForFrame in case it's maybe signaling "I'm returning something interesting in ebx".
Assignee

Comment 7

2 years ago
Assignee: nobody → dmajor
I'm going to land this on m-c with rs=me to stop the bleeding for now, but I think Jonathan should probably take a look too based on the investigation in comment 6.
Component: Build Config → Layout: Text
Flags: needinfo?(jfkthame)
Keywords: leave-open

Comment 9

2 years ago
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/mozilla-central/rev/6100472d3aa8
De-optimize some font functions to work around an MSVC PGO crash. r+a=RyanVM
Assignee

Comment 10

2 years ago
(In reply to David Major [:dmajor] from comment #6)
> 6612554f 8b08            mov     ecx,dword ptr [eax]
> But that pointer is now null! The intervening instruction had clobbered it:

I wasn't clear about this: eax is still the same value as before, but the memory at that address (in other words, dword ptr [eax], aka dword ptr [ebp-4]) is what is now null.
Assignee

Comment 11

2 years ago
I spot-checked some other call sites that do GetFontMetricsForFrame()->GetThebesFontGroup(), and they don't have this clobber business going on.

xul!mozilla::a11y::TextAttrsMgr::FontWeightTextAttr::GetFontWeight+0x14 [z:\build\build\src\accessible\base\textattrs.cpp @ 638]:
6806e74e e80b8325fe      call    xul!nsLayoutUtils::GetFontMetricsForFrame (662c6a5e)
6806e753 6a20            push    20h
6806e755 8b38            mov     edi,dword ptr [eax]
6806e757 832000          and     dword ptr [eax],0
6806e75a 8b4f3c          mov     ecx,dword ptr [edi+3Ch]

Also I checked that GetFontMetricsForFrame pops the correct amount of words to return esp to the right place. So it's looking more and more like GetFontGroupForFrame is the culprit.
(In reply to Gregory Szorc [:gps] from comment #3)
> That "error: [Errno 10061] No connection could be made because the target
> machine actively refused it" is likely masking a startup crash. If so, I

That's correct. I will fix the broken reporting by mochitests and reftests on bug 1403616. With the patch landed we will see the PROCESS_CRASH log output again.
No longer blocks: 1352671
Assignee

Comment 13

2 years ago
If enough time passes and nothing catches fire, I'd like to remove the pragma from GetFontMetrics, and limit the pragma for GetFontGroup to defined(_M_IX86).
Comment hidden (Intermittent Failures Robot)
this is looking to be fixed right now.
Whiteboard: [stockwell disable-recommended] → [stockwell fixed:product]

Comment 16

2 years ago
Pushed by dmajor@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c6edffdecb57
followup: Limit the PGO workaround to just where it's needed.
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
Assignee

Updated

Last year
Status: NEW → RESOLVED
Closed: Last year
Keywords: leave-open
Resolution: --- → FIXED
Flags: needinfo?(jfkthame)
You need to log in before you can comment on or make changes to this bug.