Intermittent mozmake.EXE: *** [check] Error 1 after testsuite-targets.mk:269: recipe for target 'check' failed

RESOLVED FIXED in Firefox 56

Status

()

Core
Build Config
RESOLVED FIXED
4 months ago
a month ago

People

(Reporter: aryx, Assigned: ted)

Tracking

({intermittent-failure})

unspecified
mozilla57
intermittent-failure
Points:
---

Firefox Tracking Flags

(firefox56 fixed, firefox57 fixed)

Details

(Whiteboard: [stockwell fixed:other])

MozReview Requests

()

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

https://treeherder.mozilla.org/logviewer.html#?job_id=106093705&repo=autoland

16:42:18     INFO - TEST-PASS | z:\task_1497110818\build\src\python\mozbuild\mozbuild\test\configure\test_toolchain_configure.py | WindowsToolchainTest.test_cannot_cross
16:42:18     INFO - TEST-PASS | z:\task_1497110818\build\src\python\mozbuild\mozbuild\test\configure\test_toolchain_configure.py | WindowsToolchainTest.test_clang
16:42:18     INFO - TEST-PASS | z:\task_1497110818\build\src\python\mozbuild\mozbuild\test\configure\test_toolchain_configure.py | WindowsToolchainTest.test_clang_cl
16:42:18     INFO - TEST-PASS | z:\task_1497110818\build\src\python\mozbuild\mozbuild\test\configure\test_toolchain_configure.py | WindowsToolchainTest.test_gcc
16:42:18     INFO - TEST-PASS | z:\task_1497110818\build\src\python\mozbuild\mozbuild\test\configure\test_toolchain_configure.py | WindowsToolchainTest.test_msvc
16:42:18     INFO - TEST-PASS | z:\task_1497110818\build\src\python\mozbuild\mozbuild\test\configure\test_toolchain_configure.py | WindowsToolchainTest.test_overridden_unsupported_clang
16:42:18     INFO - TEST-PASS | z:\task_1497110818\build\src\python\mozbuild\mozbuild\test\configure\test_toolchain_configure.py | WindowsToolchainTest.test_overridden_unsupported_gcc
16:42:18     INFO - TEST-PASS | z:\task_1497110818\build\src\python\mozbuild\mozbuild\test\configure\test_toolchain_configure.py | WindowsToolchainTest.test_unsupported_msvc
16:42:18     INFO - Return code from mach python-test: 1
16:42:18     INFO - 1
16:42:18     INFO - z:/task_1497110818/build/src/testing/testsuite-targets.mk:269: recipe for target 'check' failed
16:42:18     INFO - mozmake.EXE: *** [check] Error 1

Comment 1

4 months ago
3 failures in 892 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 2
* autoland: 1

Platform breakdown:
* windows8-64: 1
* windows2012-64: 1
* windows2012-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-06-19&endday=2017-06-25&tree=all

Comment 2

4 months ago
2 failures in 718 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1
* autoland: 1

Platform breakdown:
* windows2012-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-06-26&endday=2017-07-02&tree=all

Comment 3

3 months ago
4 failures in 656 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 2
* autoland: 2

Platform breakdown:
* windows2012-32: 3
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-07-03&endday=2017-07-09&tree=all

Comment 4

3 months ago
2 failures in 720 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 2

Platform breakdown:
* windows2012-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-07-10&endday=2017-07-16&tree=all

Comment 5

3 months ago
1 failures in 822 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows2012-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-07-17&endday=2017-07-23&tree=all

Comment 6

3 months ago
6 failures in 1008 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 5
* try: 1

Platform breakdown:
* windows2012-32: 5
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-07-24&endday=2017-07-30&tree=all

Comment 7

2 months ago
9 failures in 888 pushes (0.01 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 6
* autoland: 3

Platform breakdown:
* windows2012-32: 7
* windows2012-64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-07-31&endday=2017-08-06&tree=all

Comment 8

2 months ago
2 failures in 901 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 2

Platform breakdown:
* windows2012-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-08-07&endday=2017-08-13&tree=all

Comment 9

2 months ago
22 failures in 160 pushes (0.138 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 11
* autoland: 11

Platform breakdown:
* windows2012-32: 11
* windows2012-64: 10
* windows2012-32-noopt: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-08-18&endday=2017-08-18&tree=all

Comment 10

2 months ago
32 failures in 949 pushes (0.034 failures/push) were associated with this bug in the last 7 days. 

This is the #50 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 14
* mozilla-inbound: 12
* try: 3
* mozilla-central: 2
* mozilla-beta: 1

Platform breakdown:
* windows2012-32: 16
* windows2012-64: 14
* windows2012-32-noopt: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-08-14&endday=2017-08-20&tree=all
this picked up in frequency on August 18th on win32/64 opt/debug.

:gps, I see you as the triage owner for the build config component.  Is this something you or another build peer can look into and either classify this correctly or work on fixing it?
Flags: needinfo?(gps)
Whiteboard: [stockwell needswork]

Comment 12

2 months ago
19 failures in 143 pushes (0.133 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 9
* autoland: 8
* try: 1
* mozilla-beta: 1

Platform breakdown:
* windows2012-32: 13
* windows2012-64: 3
* windows2012-32-noopt: 2
* windows2012-64-noopt: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-08-22&endday=2017-08-22&tree=all
(Assignee)

Comment 13

2 months ago
From looking at a few logs, these all seem to be failures in the mozlint Python tests while shutting down multiprocessing:

17:01:17     INFO - ..\python\mozlint\test\test_types.py::test_no_filter FAILED
17:01:17     INFO - ================================== FAILURES ===================================
17:01:17     INFO - _______________________________ test_no_filter ________________________________
17:01:17     INFO - lint = <mozlint.roller.LintRoller object at 0x0333E770>
17:01:17     INFO - lintdir = 'z:\\build\\build\\src\\python\\mozlint\\test\\linters'
17:01:17     INFO - files = ['z:\\build\\build\\src\\python\\mozlint\\test\\files\\foobar.js', 'z:\\build\\build\\src\\python\\mozlint\\test\\files\\foobar.py', 'z:\\build\\build\\src\\python\\mozlint\\test\\files\\no_foobar.js']
17:01:17     INFO -     def test_no_filter(lint, lintdir, files):
17:01:17     INFO -         lint.read(os.path.join(lintdir, 'explicit_path.yml'))
17:01:17     INFO - >       result = lint.roll(files)
17:01:17     INFO - ..\python\mozlint\test\test_types.py:45:
17:01:17     INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
17:01:17     INFO - ..\python\mozlint\mozlint\roller.py:159: in roll
17:01:17     INFO -     m.shutdown()
17:01:17     INFO - c:\mozilla-build\python\Lib\multiprocessing\util.py:207: in __call__
17:01:17     INFO -     res = self._callback(*self._args, **self._kwargs)
17:01:17     INFO - c:\mozilla-build\python\Lib\multiprocessing\managers.py:625: in _finalize_manager
17:01:17     INFO -     process.terminate()
17:01:17     INFO - c:\mozilla-build\python\Lib\multiprocessing\process.py:137: in terminate
17:01:17     INFO -     self._popen.terminate()
17:01:17     INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
17:01:17     INFO - self = <multiprocessing.forking.Popen object at 0x03331F90>
17:01:17     INFO -     def terminate(self):
17:01:17     INFO -         if self.returncode is None:
17:01:17     INFO -             try:
17:01:17     INFO - >               _subprocess.TerminateProcess(int(self._handle), TERMINATE)
17:01:17     INFO - E               WindowsError: [Error 5] Access is denied
17:01:17     INFO - c:\mozilla-build\python\Lib\multiprocessing\forking.py:312: WindowsError
17:01:17     INFO - ===================== 1 failed, 4 passed in 37.74 seconds =====================
17:01:17     INFO - Setting retcode to 1 from z:\build\build\src\python\mozlint\test\test_types.py

A few random thoughts while poking around docs looking into this:
1) multiprocessing.Manager's shutdown method's docs say "This is only available if start() has been used to start the server process." I don't actually see a call to start in the mozlint code, is that an issue?
   https://docs.python.org/2/library/multiprocessing.html#multiprocessing.managers.BaseManager.shutdown
   https://dxr.mozilla.org/mozilla-central/rev/1867d7931c0a70ab90edf4aa84876525773a7139/python/mozlint/mozlint/roller.py#128
2) We seem to be creating that Manager purely to get a Queue out of it, but we could instead simply create a multiprocessing.Queue directly. Am I missing anything there?
   https://docs.python.org/2/library/multiprocessing.html#pipes-and-queues

Comment 14

2 months ago
25 failures in 194 pushes (0.129 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 20
* mozilla-inbound: 4
* mozilla-beta: 1

Platform breakdown:
* windows2012-32: 13
* windows2012-64: 10
* windows2012-64-noopt: 1
* windows2012-32-noopt: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-08-24&endday=2017-08-24&tree=all

Comment 15

2 months ago
74 failures in 908 pushes (0.081 failures/push) were associated with this bug in the last 7 days. 

This is the #22 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 46
* mozilla-inbound: 22
* try: 3
* mozilla-beta: 2
* mozilla-central: 1

Platform breakdown:
* windows2012-32: 46
* windows2012-64: 20
* windows2012-32-noopt: 6
* windows2012-64-noopt: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-08-21&endday=2017-08-27&tree=all
:ted, could you help find someone to look at this?  we are seeing many failures here
Flags: needinfo?(gps) → needinfo?(ted)
Comment hidden (mozreview-request)
(Assignee)

Comment 18

2 months ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4e0320a7d3fdcb9c39d4afaa7fdbccf90258c1fb
(Assignee)

Comment 19

2 months ago
I ran `mach lint` locally on my Windows machine and it seems to work fine with this change, including raising an exception in the wpt linter because I don't have node installed (the mach command exited fine, though). I also ran `mach python-test python/mozlint` and all the tests passed locally. We'll see what try has to say.
Assignee: nobody → ted
Flags: needinfo?(ted)

Comment 20

2 months ago
mozreview-review
Comment on attachment 8902264 [details]
bug 1371992 - make mozlint's LintRoller use concurrent.futures.

https://reviewboard.mozilla.org/r/173802/#review179102

Thanks for doing this, this looks much better! Works for me locally too, and the SIGINT handling also seems to be improved.
Attachment #8902264 - Flags: review?(ahalberstadt) → review+
(Assignee)

Comment 21

2 months ago
I ran 7 each of Win32 and Win64 debug builds on that try push and they're all green, so it's at least not completely broken. I triggered a few more just to check, but this failure mode is just infrequent enough that it's probably hard to catch on try. (I wish we had standalone Python test jobs on Windows!)

Comment 22

2 months ago
Pushed by tmielczarek@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f9885a560f3c
make mozlint's LintRoller use concurrent.futures. r=ahal

Comment 23

2 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/f9885a560f3c
Status: NEW → RESOLVED
Last Resolved: 2 months ago
status-firefox57: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
status-firefox56: --- → affected

Comment 24

2 months ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/6664fab5dea9
status-firefox56: affected → fixed
Whiteboard: [stockwell needswork] → [stockwell fixed]
Whiteboard: [stockwell fixed] → [stockwell fixed:other]

Comment 25

a month ago
13 failures in 939 pushes (0.014 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 8
* mozilla-inbound: 5

Platform breakdown:
* windows2012-64: 6
* windows2012-32: 4
* windows2012-64-noopt: 2
* windows2012-32-noopt: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-08-28&endday=2017-09-03&tree=all
(Assignee)

Comment 26

a month ago
Looking at the orangefactor link for this bug it looks like this patch fixed this:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1371992&startday=2017-08-28&endday=2017-09-05&tree=all

There are no occurrences since August 29th, which is when this merged to central.
You need to log in before you can comment on or make changes to this bug.