pymake hits recursion limit (frequently during PGO builds of WebIDL)

RESOLVED FIXED in mozilla28

Status

Firefox Build System
General
--
critical
RESOLVED FIXED
5 years ago
3 months ago

People

(Reporter: gps, Assigned: Ziga Seilnacht)

Tracking

Trunk
mozilla28
x86_64
Windows 7
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments)

(Reporter)

Description

5 years ago
Cloning to hopefully one day fix the underlying issue instead of wallpapering over it.

+++ This bug was initially created as a clone of Bug #872856 +++

This contributed to a long tree closure today, and I know it's also contributed to backouts in the past. As spontaneously as it appeared, it seems to have disappeared again. However, I figured we'd at least want an official bug on file for it.

https://tbpl.mozilla.org/php/getParsedLog.php?id=22999366&tree=Mozilla-Inbound

WINNT 5.2 mozilla-inbound pgo-build on 2013-05-15 14:10:42 PDT for push 060dd308b234
slave: w64-ix-slave110

evaluation from e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\config\rules.mk:1674:4:4:0$ nsinstall nsinstall -t -m 644 "AnimationEventBinding.h" "../../dist/include/mozilla/dom"
make.py[8]: Leaving directory 'e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\obj-firefox\dom\system\windows'
make.py[8]: Entering directory 'e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\obj-firefox\dom\system\tests'
make.py[8]: Leaving directory 'e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\obj-firefox\dom\system\tests'
make.py[7]: Leaving directory 'e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\obj-firefox\dom\system'
Traceback (most recent call last):
  File "e:/builds/moz2_slave/m-in-w32-pgo-00000000000000000/build/build/pymake/make.py", line 21, in <module>
    pymake.process.ParallelContext.spin()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\process.py", line 526, in spin
    c.run()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\process.py", line 456, in run
    cb(*args, **kwargs)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 868, in _depfinishedparallel
    self.resolvecb(error=self.error, didanything=self.didanything)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 762, in resolvecb
    self.runnext()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 782, in runnext
    self.rlist.pop(0).runcommands(self.indent, self.commandscb)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 910, in runcommands
    cb(error=False)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 792, in commandscb
    self.runnext()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 782, in runnext
    self.rlist.pop(0).runcommands(self.indent, self.commandscb)

...and then the 910, 792, 782 repeated until it eventually dies.

  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 910, in runcommands
    cb(error=False)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 792, in commandscb
    self.runnext()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 775, in runnext
    self.target.notifydone(self.makefile)
RuntimeError: maximum recursion depth exceeded
program finished with exit code 1

Updated

5 years ago
Keywords: intermittent-failure
(Reporter)

Updated

5 years ago
Depends on: 921198
(Assignee)

Comment 1

5 years ago
Created attachment 828545 [details] [diff] [review]
fix pymake stack depth issues when many targets and prerequisites are present

The attached patch reduces pymake's maximum stack depth from 505 to 131 for me (stack depth for PGO build was the same as for regular build for me). Not sure who to flag for review, this part of pymake seems a bit brittle and I'm not sure if the patch doesn't change some silent invariant in the code.

In any case, clobber and pgo builds completed successfully with this patch, pymake's tests also pass.
Attachment #828545 - Flags: review?
(Assignee)

Comment 2

5 years ago
Created attachment 828548 [details] [diff] [review]
debug patch used for measuring maximum stack depth

I used the attached patch to measure pymake's stack depth. It's only useful for debugging purposes, it includes a hardcoded absolute path for output.

Comment 3

5 years ago
Comment on attachment 828545 [details] [diff] [review]
fix pymake stack depth issues when many targets and prerequisites are present

(Setting explicit review for Ziga)

(In reply to Ziga Seilnacht from comment #1)
> The attached patch reduces pymake's maximum stack depth from 505 to 131 for
> me (stack depth for PGO build was the same as for regular build for me). Not
> sure who to flag for review, this part of pymake seems a bit brittle and I'm
> not sure if the patch doesn't change some silent invariant in the code.
> 
> In any case, clobber and pgo builds completed successfully with this patch,
> pymake's tests also pass.
Attachment #828545 - Flags: review?(mh+mozilla)
Attachment #828545 - Flags: review?(benjamin)
Attachment #828545 - Flags: review?
Comment on attachment 828545 [details] [diff] [review]
fix pymake stack depth issues when many targets and prerequisites are present

Greg knows this part better than i do.
Attachment #828545 - Flags: review?(mh+mozilla) → review?(gps)
(Reporter)

Comment 5

5 years ago
Comment on attachment 828545 [details] [diff] [review]
fix pymake stack depth issues when many targets and prerequisites are present

Review of attachment 828545 [details] [diff] [review]:
-----------------------------------------------------------------

I would expect things to blow up loudly if this change is wrong. Since apparently things don't blow up, it should be safe :)

Thanks for looking into this.
Attachment #828545 - Flags: review?(gps)
Attachment #828545 - Flags: review?(benjamin)
Attachment #828545 - Flags: review+
(Assignee)

Comment 6

5 years ago
Created attachment 828978 [details] [diff] [review]
[gecko] fix pymake stack depth issues (for checkin)
Attachment #828978 - Flags: checkin?
Keywords: checkin-needed
(Assignee)

Comment 7

5 years ago
Created attachment 828979 [details] [diff] [review]
[pymake] fix pymake stack depth issues (for checkin)

Same as for bug 782037 and bug 935050, could someone please check these two patches in, or set the checkin-needed flag. The [pymake] patch needs to land in
https://hg.mozilla.org/users/bsmedberg_mozilla.com/pymake.
Attachment #828979 - Flags: checkin?
Attachment #828978 - Flags: checkin? → checkin+
Attachment #828979 - Flags: checkin? → checkin+
https://hg.mozilla.org/mozilla-central/rev/41bec0e7bc30
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla28

Updated

3 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.