874210 - pymake hits recursion limit (frequently during PGO builds of WebIDL)

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

Cloning to hopefully one day fix the underlying issue instead of wallpapering over it.

+++ This bug was initially created as a clone of Bug #872856 +++

This contributed to a long tree closure today, and I know it's also contributed to backouts in the past. As spontaneously as it appeared, it seems to have disappeared again. However, I figured we'd at least want an official bug on file for it.

https://tbpl.mozilla.org/php/getParsedLog.php?id=22999366&tree=Mozilla-Inbound

WINNT 5.2 mozilla-inbound pgo-build on 2013-05-15 14:10:42 PDT for push 060dd308b234
slave: w64-ix-slave110

evaluation from e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\config\rules.mk:1674:4:4:0$ nsinstall nsinstall -t -m 644 "AnimationEventBinding.h" "../../dist/include/mozilla/dom"
make.py[8]: Leaving directory 'e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\obj-firefox\dom\system\windows'
make.py[8]: Entering directory 'e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\obj-firefox\dom\system\tests'
make.py[8]: Leaving directory 'e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\obj-firefox\dom\system\tests'
make.py[7]: Leaving directory 'e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\obj-firefox\dom\system'
Traceback (most recent call last):
  File "e:/builds/moz2_slave/m-in-w32-pgo-00000000000000000/build/build/pymake/make.py", line 21, in <module>
    pymake.process.ParallelContext.spin()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\process.py", line 526, in spin
    c.run()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\process.py", line 456, in run
    cb(*args, **kwargs)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 868, in _depfinishedparallel
    self.resolvecb(error=self.error, didanything=self.didanything)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 762, in resolvecb
    self.runnext()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 782, in runnext
    self.rlist.pop(0).runcommands(self.indent, self.commandscb)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 910, in runcommands
    cb(error=False)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 792, in commandscb
    self.runnext()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 782, in runnext
    self.rlist.pop(0).runcommands(self.indent, self.commandscb)

...and then the 910, 792, 782 repeated until it eventually dies.

  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 910, in runcommands
    cb(error=False)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 792, in commandscb
    self.runnext()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 775, in runnext
    self.target.notifydone(self.makefile)
RuntimeError: maximum recursion depth exceeded
program finished with exit code 1

Comment 1

[Ziga Seilnacht]

Attachment: fix pymake stack depth issues when many targets and prerequisites are present

The attached patch reduces pymake's maximum stack depth from 505 to 131 for me (stack depth for PGO build was the same as for regular build for me). Not sure who to flag for review, this part of pymake seems a bit brittle and I'm not sure if the patch doesn't change some silent invariant in the code.

In any case, clobber and pgo builds completed successfully with this patch, pymake's tests also pass.

Comment 2

[Ziga Seilnacht]

Attachment: debug patch used for measuring maximum stack depth

I used the attached patch to measure pymake's stack depth. It's only useful for debugging purposes, it includes a hardcoded absolute path for output.

Comment 3

[Ed Morley [:emorley]]

Comment on attachment 828545 [details] [diff] [review]
fix pymake stack depth issues when many targets and prerequisites are present

(Setting explicit review for Ziga)

(In reply to Ziga Seilnacht from comment #1)
> The attached patch reduces pymake's maximum stack depth from 505 to 131 for
> me (stack depth for PGO build was the same as for regular build for me). Not
> sure who to flag for review, this part of pymake seems a bit brittle and I'm
> not sure if the patch doesn't change some silent invariant in the code.
> 
> In any case, clobber and pgo builds completed successfully with this patch,
> pymake's tests also pass.

Comment 4

[Mike Hommey [:glandium]]

Comment on attachment 828545 [details] [diff] [review]
fix pymake stack depth issues when many targets and prerequisites are present

Greg knows this part better than i do.

Comment 5

[Gregory Szorc [:gps]]

Comment on attachment 828545 [details] [diff] [review]
fix pymake stack depth issues when many targets and prerequisites are present

Review of attachment 828545 [details] [diff] [review]:
-----------------------------------------------------------------

I would expect things to blow up loudly if this change is wrong. Since apparently things don't blow up, it should be safe :)

Thanks for looking into this.

Comment 6

[Ziga Seilnacht]

Attachment: [gecko] fix pymake stack depth issues (for checkin)

Comment 7

[Ziga Seilnacht]

Attachment: [pymake] fix pymake stack depth issues (for checkin)

Same as for bug 782037 and bug 935050, could someone please check these two patches in, or set the checkin-needed flag. The [pymake] patch needs to land in
https://hg.mozilla.org/users/bsmedberg_mozilla.com/pymake.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/41bec0e7bc30
https://hg.mozilla.org/users/bsmedberg_mozilla.com/pymake/rev/f99dd2f10896

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/41bec0e7bc30