872856 - Intermittent Windows PGO RuntimeError: maximum recursion depth exceeded while compiling WebIDL bindings

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

This contributed to a long tree closure today, and I know it's also contributed to backouts in the past. As spontaneously as it appeared, it seems to have disappeared again. However, I figured we'd at least want an official bug on file for it.

https://tbpl.mozilla.org/php/getParsedLog.php?id=22999366&tree=Mozilla-Inbound

WINNT 5.2 mozilla-inbound pgo-build on 2013-05-15 14:10:42 PDT for push 060dd308b234
slave: w64-ix-slave110

evaluation from e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\config\rules.mk:1674:4:4:0$ nsinstall nsinstall -t -m 644 "AnimationEventBinding.h" "../../dist/include/mozilla/dom"
make.py[8]: Leaving directory 'e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\obj-firefox\dom\system\windows'
make.py[8]: Entering directory 'e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\obj-firefox\dom\system\tests'
make.py[8]: Leaving directory 'e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\obj-firefox\dom\system\tests'
make.py[7]: Leaving directory 'e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\obj-firefox\dom\system'
Traceback (most recent call last):
  File "e:/builds/moz2_slave/m-in-w32-pgo-00000000000000000/build/build/pymake/make.py", line 21, in <module>
    pymake.process.ParallelContext.spin()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\process.py", line 526, in spin
    c.run()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\process.py", line 456, in run
    cb(*args, **kwargs)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 868, in _depfinishedparallel
    self.resolvecb(error=self.error, didanything=self.didanything)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 762, in resolvecb
    self.runnext()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 782, in runnext
    self.rlist.pop(0).runcommands(self.indent, self.commandscb)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 910, in runcommands
    cb(error=False)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 792, in commandscb
    self.runnext()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 782, in runnext
    self.rlist.pop(0).runcommands(self.indent, self.commandscb)

...and then the 910, 792, 782 repeated until it eventually dies.

  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 910, in runcommands
    cb(error=False)
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 792, in commandscb
    self.runnext()
  File "e:\builds\moz2_slave\m-in-w32-pgo-00000000000000000\build\build\pymake\pymake\data.py", line 775, in runnext
    self.target.notifydone(self.makefile)
RuntimeError: maximum recursion depth exceeded
program finished with exit code 1

Comment 1

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=22993923&tree=Mozilla-Inbound

Comment 2

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=23008239&tree=Mozilla-Inbound
WINNT 5.2 mozilla-inbound pgo-build on 2013-05-15 18:06:10
slave: w64-ix-slave110

RuntimeError: maximum recursion depth exceeded

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Bumping severity on this. I fully expect this to start burning nightlies before too long, and with PGO builds taking 5+ hours these days, it will cause major disruptions to timely shipping.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #3)
> Bumping severity on this. I fully expect this to start burning nightlies
> before too long, and with PGO builds taking 5+ hours these days, it will
> cause major disruptions to timely shipping.

Not to mention timely merges from inbound to m-c.

Comment 5

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Nice, so that thing I'm looking at for bz is happening without his patch now.

Comment 6

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

On the bright side if we back out the most recent stuff that added WebIDL files and don't add any more we should be ok.

Comment 7

[Masatoshi Kimura [:emk]]

(In reply to Kyle Huey [:khuey] ([e-mail]) from comment #6)
> On the bright side if we back out the most recent stuff that added WebIDL
> files and don't add any more we should be ok.

I can't land bug 871786 until this bug is fixed? Great news...

Comment 8

[Gregory Szorc [:gps]]

Quick fix: sys.setrecursionlimit(2 * sys.getrecursionlimit()) ? Then we can patch pymake whenever.

Comment 9

[Phil Ringnalda (:philor)]

I found it extremely suspicious that today's tree closure was over two PGO builds, one on w64-ix-slave110 and one on w64-ix-slave110.

Oops, apparently I missed seeing one, there was also a third on w64-ix-slave110.

Comment 10

[Andrew McCreight [:mccr8]]

FYI bug 823512 was landed on inbound a few hours ago, in case PGO builds go south again.  It has a ton of new WebIDL.

Comment 11

[Gregory Szorc [:gps]]

Attachment: Temporarily increase Python recursion limit in pymake

Comment 12

[(not currently active) Ted Mielczarek]

Comment on attachment 751169 [details] [diff] [review]
Temporarily increase Python recursion limit in pymake

Review of attachment 751169 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/pymake/make.py
@@ +17,5 @@
>  
>    gc.disable()
>  
> +  # This is meant as a temporary workaround until issues with many targets
> +  # and prerequisites is addressed.

Put a bug number here?

Comment 13

[Gregory Szorc [:gps]]

https://hg.mozilla.org/users/bsmedberg_mozilla.com/pymake/rev/e35c8b3b0cbb
https://hg.mozilla.org/integration/mozilla-inbound/rev/aaf004a82132

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/aaf004a82132