921198 - Nightly build bustage due to pymake overflowing stack RuntimeError: maximum recursion depth exceeded

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P1)

Description

[Gregory Szorc [:gps]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=28423679&tree=Mozilla-Central&full=1#error0

You trace the log back and pymake is processing mochitest installations when it hits the error.

This is a legit bug in pymake where it's little brain can't deal with very large amounts of rules.

We can work around it by converting "large" directories to use manifests.

I performed file counts for directories:

    392     ./content/html/content/test
    431     ./intl/icu/source/data/coll
    437     ./js/src/jit-test/tests/ion
    442     ./layout/base/crashtests
    455     ./js/src/jit-test/tests/debug
    476     ./dom/webidl
    503     ./layout/reftests/svg
    518     ./layout/generic/crashtests
    532     ./dom/tests/mochitest/dom-level1-core
    560     ./extensions/spellcheck/hunspell/tests/unit/data
    588     ./content/base/test
    637     ./layout/reftests/margin-collapsing
    642     ./intl/icu/source/data/locales
    669     ./dom/tests/mochitest/dom-level2-html
    1185    ./js/src/jit-test/tests/basic
    3035    ./layout/reftests/bugs 

We know from the log that dom/tests/mochitest/dom-level1-core completes without issue. Let's convert some directories to manifests to work around this issue.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Convert large directories to mochitest manifests

Convert large directories to mochitest manifests.

[test_HTMLDocument12] is a bit wonky. It appears the inactive test
filtering in emitter.py isn't working. Will file a bug.

Comment 2

[Mike Hommey [:glandium]]

Comment on attachment 810775 [details] [diff] [review]
Convert large directories to mochitest manifests

Review of attachment 810775 [details] [diff] [review]:
-----------------------------------------------------------------

::: content/base/test/mochitest.ini
@@ +305,5 @@
> +[test_bug373181.xhtml]
> +[test_bug375314.html]
> +[test_bug378969.html]
> +[test_bug380418.html]
> +[test_bug380418.html^headers^]

aren't such files support files?

Comment 3

[Gregory Szorc [:gps]]

Attachment: Increase pymake recursion limit (again)

Alternative hack.

Comment 4

[Mike Hommey [:glandium]]

Comment on attachment 810783 [details] [diff] [review]
Increase pymake recursion limit (again)

Review of attachment 810783 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/pymake/make.py
@@ +24,2 @@
>    try:
> +      sys.setrecursionlimit(100000)

Can we go for 10000 instead?

Comment 5

[Gregory Szorc [:gps]]

https://hg.mozilla.org/mozilla-central/rev/0005c66e4de4
https://hg.mozilla.org/users/bsmedberg_mozilla.com/pymake/rev/498a23902260

With 10000 instead.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

The Windows nightly retrigger ran fine. Thanks for the quick fix!