852249 - Make pymake's includedepfiles use a dumber parser

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(not currently active) Ted Mielczarek]

If we drop mddepend.pl, that means we force make (or pymake) to parse a possibly hundreds of thousands of lines of .pp files per directory instead of roughly one. Pymake is really slow at this. Pymake has an "includedeps" feature that's indended to be used for this purpose, but it goes through the normal makefile parser so it's still slow. This patch makes includedeps use a really stupid parser that's similar to what mddepend.pl does.

This is still not quite fast enough to replace mddepend.pl, possibly because pymake is bad at stat'ing files or something. We should figure that out too.

Comment 1

[(not currently active) Ted Mielczarek]

Attachment: Make pymake's includedepfiles use a dumber parser

Comment 3

[Justin Dolske [:Dolske]]

How much slower is "really slow"? Seems like the build system is pretty broken right now, with pretty painful penalties for landing, bustage, clobber, waiting, etc. (As well as everyone's local builds affected from the CLOBBER ripples.) "Slow but working" might be a net-win.

Comment 4

[[e-mail]]

(In reply to Ted Mielczarek from comment #0)
> This is still not quite fast enough to replace mddepend.pl, possibly because
> pymake is bad at stat'ing files or something. We should figure that out too.

Well, one possibility is that mddepend.pl is optimised to stat as few files as possible - for each depending file, it checks all the dependent files that it's already cached first, so for instance once it stats nscore.h then it quickly realises that everything needs to be rebuilt.

Comment 5

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

(In reply to Justin Dolske [:Dolske] from comment #3)
> How much slower is "really slow"? Seems like the build system is pretty
> broken right now, with pretty painful penalties for landing, bustage,
> clobber, waiting, etc. (As well as everyone's local builds affected from the
> CLOBBER ripples.) "Slow but working" might be a net-win.

I saw a slow down of 4 minutes in a single directory.

Comment 6

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Comment on attachment 726283 [details] [diff] [review]
Make pymake's includedepfiles use a dumber parser

I made a couple changes to make it work on Windows, but other than that it looks good.  r=me

Comment 7

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

http://hg.mozilla.org/users/bsmedberg_mozilla.com/pymake/rev/d90856f43edc

Comment 8

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

http://hg.mozilla.org/integration/mozilla-inbound/rev/f59c6cbd3bc9