492882 - pymake: Integrate race detector

Status: RESOLVED INVALID

(Firefox Build System :: General, defect)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

I just had a -j build fail b/c of a corrupted shared object, and I've heard tales of other random -j build failures.  The first two suspicious candidates are makefile races and file system synchronization, with the former seemingly much more likely than the latter.

We should head these off at the pass by enforcing ordering constraints on make rules.  This should be relatively easy to add, and this could even be checked "statically".

Bug 485390 seems related.

Comment 1

[Benjamin Smedberg]

Oh please oh please be specific! I build with -j12 or -j16 all the time and haven't experienced this.

Most of our race problems in make rules are rules which incidentally create other files which are not mentioned as targets or prereqs... I don't see how static analysis of make rules will help us there.

Comment 2

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to comment #1)
> Oh please oh please be specific! I build with -j12 or -j16 all the time and
> haven't experienced this.
> 

Can't speak for the other stories (joduinn is the last person I heard complain), but what happened to me was a linker error something like:

  ld: error: can't find MOZ_Z_inflate [or incorrect visibility, don't remember]
  ... [similar errors for other MOZ_Z_... symbols]
  ld: error: lib??.so is malformed  [forgot which one this was; wasn't libmozz, though]

Sorry I didn't save the exact error; I was in a bit of a rush.  Blowing away dist/lib/* and recompiling fixed the problem.  This may not have been a -j problem.

> Most of our race problems in make rules are rules which incidentally create
> other files which are not mentioned as targets or prereqs... I don't see how
> static analysis of make rules will help us there.

And other scripts (?) then use these incidentally-generated files, without specifying them as dependencies or listing them on the rule command line?  If they're listed on the command line, we could use heuristics in a static analysis.  If not, a dynamic analysis using strace might do the trick.

Comment 3

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

See bug 493891 for the libmozz link error.

Comment 4

[Justin Wood (:Callek)]

is this wontfix/invalid? I don't see a good description of the problem that instrumentation can fix.

Comment 5

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Without more specific information there's not much we can do here.