464080 - parallel builds easily end up with incomplete .jar files

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Robert Kaiser]

Since we parallelized building suite/ (bug 462736) with the new mechanism from bug 461395, I've been running into having incomplete chrome again and again.

When investigating today, I found out that depend rebuilds that haven't much (or actually anything) to do are more likely to run into this, and removing the -j2 option from my mozconfig makes it go away.

What happens is that comm.jar ends up missing some files, probably due to multiple jar.mn files writing stuff to it and parallel runs of JarMaker racing each other and ending up overwriting each other's changes to the .jar file. We probably will run into this even in the mozilla tree when toolkit/ or browser/ are converted to the parallel build infrastructure, I think suite/ happened to be the first consumer of it that actually runs JarMaker.

Comment 1

[Axel Hecht [:Pike]]

Which platform, which version of python?

Which files are missing, and in which jar.mn are they?

Comment 2

[Benjamin Smedberg]

Yeah, we really meant for parallel builds to be mainly focused on the compiler stuff, not chrome. The locking mechanism for rebuilding chrome is probably completely broken, if it exists at all.

Comment 3

[Robert Kaiser]

I end up with that on Linux (openSUSE Factory) with Python 2.6, all the content/navigator/ files (suite/browser/jar.mn) end up missing practically every time, a also had a case where even more ended up missing but I didn't preserve it.
My habit of verifying problems in a build by nuking dist/ or dist/bin and rebuilding the same tree might help this, as no actual compilation work needs to be done there but just recreation of symlinks, some preprocessing and rebuilding chrome.

Neil, what do you think about comment #2?

Comment 4

[Axel Hecht [:Pike]]

We create the jar files with locks on at http://mxr.mozilla.org/mozilla-central/source/config/JarMaker.py#257, and it's implemented in MozZipFile and then http://mxr.mozilla.org/mozilla-central/source/config/utils.py#59.

Thus my question for more details on how to reproduce this problem.

Comment 5

[[e-mail]]

Attachment: Proposed patch

Before:
* First process opens comm.jar
* First process creates lockfile
* First process writes comm.jar
* Second process opens comm.jar (!)
* Second process waits
* First process deletes lockfile
* Second process creates lockfile
* Second process overwrites comm.jar (!!)
* Second process deletes lockfile
After:
* First process creates lockfile
* First process opens comm.jar
* First process writes comm.jar
* Second process waits
* First process deletes lockfile
* Second process creates lockfile
* Second process opens comm.jar
* Second process appends to comm.jar
* Second process deletes lockfile

Comment 6

[[e-mail]]

Comment on attachment 349152 [details] [diff] [review]
Proposed patch

Bah, silly calendar QA contact name conflict

Comment 7

[Axel Hecht [:Pike]]

Comment on attachment 349152 [details] [diff] [review]
Proposed patch

Duh.

r=me.

Comment 8

[Robert Kaiser]

Could we get this into mozilla-central ASAP please?

Comment 9

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 349152 [details] [diff] [review]
Proposed patch

a191b2=beltzner

Comment 10

[Serge Gautherie (:sgautherie)]

It happened to me a few times too on my Windows 2000.

Comment 11

[[e-mail]]

Pushed changeset 2f0fe196aa89 to mozilla-central.