915648 - Parallelize make export

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Comment 1

[Mike Hommey [:glandium]]

Attachment: Parallelize make export

Comment 2

[Gregory Szorc [:gps]]

Comment on attachment 803668 [details] [diff] [review]
Parallelize make export

Review of attachment 803668 [details] [diff] [review]:
-----------------------------------------------------------------

I think you should cancel this review and move this patch into bug 907365, as I'm pretty sure the broken webidl dependencies will land before bug 907365. I'll bug khuey IRL right now...

Comment 3

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I don't want to be bugged IRL :-(

Comment 4

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #2)
> Comment on attachment 803668 [details] [diff] [review]
> Parallelize make export
> 
> Review of attachment 803668 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> I think you should cancel this review and move this patch into bug 907365,
> as I'm pretty sure the broken webidl dependencies will land before bug
> 907365.

Unfortunately, bug 907365 is not the only problem. I already got three different issues with make export running in parallel, one of which i addressed with explicit dependencies defined in Makefile.in, and the other two that i just got now and haven't investigated.

For the record:
 0:52.13 Traceback (most recent call last):
 0:52.13   File "/home/mh/mozilla-central/js/src/config/nsinstall.py", line 180, in <module>
 0:52.13     sys.exit(_nsinstall_internal(argv[1:]))
 0:52.13   File "/home/mh/mozilla-central/js/src/config/nsinstall.py", line 145, in _nsinstall_internal
 0:52.13     copy_all_entries(args, target)
 0:52.13   File "/home/mh/mozilla-central/js/src/config/nsinstall.py", line 106, in copy_all_entries
 0:52.13     handleTarget(e, dest)
 0:52.13   File "/home/mh/mozilla-central/js/src/config/nsinstall.py", line 135, in handleTarget
 0:52.13     shutil.copy(srcpath, targetpath)
 0:52.13   File "/usr/lib/python2.7/shutil.py", line 119, in copy
 0:52.13 Creating /home/mh/build/obj-x86_64-unknown-linux-gnu/security/build/../../dist/private/dbm
 0:52.15 cd src; /usr/bin/make private_export
 0:52.16 There are no private exports.
 0:52.16 There are no private exports.
 0:52.16 cd softoken; /usr/bin/make private_export
 0:52.17 Traceback (most recent call last):
 0:52.17   File "/home/mh/mozilla-central/js/src/config/nsinstall.py", line 180, in <module>
 0:52.17     sys.exit(_nsinstall_internal(argv[1:]))
 0:52.17   File "/home/mh/mozilla-central/js/src/config/nsinstall.py", line 145, in _nsinstall_internal
 0:52.17     copy_all_entries(args, target)
 0:52.17   File "/home/mh/mozilla-central/js/src/config/nsinstall.py", line 106, in copy_all_entries
 0:52.17     handleTarget(e, dest)
 0:52.17   File "/home/mh/mozilla-central/js/src/config/nsinstall.py", line 135, in handleTarget
 0:52.17     shutil.copy(srcpath, targetpath)
 0:52.17   File "/usr/lib/python2.7/shutil.py", line 119, in copy
 0:52.17 There are no private exports.
 0:52.17 cd legacydb; /usr/bin/make private_export
 0:52.17     copyfile(src, dst)
 0:52.17   File "/usr/lib/python2.7/shutil.py", line 82, in copyfile
 0:52.17     with open(src, 'rb') as fsrc:
 0:52.17 IOError: [Errno 13] Permission denied: u'/home/mh/mozilla-central/mfbt/MathAlgorithms.h'
 0:52.18     copyfile(src, dst)
 0:52.18   File "/usr/lib/python2.7/shutil.py", line 82, in copyfile
 0:52.18     with open(src, 'rb') as fsrc:
 0:52.18 IOError: [Errno 13] Permission denied: u'/home/mh/mozilla-central/mfbt/MemoryReporting.h'

and:

 4:35.80 Generating WebIDL bindings
 4:35.93 Traceback (most recent call last):
 4:35.93   File "/home/mh/mozilla-central/config/pythonpath.py", line 56, in <module>
 4:35.93     main(sys.argv[1:])
 4:35.93   File "/home/mh/mozilla-central/config/pythonpath.py", line 48, in main
 4:35.93     execfile(script, frozenglobals)
 4:35.93   File "/home/mh/mozilla-central/dom/bindings/BindingGen.py", line 98, in <module>
 4:35.93     main()
 4:35.93   File "/home/mh/mozilla-central/dom/bindings/BindingGen.py", line 58, in main
 4:35.93     config = cPickle.load(f)
 4:35.93 ValueError: unregistered extension code 264
 4:35.97 make[5]: *** [.BindingGen] Error 1

Comment 5

[Mike Hommey [:glandium]]

Attachment: Parallelize make export

I prefer to keep this separated.

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 806963 [details] [diff] [review]
Parallelize make export

Review of attachment 806963 [details] [diff] [review]:
-----------------------------------------------------------------

::: Makefile.in
@@ +206,5 @@
>  endif
> +
> +js/xpconnect/src/export: config/makefiles/precompile/export
> +accessible/src/xpcom/export: config/makefiles/precompile/export
> +js/src/export: mfbt/export

Shouldn't this be behind an ifdef MOZ_PSEUDO_DERECURSE?

Comment 7

[Mike Hommey [:glandium]]

Attachment: Parallelize make export

Maybe I'm being over-cautious, but i prefer to provide an easy escape route.

Comment 8

[Gregory Szorc [:gps]]

Comment on attachment 807564 [details] [diff] [review]
Parallelize make export

Review of attachment 807564 [details] [diff] [review]:
-----------------------------------------------------------------

Clever.

Comment 9

[Mike Hommey [:glandium]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/9cc7634555c3

Comment 10

[Mike Hommey [:glandium]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/9eba1b4d798f

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/9cc7634555c3
https://hg.mozilla.org/mozilla-central/rev/9eba1b4d798f

Comment 12

[:Benjamin Peterson]

glandium, I see you added

DEFINES['FOO'] = '1'

to the toplevel moz.build file in 9cc7634555c3. That's not needed, right?

Comment 13

[Mike Hommey [:glandium]]

(In reply to :Benjamin Peterson from comment #12)
> glandium, I see you added
> 
> DEFINES['FOO'] = '1'
> 
> to the toplevel moz.build file in 9cc7634555c3. That's not needed, right?

Erf. No.

Comment 14

[:Benjamin Peterson]

Okay. I killed it.
https://hg.mozilla.org/integration/mozilla-inbound/rev/c87681163457

Comment 15

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/c87681163457