949086 - Parallelize `commonplace includes`

Status: RESOLVED WONTFIX

(Marketplace Graveyard :: Consumer Pages, defect, P5)

Description

[Matt Basta [:basta]]

The includes script powers our deploy processes and backs `commonplace fiddle`. This is fine, except theres quite a lot of data that needs processing. Consider:

- JS
 - Templates need to be precompiled
 - Scripts need to be read and concatenated
 - Uglify needs to process the whole bundle
- CSS
 - Each file needs to be run through stylus
 - Compiled stylus files need to be concatenated
 - clean-css needs to process the bundle(s)

All of this happens on the same thread, and in the case of Fiddle, it happens for all commonplace dependencies in the same thread. This means that a series of processes which all would independently take up to five seconds block for over a minute.

The code (which is already async) should be updated to process each parallelizable step in its own Node subprocess.

It's possible that the best way to accomplish this is to break out the includes function to accept CLI args, then have the includes function fire up child processes to itself with the appropriate args. That would allow this to work without needing to deal with forking.

Comment 1

[Matt Basta [:basta]]

I keep running this and it keeps coming back with different times. This whole bug might not be necessary. It could just be that something is secretly saturating my CPU. We may never know.

This isn't the hardest bug to fix (and there's certainly more than one way to resolve this), so marking as good first bug. If it's really bugging someone, it's fairly straightforward to resolve.