1277903 - No way to switch off parallel build processes

Status: RESOLVED FIXED

(MozReview Graveyard :: Testing / Development Environment, defect)

Description

[Māris Fogels [:mars] (please needinfo)]

There is no way to switch off the parallel build process in the dev and test environment setup programs.  The process always forks into a half-dozen or more subprocesses to speed up builds.  This causes problems during build script debugging and on low resource machines like Vagrant VMs.

On Vagrant, the concurrent builds cause the system load to spike very high when running any test or build.  The high IO can crash vitualbox-based VMs.

When debugging build problems the mixing of the log output from the parallel processes can make reading the build log very difficult.

This problem appears during resource-intense operations like building, provisioning, and configuring containers.  Keeping operations like parallel container shutdown shouldn't be a problem.

Comment 1

[Māris Fogels [:mars] (please needinfo)]

Attachment: mozreview: added new "--forks" parallel build switch (bug 1277903)

The new switch allows you to specify the number of parallel processes the mozreview and d0cker programs use when building images and containers.  The switch is named after the same switch in ansible.

Review commit: https://reviewboard.mozilla.org/r/58848/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/58848/

Comment 2

[Māris Fogels [:mars] (please needinfo)]

Attachment: mozreview: added MOZREVIEW_BUILD_FORKS environment variable (bug 1277903)

The MOZREVIEW_BUILD_FORKS environment variable controls the create-test-environment and start-local-mozreview scripts. Its value is passed to the --forks option in the underlying tools. The intention is that developers set the new environment variable in low-resource or debugging environments.

Review commit: https://reviewboard.mozilla.org/r/58850/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/58850/

Comment 3

[Gregory Szorc [:gps]]

Comment on attachment 8761734 [details]
mozreview: added new "--forks" parallel build switch (bug 1277903)

https://reviewboard.mozilla.org/r/58848/#review56020

I could bikeshed on "forks" being the proper name. But it isn't worth it for internal/support code. Nice refactor.

Comment 4

[Gregory Szorc [:gps]]

Comment on attachment 8761735 [details]
mozreview: added MOZREVIEW_BUILD_FORKS environment variable (bug 1277903)

https://reviewboard.mozilla.org/r/58850/#review56022

Comment 5

[Steven MacLeod [:smacleod]]

Comment on attachment 8761734 [details]
mozreview: added new "--forks" parallel build switch (bug 1277903)

https://reviewboard.mozilla.org/r/58848/#review57308

::: testing/vcttesting/docker.py:35
(Diff revision 1)
>  import backports.lzma as lzma
>  
>  from docker.errors import (
>      APIError as DockerAPIError,
>      DockerException,
>  )

(Bogus line comment so I open an issue)

Please wrap your commit message with line breaks

Comment 6

[Steven MacLeod [:smacleod]]

Comment on attachment 8761735 [details]
mozreview: added MOZREVIEW_BUILD_FORKS environment variable (bug 1277903)

https://reviewboard.mozilla.org/r/58850/#review57310

::: create-test-environment:1
(Diff revision 1)
>  #!/bin/bash

(Bogus line comment so I open an issue)

Please wrap your commit message with line breaks

Comment 7

[Māris Fogels [:mars] (please needinfo)]

Comment on attachment 8761734 [details]
mozreview: added new "--forks" parallel build switch (bug 1277903)

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/58848/diff/1-2/

Comment 8

[Māris Fogels [:mars] (please needinfo)]

Comment on attachment 8761735 [details]
mozreview: added MOZREVIEW_BUILD_FORKS environment variable (bug 1277903)

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/58850/diff/1-2/

Comment 9

[Māris Fogels [:mars] (please needinfo)]

Comment on attachment 8761734 [details]
mozreview: added new "--forks" parallel build switch (bug 1277903)

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/58848/diff/2-3/

Comment 10

[Māris Fogels [:mars] (please needinfo)]

Comment on attachment 8761735 [details]
mozreview: added MOZREVIEW_BUILD_FORKS environment variable (bug 1277903)

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/58850/diff/2-3/

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e5eaaa0e6cd7
mozreview: added new "--forks" parallel build switch r=gps,smacleod
https://hg.mozilla.org/hgcustom/version-control-tools/rev/d3f0d3b8a21f
mozreview: added MOZREVIEW_BUILD_FORKS environment variable r=gps,smacleod