560857 - Sisyphus - Crash Automation - optionally use separate builders

Status: RESOLVED FIXED

(Testing Graveyard :: Sisyphus, defect)

Description

[Bob Clary [:bc] (inactive)]

Attachment: patch v1

Currently, Sisyphus (and the dependent crash and unittest automation) is designed so that each worker performs builds locally on an as needed basis. This has the great benefit of decoupling each worker but has the even greater drawback that each worker must have enough disk space to do at least a debug build on all supported branches with opt builds being an option later.

This worker local build requirement means each worker must have at least 60G and perhaps as much as 100G storage available for building.

If we were to create a separate class of worker whose sole function was to perform builds and upload them to a central location, the other workers would be able to download and install the builds on as needed basis. Their disk space requirements would drop to 10-20G. For a population of 120 workers this would decrease the storage required from 7T (120*60*/1024) / 12T (120*100/1024) to 1.2T (120*10/1024) / 2.4T (120*20/1024).

After some thought, I am thinking that the following would fit our needs.

1. Change the mozconfigs to build packaged builds.

2. Change the crash and unittest workers to either optionally perform builds as needed or to download builds, crashreport symbols and tests from a central server via http. Both possibilities must be supported since some operating system/version/cpu combinations may only have one machine available and will have to perform their own builds.

3. Fork the current worker logic into a separate build worker whose sole responsibility is to perform builds, package them and upload them to the database server.

Currently when a build is needed, {crash,unittest}workers perform the build themselves. We will change that so the worker optionally performs a query against the database asking for a build. The returned document will contain  the most current build, crashreporter symbols and tests as attachments along with the associated build date and changeset for the worker's operating system and cpu type. If the returned build date is newer than the build date the worker currently posseses, it will download the new build and install it. The worker will maintain a record of its current builds in its worker document in the database. The installed builds should live under the same directory structure as the custom builds. This will ensure the worker's killTest logic is still able to distinguish test related processes from other non-test related processes.

The build worker will be responsible for performing builds on all of the branches. This may take several hours to most of a day depending on the class of vm/machine used to perform the build. It may be necessary to be able to assign a set of build workers for a specific operating system to one or more branches so that they can complete a full set of builds in a reasonable time. This would be most likely the case for Windows.

Build workers will not try to build each changeset, but will instead try to create builds just after midnight Mountain View time. When a build is ready, the build worker will update a document on the database marking the build as unavailable for that branch/os/ver, upload the build and files to the database server and after the upload is complete update the document to reflect the meta data for the new build as well as marking the build as available again.

This patch does not completely support using downloadable builds for the unittest workers as they currently use the raw tests from a build tree. I'll file a follow up bug on getting them to download builds and use packaged tests.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: patch v1

The right file this time.

Comment 2

[Bob Clary [:bc] (inactive)]

Attachment: patch to just rm objdir rather than call out to sisyphus

includes a fix for a latent bug in create-profile.sh where it used incorrect syntax during an conditional test.

Comment 3

[Bob Clary [:bc] (inactive)]

Attachment: patch to not strip symbols in packaged builds

Comment 4

[cmtalbert]

Comment on attachment 440515 [details] [diff] [review]
patch v1

r=me, this looks good.

Comment 5

[cmtalbert]

Comment on attachment 442029 [details] [diff] [review]
patch to just rm objdir rather than call out to sisyphus

always better to keep it simple :)

Comment 6

[Bob Clary [:bc] (inactive)]

http://hg.mozilla.org/automation/sisyphus/rev/3590ece55204
http://hg.mozilla.org/automation/sisyphus/rev/9d7032f6b0a1
http://hg.mozilla.org/automation/sisyphus/rev/05da38b25a92