1517831 - Enable sparse checkout in macOS worker hrgc

Status: VERIFIED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Chris Manchester (limited bugmail, email directly)]

According to :sheehan this is a matter of enabling the extension in hgrc. Given the time for a full checkout on macOS workers currently a sparse checkout will be extremely helpful to making three-tiered pgo work on this platform.

Comment 1

[Kendall Libby [:fubar] (he/him)]

connor, what's the hgrc magic for sparse checkouts (and is there a minimum hg version)? also, yay!

Comment 2

[Connor Sheehan [:sheehan]]

Enabling the extension is just

[extensions]
sparse =

The minimum version that includes sparse is 4.3. All CI clients should be on that version or newer.

Once the extension is enabled, the invocation of robustcheckout needs to include --sparseprofile <profile path>. Profiles are checked in to central at build/sparse-profiles.

Sparse checkouts come with the requirement that all consumers of that given checkout be sparse as well. From a comment in robustcheckout:

Refuse to enable sparse checkouts on existing checkouts. The reasoning
here is that another consumer of this repo may not be sparse aware. If we
enabled sparse, we would lock them out.

There is a hint attached to the raised error that says "use a separate working directory to use sparse". So I think you will also need to switch the cache over to a sparse cache within the task definition.

I have yet to use sparse checkouts/profiles/etc for anything so I don't know too much about them. Taking a look through the sparse profiles checked in to central, it doesn't look like there are any existing profiles that would be useful for speeding up build tasks. The existing profiles are used for things like only checking out mach, or only checking out the taskgraph generation code. Can you expand on what you'd like to use sparse checkouts for? If you're looking to speed up builds, I'm pretty sure you will need to use the full checkout. :/

Comment 3

[Chris Manchester (limited bugmail, email directly)]

(In reply to Connor Sheehan [:sheehan] from comment #2)

Enabling the extension is just

[extensions]
sparse =

The minimum version that includes sparse is 4.3. All CI clients should be on
that version or newer.

Once the extension is enabled, the invocation of robustcheckout needs to
include --sparseprofile <profile path>. Profiles are checked in to central
at build/sparse-profiles.

Sparse checkouts come with the requirement that all consumers of that given
checkout be sparse as well. From a comment in robustcheckout:

Refuse to enable sparse checkouts on existing checkouts. The reasoning
here is that another consumer of this repo may not be sparse aware. If we
enabled sparse, we would lock them out.

There is a hint attached to the raised error that says "use a separate
working directory to use sparse". So I think you will also need to switch
the cache over to a sparse cache within the task definition.

I have yet to use sparse checkouts/profiles/etc for anything so I don't know
too much about them. Taking a look through the sparse profiles checked in to
central, it doesn't look like there are any existing profiles that would be
useful for speeding up build tasks. The existing profiles are used for
things like only checking out mach, or only checking out the taskgraph
generation code. Can you expand on what you'd like to use sparse checkouts
for? If you're looking to speed up builds, I'm pretty sure you will need to
use the full checkout. :/

I'd like to use sparse checkout for the pgo profiling run, which needs a mach environment, the profile server script, and some test pages checked in to the tree. It should be able to run without any source files or regular test files so I expect a big improvement in checkout time.

Comment 4

[Chris Manchester (limited bugmail, email directly)]

Hi :fubar, do you know who might be able to take this? Thank you

Comment 5

[Dragos Crisan [:dragrom]]

I'll handle this bug

Comment 6

[Dragos Crisan [:dragrom]]

Attachment: Enable sparse checkout in mac os worker

Comment 7

[Dragos Crisan [:dragrom]]

I applied changes on t-yosemite-r7-380.test.releng.mdc1.mozilla.com.

:chmanchester You can use this worker to test sparse extension

Comment 8

[Chris Manchester (limited bugmail, email directly)]

(In reply to Dragos Crisan [:dragrom] from comment #7)

I applied changes on t-yosemite-r7-380.test.releng.mdc1.mozilla.com.

:chmanchester You can use this worker to test sparse extension

Thank you for doing so. Excuse my ignorance, but can you direct me on how to connect to this worker? I tried simply pushing to try, but I get the same error.

Comment 9

[Dragos Crisan [:dragrom]]

Pushed some tests on try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7112f6a9ee7fef7ceeb4c044ac72644b68da6eaa

Comment 10

[Dragos Crisan [:dragrom]]

(In reply to Chris Manchester (:chmanchester) from comment #8)

(In reply to Dragos Crisan [:dragrom] from comment #7)

I applied changes on t-yosemite-r7-380.test.releng.mdc1.mozilla.com.

:chmanchester You can use this worker to test sparse extension

Thank you for doing so. Excuse my ignorance, but can you direct me on how to connect to this worker? I tried simply pushing to try, but I get the same error.

You can redirect your pushed jobs to run on staging environment. To do this you can follow the documentation:
https://wiki.mozilla.org/ReleaseEngineering#Run_the_tasks_to_staging_pools_.28OSX.2FLinux.29

Comment 11

[Chris Manchester (limited bugmail, email directly)]

I was able to specify a build with a sparse profile without it rejecting the configuration, but I don't think it succeeded in making the checkout any faster. Connor, can you take a quick look at https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=225288340&revision=a4caf114351ffd3d9cef3c94c4ca9c0b7871171b and maybe tell me where I'm going wrong here? Thanks!

Comment 12

[Connor Sheehan [:sheehan]]

Twenty minutes in that task are spent cloning off hgmo due to not using a repo cache on the worker. The sparse checkout seems to work, as you only update 11k files instead of the ~250k in-tree.

Unless the MacOS workers don't use shared storage for some reason, I think the task is just incorrectly specifying the cache path. By using a local path we end up looking for a cache in the task-specific directory (/Users/cltbld/tasks/task_1548950850/checkouts/hg-shared). I think you want to point them at /builds/hg-shared (that seems to be the common location from a brief scan of central). I don't know much about how the macOS workers are configured, though.

Comment 13

[Chris Manchester (limited bugmail, email directly)]

Ok, thanks Connor. I'll take a look. In the meantime sparse-checkout has been correctly configure here.

Comment 14

[Dragos Crisan [:dragrom]]

:dhouse Can you please land this PR Monday morning and close the bug?

Comment 15

[:dhouse]

Thx Dragos. I was late to merge this one!
I confirmed the sparse line is added on ~cltbld/.hgrc on the mac workers (all but a few problem ones that haven't run puppet).