1279563 - Offer to clone repo during bootstrap

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement)

Description

[Gregory Szorc [:gps]]

Currently, the one-line bootstrap tool prints how to clone repos. We want to offer to just do the clone so there is one less thing for the user to get tripped up on.

This will also give us the opportunity to perform an "optimal" clone.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Bug 1279563 - Offer to clone Firefox source repo during bootstrap;

I've always been bothered that the one-line bootstrap configures your system
then leaves you on the hook to clone source code and configure the build
system. I'd like the bootstrap wizard to guide you through end-to-end.

This commit addresses part of the disconnect by offering to clone the
Mercurial source repository at the end of bootstrap.

We only offer to clone if we aren't running from a Firefox source checkout
(likely the one-line bootstrap invocation) and if we are in interactive
mode.

I'd like to eventually offer Git support here. Mercurial is the canonical
repo, so it makes sense to start with that.

Review commit: https://reviewboard.mozilla.org/r/59006/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/59006/

Comment 2

[Gregory Szorc [:gps]]

Attachment: Bug 1279563 - Clone unified Firefox repository;

https://hg.mozilla.org/firefox now exists. It is a unified Firefox
repository containing the heads of all the major Firefox repos
(mozilla-central, inbound, aurora, beta, release, esrs, etc).

Having 1 unified repository is more useful and incurs less overhead
than N separate repos. We want to encourage the use of the unified
repository. So, we start cloning from it.

The unified repo on the server is configured in such a way that
manifest delta chains can become very long - over 30,000 deltas. This
can make manifest reading very slow and slow down many Mercurial
operations. The server compensates for this by setting
format.maxchainlen=10000 to limit the delta chains to 10,000.
Unfortunately, this setting is not preserved when clients do a
traditional clone: the changegroup consists of a single delta chain
and the client will use its own settings (often the default) to
break the chain. This will result in the client re-creating very long
delta chains. So, as part of initializing the new repo we configure
format.maxchainlen in its .hg/hgrc so it doesn't suffer from this
performance issue.

We could achieve the same result by passing the --config option to
`hg clone`. However, the option won't be preserved in the repo's
.hg/hgrc and subsequent `hg pull` operations could result in the
creation of very long delta chains. So we need to write the config
option in the .hg/hgrc. `hg clone` is equivalent to `hg init` +
`hg pull` anyway, so we just separate out the steps and insert a
write to .hg/hgrc in between.

We also set the "default" path (like `hg clone` would do).

Review commit: https://reviewboard.mozilla.org/r/59008/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/59008/

Comment 3

[Mike Hommey [:glandium]]

Comment on attachment 8762132 [details]
Bug 1279563 - Offer to clone Firefox source repo during bootstrap;

https://reviewboard.mozilla.org/r/59006/#review56284

::: python/mozboot/mozboot/bootstrap.py:115
(Diff revision 1)
> +CLONE_MERCURIAL = '''
> +If you would like to clone the canonical Mercurial repository, please
> +enter the destination path below.
> +
> +(If you prefer to use Git, leave this blank.)
> +
> +Destination directory for Mercurial clone (leave empty to not clone): '''.lstrip()

Random thought: hg clone repo-url will use a default directory. It's kind of sad that the user now has to enter one.

::: python/mozboot/mozboot/bootstrap.py:393
(Diff revision 1)
> +        # TODO check git remotes for signs of Firefox.
> +        elif os.path.exists(git_dir):
> +            return 'git'

... or you could just do the same as for mercurial and check the existence of certain commits.
The equivalent of 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29 in cinnabar is e18f9a37a98d6cba00d7e4115810c6ca19588751 (which is true whether it's grafted or not). A non grafted clone of gecko-dev will have 3b56a9af51519d2e77e05efa672a13e6be2e9ebc (free the lizard commit from 1998)

You can use git rev-parse -q --verify $sha1^{commit}

Comment 4

[Mike Hommey [:glandium]]

Comment on attachment 8762133 [details]
Bug 1279563 - Clone unified Firefox repository;

https://reviewboard.mozilla.org/r/59008/#review56286

::: python/mozboot/mozboot/bootstrap.py:370
(Diff revision 1)
> +        return False
> +
> +    # Strictly speaking, this could overwrite a config based on a template
> +    # the user has installed. Let's pretend this problem doesn't exist
> +    # unless someone complains about it.
> +    with open(os.path.join(dest, '.hg', 'hgrc'), 'wb') as fh:

why not simply open the file for append? If there was a template, it's unlikely to contain paths.default. If there wasn't, then we end up creating the file here.

::: python/mozboot/mozboot/bootstrap.py:381
(Diff revision 1)
> +        # length. This can cause performance to tank due to delta chains being
> +        # too long. Limit the delta chain length to something reasonable
> +        # to bound revlog read time.
> +        fh.write('[format]\n')
> +        fh.write('# This is necessary to keep performance in check\n')
> +        fh.write('maxchainlen = 10000\n')

is 10000 actually reasonable?

Comment 5

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/59008/#review56286

> is 10000 actually reasonable?

I think it is. I can talk to you about it IRL. We can always tune it later if data says we need to.

Comment 6

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/59006/#review56284

> Random thought: hg clone repo-url will use a default directory. It's kind of sad that the user now has to enter one.

We can improve this as a follow-up. Perhaps a prompt for 1. Clone to <pwd>/firefox 2. Clone to custom path. 3. Don't clone.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/ffd61bc024b2
Offer to clone Firefox source repo during bootstrap; r=glandium
https://hg.mozilla.org/mozilla-central/rev/b9f4f3806395
Clone unified Firefox repository; r=glandium