869773 - consider adding local mirror support to config.sh

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: GonkIntegration, enhancement)

Description

[Ben Kelly [:bkelly, not reviewing]]

The repo tool supports downloading git repositories once to a local mirror
and then pulling from those repositories for separate project working areas.
This is done by passing either --mirror or --reference= to repo init. When
working with more than one project or branch this can greatly speed up the
sync process.

It would be nice if config.sh supported creating a local mirror and then using it for subsequent config.sh runs in separate B2G directories.  For example:

    cd /some/path/b2g-mirror
    BRANCH=master ./config.sh mirror

Then:

    cd /some/other/path/B2G
    B2G_LOCAL_MIRROR=/some/path/b2g-mirror ./config.sh hamachi

Pull requests implementing this feature to follow shortly.

Comment 1

[Ben Kelly [:bkelly, not reviewing]]

Attachment: Pointer to Github pull request: https://github.com/mozilla-b2g/B2G/pull/239

Pointer to Github pull-request

Comment 2

[Ben Kelly [:bkelly, not reviewing]]

Attachment: Pointer to Github pull request: https://github.com/mozilla-b2g/b2g-manifest/pull/73

Pointer to Github pull-request

Comment 3

[Michael Wu [:mwu]]

I think this is too fragile to really be useful. Copying the .repo directory works just as well if you have enough disk space and doesn't involve as many complications.

Comment 4

[Ben Kelly [:bkelly, not reviewing]]

Thanks for taking the time to review!

I think I'm going to continue down this road in my local environment for now because I really like the idea of being able to |repo sync| in one directory overnight and then update my various checkouts without a network penalty later as needed.

I'll leave this open in case anyone else might find it useful.  Or we can close if you prefer.  Thanks again.

Comment 5

[Vasil Dimov [:vd]]

Hi,

Indeed just doing ./repo init and ./repo sync over the internet frequently is not feasible, it downloads many gigabytes and takes many hours!

What I did to overcome this was to run a git server on the localhost, mirror everything into it and hijacked connections to the relevant hosts to localhost by putting e.g. "127.0.0.1 github.com" in /etc/hosts. That's cumbersome.

I did not realize I could copy the .repo directory to minimize the internet traffic.

Ben, my understanding is that you have created a manifest that is cumulative from all the other manifests, right? If that is the case, does that mean that, in the future, each change to a any manifest would have to be replayed over the cumulative manifest? If yes, then maybe it would be more flexible to have the script traverse all the manifests at runtime and create the cumulative one on the fly when --mirror is used?

Comment 6

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Vasil Dimov [:vd] from comment #5)
> Ben, my understanding is that you have created a manifest that is cumulative
> from all the other manifests, right? If that is the case, does that mean
> that, in the future, each change to a any manifest would have to be replayed
> over the cumulative manifest? If yes, then maybe it would be more flexible
> to have the script traverse all the manifests at runtime and create the
> cumulative one on the fly when --mirror is used?

Yes, the mirror.xml would have to be updated manually to get any new repositories.  If its missing from the mirror, though, repo sync will just pull from the internet again.  So it won't be a hard failure.

I think it might be hard to generate the manifest on the fly.  One of the issues I ran into is many of the platforms pull the same git repositories, but from different sources.  I defaulted the mirror.xml to use the most common source.  So any automated tool would really need to parse the manifests and deconflict sources, etc.

In any case, the --mirror/--reference support in repo is not perfect.  For example, when the CAF git endpoints were down last night I was hoping I could just run locally from my mirror, but it didn't work.  The repo tool still hits the internet to download meta data for each repo.  It only seems to pull the gigs of change sets from the --reference setting.  In that case I ended up use the .repo copy as mwu suggested and it worked well.

I've also considered symlinking all my .repo/projects directories to one shared directory.  That might run into issues with the different git sources, though.

Comment 7

[Ben Kelly [:bkelly, not reviewing]]

To clarify, it appears repo init --reference sets up each project in the manifest with a .git/objects/info/alternative file.  So git will attempt to grab objects from there before going to the main URL.  Refs are still pulled from the main URL.

Still, this does speed up the normal ./config.sh process.  I can configure a fresh b2g directory using repo init --reference in about 1.5 minutes vs the normal hour+.  It just means full offline support is not supported.

Comment 8

[Ben Kelly [:bkelly, not reviewing]]

Sorry, that should be:  .git/objects/info/alternates

Comment 9

[BMO Automation]

Firefox OS is not being worked on