1161909 - [B2G][Build] Adopt shallow cloning in B2G repo

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[Greg Weng [:snowmantw][:gweng][:λ]]

Currently |repo init && repo sync| always suffers, especially when you're not using MoCo network. I've tried several times at different locations, that the networks are totally fine to clone other git repors, but when |repo init && repo sync| while building B2G, it usually failed and need to retry manually several times (#1). Even without the failures, to clone all histories that user may never use it, also takes too long to complete (#2).

I think the issue could be solved by add some auto-retry mechanism in the |config.sh| script, so we could solve issue#1. And if we add a shallow cloning option to prevent cloning all histories, we could greatly reduce the time it takes to solve the issue#2.

I have already do some experiments and it works well. Before I submit the patch, I want to know owners' opinion about this. So I would do some NI.

(And I put it in general component since I don't find a Build component for B2G. Should we fire a component and a module for it? Since even B2G repo is simple than Gecko/Gaia build system, it still a standalone feature).

Comment 1

[Greg Weng [:snowmantw][:gweng][:λ]]

NI Michael since from the history the config.sh is originally and majorly contributed by him.

Comment 2

[Greg Weng [:snowmantw][:gweng][:λ]]

Attachment: Proposal

Hi, this is a WIP patch. Since I don't know how to and where to test the B2G build system, I tested them all on my local console, manually. So I set feedback first rather than review.

I've do some changes to the file:

1. organize all features in functions, so it's easier to test and debug them
2. add --shallow option
3. add --retry <Num> option
4. pull all functions from |config.sh|, which is the main script, and put them in the new library file.

I still don't know who is the proper reviewer/owner for the B2G repo build system, so I set it as Bugzilla suggested me.

Comment 3

[Michael Wu [:mwu]]

Too many changes. Focus on one thing at a time. Shallow cloning should be optional. Auto retry isn't interesting - we just want reliable mirrors. What mirrors are you having problems with?

Comment 4

[Greg Weng [:snowmantw][:gweng][:λ]]

Yes, I showed somewhat too far changes in this version, since I also wish to know your opinions about if in the future I do some isolation & rearrangement for |config.sh| (while you can see I almost keep all statements as the same, just to put them in functions, and to rename some variables to make them meaingful), approach like this is acceptable or not. I'm not going to land such many changes at one time. And according to your comment, I would submit a |--shallow| version first.

The mirroring issue is somewhat ghost-like. Since at different times it may just go well or get broken. Even with the latest master and I were sure it's just finished one session perfectly, I could still encounter lots of 404 errors, connection reset by peer, and sometimes it just cloned to some progress and got totally stock, like hanging at (66/138). Most of these, in my experiences, could be solved by retrying it several times. However, I know it should be solved completely by mirroring them. So if you want I could start to do some statics to collect the data and report which parts are most unstable. I mentioned that to add this function is because to do mirroring and other things are apparently beyond do some code change, and I don't know how to make that happens (compare to a patch).

By the way, how could I test the patch? From the past landed commits and comments on Bugzilla I didn't find any words about test. Since it has no Travis as well, does I only need to check the result is backing out or not on inbound?

Comment 5

[Greg Weng [:snowmantw][:gweng][:λ]]

Attachment: Patch

As I mentioned, this is the small and only change one option version.
Changes:

- Add one flag to control whether it should generate |--depth=1| option for |repo init|

Comment 6

[Greg Weng [:snowmantw][:gweng][:λ]]

Since from the discussion at dev-b2g:

https://groups.google.com/forum/#!topic/mozilla.dev.b2g/MSC7JsJDLHg

There are some reasonable opinions supports to set the shallow cloning as the *default* option, I would submit a version for that. And maybe Michael could give some opinions to see what is reasonable.

Comment 7

[Greg Weng [:snowmantw][:gweng][:λ]]

Attachment: Patch - shallow as default option

In this patch I set the default one |repo init| with 'depth=1', and add a --full to perform full cloning.

Comment 8

[Michael Wu [:mwu]]

Comment on attachment 8605039 [details] [review]
Patch - shallow as default option

No.

Comment 9

[Greg Weng [:snowmantw][:gweng][:λ]]

May you also take a look at the patch not set it as a default option? According to your opinion at Comment 3, and since you oppose to the second version (although without any reasons makes me don't know what's your concerns), I think it could be reviewed.

Comment 10

[Gabriele Svelto [:gsvelto]]

A quick note: I've been experimenting with shallow clones and they don't seem to work correctly for non-master branches so I fear we can't use them for that. That's a pity and I don't know why they don't work exactly.

Comment 11

[Michael Wu [:mwu]]

To clarify, B2G is a tool for developers. We're not optimizing for people who are merely interested in pulling and building, and the defaults should not be for people who merely want to build. If we care about those users, than we should directly provide builds to the degree we're allowed to.

Comment 12

[Dietrich Ayala (:dietrich)]

Lots of developers are just pulling and building. For example, the developers who filed this bug and are working on this change.

Maybe we don't change the default, but we should at least can provide an option for this.

Comment 13

[Greg Weng [:snowmantw][:gweng][:λ]]

Even as a developer, and I asked Gecko guys that almost sync-up their B2G repo everyday, they almost never need to use the older histories. You may raise counter examples to say to keep histories is very important for some critical usages, but I don't think there would be any developer would be unhappy with a faster and small pulling & building process, especially when there are options to let those whom need that could keep the work perfect.

Comment 14

[an]

(In reply to Michael Wu [:mwu] from comment #11)
> To clarify, B2G is a tool for developers. We're not optimizing for people
> who are merely interested in pulling and building, and the defaults should
> not be for people who merely want to build. If we care about those users,
> than we should directly provide builds to the degree we're allowed to.

Yes but B2G (a.k.a. Firefox OS) is a platform for users as well. I am one of them. And you definitely should care about those users for multiple reasons, no need to expand on this, I hope. I don't need ready-made builds because using source allows me to patch files thus improving v2.1 that I'm currently using and I'm sure I'm not an exception. Why must I or anyone else spend hours downloading TB of stuff I don't need and who benefits from this is hard to comprehend. Please reconsider, if it doesn't hurt anyone, on the contrary, it satisfies a general need, shallow cloning should exist as an option.

Comment 15

[[:fabrice] Fabrice Desré]

My take is that for many developers, the history of all the android related repositories is actually useless. However I don't see a good reason to not do full clones of gecko & gaia.

Would that satisfy everyone to have an option that does that?

Comment 16

[Gabriele Svelto [:gsvelto]]

(In reply to Fabrice Desré [:fabrice] from comment #15)
> My take is that for many developers, the history of all the android related
> repositories is actually useless. However I don't see a good reason to not
> do full clones of gecko & gaia.

Personally I build from a shallow repository with a .userconfig file which points to my external gecko & gaia clones where I do actual work. I'd wager that most people do something similar as working directly on the cloned repository doesn't play well with repo. Just my 2p, the fact that I cannot get a shallow clone for a non-master branch is a blunder though and that alone would prevent this from being a default. Offering an easy-to-use and easy-to-discover option though is definitely something we should do.

Comment 17

[Greg Weng [:snowmantw][:gweng][:λ]]

Since Comment 11 is what B2G team concerns and insists, it's meaningless to do anything. De-assigning and hope this pain would be eased in the future, by any other chances.

Comment 18

[Greg Weng [:snowmantw][:gweng][:λ]]

Comment on attachment 8602458 [details] [review]
Patch

Bother B2G team not anymore.

Comment 19

[BMO Automation]

Firefox OS is not being worked on