support non-git cinnabar users with `mach try`

NEW
Unassigned

Status

Taskcluster
Task Configuration
2 months ago
20 days ago

People

(Reporter: froydnj, Unassigned)

Tracking

(Blocks: 1 bug)

Details

I've been using mozilla/gecko-dev to manage my Mozilla development for...well, as long as gecko-dev has existed.  My workflow for pushing to try is basically:

1. import commits into an m-c clone;
2. modify top commit to append the try syntax to the first line of the commit message;
3. push to try

I'm sure there are more elegant ways to get the job done, but this is what works for me.

I have a script to do the above, and maybe the workflow in the `mach try`-required world is to just shoehorn `mach try` into that.  But I'm interested if there are other non-git cinnabar users out there and what their workflows are like.
I maintain a mozilla-central mirror at https://github.com/mozilla/gecko.  I sync it using git-cinnabar via my local clone, which means I can push to tryserver using git-cinnabar.  But the data that git-cinnabar stores in .git doesn't get synced to GitHub, nor from GitHub to any other clone. So others who clone that repo can't push to tryserver using git-cinnabar (at least not without also fetching mozilla-central themselves using git-cinnabar).

Ideally, developers who clone that repo would be able to push to tryserver without also having to fetch mozilla-central.
I am using moz-git-tools as a way to push to try, and to push to other branches.

Currently moz-git-tools is using mq to import the patches and apply them, then it creates a new mq-commit on top with the try syntax.

Switchting to "./mach try" implies that the git-push-to-try script uses "mach try" as a backend, on a non-(mach-)bootstrapped work directory.

Does "python ./mach try" from a mercurial repository:
 1/ Work on a non configured directory?
 2/ Work with mq?

For the record, I did not switch to git-cinnabar, mostly because I am based out of gecko-dev, and I sometimes relies on github to share my work with others.  I also do not used the grafting mode from git-cinnabar because of the racing issue [1] on pushes, and I do not want to write (once-more) a Shell loop for making pushes.

[1] https://github.com/glandium/git-cinnabar/wiki/Mozilla:-Using-a-git-clone-of-gecko%E2%80%90dev-to-push-to-mercurial#following-gecko-dev

Comment 3

2 months ago
(In reply to Myk Melez [:myk] [@mykmelez] from comment #1)
> I maintain a mozilla-central mirror at https://github.com/mozilla/gecko.  I
> sync it using git-cinnabar via my local clone, which means I can push to
> tryserver using git-cinnabar.  But the data that git-cinnabar stores in .git
> doesn't get synced to GitHub, nor from GitHub to any other clone. So others
> who clone that repo can't push to tryserver using git-cinnabar (at least not
> without also fetching mozilla-central themselves using git-cinnabar).
> 
> Ideally, developers who clone that repo would be able to push to tryserver
> without also having to fetch mozilla-central.

FWIW I at least regularly point new contributors to this workflow if they're more comfortable using git than hg, and I think it would be quite terrible to require fetching mozilla-central through git-cinnabar when they get to a stage when they can push to try.

I personally have a local clone that is a mixture of gecko-dev and git-cinnabar and I have branches on top of both lines of history so I still find myself using moz-git-tools every once in a while although it's rare for me personally these days.

Comment 4

2 months ago
Given how much more work it is for new contributors to get started volunteering on Mozilla than on other open-source projects, it does seem like things that are likely to increase new-contributor funnel drop off should be avoided whenever practical, especially given that people in the existing open-source community almost certainly have existing github-based workflows for development that  keep that barrier-to-entry lower...

Comment 5

2 months ago
A clone of https://github.com/mozilla/gecko-dev.git is ~3.5 GB, which is 10-100x larger than most open source projects. The size alone already excludes a large potential contributor base.

Then it takes 20+ minutes to compile Firefox on hardware that the typical developer has access to.

Then there are hard-to-use workflows revolving around Bugzilla, etc.

I don't think the flavor of version control is as significant a concern as other areas. I think short to medium term solutions to new-contributor funnel problems need to revolve around workflows that enable contributors to only interact with a narrow and focused subset of mozilla-central.

(Note that I'm explicitly not saying anything about git-cinnabar here: I just want to put problems in perspective.)
Duplicate of this bug: 1400907

Comment 7

2 months ago
(In reply to Gregory Szorc [:gps] from comment #5)
> A clone of https://github.com/mozilla/gecko-dev.git is ~3.5 GB, which is
> 10-100x larger than most open source projects. The size alone already
> excludes a large potential contributor base.

My experience has been with people who want to contribute to an open source web browser.  The other notable alternative, Chromium's repository is in the same order of magnitude, and is a bigger pain to clone.

> Then it takes 20+ minutes to compile Firefox on hardware that the typical
> developer has access to.

Much better than the competition (Chromium) which would typically take an hour on similar hardware.

> Then there are hard-to-use workflows revolving around Bugzilla, etc.

Which is why we walk new contributors through these processes all the time.  I think conflating unrelated issues here is unhelpful.

> I don't think the flavor of version control is as significant a concern as
> other areas. I think short to medium term solutions to new-contributor
> funnel problems need to revolve around workflows that enable contributors to
> only interact with a narrow and focused subset of mozilla-central.
> 
> (Note that I'm explicitly not saying anything about git-cinnabar here: I
> just want to put problems in perspective.)

Let me try to elaborate the problem.  New contributors who come to us who use git are usually in one of the two categories:

1. They have already found the gecko-dev repo on github.  If you look at https://github.com/mozilla/gecko-dev you will see that page has enough information to help get people started on making a build, creating a patch, etc.  So by the time they come and ping someone like me, they already have a local checkout and local changes on top of it which they may need to send to the try server.

2. They don't have a checkout because they haven't found the gecko-dev repo on github (because we call it "gecko-dev" and not "firefox"...).  They ask for help at this stage.


For the first category of people, if we require them to use git-cinnabar to push to try, they'd have to install and learn what git-cinnabar is, clone another repo into their local repo, figure out how to graft that into the gecko-dev repo, figure out what to do with their existing local branches, etc. etc.  Not a great solution.

For the second category of people, we'd need to tell them yes they can use git, but first thing they need to install and learn this new way to use git before they can get our code and build it.  How would you feel if the tables were turned around and we'd need Mercurial users to install and learn a new tool before they could type "hg clone https://hg.mozilla.org/mozilla-central"?  But more importantly, adding a road bump as the very first step seems wrong.


Therefore, irrespective of whether we have other issues attracting and retaining new contributors and when/if we're going to solve those issues, and given the popularity of git, this seems like an important issue to have a solution for.

Note that I'm not advocating for a specific solution here.  One possible solution, for example, would be publishing a github/mozilla/firefox repo that contains all of the git-cinnabar metadata required as a superset of the current gecko-dev, if that is (or can be made possible.)  But let's not erase the problem for now.  :-)
(In reply to Gregory Szorc [:gps] from comment #5)
> I don't think the flavor of version control is as significant a concern as
> other areas. I think short to medium term solutions to new-contributor
> funnel problems need to revolve around workflows that enable contributors to
> only interact with a narrow and focused subset of mozilla-central.

I've been kicking around the idea of a Firefox-focused Git repository that includes just browser/ and toolkit/, which developers could use with artifact builds to hack on Firefox.  The idea is to provide the benefits of artifact builds with the Git/GitHub-centric workflow that many web developers are familiar with.

That's worth pursuing, but somewhat orthogonal to this issue, which even experienced developers will encounter.

Comment 9

2 months ago
(In reply to Myk Melez [:myk] [@mykmelez] from comment #8)
> I've been kicking around the idea of a Firefox-focused Git repository that
> includes just browser/ and toolkit/, which developers could use with
> artifact builds to hack on Firefox.  The idea is to provide the benefits of
> artifact builds with the Git/GitHub-centric workflow that many web
> developers are familiar with.
> 
> That's worth pursuing, but somewhat orthogonal to this issue, which even
> experienced developers will encounter.

I am actively working on infrastructure to support these workflows. See my posts on the gofaster mailing list.
(In reply to :Ehsan Akhgari (needinfo please, extremely long backlog) from comment #7)
> Note that I'm not advocating for a specific solution here.  One possible
> solution, for example, would be publishing a github/mozilla/firefox repo
> that contains all of the git-cinnabar metadata required as a superset of the
> current gecko-dev, if that is (or can be made possible.)

I went back and read bug 1369562, and glandium notes in bug 1369562, comment 11:

> With next release of git-cinnabar, it will even be possible to publish the cinnabar metadata, so that one could even clone gecko-dev and get a cinnabar-ready clone.

Assuming we can graft cinnabar commits onto gecko-dev branches and start syncing gecko-dev using cinnabar, it should be possible to create a superset of the current gecko-dev that not only contains all of the cinnabar metadata but also copies it to clones.  Users would still need to install cinnabar, but the burden of using it would be significantly reduced.

Comment 11

2 months ago
(In reply to Myk Melez [:myk] [@mykmelez] from comment #10)
> Assuming we can graft cinnabar commits onto gecko-dev branches and start
> syncing gecko-dev using cinnabar, it should be possible to create a superset
> of the current gecko-dev that not only contains all of the cinnabar metadata
> but also copies it to clones.  Users would still need to install cinnabar,
> but the burden of using it would be significantly reduced.

From a technical perspective, that would eliminate the problem of having to install and learn git-cinnabar before being able to clone the repository, which is good.  For pushing to try, assuming the goal is to abstract things behind ./mach try then the complexity of installing and setting up cinnabar could be hidden inside ./mach try as well, I'd expect, but if users would need to install something then that's not the end of the world, since currently the same is true with moz-git-tools anyway.

But firstly we need to see what solution Mozilla is actually willing to implement and support.  gecko-dev started as an official Mozilla supported mirror with monitoring etc. and sometime between its inception and bug 1369562 comment 4 apparently somebody decided that it's not that important to keep maintaining, without announcement or discussion.  It would be nice to not end up in a similar position if we go with some new solution a few years down the road, which is why I am reluctant to propose solutions that Mozilla isn't interested in supporting.
(In reply to :Ehsan Akhgari (needinfo please, extremely long backlog) from comment #11)

> For pushing to try, assuming the goal is to abstract things
> behind ./mach try then the complexity of installing and setting up cinnabar
> could be hidden inside ./mach try as well

Yes, or ./mach bootstrap, which developers already run to acquire dependencies.


> But firstly we need to see what solution Mozilla is actually willing to
> implement and support.  gecko-dev started as an official Mozilla supported
> mirror with monitoring etc. and sometime between its inception and bug
> 1369562 comment 4 apparently somebody decided that it's not that important
> to keep maintaining, without announcement or discussion.  It would be nice
> to not end up in a similar position if we go with some new solution a few
> years down the road, which is why I am reluctant to propose solutions that
> Mozilla isn't interested in supporting.

Indeed, and I think the best route to get Mozilla interested in supporting a solution is to actually propose it, accompanying it with an explanation of why it's important and should be prioritized.

Sure, that might not work, or it might prompt Mozilla to support the solution for a time and then lose focus on it, as appears to have happened to gecko-dev (although perhaps others have a different perspective).

Nevertheless, a concrete proposal (which can include measures to ensure the solution remains supported) is still the best way to start a conversation (which can include a review of what happened to gecko-dev, and how to avoid that happening in the future).
Bug 1406792 is filed to make a more "official" cinnabar workflow.
See Also: → bug 1406792
You need to log in before you can comment on or make changes to this bug.