743626 - Provide git repo for mozilla-central/comm-central & other relevant repos

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, enhancement, P5)

Description

[Ben Bucksch (:BenB)]

I contribute to a number of projects, and have many of my own. Many Open-Source projects are using SVN, others (maybe even most by now) have advanced to git. hg is the only project I contribute to that uses hg.
Different VCS, while generally modeled around very similar concepts, are very different in subtle way, which make the daily use very different. I find myself struggling *for hours* each time I want to commit something to mozilla-central, and it doesn't seem to get better for me. I am just fed up with having to deal with 3-4 similar, yet different tools for the same thing.

hg and git are similar enough to make an automated sync possible:
* A public mozilla-central git repository that contains all changesets and history of hg mozilla-central. Ditto the other important branches like mozilla-inbound, releases/ and comm-central.
* A public mozilla-inbound git repository that, when pushed to, invokes a hook to commit the same changeset to hg mozilla-inbound, using the same committer, so that I can contribute using git.

This should be possible without much trouble. In fact, somebody [1] has already done the first part.
The advantage is:
1) Giving much easier access to Mozilla for new contributors (who probably know git, but not hg)
2) Allowing a smooth transition to git, or co-existance, allowing everybody to use the tool and workflow they prefer, forcing neither hg nor git on anybody.
3) Allowing better re-integration of those Mozilla lab projects that are already using git, e.g. Jetpack and others.

[1] http://www.bluishcoder.co.nz/2011/02/10/git-conversion-of-mozilla-central.html
http://bholley.wordpress.com/2010/10/23/using-git-with-mozilla/
http://www.bluishcoder.co.nz/2011/04/16/my-git-workflow-for-mozilla-development.html

Comment 2

[Ben Bucksch (:BenB)]

I don't think this is a DUP. That bug asks to have a git server in production.
This bug here asks to have mozilla-central in git, including commit ability.

Comment 3

[Justin Lebar (not reading bugmail)]

> That bug asks to have a git server in production.
> This bug here asks to have mozilla-central in git, including commit ability.

Honest question: Supposing we fixed this bug, what would be left to do in bug 713782?

Comment 4

[Justin Lebar (not reading bugmail)]

See also http://jlebar.com/2012/3/18/Git_tools_for_Mozilla.html

Comment 5

[Ben Bucksch (:BenB)]

> Supposing we fixed this bug, what would be left to do in bug 713782?

This bug depends on bug 713782. First a git server, then the m-c sync described here.

Comment 6

[(no longer active)]

While this bug gets fixed, you can use https://github.com/mozilla/mozilla-central.  As a bonus, you'll get all of the history of the project (CVS + Mercurial history) in that repository.  :-)

Comment 7

[Ben Bucksch (:BenB)]

Whoo! :-) Thanks!

(I can't commit there, though, right?)

Comment 8

[Justin Lebar (not reading bugmail)]

(In reply to Ben Bucksch (:BenB) from comment #5)
> > Supposing we fixed this bug, what would be left to do in bug 713782?
> 
> This bug depends on bug 713782. First a git server, then the m-c sync
> described here.

I'm not sure that hwine sees bug 713782 as "make a git server".  Indeed, he owns the metabug and he marked this bug as a dupe of that one, so that's pretty strong evidence that he sees bug 713782 as encompassing the functionality you described in comment 0.

Perhaps we could explain in bug 713782 that, by "support git on parity with hg", we mean a superset of the functionality described in comment 0.  Then we could re-dupe this bug.  Would you be OK with that, Ben?

Comment 9

[Justin Lebar (not reading bugmail)]

> (I can't commit there, though, right?)

Right, I use the link in comment 4 for pushing from git to hg.  It's not hard, and I'm happy to help you get started.

Comment 10

[Ben Bucksch (:BenB)]

BTW: ehsan, who's pushing to https://github.com/mozilla/mozilla-central ? Is this described somewhere? I'd like the same for comm-central.

Several bugs linked to bug 713782 state "r/o mozilla-central".
This bug here is not just about checkout, but commit to git.

Thanks for the offer, Justin, I really appreciate it and the spirit. However, I didn't want want a git-hg bridge on my machine and push to a hg repo, as I'd still deal with hg, and that wouldn't help me, but I think only complicate matters. My problems are mostly the "you're not uptodate", "2 heads" stuff and other arcane hg particularities that I keep running into. I am somewhat familiar with how git manages branches and commit etc., but with a git-to-hg bridge, I need to know git branches, hg branches, and how the bridge works, so it just adds complexity.

I'd like to work exactly as with git, not a hg that looks like git, but is somewhat different in various complex situations.

Comment 11

[Justin Lebar (not reading bugmail)]

> I'd like to work exactly as with git, not a hg that looks like git, but is somewhat different in 
> various complex situations.

I understand.  To be clear, I'm trying to give you something you can use now, rather than something you might be able to use in six months.

> BTW: ehsan, who's pushing to https://github.com/mozilla/mozilla-central ? Is this described 
> somewhere?

It's updated via a script.

> Several bugs linked to bug 713782 state "r/o mozilla-central".
> This bug here is not just about checkout, but commit to git.

Perhaps hwine could clarify for us what he wants bug 713782 to track.  To me, "support git on parity with hg" does not mean "support read-only git repositories", and I thought hal was on the same page as me.  The fact that his metabug depends on bugs for r/o m-c doesn't mean anything -- read-only is a step towards read-write.

Comment 12

[Ben Bucksch (:BenB)]

> > BTW: ehsan, who's pushing to https://github.com/mozilla/mozilla-central ? Is this described 
> > somewhere?
>
> It's updated via a script.

I figured :). But
1) who wrote it?
2) on which machine does it run?
3) where can I get the script?

I'm asking because of bug 743760.

> bug 713782 to track

... and this bug is a blocker for bug 713782, so bug 713782 does track :)

Whatever - as long as it gets done...

Comment 13

[Hal Wine [:hwine] use NI!]

Ben - the (long term) plan is to have only one writable repository to avoid many nasty race conditions. The shorter term goal in bug 713782 is to make those hg nasties as invisible as possible.

Re: comm-central - please put that in a separate bug that is blocked by bug 713782 - we (release engineering) will be rolling out additional repos shortly (within Q2).

Comment 14

[Ben Bucksch (:BenB)]

> Re: comm-central - please put that in a separate bug that is blocked by bug 713782

Already done so: bug 743760. ;-)

Comment 15

[(no longer active)]

(In reply to Ben Bucksch (:BenB) from comment #12)
> > > BTW: ehsan, who's pushing to https://github.com/mozilla/mozilla-central ? Is this described 
> > > somewhere?
> >
> > It's updated via a script.
> 
> I figured :). But
> 1) who wrote it?

I did.

> 2) on which machine does it run?

On my own machine!  :-)

> 3) where can I get the script?

See <https://github.com/ehsan/mozilla-history-tools/tree/master/initial_conversion> for how I did the initial conversion, and <https://github.com/ehsan/mozilla-history-tools/blob/master/updates/update.sh> for how I keep it updated.

Please let me know if you have any questions.

FWIW, you can use Justin's script to push to the hg m-c repo, but for the time being https://github.com/mozilla/mozilla-central is a read-only copy.

Comment 16

[Ben Bucksch (:BenB)]

Thanks, reed! Also thanks for setting this up! :-)

---

Talked with hwine and jlebar on IRC, and we agreed to keep this bug as blocker for bug 713782.

Comment 17

[Reed Loden [:reed]]

(In reply to Ben Bucksch (:BenB) from comment #16)
> Thanks, reed! Also thanks for setting this up! :-)

I think you meant somebody else...

Comment 18

[Ben Bucksch (:BenB)]

Indeed, I meant ehsan, sorry. Thanks, ehsan!

Comment 19

[Nicolas B. Pierron [:nbp]]

I have a server which should only be visible to employee (for security reason, at least for now), If you send me your public key and your hg-username (email) you should be able to interract with it with the following instructions:

- To clone:
git clone [e-mail]:mozilla-central.git

- To push to the try server:
git remote add try [e-mail]:try.git
git commit -m "try: …"
git push -f try HEAD:master

- To get the corresponding changeset/commit
ssh hydra.mv.mozilla.com commit-to-changeset <git-commit>
ssh hydra.mv.mozilla.com changeset-to-commit <hg-changeset>

- To update the remote repository (done automatically every 20 minutes):
ssh hydra.mv.mozilla.com hg-fetch

- List all supported repositories:
ssh hydra.mv.mozilla.com list

As of today, this give you pull/push for ionmonkey, mozilla-inbound, mozilla-central and the try server.

Sources:
https://github.com/nbp/spidermonkey-dev-tools/tree/master/git-hg-bridge

Comment 20

[Shyam Mani [:fox2mike]]

(In reply to Nicolas B. Pierron [:pierron] from comment #19)
> I have a server which should only be visible to employee (for security
> reason, at least for now), If you send me your public key and your
> hg-username (email) you should be able to interract with it with the
> following instructions:

No he can't.

[root@mvadm01 ~]# host hydra.mv.mozilla.com
hydra.mv.mozilla.com has address 10.250.1.241

That is an internal IP, not accessible to the world.

Comment 21

[Ben Bucksch (:BenB)]

Yes, no worries, I'll wait for this bug here to be implemented as specced in comment 0.

Comment 23

[Mark Banner (:standard8)]

Adding comm-central to the title per bug 743760. Also expanding this as if we do this, then it should just cover whichever repos are relevant at the time, rather than one bug per repo.

Comment 24

[Shyam Mani [:fox2mike]]

Moving this to releng/hwine.

Comment 25

[Hal Wine [:hwine] use NI!]

(In reply to Mark Banner (:standard8) from comment #23)
> Adding comm-central to the title per bug 743760. Also expanding this as if
> we do this, then it should just cover whichever repos are relevant at the
> time, rather than one bug per repo.

Define "relevant", please - there are 681 non-user repos on hg.m.o. :) I opened bug 770592 for that list for the ThunderBird project. ("project/product" feels like the right granularity for these lists.)

Comment 26

[Hal Wine [:hwine] use NI!]

This bug has wandered all over the place. As the reporter has said in comment #21:
> Yes, no worries, I'll wait for this bug here to be implemented as specced in
> comment 0.

Based on that, this doesn't block any of the work to date, and is an independent request to be prioritized. Removing blockers

Comment 27

[Ben Bucksch (:BenB)]

hwine, with "I'll wait", I didn't mean to say that I don't care about this bug. The "no worries" was pierron and fox2mike only.

If you're not intending to do this, please allow somebody else do it. I've offered before that I can set this up. Give me 2 empty Linux machines, and I'll do it.

Comment 28

[Ben Bucksch (:BenB)]

(This is an open project after all, isn't it?)

Comment 29

[Mark Banner (:standard8)]

(In reply to Mark Banner (:standard8) from comment #23)
> Adding comm-central to the title per bug 743760. Also expanding this as if
> we do this, then it should just cover whichever repos are relevant at the
> time, rather than one bug per repo.

Considering it is just a list of repositories, I don't see any point in project specific bugs when both are desired from the start... but if you want to track it that way, that's fine.

Comment 30

[Ben Bucksch (:BenB)]

Mark, I had already filed bug 743760 back then. It's also in the deps list here.

Comment 31

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage.

Comment 32

[Hal Wine [:hwine] use NI!]

git mirror of comm-central (without cvs history) is available at https://github.com/mozilla/releases-comm-central

Assuming that addresses this bug. Please open a new bug for any changes.

Comment 33

[Ben Bucksch (:BenB)]

> Assuming that addresses this bug.

It doesn't. This ticket was specifically asking for commit/push.

It fails this spec in comment 0:
> * A public mozilla-inbound git repository that, when pushed to, invokes a hook to commit
> the same changeset to hg mozilla-inbound, using the same committer, so that I can
> contribute using git.

Therefore, it fails these important stated goals:
> 1) Giving much easier access to Mozilla for new contributors (who probably know git, but not hg)
> 2) Allowing a smooth transition to git, or co-existance, allowing everybody to use the tool and
> workflow they prefer, forcing neither hg nor git on anybody.

Contributors can still not use git, because they cannot commit/push.

Comment 34

[Ben Bucksch (:BenB)]

Filed bug 1271013 about push.