820955 - The master branch of gecko.git does not seem to be updated

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[(no longer active)]

The latest commit <http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=5bad6f1d595945a4b4acc1125447ec0669bf6474> corresponds to <http://hg.mozilla.org/mozilla-central/rev/99c2b5cf341c> which has been pushed two days ago <http://hg.mozilla.org/mozilla-central/pushloghtml?changeset=99c2b5cf341c>.

Comment 1

[u429623]

Ehsan - my understanding is that this is not affecting b2g work. As such, it will be a while before I can spend time on it.

If I misunderstand, please add the b2g impacting details.

Thanks!

Comment 2

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

It affects b2g only in that b2g 'master' builds are using the master branch.  These aren't the nightly builds (those are off the 18 branch), but there are people who are testing patches and doing development on mainline Gecko that need to do b2g-related work before porting.

We need to figure out the final full support policy for Mozilla's git infrastructure.

Comment 3

[(no longer active)]

I think Vlad is right, and this could in fact be b2g developers.  In general, I think we should assume that the gecko-18 branch (and any future similar ones) are special only in that build infrastructures may target them more frequently, but maintaining all branches updated should be a goal at all times.

Hal, my experience maintaining the current github mirror showed that it is valuable to move the update scripts towards working on each branch the same way and model the entire update job as updating the entire repository as a whole makes a lot more sense in attaining this goal.  That also helps with simplifying the entire update process which I think is a valuable goal to have especially when we're working towards a team the size of RelEng maintaining this infrastructure.

Pursuing this goal, I managed to decrease the number of scripts involved from 1 per branch to 1 total.  I would be more than happy to provide you with details on how that works now.

Comment 4

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Any movement on fixing this?  This does affect b2g, as we stated in comment #2 above, and bug 822970 reflects this.

Comment 5

[Justin Lebar (not reading bugmail)]

Note that B2G developers must develop patches which apply to m-c, aurora, and b2g18.

If the repository we're developing against doesn't have an up-to-date m-c, we're obviously affected.

So basically anyone who tries to develop for B2G using git is affected here, unless they use Ehsan's repository.  I suppose as a practical matter that most of us use Ehsan's repository, since that has been the most consistently reliable over the past few months.  But if the idea is to get us to switch, letting problems like this sit certainly won't inspire confidence in your constituents...

Comment 6

[u429623]

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #4)
> Any movement on fixing this?  This does affect b2g, as we stated in comment
> #2 above, and bug 822970 reflects this.

Vlad - thanks for clarifying that you do see this as a b2g issue. (I'd read comment 2 as "could impact", not "does impact".)

bug 822970 shows there are some communication gaps - we were told b2g was on "nightly" quite a while back.  I'll get that clarified, and get a prioritization on this work, then update the bug.

Comment 7

[Justin Lebar (not reading bugmail)]

I sent e-mail, but to say the same thing publicly:

The master and aurora branches on git.m.o appear to be three weeks out of date now.

Unless there's a plan to fix this bug within the next few days, it seems to me that we could clear up some confusion about the purpose of git.m.o/releases/gecko.git by deleting these unmaintained branches.  I think we should also write to dev.platform and dev.b2g explaining the situation and how to work around it.

I'm happy to do the latter if that helps free you guys up to do the former.

Comment 8

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Is something disastrously wrong with the git setup that it would take days to fix?  It seems like we've collectively spent hours on this now, for what could potentially have taken 15-30 minutes to fix.

Comment 9

[u429623]

My bad for not posting the prior email here - I'm doing that first, then I'll ask another question for clarification.

:joduinn sent an email on Dec 21 updating the status of the git related bugs:

-------- Original Message --------
Subject: 	priorities with git repos
Date: 	Fri, 21 Dec 2012 14:31:16 -0800
From: 	John O'Duinn <[e-mail]>
Reply-To: 	[e-mail]
To: 	Ehsan Akhgari <[e-mail]>, Vladimir Vukicevic <[e-mail]>, Justin Lebar <[e-mail]>, Hal Wine <[e-mail]>, Chris AtLee <[e-mail]>, Dave Hylands <[e-mail]>


hi ehsan, vlad, jlebar, hwine, catlee, dhyland;


There's been a lot of back/forth, in bugs (bug#820955, bug#822970 are 
recent examples), various email threads and also in irc; I'm noticing 
that the tone/frustrations are degrading, between people who all care 
about Mozilla, so am stepping in here. I note that I'm also hearing 
various different incompatible assertions being made, so I thought it 
would be helpful to step back, pause, and make sure we are all on the 
same page.


Priorities as I see them:

1) all B2G shipping requirements for jan15. These criteria change 
frequently and include:
* gecko/gonk/gaia repos on git.m.o for partner handover - completed 
w/partner last night (20dec).
* b2g v1.0 builds for all devices - wrapping up gecko+gaia replication
* b2g v1.1 builds for all devices - wrapping up gecko+gaia replication
* l10n repos for v1.0 and v1.1 - in progress
* repos for global apps included w/b2g v1.0 and v1.1 - starting

2) any git repo setup for b2g developers
* noting that all gaia developers do checkins on github
* noting that all gecko developers do checkins on hg
* noting that gecko developers who have a preference for using git for 
development, would find git replications from hg helpful to streamline 
their development process, before they commit to hg.

3) any git repo setup for mozilla-wide developers
* we need a long term owner for the hg->github repo replications. This 
is something that RelEng is happy to do and support in a scalable way. 
I've already said that before, but repeating here for the record.
* we've already spent the time to detangle all the cvs-hg merge history, 
to make a clean setup that everyone on this thread agrees is an 
improvement on what we've had before.


As I see it, meeting all the shipping criteria for B2G v1.0 (priority#1) 
is more important to Mozilla then anything in priority#2 or priority#3. 
Once we have cleared all shipping criteria, I'm more then happy to have 
RelEng switch attention to setup systems which help 
git-preferenced-developers streamline their workflow. However, I believe 
nothing in #2 or #3 actually *blocks* b2g or wider Firefox development 
work.

If I am missing something, and these priorities need to change, please 
let me know.

If the above summary is correct, then I would ask we all focus together 
on clearing ship-blocking issues in priority#1. Any and all help here 
would be great. I'd be happy to post status on priority #1 work every 
week if that helps keep people in sync with our workload, and help set 
expectations.


thanks
John.

Comment 10

[(no longer active)]

(In reply to comment #9)
> My bad for not posting the prior email here - I'm doing that first, then I'll
> ask another question for clarification.
> 
> :joduinn sent an email on Dec 21 updating the status of the git related bugs:

Hmm,,...  is that an answer to comment 8?

Comment 11

[Justin Lebar (not reading bugmail)]

I spoke with John and Hal today over vidyo.

One of the things that they said was that they intentionally stopped updating the master and aurora branches in git.m.o/releases/gecko.git.

So as I understood our conversation, this bug did not occur by accident, but was instead caused intentionally.  But if that's true, I can't figure out why Hal didn't mention this in comment 1 and why John didn't mention it in the e-mail from comment 9.  Perhaps I'm misunderstanding something.

Comment 12

[(no longer active)]

Hmm, I never got the impression that this has been intentional given the prior comments in this bug.  Even in that case, these branches should be deleted if they're not going to be kept updated.  Otherwise this will continue to confuse the developers who come across this repository.

Comment 13

[Justin Lebar (not reading bugmail)]

> Even in that case, these branches should be deleted if they're not going to be kept updated.

I totally agree, but we need to ensure that we don't break things further when doing so.  The "master" branch of B2G's manifests points to the master branch of git.m.o/releases/gecko.git -- if we delete the branch from gecko.git, will that break those using this manifest branch further?  Should we delete the manifest branch?

Frankly, I'm inclined to let sitting dogs lie.  Every change we ask for here has turned into an ordeal, and the potential gain here by reducing confusion is modest.

Comment 14

[(no longer active)]

(In reply to comment #13)
> I totally agree, but we need to ensure that we don't break things further when
> doing so.  The "master" branch of B2G's manifests points to the master branch
> of git.m.o/releases/gecko.git -- if we delete the branch from gecko.git, will
> that break those using this manifest branch further?  Should we delete the
> manifest branch?

The "right" thing to do here would be to keep all of the existing branches updated, of course.

> Frankly, I'm inclined to let sitting dogs lie.  Every change we ask for here
> has turned into an ordeal, and the potential gain here by reducing confusion is
> modest.

I'm really interested to know why the decision to stop updating these branches was made, and why it was never communicated in this bug.

This all being said, I'm personally going to stop pursuing things further here.  The fact that it took us three weeks to figure out that this was intentional is an indication that it's probably not worth trying to improve things here in a timely manner.

Comment 15

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Justin Lebar [:jlebar] from comment #11)
> I spoke with John and Hal today over vidyo.
(thanks again jlebar, for meeting us on Friday evening!)


> One of the things that they said was that they intentionally stopped
> updating the master and aurora branches in git.m.o/releases/gecko.git.
> 
> So as I understood our conversation, this bug did not occur by accident, but
> was instead caused intentionally.  But if that's true, I can't figure out
> why Hal didn't mention this in comment 1 and why John didn't mention it in
> the e-mail from comment 9.  Perhaps I'm misunderstanding something.
Sorry for not communicating this better, it has been super-hectic here with all the B2G changes. As hwine+myself said with jlebar in that Friday night meeting, we disabled this intentionally at the time to avoid potential problems with the replication/convertion scripts for anything not immediately blocking B2G. I should have communicated that more widely. 

As an example of this: 48 hours after this meeting, we discovered that the replication had broken, causing tree-closing disruption at the start of the B2G workweek in Berlin. As this was the same weekend that we were doing the migrations for FF18 release, it was very helpful for us to be able to clearly rule out any disruption caused by train-migrations-in-progress. With this reduced scope, we found and fixed the problem (caused by a network hiccup) in a few hours. 

We've had time to make the conversion scripts, and error reporting, be more robust. Additionally, since branching last Friday for 1.0.0, this "master" branch in gecko.git is now needed as part of our handover for v1.1 builds, so is blocking bug#820955. 

Therefore we're re-enabling this today. Expect to see this caught up to tip of master within the next 24hours, at which point "master" would be maintained with the same level of alert+monitoring as the other branches.

Comment 16

[John Ford [:jhford] CET/CEST Berlin Time]

Thanks for the update and resuming the mirroring!

Comment 17

[Panos Astithas (he/him) [:past] (please ni?)]

(In reply to John O'Duinn [:joduinn] from comment #15)
> Therefore we're re-enabling this today. Expect to see this caught up to tip
> of master within the next 24hours, at which point "master" would be
> maintained with the same level of alert+monitoring as the other branches.

Will the workaround from bug 822970 be reverted at that point?

Comment 18

[u429623]

In production and stable for several days now. Nothing left to do here.

Comment 19

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Panos Astithas [:past] from comment #17)
> (In reply to John O'Duinn [:joduinn] from comment #15)
> > Therefore we're re-enabling this today. Expect to see this caught up to tip
> > of master within the next 24hours, at which point "master" would be
> > maintained with the same level of alert+monitoring as the other branches.
> 
> Will the workaround from bug 822970 be reverted at that point?

Good question, and yes! This has already been done in https://bugzilla.mozilla.org/show_bug.cgi?id=835467#c9.