799843 - github copy of releases-mozilla-aurora is not updating

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

The copy of mozilla-aurora that is being maintained by releng does not currently have Gecko version 18.  My understanding is that this repository is supposed to be updated very frequently, but this repository hasn't updated for around 12 hours.

The new version of aurora landed on hg.m.o at around noon:
http://hg.mozilla.org/releases/mozilla-aurora/rev/3bfbc73d1b58

The version of aurora on releases-mozilla-aurora is from 6 weeks ago:
https://github.com/mozilla/releases-mozilla-aurora/commit/0ed85425bb4f07d7ca6b622a9114121636293117

My understanding is that the repository syncing for this is supported and alerts on failure with intervals of minutes, not days.  Please let me know if this is intended or a problem with the syncing.  My concern is both the delay and the possibility the mirroring is not producing a true copy of the hg.m.o contents.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

hwine knows this area best, so lets start by triaging to him.

Comment 2

[Justin Lebar (not reading bugmail)]

We need this pretty badly; at the moment, every B2G dev is developing against the wrong branch.

If you guys can't fix this in the next day or two, we can switch to Ehsan's repository, which has an up-to-date m-a clone.  Switching would be painful because the releng repository does not match the hashes in Ehsan's repository, but would have ergonomic benefits due to the fact that Ehsan's one repository contains all our branches.  (Among the benefits of this: We are getting somewhat bewildered complaints from our partners about the fact that our branches are spread out among multiple repositories, which is highly non-canonical for git, and switching to Ehsan's repository would mitigate that confusion for our partners as well as for Mozilla devs.)

Given that Ehsan has in general been more responsive to ergonomic concerns from developers than hwine (with respect specifically to full CVS history and the canonical one-repository setup) I personally wouldn't object to eating the pain of switching back to Ehsan's repository, if it's going to take a few days for releng to get their m-a repository back up and running.

Please let us know what time-frame we can expect here, since that will inform our decision.

Comment 3

[Hal Wine [:hwine] use NI!]

Problem has been identified: system is still processing the gexport from changes on merge day. (Working since Oct 8.) Delay is due to repository being on NFS - will work with IT to see what can be done.

Bumping priority

Comment 4

[Hal Wine [:hwine] use NI!]

Verified with IT that neither process nor NFS access throttled.

Beginning process of moving repository to local disk on another machine. This was going to be the long term solution in any case. 

Conversion will continue until final steps - I'll update bug before doing "cutover".

Comment 5

[Hal Wine [:hwine] use NI!]

update: expected repository move to be completed approx 1500PT

Will update more at that time.

Comment 6

[Hal Wine [:hwine] use NI!]

update: initial rsync completed, initial conversion still underway. Decision taken to halt initial conversion and move to new machine on local disk.

Final rsync underway, will update when conversion restarted on new machine.

Comment 7

[Hal Wine [:hwine] use NI!]

Conversion restarted - data on github in sync with data on hg.m.o