1362350 - Git mirror https://github.com/mozilla/gecko-dev stuck at two days ago

Status: RESOLVED FIXED

(Developer Services :: Git, enhancement)

Comment 1

[Kartikaya Gupta (email:[e-mail])]

https://github.com/mozilla/gecko-projects (graphics branch) is also stuck

Comment 2

[:dhouse]

The vcssync of beagle is timing out. It looks like the cleanup to remove b2g from that job did not apply on the sync server's beagle. So first I'm checking/applying that to reduce the workload for the job. If that doesn't work, I'll manually increase the timeout until we catch up.

Comment 3

[:dhouse]

Production mozharness had some changes (looks like http->https) which were not in default and that appears to have prevented the b2g/etc cleanup from sticking. I've merged production into default and reapplied the cleanup. https://hg.mozilla.org/build/mozharness/rev/f363fe059fb8808e5452c802e3b7948fbbf2430c

The latest sync run is using this updated configuration; We'll see if it completes before the timeout. If not, I'll run the next sync with a longer timeout.

Comment 4

[:dhouse]

The cleanup alone did not give it enough time. Extending the timeout, I'm seeing changesets converted. The prep for export from hg took 40 minutes. The conversion to git has reached May 1st and continues running. However, I doubled the timeout from 30 minutes to 60 minutes and so it will soon run out of time. On the next run I expect for it to resume from there and catch up, but we could still expect a long prep (hopefully a lot less than 40 minutes) so I'm doubling the timeout to 120min for the next run.

```
17:15:27     INFO - Calling [[...] 'gexport'] with output_timeout 3600
17:56:55     INFO -  exporting hg objects to git
17:56:55     INFO -  converting revision 69ceb4c8dbb84e0e9c1a0869eb60499681aca92d [2017-04-18 12:46 +0100]
[...]
17:56:55     INFO -  converting revision c06d46d320545f2ee43bc3b99ce19bea6e1e5d69 [2017-05-01 14:22 -0400]
```

Comment 5

[:dhouse]

Now it has caught up to commits from yesterday:
```
[...]
18:32:06     INFO -  converting revision 5f2914fb211dafdc43d26087e68f174384526156 [2017-05-04 09:38 +0100]
[...]
18:33:08     INFO -  converting revision d3b488ac423bd50913e14ea5340c98387ece9523 [2017-05-04 08:04 -0700]
```
Once it finishes all of the conversion the push to github will happen.

Comment 6

[Marco Castelluccio [:marco]]

Any news? It's still stuck for now.

Comment 7

[Marco Castelluccio [:marco]]

I'm seeing newer commits now.

Comment 8

[:dhouse]

:) It took a long time for it to catch up, but the cleanup (removal of old esr branches and b2g, which still took some time despite no updates) and the extended timeouts allowed the sync enough time.

I'm closing this bug and let's hope this doesn't repeat in a few weeks.

Comment 9

[:dhouse]

We are seeing the same timeouts on gecko-projects today. I'm reopening this bug to perform the same cleanup and timeout changes on gecko-projects to try to fix the problem (short term).

Comment 10

[:dhouse]

I extended the timeout limits for the conversion and push. It looks like some of the branch conversions are taking more than an hour (standard timeout is 30 minutes). So they are catching up -- waiting to see and confirm that we get the graphics branch caught up.

Comment 11

[Kartikaya Gupta (email:[e-mail])]

Seems to have caught up now, thanks!

Comment 12

[:dhouse]

:kats, Thanks for checking! Also the sync logs don't show any more timeouts for the project branches. I'm closing this for now.