1369562 - github gecko-dev sync job seems to be down

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[(no longer active)]

The last update is https://github.com/mozilla/gecko-dev/commit/972b7a5149bbb2eaab48aafa87678c33d5f2f045 from ~12 hours ago.  I think this corresponds to https://hg.mozilla.org/mozilla-central/rev/0bcea6bac1797e14b00af45cc7c368d12460ab7f which would mean that update jobs haven't been running since some time this morning.

Comment 1

[:dhouse]

I'm seeing updates from 3 hours ago in gecko-dev.

3efd1caff252061946f0773df68bdd50de5eb756 - no bug - Bumping Fennec l10n changesets r=release a=l10n-bump
from:
d69d09cff6dc 2017-06-02 06:00 -0700 	L10n Bumper Bot - no bug - Bumping Fennec l10n changesets r=release a=l10n-bump

efccdfdb0a86ef5a10790f8c501b416a6384a3ef - merge mozilla-inbound to mozilla-central a=merge
from:
2a8478029a0c 2017-06-02 14:22 +0200 	Carsten "Tomcat" Book - merge mozilla-inbound to mozilla-central a=merge

Comment 2

[:dhouse]

There were failures pushing changes yesterday, but it recovered automatically (slowly, but eventually). First failure was with mozilla-inbound at 7:55 pacific time yesterday. m-c then failed at 12:18 pacific. Recovery completed at 21:28 pacific (last night).

Comment 3

[(no longer active)]

Yeah, looks like it, thanks.

Is there no automatic failure reporting on the conversion jobs?  I was blocked on this for a few hours since I was trying to pull some code from github...

Comment 4

[Gregory Szorc [:gps]]

gecko-dev github sync is not a very well monitored service and it seems to have a best effort SLA. If you want a robustly monitored service with an SLA, you should use hg.mozilla.org via git-cinnabar.

If you want this service to be better monitored and have an SLA, I /think/ Amy Rich is the person you should take it up with.

Comment 5

[:dhouse]

gecko-dev is again lagging behind since last night. github is closing the push connections before they complete. I've contacted github support and asked if they can check on their side.

:kats raised the issue in #vcs
we discussed git-cinnabar according to glandium's gecko-dev workflow notes: https://github.com/glandium/git-cinnabar/wiki/Mozilla:-Using-a-git-clone-of-gecko%E2%80%90dev-to-push-to-mercurial

Comment 6

[Kartikaya Gupta (email:[e-mail])]

For the record, I tried setting up git-cinnabar today and ran into problems. I filed https://github.com/glandium/git-cinnabar/issues/132 for it, but for now I'm blocked on gecko-dev.

It might be a good idea to just switch gecko-dev to being updated by cinnabar, if that would be more reliable in the long term.

Comment 7

[:dhouse]

I'll stop the sync and try a manual push. I'm letting the current sync run until complete(failure) and then I'll do the manual push to see if github does not disconnect then (*should be the same and fail, but worth trying I think).

Comment 8

[Marco Castelluccio [:marco]]

For code coverage, we are using codecov.io and coveralls.io which need a git repository on GitHub in order to show source files. So I guess we can't use git-cinnabar for this use case.

Comment 9

[Mike Hommey [:glandium]]

The point kats was making is to use git-cinnabar to update gecko-dev forward, instead of hg-git. That would have multiple advantages.

Comment 10

[Marco Castelluccio [:marco]]

(In reply to Mike Hommey [:glandium] from comment #9)
> The point kats was making is to use git-cinnabar to update gecko-dev
> forward, instead of hg-git. That would have multiple advantages.

Yeah, I completely missed his last sentence. If it's more reliable, it makes perfect sense. I should note that over the past few months we haven't had many problems with the sync (and I'm basically checking almost every day), but after the first failure (around one month ago) we started seeing failures more often (this is an empirical observation, I might be wrong).

Comment 11

[Mike Hommey [:glandium]]

There are other advantages than reliability. It would likely be faster, and it would make the world unified for people using gecko-dev + git-cinnabar (that is, they wouldn't diverge from gecko-dev anymore). With next release of git-cinnabar, it will even be possible to publish the cinnabar metadata, so that one could even clone gecko-dev and get a cinnabar-ready clone.

Comment 12

[:dhouse]

My manual push of gecko-dev worked.

I moved the vcssync to a larger instance to see if more cpu/memory/shares would change something. I left gecko-projects for the automatic sync for comparison since both were failing before the upgrade.

Comment 13

[:dhouse]

gecko-projects succeeded also (on cron). So perhaps the larger instance fixed things, or stopping the retries on the push got past github throttling (every 20-30 minutes does not seem often enough to throttle however). I'd like to see the github side of things :/ to know.

I'll leave vcssync running on the larger instance for a couple of weeks to see if that avoids the problems we have been seeing.

Comment 14

[:dhouse]

The sync looks good from the runs so far on the larger instance. System monitoring shows higher network out when we push (up from spikes at 40mb/s to 80mb/s). We'll see if, when there are more commits to push, it finishes faster and does not hit a possible timeout on github's side. Here's from our logs, showing a minimal push this morning taking 5 seconds:

14:38:35     INFO - Calling ['git', 'push', '[e-mail]:mozilla/gecko-dev.git', '+refs/heads/master:refs/heads/master'] with output_timeout 5400
14:38:40     INFO -  To [e-mail]:mozilla/gecko-dev.git
14:38:41     INFO -     63b01ad..186c6d7  master -> master

Comment 15

[:dhouse]

github support replied to me yesterday morning: They explained that they do not keep logs for more than a couple of hours. So they could not review the failures. Yesterday I set up a smaller (matching from before I changed it last Friday) vcssync machine to run the sync. When the push fails again, I will notify github and leave it running for them to review their recent logs to see if they are closing the connection.

Comment 16

[:dhouse]

No failures yet on the larger host. I'll close this out for now until we hit a limit or timeout again.