853124 - Consistent failures talking to GitHub from phx

Status: RESOLVED INVALID

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[James Socol [:jsocol, :james]]

SUMO has been experiencing a consistent failure pattern talking to GitHub from both CI[1] and Chief[2]. We try, get a failure, try again, and it works. It looks like the failure is the same every time and is a failed `git fetch`.

[1] https://ci.mozilla.org/job/sumo-master/2265/console
[2] http://supportadm.private.phx1.mozilla.com/chief/support.stage/logs/66a3dc0eeb46ddca980d4f641e85a10ac7d6788c (The first 8 lines are the first attempt failing, then the rest is the successful second attempt.)

Comment 1

[James Socol [:jsocol, :james]]

:rbryce informs me that this is probably GitHub's fault and there's nothing we can do. I wish their status page would include info like this.

Maybe we should file a GitHub support ticket.

Comment 2

[casey ransom [:casey]]

I don't have a great deal of information to go on as we dont' do a large amount of traffic to github, but i don't have any network based explanation for your failures.  network traffic between jenkins1.dmz.phx1 and github has been steady for the past week and if its just one repo thats failing (i assume there is more than just kitsune), it's likely to be a very localized github issue.  

http://netops2.private.scl3.mozilla.com/nfsen/nfsen.php?bookmark=MnwwfHBoeDEvamVua2lucy1naXRodWJ8amVua2luczEtaW4hamVua2luczEtb3V0fGFueXxwYWNrZXRzfDV8MHwxMzYzODE1MzAwfDEzNjM4MTUzMDB8MTM2MzgxNTMwMHwwfDF8MA%3D%3D

Comment 3

[James Socol [:jsocol, :james]]

Github support helped us walk through the issue, it was with our repo.

Thanks and sorry for the noise.