API Rate Limit Exceeded for 63.245.214.162 for requests to https://api.github.com

RESOLVED FIXED

Status

RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: jgriffin, Assigned: cransom)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

6 years ago
Autolog (http://brasstacks.mozilla.com/autolog/) performs queries against github for the B2G project.  Currently, these are failing with this message from github:

curl https://api.github.com/repos/mozilla/releases-mozilla-central/git/commits/
{
  "message": "API Rate Limit Exceeded for 63.245.214.162"
}

What is 63.245.214.162?  Brasstacks is 63.245.215.33.  Can we figure out what the source of these excessive queries is?

Comment 1

6 years ago
Lawrence, Dietrich, any ideas on what could be automatically hitting github from that IP address?

Hwine, could this be one of the releng mirrors?
(Assignee)

Comment 2

6 years ago
I'll be working on this momentarily. The source address is a nat address on the scl3 firewall so it could be any number of machines.
The only code that I know hits github is the daily pull of basecamp+ github issues to generate the csv files that are used in the basecamp dashboard.
(Assignee)

Comment 4

6 years ago
Created attachment 672814 [details]
traffic seen for brasstacks to api.github.com

Just a quick note to understand the traffic flow - 
For brasstacks, incoming traffic is via a load balancer for external 80/443 traffic and outgoing traffic is being natted to the general NAT address (.162).  This is why you are seeing connections from brasstacks coming from a different adress than what it is addressed as externally.

I looked at netflow data and the only host hitting api.github.com from the NAT address is brasstacks.

I attached the traffic seen from the firewall for timestamp comparisons, stamps are in pacific time.  Let me know how you want to proceed.
(Assignee)

Updated

6 years ago
Assignee: network-operations → cransom
(Reporter)

Comment 5

6 years ago
Perhaps autolog is getting more hits than it has in the past.  If that's the issue, the only way around it that I can think of is to implement a local cache of the github queries.
(Assignee)

Comment 6

6 years ago
this request is complete from the netops end.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.