Last Comment Bug 459919 - GeoIP Bouncer, turn on for world
: GeoIP Bouncer, turn on for world
12/17/2009 @ 7pm
Product: Graveyard
Classification: Graveyard
Component: Server Operations (show other bugs)
: other
: All Other
: P2 minor (vote)
: ---
Assigned To: Jeremy Orem [:oremj]
: matthew zeier [:mrz]
Depends on: 395950
  Show dependency treegraph
Reported: 2008-10-14 14:07 PDT by matthew zeier [:mrz]
Modified: 2015-03-12 08:17 PDT (History)
18 users (show)
mzeier: needs‑downtime+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---

Average transferred bytes per sec in each hour from (620 bytes, text/plain)
2009-12-20 18:15 PST, Kazuhiro Fujieda
no flags Details

Description matthew zeier [:mrz] 2008-10-14 14:07:46 PDT
Tracking bug.

As far as I know, geoip bouncer is only on for China.  Not sure what it takes to turn it on globally.
Comment 1 Dave Miller [:justdave] ( 2008-10-15 14:24:45 PDT
I'm not sure it's ready yet.  See bug 353237 comment 14.
Comment 2 matthew zeier [:mrz] 2008-10-22 15:09:05 PDT
Dave - I think bug 353237 is a different issue.  As far as I know, geoip is ready.  That bug talks about "peer network patch" and the sentry issue you mentioned is still an issue regardless.

oremj, any clue what has to happen to turn this on globally?
Comment 3 Jeremy Orem [:oremj] 2008-10-22 15:48:34 PDT
The GeoIP section of code is already turned on, so someone just need to adjust the Region -> GeoIP throttle for each region in the admin interface.
Comment 4 matthew zeier [:mrz] 2008-11-03 15:44:17 PST
(In reply to comment #3)
> The GeoIP section of code is already turned on, so someone just need to adjust
> the Region -> GeoIP throttle for each region in the admin interface.

What's involved in that?  Should/could that be automated?  Is that something Lars would be able to do?  (oremj, you see like you know...)
Comment 5 Jeremy Orem [:oremj] 2008-11-03 16:17:41 PST
It's pretty simple.  You just go to and adjust the GeoIP throttles for the different regions.

If the throttle is at 0 GeoIP isn't on for that mirror.  If the throttle is at 100 all requests will go to a mirror in its region.  If the throttle is at 25 then 25% of all the requests will go to a mirror in its region, and the other 75% will go to any of the mirrors.
Comment 6 matthew zeier [:mrz] 2008-11-04 13:14:06 PST
Reed owns this now.
Comment 7 Reed Loden [:reed] (use needinfo?) 2008-11-06 21:33:41 PST
So, schedule this for the next downtime window (in case something goes wrong), and we'll give this a shot.

(In reply to comment #0)
> As far as I know, geoip bouncer is only on for China.  Not sure what it takes
> to turn it on globally.

Bouncer doesn't seem to have any working China mirrors, so even though the geoip throttle is 25 for China, it isn't working, as there aren't any mirrors.
Comment 8 chizu 2008-12-15 17:48:31 PST
Is 395950 still an issue for this?
Comment 9 matthew zeier [:mrz] 2008-12-17 22:36:03 PST
Gental reminder to add in why this is blocked :)
Comment 10 matthew zeier [:mrz] 2008-12-18 20:11:34 PST
ping ping ping?
Comment 11 chizu 2008-12-18 22:55:49 PST
This is blocked for a couple reasons:

Bug 395950 still affects production.
Bug 353237 comment 14 limits the usefulness without some way to discover mirror availability local to the mirror. It could degrade download response in areas where mirror coverage doesn't match the user base (like the US).
Comment 12 matthew zeier [:mrz] 2009-03-12 09:50:40 PDT
I think this encapsulates two issues:

1. distributed sentry checks
2. fixing dependencies for turning on geoip bouncer full time.

Tagging as summer intern '09 project.
Comment 13 David Humphrey (:humph) 2009-03-12 11:02:57 PDT
Re-assigning to nobody so it's clear that this is open for a student to take.
Comment 14 Dave Miller [:justdave] ( 2009-11-14 21:49:01 PST
mrz: I see you pulled this back to server-ops... what's the plan?
Comment 15 matthew zeier [:mrz] 2009-11-15 09:28:14 PST
Tracking a webdev Q4 goal (replaced pushing mirrorbrain in favor of this).
Comment 16 matthew zeier [:mrz] 2009-12-11 09:33:41 PST
Let's shoot for next Tuesday.
Comment 17 Jeremy Orem [:oremj] 2009-12-17 19:35:00 PST
All regions are set at 5% right now.
Comment 18 Justin Fitzhugh 2009-12-17 22:11:22 PST
doesn't seem to be working - just got Netherlands then japan from Menlo Park.
Comment 19 Jeremy Orem [:oremj] 2009-12-17 22:24:38 PST
(In reply to comment #17)
> All regions are set at 5% right now.

So you'll get your region 5% or the time minimum.
Comment 20 Justin Fitzhugh 2009-12-17 22:26:22 PST
that seems useless - can we up the percentage to something meaningful?  or is there a plan/timeline to do such?
Comment 21 Channy Yun [:channy] 2009-12-17 22:31:53 PST
I think this problem always will be in future. 

How about measuring ping results to mirror servers? I guess it can be measured in download page before clicking button or recommending mirror servers based on pre-measured data.

I experienced some trouble to update it from Sweden server in Korea. (I preferred Korean mirror.)

It's just idea.
Comment 22 Kazuhiro Fujieda 2009-12-19 02:03:38 PST
Please set the throttle for Japan to 100%. Mirror servers in Japan can handle all requests from Japan.
Comment 23 matthew zeier [:mrz] 2009-12-19 09:38:58 PST
Do we have a sense of how much bandwidth is required to serve all .jp users?  

Right now it's broken down into geo regions, not countries.  Asia is listed as having 33 mirrors.
Comment 24 Gen Kanai [:gen] 2009-12-19 09:45:37 PST
(In reply to comment #23)
> Do we have a sense of how much bandwidth is required to serve all .jp users?  

I'll have Kenji provide a snapshot of the recent traffic to early next week.
Comment 25 Reed Loden [:reed] (use needinfo?) 2009-12-19 09:50:20 PST
Also, what about CDNs that have connectivity in multiple regions? How are they going to be handled? Seems like an easy way would be to make the region a multi-select option so multiple regions can be selected if needed.
Comment 26 Kazuhiro Fujieda 2009-12-20 18:15:08 PST
Created attachment 418583 [details]
Average transferred bytes per sec in each hour from
Comment 27 Kazuhiro Fujieda 2009-12-20 18:57:53 PST
(In reply to comment #26)
This data shows average bytes per second in each hour for .jp users transferred from
The peak is 7.52 Mbytes per second. handled 4.58% of all requests. The bandwidth required to serve all .jp users is estimated at 1.31 Gbps. Our server can serve this bandwidth without any trouble.
Comment 28 MIZUKOSHI Kenji 2009-12-21 01:09:05 PST
NAIST kyoto mirror sever ( got 700Mbps requests in peak for fx 3.6 release. (20091217)
About 20% of requests are from .jp and rest are from world wide.
Load is 60% in peak. 
Usually less 10%, yes.

JAIST have big pipe and servers.
So they can handle all request from .jp.

But problems are request from outside of .jp.
I Think best way is Jaist raize up tier1, not only .jp.
Comment 29 MIZUKOSHI Kenji 2009-12-21 01:23:20 PST
Solly.  s/3.6/3.5.6/
Comment 30 Kazuhiro Fujieda 2009-12-21 05:22:14 PST
Our server ( used 2.4 Gbps of bandwidth at the peak on 17th Dec. about 5 am in JST. The usage of CPU is 75%. The requests from Japan account for only 0.2%.

Most of all requests came from other countries and consumed our bandwidth for non-academic overseas networks (2 Gbps). Our server can't take more requests from other countries, but it can take much more requests inside Japan.
Comment 31 Kohei Yoshino (was: 2010-04-07 19:00:58 PDT
How much have you set throttle for Japan? Still 5%?
Comment 32 matthew zeier [:mrz] 2010-04-08 14:57:43 PDT
It's currently set for 15% but we're also in the midst of pushing out an updated version of bouncer which does a better job at locale redirection.

Note You need to log in before you can comment on or make changes to this bug.