Closed Bug 459919 Opened 16 years ago Closed 15 years ago

GeoIP Bouncer, turn on for world

Categories

(mozilla.org Graveyard :: Server Operations, task, P2)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
All
Other
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mrz, Assigned: oremj)

References

Details

(Whiteboard: 12/17/2009 @ 7pm)

Attachments

(1 file)

Average transferred bytes per sec in each hour from ftp.jaist.ac.jp
620 bytes, text/plain
Details 
Tracking bug.

As far as I know, geoip bouncer is only on for China.  Not sure what it takes to turn it on globally.
I'm not sure it's ready yet.  See bug 353237 comment 14.
Depends on: 353237
Assignee: server-ops → mrz
Dave - I think bug 353237 is a different issue.  As far as I know, geoip is ready.  That bug talks about "peer network patch" and the sentry issue you mentioned is still an issue regardless.

oremj, any clue what has to happen to turn this on globally?
The GeoIP section of code is already turned on, so someone just need to adjust the Region -> GeoIP throttle for each region in the admin interface.
(In reply to comment #3)
> The GeoIP section of code is already turned on, so someone just need to adjust
> the Region -> GeoIP throttle for each region in the admin interface.

What's involved in that?  Should/could that be automated?  Is that something Lars would be able to do?  (oremj, you see like you know...)
It's pretty simple.  You just go to https://download.mozilla.org/admin/regions.php and adjust the GeoIP throttles for the different regions.

If the throttle is at 0 GeoIP isn't on for that mirror.  If the throttle is at 100 all requests will go to a mirror in its region.  If the throttle is at 25 then 25% of all the requests will go to a mirror in its region, and the other 75% will go to any of the mirrors.
Reed owns this now.
Assignee: mrz → reed
No longer depends on: 353237
So, schedule this for the next downtime window (in case something goes wrong), and we'll give this a shot.

(In reply to comment #0)
> As far as I know, geoip bouncer is only on for China.  Not sure what it takes
> to turn it on globally.

Bouncer doesn't seem to have any working China mirrors, so even though the geoip throttle is 25 for China, it isn't working, as there aren't any mirrors.
Flags: needs-downtime+
Assignee: reed → thardcastle
Is 395950 still an issue for this?
Gental reminder to add in why this is blocked :)
ping ping ping?
This is blocked for a couple reasons:

Bug 395950 still affects production.
Bug 353237 comment 14 limits the usefulness without some way to discover mirror availability local to the mirror. It could degrade download response in areas where mirror coverage doesn't match the user base (like the US).
Assignee: thardcastle → mrz
Component: Server Operations → Server Operations: Projects
I think this encapsulates two issues:

1. distributed sentry checks
2. fixing dependencies for turning on geoip bouncer full time.

Tagging as summer intern '09 project.
Assignee: mrz → oremj
Keywords: student-project
Priority: -- → P2
Whiteboard: intern09
Re-assigning to nobody so it's clear that this is open for a student to take.
Assignee: oremj → nobody
Flags: needs-downtime+
Assignee: nobody → dtran
Assignee: dtran → server-ops
Component: Server Operations: Projects → Server Operations
Keywords: student-project
Whiteboard: intern09
mrz: I see you pulled this back to server-ops... what's the plan?
Tracking a webdev Q4 goal (replaced pushing mirrorbrain in favor of this).
Assignee: server-ops → mrz
Depends on: 395950
Let's shoot for next Tuesday.
Assignee: mrz → server-ops
Flags: needs-downtime+
Whiteboard: 12/15/2009 @ 7pm
Whiteboard: 12/15/2009 @ 7pm → 12/17/2009 @ 7pm
Assignee: server-ops → jeremy.orem+bugs
All regions are set at 5% right now.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
doesn't seem to be working - just got Netherlands then japan from Menlo Park.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to comment #17)
> All regions are set at 5% right now.

So you'll get your region 5% or the time minimum.
Status: REOPENED → RESOLVED
Closed: 15 years ago → 15 years ago
Resolution: --- → FIXED
that seems useless - can we up the percentage to something meaningful?  or is there a plan/timeline to do such?
I think this problem always will be in future. 

How about measuring ping results to mirror servers? I guess it can be measured in download page before clicking button or recommending mirror servers based on pre-measured data.

I experienced some trouble to update it from Sweden server in Korea. (I preferred Korean mirror.)

It's just idea.
Please set the throttle for Japan to 100%. Mirror servers in Japan can handle all requests from Japan.
Do we have a sense of how much bandwidth is required to serve all .jp users?  

Right now it's broken down into geo regions, not countries.  Asia is listed as having 33 mirrors.
(In reply to comment #23)
> Do we have a sense of how much bandwidth is required to serve all .jp users?  

I'll have Kenji provide a snapshot of the recent traffic to mozilla.jp early next week.
Also, what about CDNs that have connectivity in multiple regions? How are they going to be handled? Seems like an easy way would be to make the region a multi-select option so multiple regions can be selected if needed.
Attachment #418583 - Attachment description: Transferred bytes per hour from ftp.jaist.ac.jp → Average transferred bytes per sec in each hour from ftp.jaist.ac.jp
(In reply to comment #26)
This data shows average bytes per second in each hour for .jp users transferred from ftp.jaist.ac.jp.
The peak is 7.52 Mbytes per second. ftp.jaist.ac.jp handled 4.58% of all requests. The bandwidth required to serve all .jp users is estimated at 1.31 Gbps. Our server can serve this bandwidth without any trouble.
NAIST kyoto mirror sever (kyoto-mz-dl.sinet.ad.jp) got 700Mbps requests in peak for fx 3.6 release. (20091217)
About 20% of requests are from .jp and rest are from world wide.
Load is 60% in peak. 
Usually less 10%, yes.

JAIST have big pipe and servers.
So they can handle all request from .jp.

But problems are request from outside of .jp.
I Think best way is Jaist raize up tier1, not only .jp.
Solly.  s/3.6/3.5.6/
Our server (ftp.jaist.ac.jp) used 2.4 Gbps of bandwidth at the peak on 17th Dec. about 5 am in JST. The usage of CPU is 75%. The requests from Japan account for only 0.2%.

Most of all requests came from other countries and consumed our bandwidth for non-academic overseas networks (2 Gbps). Our server can't take more requests from other countries, but it can take much more requests inside Japan.
How much have you set throttle for Japan? Still 5%?
It's currently set for 15% but we're also in the midst of pushing out an updated version of bouncer which does a better job at locale redirection.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: