GeoIP Bouncer, turn on for world

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
P2
minor
RESOLVED FIXED
9 years ago
2 years ago

People

(Reporter: mrz, Assigned: oremj)

Tracking

Bug Flags:
needs-downtime +

Details

(Whiteboard: 12/17/2009 @ 7pm)

Attachments

(1 attachment)

(Reporter)

Description

9 years ago
Tracking bug.

As far as I know, geoip bouncer is only on for China.  Not sure what it takes to turn it on globally.
I'm not sure it's ready yet.  See bug 353237 comment 14.
(Reporter)

Updated

9 years ago
Depends on: 353237
Assignee: server-ops → mrz
(Reporter)

Comment 2

9 years ago
Dave - I think bug 353237 is a different issue.  As far as I know, geoip is ready.  That bug talks about "peer network patch" and the sentry issue you mentioned is still an issue regardless.

oremj, any clue what has to happen to turn this on globally?
(Assignee)

Comment 3

9 years ago
The GeoIP section of code is already turned on, so someone just need to adjust the Region -> GeoIP throttle for each region in the admin interface.
(Reporter)

Comment 4

9 years ago
(In reply to comment #3)
> The GeoIP section of code is already turned on, so someone just need to adjust
> the Region -> GeoIP throttle for each region in the admin interface.

What's involved in that?  Should/could that be automated?  Is that something Lars would be able to do?  (oremj, you see like you know...)
(Assignee)

Comment 5

9 years ago
It's pretty simple.  You just go to https://download.mozilla.org/admin/regions.php and adjust the GeoIP throttles for the different regions.

If the throttle is at 0 GeoIP isn't on for that mirror.  If the throttle is at 100 all requests will go to a mirror in its region.  If the throttle is at 25 then 25% of all the requests will go to a mirror in its region, and the other 75% will go to any of the mirrors.
(Reporter)

Comment 6

9 years ago
Reed owns this now.
Assignee: mrz → reed
No longer depends on: 353237
So, schedule this for the next downtime window (in case something goes wrong), and we'll give this a shot.

(In reply to comment #0)
> As far as I know, geoip bouncer is only on for China.  Not sure what it takes
> to turn it on globally.

Bouncer doesn't seem to have any working China mirrors, so even though the geoip throttle is 25 for China, it isn't working, as there aren't any mirrors.
Flags: needs-downtime+
(Reporter)

Updated

9 years ago
Assignee: reed → thardcastle

Comment 8

9 years ago
Is 395950 still an issue for this?
(Reporter)

Comment 9

9 years ago
Gental reminder to add in why this is blocked :)
(Reporter)

Comment 10

9 years ago
ping ping ping?

Comment 11

9 years ago
This is blocked for a couple reasons:

Bug 395950 still affects production.
Bug 353237 comment 14 limits the usefulness without some way to discover mirror availability local to the mirror. It could degrade download response in areas where mirror coverage doesn't match the user base (like the US).
Assignee: thardcastle → mrz
(Reporter)

Updated

9 years ago
Component: Server Operations → Server Operations: Projects
(Reporter)

Comment 12

8 years ago
I think this encapsulates two issues:

1. distributed sentry checks
2. fixing dependencies for turning on geoip bouncer full time.

Tagging as summer intern '09 project.
Assignee: mrz → oremj
Keywords: student-project
Priority: -- → P2
Whiteboard: intern09
Re-assigning to nobody so it's clear that this is open for a student to take.
Assignee: oremj → nobody
(Reporter)

Updated

8 years ago
Flags: needs-downtime+

Updated

8 years ago
Assignee: nobody → dtran
(Reporter)

Updated

8 years ago
Assignee: dtran → server-ops
Component: Server Operations: Projects → Server Operations
Keywords: student-project
Whiteboard: intern09
mrz: I see you pulled this back to server-ops... what's the plan?
(Reporter)

Comment 15

8 years ago
Tracking a webdev Q4 goal (replaced pushing mirrorbrain in favor of this).
Assignee: server-ops → mrz
(Reporter)

Updated

8 years ago
Depends on: 395950
(Reporter)

Comment 16

8 years ago
Let's shoot for next Tuesday.
Assignee: mrz → server-ops
Flags: needs-downtime+
Whiteboard: 12/15/2009 @ 7pm
(Reporter)

Updated

8 years ago
Whiteboard: 12/15/2009 @ 7pm → 12/17/2009 @ 7pm
(Reporter)

Updated

8 years ago
Assignee: server-ops → jeremy.orem+bugs
(Assignee)

Comment 17

8 years ago
All regions are set at 5% right now.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED

Comment 18

8 years ago
doesn't seem to be working - just got Netherlands then japan from Menlo Park.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 19

8 years ago
(In reply to comment #17)
> All regions are set at 5% right now.

So you'll get your region 5% or the time minimum.
Status: REOPENED → RESOLVED
Last Resolved: 8 years ago8 years ago
Resolution: --- → FIXED

Comment 20

8 years ago
that seems useless - can we up the percentage to something meaningful?  or is there a plan/timeline to do such?
I think this problem always will be in future. 

How about measuring ping results to mirror servers? I guess it can be measured in download page before clicking button or recommending mirror servers based on pre-measured data.

I experienced some trouble to update it from Sweden server in Korea. (I preferred Korean mirror.)

It's just idea.

Comment 22

8 years ago
Please set the throttle for Japan to 100%. Mirror servers in Japan can handle all requests from Japan.
(Reporter)

Comment 23

8 years ago
Do we have a sense of how much bandwidth is required to serve all .jp users?  

Right now it's broken down into geo regions, not countries.  Asia is listed as having 33 mirrors.

Comment 24

8 years ago
(In reply to comment #23)
> Do we have a sense of how much bandwidth is required to serve all .jp users?  

I'll have Kenji provide a snapshot of the recent traffic to mozilla.jp early next week.
Also, what about CDNs that have connectivity in multiple regions? How are they going to be handled? Seems like an easy way would be to make the region a multi-select option so multiple regions can be selected if needed.

Comment 26

8 years ago
Created attachment 418583 [details]
Average transferred bytes per sec in each hour from ftp.jaist.ac.jp

Updated

8 years ago
Attachment #418583 - Attachment description: Transferred bytes per hour from ftp.jaist.ac.jp → Average transferred bytes per sec in each hour from ftp.jaist.ac.jp

Comment 27

8 years ago
(In reply to comment #26)
This data shows average bytes per second in each hour for .jp users transferred from ftp.jaist.ac.jp.
The peak is 7.52 Mbytes per second. ftp.jaist.ac.jp handled 4.58% of all requests. The bandwidth required to serve all .jp users is estimated at 1.31 Gbps. Our server can serve this bandwidth without any trouble.

Comment 28

8 years ago
NAIST kyoto mirror sever (kyoto-mz-dl.sinet.ad.jp) got 700Mbps requests in peak for fx 3.6 release. (20091217)
About 20% of requests are from .jp and rest are from world wide.
Load is 60% in peak. 
Usually less 10%, yes.

JAIST have big pipe and servers.
So they can handle all request from .jp.

But problems are request from outside of .jp.
I Think best way is Jaist raize up tier1, not only .jp.

Comment 29

8 years ago
Solly.  s/3.6/3.5.6/

Comment 30

8 years ago
Our server (ftp.jaist.ac.jp) used 2.4 Gbps of bandwidth at the peak on 17th Dec. about 5 am in JST. The usage of CPU is 75%. The requests from Japan account for only 0.2%.

Most of all requests came from other countries and consumed our bandwidth for non-academic overseas networks (2 Gbps). Our server can't take more requests from other countries, but it can take much more requests inside Japan.

Comment 31

7 years ago
How much have you set throttle for Japan? Still 5%?
(Reporter)

Comment 32

7 years ago
It's currently set for 15% but we're also in the midst of pushing out an updated version of bouncer which does a better job at locale redirection.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.