656366 - Bouncer needs an updated geoip database

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task, P1)

Description

[Dave Miller [:justdave]]

The current geoip database in use in Bouncer is the same one that was dropped in when Bouncer first got the capability to do GeoIP.  It's now vastly out of date, and we're continually getting reports of reassigned IP blocks that no longer belong to the same company they're listed in, or IP blocks which aren't in the database to begin with.

We should probably look into some way to get this updated on a regular basis.

We also have custom changes to it for specific regions which need to survive (build network at MPT for example) a database reload.

Comment 1

[Arzhel Younsi [:XioNoX]]

It's also important for countries that have a better national connectivity than international.
For example Tanzania: (Mirror connection speed: 2mbps (International), 60mbps (Local - Tanzania Internet Exchange)

We can use that page to confirm the freshness of the new database :
http://apps.aptus.co.tz/tixfeed.php (Live BGP feed of TZ prefixes).

Comment 2

[Anthony Ricaud (:rik)]

Dave: About the custom changes, could we list them in the script in bug 677640 so that they are preserved everytime we run it?

Comment 3

[Dave Miller [:justdave]]

Probably could.  I have no clue what they are.  RelEng set them up.  I'm not sure if they're even being used at the moment, I think that original plan backfired.

Comment 4

[[e-mail] (:bhearsum)]

They're not being used just at the moment, but once bug 613620 is fixed we'll be looking at using them again. Do you need anything from us to proceed?

Comment 5

[Anthony Ricaud (:rik)]

So the script is now done.

In order to run this script on the actual servers, we need to change the table to InnoDB. (because it relies on transactions)

Comment 6

[Anthony Ricaud (:rik)]

CCing Sheeri for the InnoDB migration of geoip_ip_to_country.

Comment 7

[Sheeri Cabral [:sheeri]]

The bouncer.geoip_ip_to_country table has been converted to InnoDB.

Comment 8

[Anthony Ricaud (:rik)]

Oops, looks like I forgot to cc Sheeri :( Thanks!

Two things left: run "python manage.py updategeoip" to update it.

And setup a cron to run this. GeoCountryLite is updated once per month (Jan 3rd, Feb 7th). Running it around the 10th each month should be good.

Comment 9

[Arzhel Younsi [:XioNoX]]

Moving to server ops

Comment 10

[Dumitru Gherman [:dumitru]]

cd /var/www/django/tuxedo
source /data/virtualenvs/tuxedo/bin/activate
./manage.py updategeoip
Downloading http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip downloaded
Deleting previous records
Previous records deleted
Inserting new records
New records inserted

Comment 11

[Dumitru Gherman [:dumitru]]

crontab entry added.

Comment 12

[Anthony Ricaud (:rik)]

Thanks Dumitru. What's the crontab schedule?

Comment 13

[Dumitru Gherman [:dumitru]]

What you said :)
On the 10th every month at 5PM PST.