1135717 - [bedrock] Install MaxMind database and python library maxminddb

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Description

[Paul [:pmac] McLanahan]

We need server side geolocation by IP address for bedrock. Please install the MaxMind GeoIP2 Country database on the webheads and let us know the full path to the database file.

We also need IT to install the maxminddb python library as there are compiled C extensions that make it fast. To get the fast version requires installation at the OS level of libmaxminddb.

We'd like to use this on some work that is coming up very soon. In order that we can plan whether or not we should count on having this by launch of this work, please provide an estimated completion date for this before starting work on it.

Thanks!

Comment 1

[Jennifer Bertsch [:jbertsch]]

Hi-

When do we think we'll have an estimated completion date for this?

Thanks.

Comment 2

[Jake Maul [:jakem]]

Had a meeting on this just now...

TL;DR is: we'll have the admin node maintain the authoritative copy, update it periodically, and (one way or another) sync it out to the web nodes. Bedrock's code will include the proper python bindings, and we'll install the system package.

We already have a geoip puppet module suitable for putting on an admin node that handles all the configuration, updating, and licensing.

Syncing the data to the web nodes can be done in a number of ways The prevailing idea is to update the data from MaxMind on the admin node, then copy it into each app's /data/<env>/src/<app>/ directory. Leave it alone, and let it get pushed out whenever a deploy happens. Drift is purported to be around 1.5% per month according to MaxMind, so this should be just fine. Alternatively we can handle it like locales, and sync out on a regular basis, outside of normal code deploys.

Comment 3

[Jake Maul [:jakem]]

Started work on this, ran into some yaks to shave.

Turns out all our stuff for GeoIP is very old... even upstream (EPEL, rpmforge). It's all for the older, legacy "GeoIP" format. The current hotness is "GeoIP2".

I've packaged up some modern stuff for this. 1) the current upstream "geoipupdate" package - which ultimately boils down to just a single binary and a manpage - and 2) libmaxminddb, the C library (and command line tool for looking up IPs).


Next up, I need to modify the puppet "geoip" module to ship these packages instead of the legacy "GeoIP" package. The admin node needs both (including libmaxminddb-devel, so really 3 packages), the web nodes need just libmaxminddb. This will have to be some sort of flag in the puppet module so it doesn't break anything that's already out there using the legacy stuff.


Simultaneously, @pmac, you should be good to add https://pypi.python.org/pypi/geoip2 to your requirements.txt. We'll put it in the virtualenv's for all the environments. Of course we can't actually install it in the virtualenv's until libmaxminddb is installed...

Comment 4

[Paul [:pmac] McLanahan]

https://pypi.python.org/pypi/maxminddb/ is actually all we need.

Comment 5

[Jennifer Bertsch [:jbertsch]]

Hi Jake-

How is this stuff coming along?  Is it on track be done early next week so that we can have our Firefox OS pages live by March 26?

Thanks,
Jen

Comment 6

[Jake Maul [:jakem]]

This is in place. All that's left is for you to add this library to the requirements/compiled.txt file and for us to install it into each virtualenv.

Let us know when it's added and I'll loop through them all to do the install. Don't forget the hash for peep. :)

Comment 7

[Paul [:pmac] McLanahan]

WOO! Thanks Jake. I've already added it to the requirements in the PR[0], so it'll be ready as soon as that is merged and pushed. I'll ping you when that happens.

[0] https://github.com/mozilla/bedrock/pull/2840

Comment 8

[Jennifer Bertsch [:jbertsch]]

Thanks, Jake!  This is great!!!

Comment 9

[:Atoll]

(In reply to Paul McLanahan [:pmac] from comment #7)
> WOO! Thanks Jake. I've already added it to the requirements in the PR[0], so
> it'll be ready as soon as that is merged and pushed. I'll ping you when that
> happens.
> 
> [0] https://github.com/mozilla/bedrock/pull/2840

Hi :pmac, this was merged 11 days ago - has it been pushed?

Comment 10

[Paul [:pmac] McLanahan]

Yes! Thanks for the ping :atoll. Forgot to close this. We added the maxminddb peep install to the deploy script, so we're fully done with this. Thanks again all!