1179534 - [research] geo for heartbeat

Status: RESOLVED FIXED

(Input Graveyard :: Submission, defect)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We want us some geo for the heartbeat.

The description here is vague. I promise to flesh it out before we do anything.

Comment 1

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Goal:  
1. untangle locale from locality.  In particular, en-US & US, and es-* & US.  

Suggested Implmentation

1.  do whatever else other mozilla packet recording services do.

I believe that we have some license for a geocoding service already on all *.mozilla domains.  National level geocoding is acceptable.  Edge cases like US territories, other disputed zones should 'do something sensible'.  IP geolocation is fine.

Comment 2

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

(Note: this implies something about adding a new field to heartbeat tables for national-level GEO).

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Mmm... We definitely need to add a field to the heartbeat_answer table. Beyond that, I don't know what's involved here technically because I don't know the specifics of the options as they pertain to whatever licenses we have to the geo data.

I'll figure out what Mike did with SUMO (pretty sure they have something like this) and follow up with Lonnen who will know about how other sites work or know who to talk to.

Grabbing this to flesh it out this week. Making it a research bug for now--we'll spin off other bugs later as needed. Making it 2 points since I think that's about how much work is required. Pretty sure it's mostly tracking down people and figuring out how they did it.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Removing extra [ from summary. Oops.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

I added a field to the hb table in bug #1183764.

Gregg is experimenting with doing the country lookup from the hb client. If that doesn't work out for whatever reasons, we'll look into doing it server side, but I'm pretty sure that gets a lot more complicated.

Need to double-check with Lonnen architecture and data issues here.

Comment 6

[Osmose [:osmose, :mkelly]]

DID SOMEONE SAY GEO

At the behest of gregglind I am here to share knowledge.

geo.mozilla.org is an instance of https://github.com/mozilla/geodude that can give you the current user's country. It's hosted on Mozilla infra and we load tested it (bug 850784, comment 2) up to about 5000 requests per second, and are comfortable with about 2000 per second sustained. about:home, mozilla.org, and I think SUMO all use it.

about:home in particular hits it once a month per user and caches their country in IndexedDB; I'd suggest a similar rate for whatever you're working on, maybe once every 14 days if you're really worried about the value changing.

If you do end up using it, and you're worried about overloading it, run your numbers by jakem to see how much he cares.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

We pushed out the geo for hb and it "melted the servers" at 3.3k/s.

Quotes from IRC:

Aug 03 14:55:31 <atoll> the service previously was provision to support ongoing load of 20k hits/minute
Aug 03 14:55:36 <atoll> and is, thus, entirely melting down and failing you.

Aug 03 16:26:16 <atoll> observed was 205k/min which is 3300/sec by my earlier math
Aug 03 16:26:38 <atoll> 200/sec is 12k/min which is only 50% higher than current load (which makes sense, given two prior consumers using the same codebase)


Those statements seem to suggest the load test wasn't done right or that the infrastructure has changed significantly since then.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think this is all set now. We added a db field. The HB client added some code to do the lookup and sends the data along. We don't need to do anything extra on the Input side.

Going to mark this as FIXED now.