1223199 - Create dataset for web map of FF use

Status: RESOLVED WONTFIX

(Data Platform and Tools :: General, defect, P5)

Description

[Sam Penrose]

We'd like to create a map similar to [1] that shows Firefox use by Geo. The dataset will be a derived stream looking something like:

(session-start-timestamp, session-end-timestamp, geo)

details TBD, second-level resolution probably preferred. We may need to deliver the work in the first week of December.

[1] https://webwewant.mozilla.org

Comment 1

[Sam Penrose]

Mark, here's the format I've got working. Given pings with pseudo-structure:

   {sessionStart: <stamp1>, sessionLength: <seconds1>, location: <place1>, ...}
   {sessionStart: <stamp2>, sessionLength: <seconds2>, location: <place2>, ...}

I would like to end up with JSON of the form:
{
  "starts": [
    {"when": stamp1, "lat": <lat1>, "lon": <lon1>},
    {"when": stamp2, "lat": <lat2>, "lon": <lon2>},
  ],
  "ends": [
    {"when": stamp2+elapsed2, "lat": <lat2>, "lon": <lon2>},
    {"when": stamp1+elapsed1, "lat": <lat1>, "lon": <lon1>},
  ]
}

Can you do that, with the arrays sorted by when ascending? Do I need to tackle GEO->lat/lon?

Comment 2

[Mark Reid [:mreid]]

Definitely looks doable.

If you want to tackle GEO -> lat/long, go for it!  If not, I will just use something like this:
http://www.csgnetwork.com/llinfotable.html

to map countries to a representative lat/long.

Comment 3

[John Jensen]

IIRC the MaxMind DB we use has lat/long. No idea how to get access to it though.

Comment 4

[Mark Reid [:mreid]]

How does this look?

{
  "starts": [
    { "when": 1447113708, "lon": 138, "lat": 36 },
    { "when": 1447120359, "lon": -95, "lat": 60 },
    { "when": 1447120364, "lon": -95, "lat": 60 },
    { "when": 1447121288, "lon": 105, "lat": 35 },
    { "when": 1447128998, "lon": 105, "lat": 35 },
    { "when": 1447128999, "lon": 105, "lat": 35 },
    { "when": 1447132482, "lon": -97, "lat": 38 },
    { "when": 1447132482, "lon": -97, "lat": 38 },
    { "when": 1447137010, "lon": 120, "lat": -5 },
    { "when": 1447137017, "lon": 120, "lat": -5 }
  ],
  "ends": [
    { "when": 1447113708, "lon": 138, "lat": 36 },
    { "when": 1447120359, "lon": -95, "lat": 60 },
    { "when": 1447120364, "lon": -95, "lat": 60 },
    { "when": 1447121288, "lon": 105, "lat": 35 },
    { "when": 1447128998, "lon": 105, "lat": 35 },
    { "when": 1447128999, "lon": 105, "lat": 35 },
    { "when": 1447132482, "lon": -97, "lat": 38 },
    { "when": 1447132482, "lon": -97, "lat": 38 },
    { "when": 1447137010, "lon": 120, "lat": -5 },
    { "when": 1447137017, "lon": 120, "lat": -5 }
  ]
}

Comment 5

[Mark Reid [:mreid]]

(In reply to John Jensen from comment #3)
> IIRC the MaxMind DB we use has lat/long. No idea how to get access to it
> though.

It does, but we discard the IP address as soon as possible during the processing. We'd have to change the upstream processing to put the lat/long info in while we still have the IP.

Comment 6

[Mark Reid [:mreid]]

Let me know if that looks alright. It's going to be a fairly large data set in that format, depending on how much time we want to cover.

Comment 7

[Sam Penrose]

(In reply to Mark Reid [:mreid] from comment #6)
> Let me know if that looks alright. It's going to be a fairly large data set
> in that format, depending on how much time we want to cover.

That looks great, thanks. This is no longer a P1.