994927 - [geolocation] Mock up a general purpose map

Status: RESOLVED INVALID

(Participation Infrastructure :: Phonebook, defect)

Description

[Justin Crawford [:hoosteeno] [:jcrawford]]

Let's build a map view that we can use on pages where we have lists of Mozillians. For example, search results or group members. 

* The goal is visualization, not getting directions to someone's house. Knowing this may help decide between various visualization options.

* The map should consider that at any zoom level there will be some people for whom we do not have (or cannot show) location data at that specificity. Region and city are optional; all fields are subject to per-field privacy levels which may mean they cannot be shown to a public viewer of the map. 

* It may be the case that certain zoom levels could conceivably plot people with specific location information (New York, New York) and generic location information (USA). There may be an elegant solution for this but the easiest solution is probably to eliminate the generic points after a certain zoom level.

* We should not plot mozillians with no viewable data. If an anonymous viewer is looking at a map of France, they should see the number of mozillians in France with _public_ country data. Mozillians without public country data should not be counted in aggregate numbers nor plotted on the map.

Comment 1

[Wray Bowling [:wbowling]]

Attachment: mockup.jpg

Option 1 mocked up: create a simple block that shows the map above the list of users on a group. (some buttons were moved around to make space)

Comment 2

[Wray Bowling [:wbowling]]

Attachment: Screen Shot 2014-04-17 at 11.04.56 AM.png

For the map tiles themselves, we can create our own set with TileMill. The resulting tiles will be about 5Mb in total for 5 zoom levels. Keeping the zoom levels low will obfuscate the specifics/privacy of each user and keep load down on rendering which took 4 seconds on my macbook.

TileMill can be run serverside on a cronjob if we decide we'd like to use it that way. Since specific data will be handled by markerclusters in realtime though, the cron wouldn't need to run all that often.

Comment 3

[Wray Bowling [:wbowling]]

Attachment: at-zoom-level-5.jpg

Here is a mockup of the map at zoom level 5 (as close as it goes) some example markers made at different cities in NC are clustered together in screen-space. Clicking on the cluster makes the markers expand. Any marker from there reveals full details, in this case "Texas, United States"

Comment 4

[Wray Bowling [:wbowling]]

Attachment: mockup2.jpg

Here's a second mockup mimicking something more Pinterest-like

Comment 5

[Wray Bowling [:wbowling]]

Attachment: markers.png

custom graphics for marker clusters using Open Sans Light from Mozillians' header.

Comment 6

[Wray Bowling [:wbowling]]

Started a branch on Github for this https://github.com/caktus/mozillians/tree/geo-visualization that displays the map on a group page.

There are a lot of things to hook up on the back, and front ends.

- provide data that the map can use as markers. A simple array of javascript objects will do.
example:

[-14.235004,-51.925280, 'Pedro Smith (Brazil)'],
[31.968599, -99.901813, 'Jane Doe (Texas, United States)'],

How that happens can be in one of multiple ways
1. We can serve a js file generated by python's backend up front
2. We can render the Javascript directly with a jinja template

Either way, until there's real geographic data to play with, I'll keep working on this with a mocked dataset of the Python / Firefox groups so I can start to catch edge cases. Mapbox has a few tutorials for 
using Google Docs to convert a list of addresses to lat/long quickly and i'll be able to get by with that and some more manual entry for the time being as I style the tiles and markers.
https://www.mapbox.com/tilemill/docs/guides/google-docs/

Lastly, to generate our own tiles for the broader data, I can work with a local tilemill export, but eventually we should attempt running tilemill on the server.
Guides like this are floating around:
https://www.mapbox.com/tilemill/docs/guides/ubuntu-service/ (tutorial)
https://www.mapbox.com/tilemill/docs/manual/exporting/ (fill list of cli flags)

Comment 7

[Wray Bowling [:wbowling]]

Attachment: Screen Shot 2014-04-22 at 1.48.36 PM.png

Today's experiment with importing what available data we already have into a spreadsheet, geocoding it (mocking what we'll actually do), and displaying that data in the map tiles themselves.

Comment 8

[Wray Bowling [:wbowling]]

Attachment: mozilla-experimental-render.png

a quick render of the whole mercatur map of mozillians geo data

Comment 9

[Wray Bowling [:wbowling]]

Attachment: Screen Shot 2014-04-22 at 3.46.57 PM.png

Woohoo! I got the two connected together and showing up on the page. A good next step would be to pass the right data to the map based on what page you're looking at. I'll need back-end help to do that. We can probably get by not updating the map tiles until later since they aren't interactive anyway.

Comment 10

[Wray Bowling [:wbowling]]

The data I'll need from the backend ideally would be as follows:

Name,
Latitude,
Longitude,
Country,
Region (if available),
City (if available),
some sort of URI to point to the user's gravitar or a short URL.

The gravitar isn't necessary, but I think would be nice. With a lot of users, the response might get big with full gravitar URLs.

Comment 11

[Wray Bowling [:wbowling]]

Formatting of that data will eventually get looped through in Javascript. A simple array of objects is fine.

Comment 12

[Wray Bowling [:wbowling]]

Attachment: Screen Shot 2014-05-08 at 11.33.48 AM.png

A screenshot of progress up to this point sancus and I have made

Comment 13

[Wray Bowling [:wbowling]]

Attachment: Screen Shot - marker labels

Got marker labels working today.

Comment 14

[Wray Bowling [:wbowling]]

There seems to be a cross-domain issue that's being handled by the Firefox browser even after we get a response from api.tiles.mapbox.com

Looking in the Style editor, the map styles are just plain gone. If I paste in the response from mapbox.css into a new stylesheet, suddenly the map looks fine. The GUI images (zoom in, zoom out, the little `i` for info) are also missing which further reinforces that it's related to a cross-domain thing.

All the settings in base.py are giving plenty of the correct permissions so I'm not sure what the problem is.

Comment 15

[Giorgos Logiotatidis [:giorgos]]

Moving here comments from github so we can continue the discussion:

Pull request: https://github.com/mozilla/mozillians/pull/1054


-- glogiotatidis

I have same serious concerns about this implementation:

    We drastically increase processing time depending on the size of the group. And we do re-generate the same geodata on every page of the same group
    Similarly for the size of the downloaded html.
    To measure the difference:
        the 'sf-momument' group needs 8+ seconds on my local machine to render compared to 1+ seconds without this change
        every page of the 'sf-momument' group has 400Kb more data

    One option is to load the map on demand, using ajax or on a separate group view.

    On another note the group page needs some beatification now that we have a map as well. @wraybowling I'm sure you can do miracles :)


-- sancus

@glogiotatidis Why don't we cache the data for groups larger than 100 members by simply sticking the json files on disk? There's only 20 or so such groups now, so it won't take much space, and I don't think it's a performance issue to just do it dynamically for groups smaller than 100. We could even put the threshold down at 20 or 50 members.

We could regenerate these big groups every 24 hours. And since it'll be a JSON file that they link to, they'll take full advantage of browser caching and won't need to redownload the data on every page.


-- wbowling

I second the caching idea.


-- glogiotatidis

Caching is a neat idea. It doesn't have to be a file, it can be a django.CACHE. We can have a special url for all groups like /group/foo/locations.json that will return the json object that we're currently embedding into the html. This view can do all short of fancy caching and a url can get better browser caching.

But bytes transferred are bigger concern than cpu time in this case: Caching will help our servers mostly but will not help users, unless the same user is visiting the same group more than once per day. Also clients are downloading N*2 times the avatars of all members of the group, before they even zoom in. This should be done only at a level that avatars appear.

Caching included I still believe that displaying the map directly on every group page is a bad idea. We should bring the map on demand or have it only in the first page of the group. On demand will help with both bytes transferred and rendering time especially on mobile devices. In both cases the UI needs love, it's just like we threw a (really cool) map in a page that already needed UI love.

I understand that we currently have only a few groups with more than 100 members but they are also the most visited groups. Most -if not all- of the functional areas have more than 100 members, so this is our common case.

Also currently the biggest concern is that we don't respect the "save region" and "save city" settings atm. If a user doesn't share city and/or region then we still feed their lat/lon to the clients.

Also this is based on lat/lon which I understand they are a temp solution? https://bugzilla.mozilla.org/show_bug.cgi?id=1022014

Comment 16

[Wray Bowling [:wbowling]]

It seems best to keep both the lat/long coordinates *and* the mapbox IDs. It's the only way we're going to be able to keep accurate records (mapbox IDs) and have fast location recall for the visualizations (coordinates). If calling the coordinates, placenames, and user name is bogging us down, cache it.

Comment 17

[Giorgos Logiotatidis [:giorgos]]

(In reply to Wray Bowling [:wbowling] from comment #16)
> It seems best to keep both the lat/long coordinates *and* the mapbox IDs.
> It's the only way we're going to be able to keep accurate records (mapbox
> IDs) and have fast location recall for the visualizations (coordinates). If
> calling the coordinates, placenames, and user name is bogging us down, cache
> it.

Keeping lat/lng for backend use and displaying it only to the user which it belongs to it's fine. But the pending pull requests [0] expose the lat/lng of _all_ users to all other vouched mozillians, disregarding the privacy settings for each profile. This is a no-go in my opinion.

[0] https://github.com/mozilla/mozillians/pull/1054

Comment 18

[Andrei Hajdukewycz [:sancus]]

Well privacy settings aren't really disregarded because you can't hide location info from vouched mozillians anyway, there's no more restrictive privacy setting.

There is an issue where we're storing overly-precise lat/lngs instead of the "generic" lat/lng for your region/country if you've selected not to use city. There should be no absolutely no way to set a lat/lng that is more precise than the generic one for your city -- the map pointer snaps to the closest city bounding box even if you drag it around.

Given this, I think there is no privacy concern as long as the person in question has chosen to use City-level precision in their profile.

The precision issue is that when I remove the checkbox for saving City, the code currently does not reset my lat/lng to the next higher lat/lng(the one for California). It should. This would also require a migration to correct current profiles with this overly-precise lat/lng issue. IMO this is a separate bug.

What I can do for this bug is change the code so that you are only included in the map if you have set your profile to save City. This should alleviate any privacy concerns -- at that point you are sharing the lat/lng for your City with other vouched mozillians, which is not at all private data, anyone who knows your City can look it up for themselves via the Mapbox API.

Comment 19

[Giorgos Logiotatidis [:giorgos]]

Andrei please verify the following scenario:
 - In user/edit search for a city e.g. "Athens, Greece"
 - Select to save only country, uncheck region and city
 - Save()

The lat/lon saved, points to the specific city not to the country. The same lat/lon is used by the group map to render the pointers. Therefore one with a vouched account can figure out all locations on city level of all other vouched mozillians.

Can you confirm this?

Comment 20

[Andrei Hajdukewycz [:sancus]]

That's correct. That's exactly what I said happens in Comment #18...

Comment 21

[Giorgos Logiotatidis [:giorgos]]

Therefore there is a privacy concern since a user can chose not to share city and the city is being shared.

Comment 22

[Giorgos Logiotatidis [:giorgos]]

(In reply to Andrei Hajdukewycz [:sancus] from comment #18)
> What I can do for this bug is change the code so that you are only included
> in the map if you have set your profile to save City. This should alleviate
> any privacy concerns -- at that point you are sharing the lat/lng for your
> City with other vouched mozillians, which is not at all private data, anyone
> who knows your City can look it up for themselves via the Mapbox API.

Yes that probably will work but I think a better solution is to store the lat/lon in the level you chose to share. I.e if I chose to share only country, the lat/lon should be the same as the one returned when I search for country. 

Do you think what will work without breaking something else?

Comment 23

[Andrei Hajdukewycz [:sancus]]

Yes, I think we should fix the underlying issue. I just don't think that the group maps page bug is the place to do it. It will involve a data migration to correct all the profiles that have overly-precise lat/lngs and there may be some complexity/workarounds needed, because mapbox doesn't always properly return lat/lngs for regions and countries last I checked.

I think restricting the map to City-enabled profiles is a reasonable stopgap until someone has time to work on that issue.

Comment 24

[Andrei Hajdukewycz [:sancus]]

https://github.com/Sancus/mozillians/commit/96b10b0c98e050af35342527205136a13936eaf4

I've updated the code accordingly.

Comment 25

[Andrei Hajdukewycz [:sancus]]

I actually mean: https://github.com/Sancus/mozillians/commit/483457ed6ba58cbe9b658bb24f026632f4ced499

Comment 26

[Greg Cox [:gcox]]

This is an RFE specific to mozillians, which is decom'ed.
If this is found to pertain to people.mozilla.org then please open a new issue.