Closed Bug 1200358 Opened 11 years ago Closed 10 years ago

Elevated Keys for Community Dashboard

Categories

(Participation Infrastructure :: API Requests, task)

Product:
Participation Infrastructure
Component:
API Requests
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mkohler, Unassigned)

References

(
URL
)

Details

For https://discourse.mozilla-community.org/t/gathering-country-data-from-mozillians-org-needs-help/3716 we will need an elevated key to access the skills. Since there are multiple devs involved in this, how can we manage this?
Flags: needinfo?(jgiannelos)
Flags: needinfo?(nikos)
Hi :mkohler In order to create a new API key with elevated access, first of all the owner should create a new API key through mozillians.org. Then you can tell us the app name and we grant access to that app. Just some notes on that: * API v2 provides the same schema in both public/mozillians/privileged access level. It's the API results that would be different (respecting user privacy per field). For example, for "skills" you can try this: curl -XGET "https://mozillians.org/api/v2/skills/" -H "X-API-KEY: <PUBLIC_API_KEY>" * In order to grant you access to non public data we first need to make sure there is no data leak (eg. for mozillians only data we need to make sure AUTHN/AUTHZ exists and is based on mozillians.org). * API requests should be proxied in order to avoid API key leaks (API keys *should not* be available in client side JS code). What we usually do is have some backend code to do the authentication and the API requests and then show data only to authenticated users. Also please use our latest API version (v2 not v1). (cc'ed pierros to add him in the loop)
Flags: needinfo?(nikos)
Flags: needinfo?(jgiannelos)
(In reply to John Giannelos [:nemo] from comment #1) > In order to create a new API key with elevated access, first of all the > owner should create a new API key through mozillians.org. Then you can tell > us the app name and we grant access to that app. CommunityDashboard thanks!
Flags: needinfo?(jgiannelos)
Thanks :mkoehler. You can continue the development with the same API key since the results won't change after we grant elevated access. Is there any code so far that we can review? Or notes with your project's architecture to know how are you going to handle the data?
Flags: needinfo?(jgiannelos)
Hi :nemo, I have made a Django app which will make this work done. Can you please review the app in order to issue a Elevated Keys? Here is the App Source: https://github.com/safwanrahman/community Let me know if any improvements or change is needed in the app.
Flags: needinfo?(jgiannelos)
Hey all, Please tag a release on the code, so we can do a review within this week to assign elevated API access keys. Post the link of the release here for review.
(In reply to Pierros Papadeas [:pierros] from comment #5) > Hey all, > > Please tag a release on the code, so we can do a review within this week to > assign elevated API access keys. Post the link of the release here for > review. Once the front end is ready within tonight, I will ship a release! :) Thanks
(In reply to Pierros Papadeas [:pierros] from comment #5) > Hey all, > > Please tag a release on the code, so we can do a review within this week to > assign elevated API access keys. Post the link of the release here for > review. Hi, here I have made a release 0.1.0. https://github.com/safwanrahman/community/releases/tag/0.1.0 Please review as soon as possible. The information need to get ready by 11th September. so please concern this.
Flags: needinfo?(pierros)
Some notes here regarding your project: * Please *don't* distribute API keys even for public data * https://github.com/safwanrahman/community/blob/master/mozillians/settings.py#L26 * Disabled it for now in mozillians.org and we need to delete it soon. * Let's make sure no API keys get pushed to public repos again. * I don't see any authentication happening. Is it going to be public for everyone? * Since you don't need the usernames for any other reason except of associating skills-users-countries and its not good idea to share publicly usernames I suggest you either anonymize it or scrap it after the data collection (no need to be in the database in order to render the dashboard).
Flags: needinfo?(jgiannelos)
I've contacted Safwan to remove the API code from the repo ASAP.
Flags: needinfo?(safwan.rahman15)
(In reply to John Giannelos [:nemo] from comment #8) > Some notes here regarding your project: > > * Please *don't* distribute API keys even for public data > * > https://github.com/safwanrahman/community/blob/master/mozillians/settings. > py#L26 > * Disabled it for now in mozillians.org and we need to delete it soon. > * Let's make sure no API keys get pushed to public repos again. Sorry about this. I did not notice that it was published through the settings.py. I apologies. I have removed it from the repo and rest other history. So it will no longer be there > * I don't see any authentication happening. Is it going to be public for > everyone? The dashboard will be public for anyone. As its only showing the stats, and nothing more confidential which violets the individual user privacy, I think its ok to have show the stat public. > * Since you don't need the usernames for any other reason except of > associating skills-users-countries and its not good idea to share publicly > usernames I suggest you either anonymize it or scrap it after the data > collection (no need to be in the database in order to render the dashboard). As the data collection is really huge (about 9000 skills API need to be called), without storing the usernames into the database, it seems bit critical to get the work done. ""Please be informed that, though the usernames are stored in the database, it would not show in the dashboard. So there is no way to share the usernames publicly. We are showing the stats only!
Flags: needinfo?(safwan.rahman15)
I know but since the dashboard calculations don't require the username and _url fields you can keep it for the data collection and then scrap it (eg. delete the columns).
(In reply to John Giannelos [:nemo] from comment #11) > I know but since the dashboard calculations don't require the username and > _url fields you can keep it for the data collection and then scrap it (eg. > delete the columns). Thanks for clearing. Yes, the data is not needed for dashboard calculation. I have shipped another release 0.1.1 which scrap the not needed data (username and URL). Here it is: https://github.com/safwanrahman/community/releases/tag/0.1.1 Please review it as soon as possible as we need it get done by tomorrow!
Flags: needinfo?(jgiannelos)
Code looks OK (in terms of data sanity). I see 2 management commands: `fetchdata.py`, `fetchskills.py`. The first one keeps usernames in the DB. I assume that you are using the second one right?
Flags: needinfo?(jgiannelos)
(In reply to John Giannelos [:nemo] from comment #13) > Code looks OK (in terms of data sanity). I see 2 management commands: > `fetchdata.py`, `fetchskills.py`. > The first one keeps usernames in the DB. I assume that you are using the > second one right? Yap. The first one gets the usernames and the second one gets the skills of that collected usernames in the DB and then delete the usernames wise skill data and other unused data. So, the 2 management commands are being used for fetching the data to show in the dashboard. :)
FYI, the previous key got deleted for security reasons, so for your app you need to create a new one. Also ATM I can see 2 ``community dashboard`` API keys from different owners. Do you need those?
(In reply to John Giannelos [:nemo] from comment #15) > FYI, the previous key got deleted for security reasons, so for your app you > need to create a new one. Also ATM I can see 2 ``community dashboard`` API > keys from different owners. Do you need those? Probably one of those is mine, though the name differs "CommunityDashboard". This is the one I mentioned in comment 2.
Hi, I have just registered a new app name "Community Dashboard" from my account. Please issue elevated access for that keys. My mozillians account email is same as my email. Here is my profile: https://mozillians.org/en-US/u/safwan/ So issue for this owner
(In reply to John Giannelos [:nemo] from comment #15) > FYI, the previous key got deleted for security reasons, so for your app you > need to create a new one. Also ATM I can see 2 ``community dashboard`` API > keys from different owners. Do you need those? ^^
Flags: needinfo?(jgiannelos)
:pierros should I proceed and grant permissions? Code looks OK.
Flags: needinfo?(jgiannelos)
Elevated keys will be granted for a local instance of Nukeador till we figure out hosting etc. Nuke please let us know when you have a key issued so we can elevate it.
Flags: needinfo?(pierros)
I've create a key called "Bug 1200358". I'm testing the app locally right now with normal privileges. As soon as I check everything look good, I will ping here for elevated one.
So, country data is correctly fetched, it takes some time, but I cat get results. Can I get the privileged key to fetch the real data? Thanks!
Flags: needinfo?(jgiannelos)
:Nukeadon Done. I changed the permissions to your API key.
Flags: needinfo?(jgiannelos)
/s/Nukeadon/Nukeador/g/ :)
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.