We should have detailed documentation on how to request and use a GCP service account to access bigquery for prototyping/dev work
Categories
(Data Platform and Tools :: Documentation and Knowledge Repo (RTMO), task)
Tracking
(Not tracked)
People
(Reporter: wlach, Unassigned)
Details
(Whiteboard: [firefox-data-docs-github-import])
** This is an automated import from issue #370 filed on https://github.com/mozilla/firefox-data-docs/ (see bug 1628699) **
Original report (from @wlach)):
The missioncontrol-v2 project was a bit of a pain to set up on my workstation. First I was fumbling around not really knowing what I had to do (create a service account for a new GCP project), then I had to ask @jasonthomas, @haroldwoo, and @whd and various other people for expanded permissions, and some manual fiddling was required so that this service account/project had the required permissions to the production tables in question (clients_daily and telemetry.crash).
We have a very very basic stub of instructions here, but it really needs expanding:
https://docs.telemetry.mozilla.org/cookbooks/bigquery.html#gcp-bigquery-api-access
Ideally we would have a document explaining the basics of service accounts and what's needed to request access to specific tables needed for development. The intended audience would be data science + engineering. Ideally these instructions, if properly followed, would be enough to have credentials sufficient to run things like missioncontrol-v2, bigquery-etl, and telemetry-airflow with a minimum of followup.
We may need to sort out some details of how permissions work on the backend before we can write this.
See also:
- @whd's gcp-quickstart project: https://github.com/whd/gcp-quickstart
- @fbertsch's instructions for setting up telemetry-airflow https://bugzilla.mozilla.org/show_bug.cgi?id=1553559#c1
- bigquery-etl's documentation for running tests: https://github.com/mozilla/bigquery-etl/blob/master/tests/README.md
|
Reporter | |
Comment 1•6 years ago
|
||
Another detail worth documenting: permissions granted to these service accounts is not automatic. There is an ACL-granting job which ops runs on a regular basis. Under normal conditions it should grant the permissions within one hour of the service account being created, but may take longer in some cases.
This slack conversation has some details:
https://mozilla.slack.com/archives/GE83ZMSAW/p1588018650312000
Description
•