1552950 - gcs emulator for local development

Status: RESOLVED FIXED

(Tecken :: General, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

From https://github.com/mozilla-services/tecken/issues/1247

Peter wrote:

If you, today, want to simulate GCS locally you can't. You have to actually have a real GCS bucket and real credentials.

With minio in docker-compose we were able to fake all that bucket and object put/get locally. Let's see if we can do the same for GCS.

And I later added:

Telemetry Edge has this (for pubsub): https://github.com/mozilla/gcp-ingestion/blob/master/ingestion-edge/pubsub_emulator.py

Maybe we can do something like that for GCS. We have this problem with Tecken and Socorro, so maybe that's worth spending a week of time on.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Bug #1550534 covers figuring this out for Socorro. I'm going to make this depend on that bug because once we figure out a solution, we'll need to implement it in Socorro as well as in Tecken and those implementations are separate units of work.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this a P2 and having this block GCP migration.

Comment 3

[John Whitlock [:jwhitlock]]

Testing if jwhitlock/gc-fake-storage can be used as the local GCS emulator for development.

Comment 4

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/tecken/pull/1830

Comment 5

[John Whitlock [:jwhitlock]]

PR 1830 was merged on July 9, 2019, with some additional issues. From the github comment:

This process was longer and messier than I hoped, but I think it is time to merge and move on to the next long, messy bit. A lot of discussion was threaded with the review, so to summarize:

• This adds a gcs-emulator service that provides a subset of the Google Cloud Storage API. It allows creating and listing buckets, and creating, listing, reading, and deleting objects. It does not include authentication or encryption, and several other features.
• This modifies the official GCS library to talk to the gcs-emulator instead of the official API endpoints. It does this in a way that prevents accessing the emulated and official API at the same time. I expect it will be a while, if ever, before the upsteam libraries make this change easier.
• The gcs-emulator image is currently jwhitlock/gc-fake-storage:test, which is built from the test branch of my fork of https://github.com/teone/gc-fake-storage. The long term plan is for the upstream project to merge the Docker build configuration. See https://github.com/fsouza/fake-gcs-server/issues/45 and bug 1563036.
• This PR follows the pattern of auto-creating emulated S3 buckets at server startup to create the emulated GCS buckets as well. The long term plan is to move both to make setup. See bug 1556775.
• The code that handles the difference in storage backends is spread over the code base. It may be better to create a generic object storage interface on StorageBucket, and hide the details of the backends from the client code. See bug 1564452.

Since then, the upstream project has taken over the docker image, so I can proceed on bug 1563036.

I believe this bug is closed for Tecken. The gcs emulator works for local airplane development, although full development of GCS integration should still use a real GCS bucket.

I think the Socorro work could proceed in bug 1550534, using this emulator image, but would benefit from waiting, as the GCS emulator and client code becomes more mature. I'm switching that bug to depend on this one.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Moving from Socorro product to Tecken product.