1547936 - Google Cloud Storage buckets for code coverage cache

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Bastien Abadie [:bastien]]

I'm starting to work on improving performances for the code coverage backend, built by the Release Management team, in release-services.

We currently store data in multiple places: external coverage providers (Coveralls & Codecov.io), but also in ActiveData.
We want to pre-compute and store some coverage statistics in Amazon S3 buckets, that will then be processed by the backend (instead of computing per-request).

So I need 3 S3 buckets & associated IAM accounts, one for each release-services environments:

• relman-code-coverage-testing
• relman-code-coverage-staging
• relman-code-coverage-production

The credentials will be stored in Taskcluster secrets & retrieved through taskcluster-proxy.

Thanks for your help !

Comment 1

[Jeremy Orem [:oremj]]

We are generally trying to move things over to GCP. Would it work to store this data in Google Cloud Storage instead?

Comment 2

[Bastien Abadie [:bastien]]

As Google Cloud Storage has a Python lib, and has the same base features as S3, i think we can use it.

Marco, are you OK with this ?

Comment 3

[Marco Castelluccio [:marco]]

It would be more expensive and a bit slower (but we are not so concerned about upload speed), as we are going to upload data from Taskcluster (which doesn't support GCP yet). It will get better once Taskcluster supports GCP and we can switch to that for the tasks too.

If that's OK for you, it's OK for us.

Comment 4

[Jeremy Orem [:oremj]]

How much data will be stored in these buckets?

Comment 5

[Marco Castelluccio [:marco]]

It should be a few GBs per day. Only recent data will be accessed frequently, older data will be accessed very rarely.

Comment 6

[Jeremy Orem [:oremj]]

Sounds good. I'll create these buckets in GCP and send you the service account credentials.

Comment 7

[Jeremy Orem [:oremj]]

Do you want versioning on or a lifecycle policy attached to any of these buckets?

Comment 8

[Bastien Abadie [:bastien]]

Versioning is not necessary, but this type of data should stay in the bucket at least several months.

We may change that later in the project, as we explore different caching mechanisms / strategies.

Comment 9

[Jeremy Orem [:oremj]]

I've sent you a link to the credentials in slack. The buckets are named:

relman-code-coverage-staging
relman-code-coverage-testing
relman-code-coverage-prod

Right now the buckets have no versioning, lifecycle policies or backups attached to them. Let me know if you want me to add these later.

Comment 10

[Bastien Abadie [:bastien]]

I finally got time to start the implementation: i have an issue with the credentials & bucket you provided me...
I've stored the testing credentials on Taskcluster, and read them from the codecoverage/bot to instanciate a Python client using google-cloud-storage official library

I get the following error when trying to get the bucket named relman-code-coverage-testing:

google.api_core.exceptions.Forbidden: 403 GET https://www.googleapis.com/storage/v1/b/relman-code-coverage-testing?projection=noAcl: [e-mail] does not have storage.buckets.get access to relman-code-coverage-testing.

The client seems well configured, and the project is used according to the Exception message.

Here is my simple implementation for an upload.

Am i doing something wrong ? Is there a scope issue ?

Thanks for your help Jeremy.

Comment 11

[Jeremy Orem [:oremj]]

Ah, you have access to rw objects, but not do operations on buckets. I will add those permissions.

Comment 12

[Jeremy Orem [:oremj]]

You should have the right permissions now.

Comment 13

[Bastien Abadie [:bastien]]

I confirm I was able to upload and then retrieve a sample file on the testing bucket.

Thanks Jeremy !