1415899 - Add fx-test-jenkins-s3-publisher S3 profile to qa-master.fxtest.jenkins.stage.mozaws.net

Status: VERIFIED FIXED

(Cloud Services :: FXTest-infra, enhancement)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

In order to publish to S3 from the new Jenkins host at qa-master.fxtest.jenkins.stage.mozaws.net the following profiles need to be created:

net-mozaws-stage-fx-test-activedata
net-mozaws-stage-fx-test-treeherder

These are configured in Manage Jenkins > Configure System under the Amazon S3 profiles section. The Use IAM Role checkbox should be enabled, and the permissions on the buckets may need to be modified to allow this host to write to them.

See bug 1352229 comment 3 for details on how the credentials were configured.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Sorry, a correction. I believe we only need a single profile named fx-test-jenkins-s3-publisher.

Comment 2

[Jeremy Orem [:oremj]]

Which instances need roles? Just the master or master and workers?

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Anywhere that jobs would execute may need to publish to the S3 profile.

Comment 4

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

I think we'd prefer to use different Access Keys for each profile. If we created IAM accounts in the form of qa-<host>-s3-publisher, would that cause problems for the job configs?

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We currently rely on a default profile of 'fx-test-jenkins-s3-publisher' in the shared library: https://github.com/mozilla/fxtest-jenkins-pipeline/blob/c1e22572071172418cf5bff389ba3ffea0cd8119/vars/publishToS3.groovy#L27

My understanding was the the profile name was only used by Jenkins to establish the credentials and other settings for publishing via S3. We've been using "IAM Role" for authentication, which I believed to mean the the Jenkins instance would have its own role. As far as I can tell from looking through the plugin's source code, the profile name is not used when authenticating with S3.

If this is the case, we should be able to have the same profile name on multiple Jenkins instances, but for those instances to have distinct IAM roles/accounts as you're suggesting in comment 4. I apologise if I've misunderstood, and would appreciate if you could help me to have a better understanding of how the profile names are bound to IAM accounts.

Comment 6

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

As shown here:

https://screenshots.firefox.com/qgn4H6g0SL82LIIl/fx-test-jenkins.stage.mozaws.net

The s3 publisher uses a specific access key/private key pair to publish to s3. Access keys are associated with IAM (think user) accounts. Each IAM account has a limit of 2 keys active at any one time.

Unless we will never have more than one job publishing to a S3 bucket, we will not be able to provide enough keys to cover all the job+profile+bucket combinations. For this reason, we would like to create a separate IAM account for each publishing profile. Ideally, this IAM account would differ in name between stage/prod, but this doesn't have to be the case; We can use each of the two keys in a different env as needed (one in stage, the other in prod).

Does that help?

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

As mentioned on IRC we currently have most jobs publishing to two buckets:

Logs are published to 'net-mozaws-stage-fx-test-activedata' for consumption by ActiveData
Artifacts are published to 'net-mozaws-stage-fx-test-treeherder' for display in Treeherder

These are currently hard-coded in the shared library via the submitToActiveData and submitToTreeherder steps, and both also use the hard-coded profile name of 'fx-test-jenkins-s3-publisher'.

We could make these steps accept additional arguments for profile and bucket, however in order to use the same pipelines against Jenkins instances with different profile and buckets associated we'd either need to be able to construct the names from existing environment variables, or we'd need to store these values outside of the pipelines and shared library.

If we're going to create multiple buckets for ActiveData, then we'd also need to ask Kyle to set up data injestion from each new bucket. I'm not sure how much work is involved in this.

I think what would help at this point is a list of the profile and bucket names that you're proposing. From that we can see how much work would be involved in modifying our pipelines and associated services to the new model.

In case it makes any difference, there are currently no plans to publish to any additional buckets, and the only things we currently publish are test logs and reports, which do not contain sensitive information.

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

ActiveData and Treeherder are now working for qa-master.fxtest.jenkins.stage.mozaws.net, which is what this bug was concerning. I have noticed that publishing to S3 from qa-preprod-master.fxtest.jenkins.stage.mozaws.net is failing, but I will file a separate bug for this.

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; looked through the build logs, and using https://qa-master.fxtest.jenkins.stage.mozaws.net/job/fxapom.stage/43/console as an example, this is verified.