1252141 - Deploy TestPilot to Stage/Prod

Status: RESOLVED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Benson Wong [:mostlygeek]]

This bug will track progress of creating the ops assets (jenkins, cloudformation, puppet, etc) and deployment of the stage/prod environments for TestPilot.

Comment 1

[Benson Wong [:mostlygeek]]

Stage has been created / deployed. 

 - stage deploys successfully
 - jenkins, cloudformation and puppet code has been created
 - secrets stored encrypted at rest with SOPs

It is available at https://testpilot.stage.mozaws.net. 
Still need to verify that all configurations are set correctly.

Comment 2

[Benson Wong [:mostlygeek]]

Testpilot fixed in stage.

 - uses production FxA (accounts.firefox.com) for OAuth
 - fixed configuration issues 

I was able to sign in. \o/

Comment 3

[Benson Wong [:mostlygeek]]

Production deployed. Was able to log in.

Comment 4

[Benson Wong [:mostlygeek]]

Les, could you test out production and mark this bug as verified if it is ok.

Comment 5

[Les Orchard [:lorchard]]

Okay, my FxA login works, and the admin password works.

But, I get Secure Connection Failed errors on these admin pages when I try to create a new experiment:

https://testpilot.stage.mozaws.net/admin/experiments/experiment/add/
https://testpilot.firefox.com/admin/experiments/experiment/add/

Both of those pages involve multipart/form-data file uploads for images needed in experiment content. Other admin pages with form posts work, so I suspect the image submissions are the problem

Comment 6

[Benson Wong [:mostlygeek]]

Looks like it was blowing up the max body size that nginx allowed. I've increased this to 256MB. 
Stage and prod are both updated.

Comment 7

[Les Orchard [:lorchard]]

Okay, now I can get past the form submission on stage. But, the images 404 on S3:

http://testpilot-stage.s3-website-us-east-1.amazonaws.com/experiments_experiment/2/6/26faa091bc6f0767d90031ebb3898d6c_thumbnail_1457394529_0974.gif

Code: NoSuchWebsiteConfiguration
Message: The specified bucket does not have a website configuration
BucketName: testpilot-stage

Comment 8

[Benson Wong [:mostlygeek]]

Did you want these S3 buckets to also serve HTTP? 
That is no problem, in prod we prefer to put a CDN in front so the URL will have to be https://testpilot.cdn.mozilla.net

Comment 9

[Les Orchard [:lorchard]]

Though, FWIW, it does look like the image made it to S3:

http://s3.amazonaws.com/testpilot-stage/experiments_experiment/2/6/26faa091bc6f0767d90031ebb3898d6c_thumbnail_1457394529_0974.gif

Comment 10

[Les Orchard [:lorchard]]

> Did you want these S3 buckets to also serve HTTP? 

Yes, these are image uploads for Experiment site content

Comment 11

[Benson Wong [:mostlygeek]]

Alright, buckets have webhosting enabled.

Comment 12

[Wil Clouser [:clouserw]]

Thanks Ben and Les.  This is looking great.

Comment 13

[Les Orchard [:lorchard]]

Images on staging are working, but the URLs for prod images respond with 400 Bad Request:

http://testpilot-prod.s3-website-us-east-1.amazonaws.com/experiments_experiment/2/6/26faa091bc6f0767d90031ebb3898d6c_thumbnail_1457455911_0008.JPG

Code: IncorrectEndpoint
Message: The specified bucket exists in another region. Please direct requests to the specified endpoint.
Endpoint: testpilot-prod.s3-website-us-west-2.amazonaws.com

I can confirm that the image made it up to S3, though, with a direct bucket URL:

http://testpilot-prod.s3.amazonaws.com/experiments_experiment/2/6/26faa091bc6f0767d90031ebb3898d6c_thumbnail_1457455911_0008.JPG

Comment 14

[Les Orchard [:lorchard]]

Oh yeah, and this URL works:

http://testpilot-prod.s3-website-us-west-2.amazonaws.com/experiments_experiment/2/6/26faa091bc6f0767d90031ebb3898d6c_thumbnail_1457455911_0008.JPG

So, maybe just a matter of changing the configured base URL? Though, we probably want an HTTPS CDN in front of that so we don't have mixed content issues on prod

Comment 15

[Benson Wong [:mostlygeek]]

prod is on a CDN now: 

curl -I https://d3b379h2pte65z.cloudfront.net/experiments_experiment/2/6/26faa091bc6f0767d90031ebb3898d6c_thumbnail_1457455911_0008.JPG