1357118 - Please set a correct certificate on https://release.mozilla.org/ (hosted on github)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Sylvestre Ledru [:Sylvestre]]

Reported also here:
https://github.com/mozilla/release-blog/issues/16

Thanks!

Comment 1

[Shyam Mani [:fox2mike]]

Hey guys,

Sorry this bug sat in the wrong queue. We are working out some possible solutions here and will get back to you by Austin. Thanks!

Comment 2

[Pascal Chevrel:pascalc (PTO until April 26)]

(In reply to Shyam Mani [:fox2mike] from comment #1)
> Hey guys,
> 
> Sorry this bug sat in the wrong queue. We are working out some possible
> solutions here and will get back to you by Austin. Thanks!

Hi Shiam, any progress on this one?  thanks!

Comment 3

[Pascal Chevrel:pascalc (PTO until April 26)]

Shiam, can we get a status update? Can this bug be assigned to somebody in your team? Thanks

Comment 4

[Shyam Mani [:fox2mike]]

Hey Pascal,

Webops will pick this up in our next sprint starting this Friday. Expect updates on the bug!

Comment 5

[Daniel Hartnell [:danielh]]

Hi Pascal,

I have a staging site setup which will use the gh-pages branch of the mozilla/release-blog repository to build the Jekyll blog. The site itself is hosted from S3 and provides an SSL certificate through ACM. It should pick up any changes to that branch automatically (we can validate that after the next push). Due to caching in CloudFront, you might see a little latency between your changes in GitHub and what we're hosting. I wouldn't expect more than a 10 minute wait.

For now, we're hosting the site at this domain:

https://release.allizom.org/

If you can look over that and confirm that everything is functioning, we can start work on the production site (that will be pretty quick to setup). If you have any other requirements, let me know and we can work through that.

Comment 6

[Pascal Chevrel:pascalc (PTO until April 26)]

I did several commits today to our repo and I confirm that they were picked up by https://release.allizom.org/, you can proceed with the work on the production site. Thanks!

Comment 7

[Daniel Hartnell [:danielh]]

Pascal,

Awesome! Happy to hear that it's working well. I have created a new bug to track our work around the production implementation. I'll get you an update here once that's complete. If you have any questions in the meantime, let me know.

Comment 8

[Sylvestre Ledru [:Sylvestre]]

Many thanks. This is very appreciated! :)

Comment 9

[Daniel Hartnell [:danielh]]

You're welcome, Sylvestre!

Comment 11

[Pascal Chevrel:pascalc (PTO until April 26)]

Daniel, can I get an ETA on this please? thanks

Comment 12

[Daniel Hartnell [:danielh]]

Hey Pascal,

The work for the production site was pulled into this sprint. I would like to plan to have DNS switched over next week - tentatively on Wednesday, February 28th. I have a few technical tasks to complete to provision the resources needed in our AWS account. I also need to work with our security team to review the architecture and make sure they're comfortable with what we've done. A very similar pattern is used on another website and I expect that they'll be happy with what we've done here.

Comment 13

[Pascal Chevrel:pascalc (PTO until April 26)]

Thanks for the update Daniel

Comment 14

[Daniel Hartnell [:danielh]]

Pascal,

Quick update - I see no reason that we cannot continue with my planned timeline above. We have passed our security review which means there is nothing preventing us from making the production switch on Wednesday. I expect the change to be a no-downtime event so I am comfortable making the DNS change at 11am pacific on Wednesday. If you want to schedule the change at a different time, just let me know. Once DNS propagates and new users are directed to our AWS infrastructure, they will automatically see the new SSL certificate.

release.mozilla.org is the only domain name that I am aware of. If there is another one, let me know and we can plan to update that as well.

Comment 15

[Pascal Chevrel:pascalc (PTO until April 26)]

(In reply to Daniel Hartnell [:danielh] from comment #14)
> Pascal,
> 
> Quick update - I see no reason that we cannot continue with my planned
> timeline above. We have passed our security review which means there is
> nothing preventing us from making the production switch on Wednesday. I
> expect the change to be a no-downtime event so I am comfortable making the
> DNS change at 11am pacific on Wednesday. If you want to schedule the change
> at a different time, just let me know. Once DNS propagates and new users are
> directed to our AWS infrastructure, they will automatically see the new SSL
> certificate.
> 
> release.mozilla.org is the only domain name that I am aware of. If there is
> another one, let me know and we can plan to update that as well.

Sounds good, thanks Daniel

Comment 16

[Daniel Hartnell [:danielh]]

Hey Pascal,

The changes took affect at about 11:30am pacific. HTTPS connectivity works great! I did run into an issue though - I just rolled it back because our monitoring is showing some failures in certain regions around the world (Mumbai and parts of Australia among others). It looks like the CloudFront distribution in AWS is preventing access from those regions. I'll do a little research, identify the problem, update the CloudFront configuration, validate that everything is working with our monitoring and then move the site again. I expect to have that done today but I'll send out another progress report shortly.

Comment 17

[Daniel Hartnell [:danielh]]

Pascal,

In my initial CloudFront distribution I mistakenly set geo restrictions which blocked access from a number of countries. There was a brief window of time (I'm pretty sure it was less than 30 minutes) where someone from Australia, India and a number of other countries may have received an error. As soon as I noticed the issue, I updated DNS to point users back to GitHub.

Since then, I was able to identify the cause, fix the CloudFront distribution configuration and restore DNS back to our site in AWS. Right now, it looks like it is working perfectly.

- I have global monitoring setup through New Relic to make sure the site remains available (the issue above was missed because I was only monitoring from the United States initially)
- HTTP connections are automatically redirected to HTTPS and we have an SSL certificate for the domain
- As we noted with our staging website, any updates that are pushed to the gh-pages branch in the repository will automatically take affect in about 5-10 minutes
- We're getting a B in Mozilla's Observatory as well! https://observatory.mozilla.org/analyze.html?host=release.mozilla.org I would like to bump that to an A in the near future by adding a CSP header

https://release.mozilla.org/

Go ahead and take a look. If you have any questions or concerns, let me know.

Comment 18

[Pascal Chevrel:pascalc (PTO until April 26)]

Thanks Daniel, this is working, happy to see our blog finally served from https!

Comment 19

[Daniel Hartnell [:danielh]]

Hooray! I'm happy we're serving https content as well! I'm going to close this now but I'll be around if you need anything.