Closed Bug 1528112 Opened 6 years ago Closed 6 years ago

mdn-samples.mozilla.org subdomain has gone away

Categories

(Infrastructure & Operations :: SRE, task)

Product:
Infrastructure & Operations
Component:
SRE
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: sheppy, Assigned: limed)

Details

The MDN sample server (used for a few specific examples that need special support in order to operate) is operated and maintained by the MDN writing team on AWS. It has operated under the name mdn-samples.mozilla.org for several years now, and has shared an SSL cert with the main MDN site.

I have been handling admin tasks on the server as well as doing most of the code work on it and its support scripts and such.

The purpose of this bug: the domain mdn-samples.mozilla.org has vanished. The EC2 server is still online at 52.0.70.144, but the name is not connected to it any longer. I am not clear when this happened although I don't think it could have been all that long ago.

We need the name reconnected so links to this server begin working again; it's linked to from a number of places including several popular pages on MDN.

Then, once the name is reconnected, we need to confirm that the SSL certificate is up to date. It currently claims it's out of date, but since a CNAME from this server to another is used to share the certificate, I don't know that that claim is necessarily accurate.

mdn-samples.mozilla.org is still a CNAME pointing to mdn-samples.moz.works. I believe all of the MDN related DNS entries were removed from the moz.works domain within a couple of weeks of the move of MDN to IT managed infrastructure[0], so this has likely been broken since November of last year.

The server at 52.0.70.144 is listening on port 443, but that port is not configured for HTTPS. The deleted DNS record was most likely previously pointed to an ELB with a cert issued from ACM.

[0] https://hacks.mozilla.org/2018/11/performance-and-hosting-moves-mdn-changelog-for-october-2018/#moved-mdn-to-mozit

Is there a DNS change that needs made here? I'm happy to help if there is.

I have been handling admin tasks on the server as well as doing most of the code work on it and its support scripts and such.

:sheppy since MDN was migrated to the IT managed AWS account we should probably talk about moving this server to the IT AWS account, is the code for this hosted anywhere so that we can stand up a new EC2 instance on the new account. Let me know, thanks

As a stop gap can the mozmeao group perhaps recreate the ELB with an ACM temporarily while we stand up this new instance?

Assignee: infra → nobody
Component: DNS and Domain Registration → Infrastructure: AWS

(In reply to Ed Lim [:limed] from comment #3)

I have been handling admin tasks on the server as well as doing most of the code work on it and its support scripts and such.

:sheppy since MDN was migrated to the IT managed AWS account we should probably talk about moving this server to the IT AWS account, is the code for this hosted anywhere so that we can stand up a new EC2 instance on the new account. Let me know, thanks

As a stop gap can the mozmeao group perhaps recreate the ELB with an ACM temporarily while we stand up this new instance?

The server that Sheppy is describing was never in the Marketing AWS account; only the temporary intermediate moz.works DNS entry was. It predates the move of MDN to Marketing.

Sheppy, since you are an admin of this machine, I assume you have AWS credentials to manage that EC2 instance? If so, you should be able to see it at the following url:

https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Instances:search=52.0.70.144

(I'm basing the region in the URL above on the DNS PTR record for that IP, ec2-52-0-70-144.compute-1.amazonaws.com.)

If you can see the EC2 instance on that page, please reply with the AWS account number in the upper right corner.

Flags: needinfo?(eshepherd)

I found the instance on cloudhealth, I think this is an instance in the cloudservices-aws-dev account, :sheppy if you have the ELB address I can setup the DNS for you to have this fixed in the short term. But this should probably be moved to the MDN AWS account at the end of the day

I'm going to assign this to myself so it stops alerting in IRC

Assignee: nobody → rtucker

Sorry for the delay -- wound up sick for a couple days and couldn't deal with this.

The ELB IP is 52.0.70.144

I don't see an AWS account number on the screen. I can tell you that the account is listed as cloudservices-aws-dev. Is that what you need?

The code is all on github and is hypothetically built to practically set itself up, although its first-time-startup code hasn't been tested in a while. The instance has been running for a little over three years.

Flags: needinfo?(eshepherd)

:sheppy if you have the DNS name that would be better. You will need to go to EC2 -> Load balancer (on the left) -> Select the elb -> DNS Name

Flags: needinfo?(eshepherd)
Assignee: rtucker → limed

@:limed - The only DNS names I see:

Public DNS: ec2-52-0-70-144.compute-1.amazonaws.com
Private DNS: ip-172-31-42-195.ec2.internal

I don't see anything about load balancers other than the sidebar link that takes me to a list of them, none of which seem right at a glance.

Flags: needinfo?(eshepherd)

Anyone progress on this? I have been asked again about it. Thanks!

Flags: needinfo?(limed)

Fixed dns and it loaded fine (needs a cert renewal though)

Flags: needinfo?(limed)
Flags: needinfo?(eshepherd)

How do I go about getting the cert that mdn-samples.mozilla.org shares with the other MDN servers into place? Or does it need one of its own again? In the past both jgmize and ckolos have dealt with these things, at least in part.

Flags: needinfo?(limed)
Flags: needinfo?(jmize)
Flags: needinfo?(eshepherd)
Flags: needinfo?(ckolos)

CloudOps isn't managing this, so I'm not probably not the right guy to handle the cert.

Flags: needinfo?(ckolos)

(In reply to Eric Shepherd [:sheppy] from comment #12)

How do I go about getting the cert that mdn-samples.mozilla.org shares with the other MDN servers into place? Or does it need one of its own again? In the past both jgmize and ckolos have dealt with these things, at least in part.

(In reply to Chris Kolosiwsky [:ckolos] from comment #13)

CloudOps isn't managing this, so I'm not probably not the right guy to handle the cert.

Any new ACM certs would need to be provisioned in the cloudservices-aws-dev account you described in comment #7, but I agree with :ckolos that wouldn't be an appropriate use of CloudOps resources.

I believe the best way to proceed is as :limed described in comment #3 and for you two to work together to recreate that server or service as a new ec2 instance managed by terraform, a k8s service, or whatever else :limed thinks best.

Flags: needinfo?(jmize)

(In reply to Josh Mize [:jgmize] from comment #14)

(In reply to Eric Shepherd [:sheppy] from comment #12)

How do I go about getting the cert that mdn-samples.mozilla.org shares with the other MDN servers into place? Or does it need one of its own again? In the past both jgmize and ckolos have dealt with these things, at least in part.

(In reply to Chris Kolosiwsky [:ckolos] from comment #13)

CloudOps isn't managing this, so I'm not probably not the right guy to handle the cert.

Any new ACM certs would need to be provisioned in the cloudservices-aws-dev account you described in comment #7, but I agree with :ckolos that wouldn't be an appropriate use of CloudOps resources.

I believe the best way to proceed is as :limed described in comment #3 and for you two to work together to recreate that server or service as a new ec2 instance managed by terraform, a k8s service, or whatever else :limed thinks best.

Agreed with this, lets work to get this working on the MDN account. In the meantime if you need the site up right now I would recommend setting up LetsEncrypt for the cert. If you can give me access to the server I can help set that up

Flags: needinfo?(limed)

If this is in cloudops-aws-dev, ACM could be used as a self-service option on the ELB. CloudOps doesn't maintain the systems running in -dev, but we do allow those running things in there to manage resources as they need to. A cert could easily be generated in ACM, modulo the verification of the domain, and assigned to the ELB fronting the server.

Ed: can you send me a public key for use on the server so I can set you up?

Flags: needinfo?(limed)

:ckolos -- it is in fact cloudops-aws-dev. I have no direct preference for how this gets sorted out as long as it does. Long term, I'd like for us to come up with a more permanent, less fragile solution, but let's get it up and running as-is for starters.

Where and how it's hosted doesn't matter much to me. The process of standing up the samples-server is pretty easy; nearly everything is automated once you get the base instance itself created. As long as there's a way to keep this process (or one like it) working, we're good.

Quick overview of the process of standing up the site: All of the code is on github: https://github.com/mdn/samples-server/

Standing it up works like this currently:

  1. Create an EC2 instance starting with one of the SvcOps CL7 images.

  2. Open the new instance's User Data editor. From the GitHub repository, take the contents of the file user-data.sh and paste it into the User Data editor. Save that.

  3. Boot the instance.

That should be it. The User Data script is run every time the instance is spawned, and does a bunch of stuff:

  • Automatically installs the needing components
  • Sets up permissions, creates groups, etc
  • Clones the GitHub repo into the web directory (/var/www/html)
  • Fetches the startup and update script (update.sh) from GitHub and installs it so the server will run it every time the instance is booted up
  • Finally, hands off control to the startup script to ensure that the setup/startup process is run on instance spawning

The update script does the real work:

  • Updates the system software and installed software
  • Ensures that sshd is set up correctly and starts it up
  • Pulls the latest samples-server repository code from GitHub, into the web directory
  • Installs the latest version of adapter.js in the location it's expected to be in by the web content
  • Updates permissions throughout the web tree
  • Ensures that the Apache configuration is set up to disable directory indexes and to make attempts to enter a folder named ".git" return a 404
  • Starts up or restarts Apache
  • Downloads, builds, and installs COTURN, the STUN/TURN server hosted for the WebRTC examples served by the site
  • Starts up COTURN
  • Starts up the Python app that iterates over each example's directory looking for "startup.sh" files, executing them, to allw samples to prefer any needed startup-time tasks

(In reply to Eric Shepherd [:sheppy] from comment #17)

Ed: can you send me a public key for use on the server so I can set you up?

You can use any key listed on https://github.com/limed.keys

Flags: needinfo?(limed)

:limed - you should now be able to ssh into mdn-samples.mozilla.org; any one of those keys should work. I added you to group wheel and confirmed that you can do things like use sudo to access files regular users can't. Please let me know if you need more done. At this point, pinging me in IRC probably makes more sense than the long round-trip time involved in using Bugzilla for stuff like this. :)

I setup letsencrypt for this, all remaining items sans moving to MDN AWS account should be fixed. However I have filed https://github.com/mdn/infra/issues/209 to keep track of moving the server to MDN's AWS account

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.