708035 - Setup production servers for bedrock

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[James Long (:jlongster)]

This bug tracks the progress of settings up servers in the production environment for hosting bedrock, the new Python mozilla.org site.

We have plans to get one or two simple, low-traffic pages serving by the end of Q4. Jake has said we can work towards that.

Comment 1

[Shyam Mani [:fox2mike]]

Jake, over to you? :) 

Or let one of us know the plan, we'll be happy to help.

Comment 2

[James Long (:jlongster)]

Please note that we really don't need a database for this at first. We would like one soon (maybe Q1 of next year, if that) and leverage the work you guys are doing for multi-cluster dbs, but the main site will never use a database.

The databases will be used on smaller pages and subsites, which is not an urgent need.

Comment 3

[James Long (:jlongster)]

Jake, you've mentioned that all you need is a date for this to go live. Fred and I talked and we settled on January 31st. We want to have just one or two pages by then, something like the 404 page or http://www.mozilla.org/en-US/about/participate/.

Again, no database required. Just a basic Python setup. Please let me know if this works.

Comment 4

[Jake Maul [:jakem]]

Here's the plan. For new CC's, 'bedrock' is the new Django app for www.mozilla.org.

Spin up bedrock prod nodes in PHX1. We need to scrounge up hardware for this. This will become a sole source for www.mozilla.org for a couple weeks (apart from caching nodes, which my understanding is a large portion of the content will be cache-able. We'll do caching nodes in at least AMS1 (for EU performance) and probably SJC1 (for redundancy). This should let us hit the 1/31 date.

We'll spin up a new prod cluster in SCL3, as soon as we can. This will probably be mid-to-late February. This will replace the caching node in SJC1.

Once we have prod stuff in SCL3, we'll start work on the database replication issues. Unknown exactly how long this may take, but hopefully not too long. We have a DB set up in PHX1 already, we'll need to set up the SCL3 end and the replication between them... so, the hard part. :)

By sometime in March, we aim to be completely "full-production" in both places.



Assigning to cshields for node allocation for PHX1 prod cluster. oremj mentioned it would be okay to take 3 nodes from the new rack of 64 that should hopefully be online soon-ish. There is already an admin/dev/stage cluster to plug this into, but might need a little bit of puppet tweaking for prod.

Adding WebOps folks to CC list so they know what's up with this project, just so they're up to speed on the overall plan.

Comment 5

[Jake Maul [:jakem]]

The new rack of blades is online, we should be able to allocate the 3 we need shortly. I believe Phong's team is still working on getting the chassis's all squared away first (HP Onboard Admin stuff), then we can do snag 3 and get to work.

Comment 6

[Jake Maul [:jakem]]

Assigning to Phong for action:

cshields says we can re-allocated pp-zlb03 and pp-zlb04 for this, as well as spin up a VM in PHX1 (really want to have 3 nodes). The sooner we can have these re-kicked and puppetized, the better: we want to have live content on them by Monday.

OS should be RHEL6, 64-bit

Names should be:
bedrock[123].webapp.phx1.mozilla.com

Thanks!

Comment 7

[James Long (:jlongster)]

Just a ping here. We want to push this next week. We were thinking Monday, but with the Firefox release on Tuesday, maybe we should push it to Thursday or something. What do you think?

Comment 8

[Dan Parsons [:lerxst]]

I've created DNS entries for all 3 hosts and created a VM for the third one:

bedrock1.webapp.phx1.mozilla.com has address 10.8.81.86
bedrock2.webapp.phx1.mozilla.com has address 10.8.81.87
bedrock3.webapp.phx1.mozilla.com has address 10.8.81.88 (this is the VM)

Comment 9

[James Long (:jlongster)]

\o/ Thanks guys. Thursday's looking good. The push bug is the blocked bug.

Comment 10

[Michael Burns [:mburns]]

After a bit of poking and prodding, these hosts have been all setup with rhel6+puppet.

Comment 11

[James Long (:jlongster)]

How much power do these servers have? Zeus will cache these pages for 15 minutes, so I'm assuming we can still handle a good bit of load.

Comment 12

[Jake Maul [:jakem]]

Should be quite powerful. Only 3 nodes, but 2 of them are ex-Zeus nodes with ridiculous amounts of RAM and CPU power.

If we need more, we should be able to get a blade or two like we were originally planning... but I don't see that being very likely.