969533 - Please install Python 2.7 on the machines in the bedrock cluster

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Description

[Paul [:pmac] McLanahan]

I've heard that this is now doable, and it would be very nice to have for bedrock. This would include the dev/demo server, stage, prod, and the admin node.

There's no rush, but will be quite nice to have.

Thanks for looking into this.

Comment 1

[Daniel Maher [:phrawzty]]

(See also bug 969183 and bug 969182)

Comment 2

[Jake Maul [:jakem]]

Where did you hear this is now doable? I don't think it is, at least on our normal clusters. It might be possible to side-install 2.7, like we did with 2.6 back when our OS's had 2.4 as the default... but that would mean recompiling and side-installing a lot of other packages as well. Maybe someone has already done this... not sure.

PaaS can use 2.7, maybe that's where this came from?

My info could be out of date... ;)

Comment 3

[Paul [:pmac] McLanahan]

(In reply to Jake Maul [:jakem] from comment #2)
> Where did you hear this is now doable? I don't think it is, at least on our
> normal clusters.

A side-install would be just fine, and is really what I should have said. "Upgrade" is not what I meant. "Make available" is a better way to put it. I heard this based on other webdev teams saying they currently have 2.7 in production, and hearing that Sumo and Input were requesting 2.7 right now. I was told that getting it on generic cluster is likely a non-starter, but for hardware dedicated to a single site (like bedrock) might be doable. I'm quite happy to update our references to use "python2.7" in our deployment scripts. All that would need doing by Ops would be installing 2.7 and configuring apache to use the 2.7 mod_wsgi (I think).

Comment 4

[Josh Mize [:jgmize]]

According to https://bugzilla.mozilla.org/show_bug.cgi?id=785137#c16 we have had the ability to use python 2.7 since at least 2013-10-17. Further comments on that bug indicate that solitude and webpay are already on python 2.7.

Comment 5

[Paul [:pmac] McLanahan]

And as bug 1005361 says, Python 2.6 has been unsupported (no security patches) for 6 months now. I know we have to keep 2.6 around because RHEL, but for public facing stuff we should be running a more recent and supported version.

Comment 6

[Paul [:pmac] McLanahan]

I've updated the title and added the full description to the "user story" field. Any update on this one would be appreciated as we're increasingly hitting a wall on feature development due to being unable to use 2.7 dependent libraries.

Comment 7

[Jake Maul [:jakem]]

@pmac: the bedrock/requirements/compiled.txt file does not appear to include peep hashes. Care to add them or shall I proceed without?

Comment 8

[Paul [:pmac] McLanahan]

This is for setting up the new virtualenvs? It's probably safe for now without, but I can add them tomorrow. We'll be moving toward peep and away from vendor soon, but not quite yet.

Comment 9

[Jake Maul [:jakem]]

We have completed moving over the www-dev and demoX environments!

So far all seems well. Let's aim for doing stage and prod next week. We should be able to do one datacenter at a time and have zero downtime.

Comment 10

[Jake Maul [:jakem]]

Stage was completed without significant issue. (writing this after the fact, so I may be remembering optimistically)

Prod had a few snags, but nothing really horrible. SCL3 upgrade is still in progress, but should be completed momentarily.


1) When doing PHX1 (first), WebQA folks were getting 500 ISE errors being served from PHX1. The errors were expected, but they should have been getting data from SCL3 (which was not broken) and not PHX1. We believe this may be some sort of DNS caching or authoritative-ness issue affecting Mozilla offices. This was not replicable from the outside.

2) PHX1 took longer than expected due to an odd quirk in the bedrock puppet module that I'd forgotten about. Namely, PHX1 has a different apache config file than SCL3. IIRC, this was a quick-and-dirty hack put in place a while back because they needed slightly different settings, and can likely go away. Once I made the same change in the PHX1 configs, those nodes came right up fine.

3) SCL3 went down unexpectedly while we were letting PHX1 "bake". This is entirely my fault- I disabled puppet on these nodes, but not for long enough- they re-enabled themselves, updated the apache config, and broke (because they didn't have the proper virtualenv to run with python-2.7). Luckily, it appears that Dynect caught this (as it should) and stopped sending traffic to SCL3, meaning there was only a short incident as measured by New Relic (about 2 minutes). During that time, approximately 33% of total django traffic would have failed (6 working nodes in PHX1, 2 working in SCL3 - hadn't yet broken - and 4 failed in SCL3).

Comment 11

[Jake Maul [:jakem]]

Another minor snag - this involved an upgrade to a newer version of the New Relic python library, which needed a slightly newer config file. I forgot to do this beforehand, and we forgot to do it during dev and stage as well. So, this was done *after* the update was pushed live.

Not a big issue, except it does result in an *apparent* drop in throughput to bedrock during those 2 time windows (first PHX1, then SCL3). Of course no real drop occurred, those nodes just weren't reporting their data properly.

Comment 12

[Jake Maul [:jakem]]

All done! No ill effects have surfaced as far as we can tell.