Closed Bug 1342201 Opened 8 years ago Closed 8 years ago

Update httplib2 to fix planet SNI bugs

Categories

(Infrastructure & Operations :: IT-Managed Tools, task)

Product:
Infrastructure & Operations
Component:
IT-Managed Tools
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: mhoye, Assigned: ericz)

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/4315])

In order to fix a series of SSL issues impacting Ehsan and JDM's blogs, among many others - https://bugzilla.mozilla.org/show_bug.cgi?id=1334949 - I'd like to migrate Planet to a CentOS7 machine as soon as possible. CentOS 7 Python libraries do not support SNI, among other things. A second part of this migration will involve removing an outdated, vendored-in library in the Planet source code, which I will take care of myself. Redeploying that codebase to the CentOS7 VM (after installing httplib2 from pip locally - should Just Work once that's done. Thank you.
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/4315]
To do this, we'd have to set up a third cluster (alongside generic and python) just for planet, and planet only. It would need at least 2 machines, because we can't just serve off of one. Can the Python code be easily adapted to submit its requests to the DC proxies instead? (Without CONNECT requests, so that the proxy does the SSL negotiation correctly.)
Clustering this seems like a massive overinvestment. Planet _extremely_ lightweight by modern standards, pushes text exclusively, and there's no reason it should run on real metal with a five nines SLA. Two nines and a minimal virtual machine somewhere in AWS is totally fine. I suspect you could run planet on a wristwatch, provided it had the right python libs.
Agreed - but if we're to meet your needs *today* with the tools we're permitted to use, then we either spin up a plural VM cluster, or we patch the code, or we route through proxies. I'd very much *like* to deploy this to AWS, and I understand Daniel's working on this right now, but it isn't ready yet, and if we rushed it out right now, our team wouldn't be able to repair it in an emergency. What level of urgency does this issue carry? Are we talking "Engagement is continuously upset every minute it's broken" or "we're unintentionally excluding 2 people out of 1000 people from Planet" or ..? Are you open to any other solutions between now and when we migrate it to AWS?
Hard to gauge. This is impeding some of our prominent developers and contributors from reaching a larger audience, which is not quite as hair-on-fire as "engagement is upset every minute", but I think more of an "I'd feel a lot better if we could solve this in the next five to seven days" kind of thing.
Help me understand why we can't patch the Python code here to use requests+PyOpenSSL to gain SNI support?
It's just software, so sure? I guess we could do anything with it? But we have proposed solution in-hand here that involves installing planet nearly unchanged, on the stock version of a recently-released OS. Substantially patching a legacy software stack to run correctly on an older version of that OS seems a lot more like a project than a fix, and a distant second-best choice.
The quick fix for the SNI issues is to update httplib2 in your repo Mike and using python 2.7 which we have on the current cluster. This make's Josh's blog work and all SNI-related errors disappear. Planet still does not find recent blog posts on Ehsan's blog but nothing seems TLS or SNI related in there -- there are no errors. I tested that out on CentOS 7 and got the same result. So I think no new cluster needed here, just update httplib2 and I'll make sure we're running planet with python 2.7 which I think we already are. Troubleshooting Ehsan's blog seems it should be a different issue at this point.
OK, I'll take that as a fix and continue investigating.
Great, then drop a comment when httplib2 is updated in the repo and I'll make sure it takes on the admin host where planet.py is run. I've verified it's using Python 2.7.11 to run planet.py now.
Summary: Migrate Planet to a CentOS7 VM → Update httplib2 to fix planet SNI bugs
Assignee: server-ops-webops → eziegenhorn
OK, removing httplib2 is done.
See also https://github.com/rubys/venus/pulls, which has a ton of bug fixes for the version of Planet we're running. Somebody should just fork the project and take it over.
Yeah, I'll ask him.
(In reply to Mike Hoye [:mhoye] from comment #10) > OK, removing httplib2 is done. I don't see any change with regard to httplib2, it still shows in the repo as it was before. Am I missing something?
Mike fixed this via a different route so I believe this is no longer relevant.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.