1342112 - Serve host-secrets API using HTTPS

Status: RESOLVED WONTFIX

(Taskcluster :: Services, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

The current plan is to deploy this service using HTTP.  It depends on the security of the local network anyway, so I don't think that's a blocker to deployment.  However, we should try to use https everywhere!

I think the preferred choice, based on our conversation just now, is to use the puppetmasters' SSL hierarchy.  That would just require adding that SSL certificate to the cert store on all of the workers.

With that in place, we can use the existing Apache install to do the SSL termination, proxying https://<puppetmaster>/secrets to the secrets service.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

On second thought, adding that to the system cert store is tricky and means that *every* https client will trust that ssl hierarchy, which isn't its intended use.  Hmm.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

The more I think about this, the sillier it seems.

This is parallel to the puppetization process, where we get a puppet certificate from the puppet servers:

---
import urllib2, getpass, ssl
deploypass="""$deploypass"""
puppet_server="${PUPPET_SERVER:-puppet}"
print "Contacting puppet server %s" % (puppet_server,)
if not deploypass:
    deploypass = getpass.getpass('deploypass: ')
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, 'https://'+puppet_server, 'deploy', deploypass)
handlers = [urllib2.HTTPBasicAuthHandler(password_mgr)]
try:
    # on Pythons that support it, add an SSL context
    context = ssl._create_unverified_context()
    sslhandler = urllib2.HTTPSHandler(context=context)
    handlers.insert(0, sslhandler)
except AttributeError:
    pass
opener = urllib2.build_opener(*handlers)
data = opener.open('https://%s/deploy/getcert.cgi' % (puppet_server,)).read()
open("$ROOT/certs.sh", "w").write(data)
---

In particular, note `ssl._get_unverified_context()` which does what it suggests -- it doesn't verify anything, not the hostname, not the cert: https://hg.python.org/cpython/file/2.7/Lib/ssl.py#l458  So someone with access to the network could MITM that pretty easily, provide their own cert (even self-signed!), and get the puppet cert.

Back to the host-secrets service, sure, things are a little easier in that an attacker on the network could just snarf the packets since they're cleartext.  But even easier would be to just request your own secrets from a host on the same network.  And remember this whole things starts from a totally un-encrypted, un-authenticated PXE boot.  So basically we are inescapably reliant on the security of the VLAN here.  Adding TLS would be useless (if we didn't do any validation of the cert) and/or really difficult (setting up a cert hierarchy and verifying it).

Rob, does that make sense?

Comment 3

[Rob Thijssen [:grenade (EET/UTC+0300)]]

i agree. the entire model around this service is that we implicitly trust you if you can create requests originating from somewhere on the 10.0.0.0/0 network.

snooping the http traffic, hoping to overhear someone else's secrets is harder than just sending a request to the secrets service asking for your own secrets.

it might tick a box on a clipboard somewhere to be able to say "everything is ssl" but the material security gain from requiring requests to the secrets service to be over ssl is hovering pretty much at zero from my perspective.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Thanks!  If there's resources to do it, I'd be keen to invent a better solution to the problem of hardware initialization, but until then, this is what we've got :)

Comment 5

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

:dustin - what's wrong with changing the code behavior to enable verify?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

For the PXE boot, it could. To be honest, until I looked yesterday, I thought it did! That requires getting the CA root onto the machine at installation time. And that process is itself insecure since it and PXE depend on un-encrypted, un-validated network communication.  I'll leave that to Amy.

The host-secrets service, too, could use https and could validate certificates.  I just don't see any benefit aside from the box-checking Rob mentioned.  I think a better solution for machine installation and credentialling is possible, but not easy. Furthermore, it's quite far outside the Taskcluster team's purview, and Amy's team doesn't have the bandwidth to work on it.