460020 - Build machines should fetch from hg via https or ssh

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P5)

Description

[Ben Bucksch (:BenB)]

See bug 450645

Comment 1

[[e-mail] (:bhearsum)]

I'm tossing this back in the pool for now. If someone else has time to tackle it before I do, please feel free to do so.

Comment 2

[[e-mail] (:bhearsum)]

RelEng came across this in triage and given the problems with the first deployment, and the fact that the build machines are in the same colo as hg.m.o we've decided that we will not be going ahead with this.

Comment 3

[Reed Loden [:reed]]

Yet we spent tons of time making sure our build machines pulled CVS source over SSH rather than pserver?

Comment 4

[Justin Dolske [:Dolske]]

I'm not sure what our colo network looks like, but one usual attack vector for this stuff is DNS poisoning. Being on the same network doesn't help if an attacker can make a machine thing hg.mozilla.org is an IP in Elbonia. Are we sure that can't happen?

Comment 5

[Reed Loden [:reed]]

This needs to be done for the same reason CVS builds were moved to SSH rather than pserver.

Comment 6

[Phil Ringnalda (:philor)]

Wouldn't that be "Build machines should fetch from hg via ssh"? To the best of my knowledge, that would actually have a point, unlike this, which is utterly pointless without any cert checking.

Comment 7

[matthew zeier [:mrz]]

What's the concern here?  

Since all the build infrastructure is on Mozilla owned ethernet in locked environments, what does encryption help with?  

If DNS is a real concern (and we really think that Mozilla's internal nameservers can be compromised) then the more desirable fix would be to add host entries.

Using some for of encryption will break the content caching we're using for non-local-to-MPT locations, some of which will be through an IPSec VPN and some of which are private ethernet connected.

Comment 8

[Ben Bucksch (:BenB)]

There are tons of attack vectors, yes. DNS is one. Getting access to ethernet is another. Mozilla.com network access != root access to 400 million desktops.
"Better safe than sorry"
See reed comment 3 and 5.

Comment 9

[matthew zeier [:mrz]]

There's a trade-off and balance in security vs. practicality.  

We'd be significantly increasing the costs associated with doing remote build pods (or increasing build delays if we scratch them) if we can't use caching proxies.

And by extension, if we're going to argue that every machine-to-machine communication within the locked datacenter can be compromised we should look at addons.mozilla.org and the non-encrytped mysql connections.  

And if everything's going to be encrypted we'll need to look at augmenting CPU on everything.  

And if you could tap the wire, why wouldn't you just reboot the hg servers and install something that sits before it's all encrypted?

The essence of my question is, is it worth it?

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Talked with clyon, mrz yesterday afternoon at length:

1) There's a few reasons, in addition to the potential MITM scenario
described here, for separating the repo machines from other machines in
the secured colo. I've spun off bug#567488 to track that work.

2) The pro-vs-con debate of using https/ssh within the locked-down colo
continues. Still figuring out possible infrastructure impact, and
bug#567488 will further reduce the possible attack vectors, but other
comments/concerns welcome.

3) BenB,reed: after reading comment#3, comment#5, comment#8, if you have
other security concerns, please file separate bugs with specific
actionable items.

Comment 11

[Ben Bucksch (:BenB)]

I can't read bug 567488 (despite being in security-group), so I don't know what exactly it will do or help and want remains.

> please file separate bugs with specific actionable items.

This one is actionable: s/http/ssh/ in hgrc :)

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #11)
> I can't read bug 567488 (despite being in security-group), so I don't know what
> exactly it will do or help and want remains.
I've cc'd you.

Comment 13

[Ben Bucksch (:BenB)]

Thanks. From out-of-band discussion, I misunderstood comment 10, I thought you wanted to close this bug.

Comment 14

[Chris Cooper [:coop] (he/him)]

Does using a shared source dir (bug 589885) save us some of the pain of doing a full, non-cached clone over ssh every time?

Comment 15

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Summary of meeting with RelEng (joduinn), IT (mrz) and InfraSec (clyon):

The security risks portrayed here are being addressed by architectural changes already being worked on separately in bug#567488, bug#602741, and bug#617414, therefore this bug is being WONTFIXed.

Comment 16

[Ben Bucksch (:BenB)]

Will there be a secure connection (eg. VPN) between build machines and repo machines?

Comment 17

[Ben Bucksch (:BenB)]

BTW: I can't access bug 602741. Could you maybe CC me?

Comment 18

[matthew zeier [:mrz]]

There will be some sort of end-to-end encryption, the details of which have yet to be worked out.

Comment 19

[Ben Bucksch (:BenB)]

Can we reopen this bug, then, please? Because this is exactly about that end-to-end encryption.