1058844 - Security review for proxxy

Status: RESOLVED FIXED

(Enterprise Information Security Graveyard :: Review, task)

Description

[Chris AtLee [:catlee]]

We're rolling out proxxy (a simple http cache) to more systems, so probably a good time to get a security review!

Code is https://github.com/mozilla/build-proxxy
Docs are at https://wiki.mozilla.org/ReleaseEngineering/Applications/Proxxy

Comment 1

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

assigning to yvan to assign a resrouce

Comment 2

[Julien Vehent]

This could benefit an OpSec review as well. needinfo :joes.

Comment 3

[Julien Vehent]

Attachment: proxxy-system-sec.png

I took a closer look at proxxy1.srv.releng.scl3.mozilla.com, which is running Ubuntu 14.04 trusty and is managed by puppetagain.

From the configuration, proxxy is currently caching the following destinations:
- pvtbuilds.mozilla.org.*;
- ftp.mozilla.org.*;
- runtime-binaries.pvt.build.mozilla.org.*;
- pypi.pub.build.mozilla.org.*;
- pypi.pvt.build.mozilla.org.*;
Destinations can be reached by appending the proxxy hostname to the target, like: http://ftp.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub/mozilla.org/

:catlee - we should meet on vidyo and do a risk assessment (RRA) to understand the risk level of that system. For now, I'm going to assume that this is a HIGH risk system, because as a proxy, its security impacts all the services that depend on it, and some of these systems may have HIGH security requirements. Let's confirm that during a RRA.

As a proxy, I can see two risks:
1) using it to download malicious code onto systems that would normally not have an internet access
2) tampering its cache to replace legitimate code with malicious code

1) can be solved by strict whitelisting of the clients, and of the proxy destinations. Clients should be approved by way of ACLs, in nginx or at the network level. Proxy destinations should be controlled in the configuration management tool, and require peer review for approval of a new destination. The proxy should also serve whitelisted content, and not mirror locations where anyone can upload files. I see two pypi repositories in the proxy destinations: do those serve only approved pypi packages, or mirrors the public pypi available at pypi.python.org?

2) can be solved by implementing stricted security controls on the proxxy servers. I recommend implementing the HIGH level of the IT Service and System Security Practices, described at https://mana.mozilla.org/wiki/display/ITSECURITY/IT+System+security+practices. I attached a screenshot of today's measurement of Proxxy's coverage of the system security practices, and it's fairly low. Most notably:
- Auditd should be implemented and send events to MozDef
- The mount settings should prevent execution in random directories, particularly in the proxy cache directories
- Logs should be routed to a central syslog server
- Proxy logs should contain the IP of the client, and be kept for at least 90 days
- Access to the servers should require two-factor auth

It seems that you are running logstash on the proxxy server directly. Have you considered using some of the existing services at Mozilla, instead of reimplementing a new one? I'm thinking of Bunker, operated by :phrawzty and owned by Laura's team. Aside from saving you the trouble of running a JVM, from a security perspective, storing the logs on the proxy server itself doesn't provide the assurance that logs will be available or untempered.

Have you considered using the existing proxy service in PHX1 and SCL3 and operated by netops? I don't know if these services support caching, but they do have proper security protocols in place (including network ACLs management), and are already used by ~300 servers.

Comment 4

[Daniel Maher [:phrawzty]]

(In reply to Julien Vehent [:ulfr] from comment #3)
> reimplementing a new one? I'm thinking of Bunker, operated by :phrawzty and
> owned by Laura's team. Aside from saving you the trouble of running a JVM,

Bunker (the ELK stack) was decommissioned in January 2014.

Comment 5

[Julien Vehent]

Secreview meeting was today. RRA is at https://docs.google.com/a/mozilla.com/spreadsheets/d/14oiz5ZDEJKLhMh_wA2rsCf310E4SvwhtchSuh-_3xX8/edit#gid=1112083357

Assumptions are that proxxy will be used for test and build machines. Recommendations is to hold off on using proxxy with build machines until a technical review is done (once service is puppeted and deployed).

Action items:
* standardize deployment in puppet. This will also add a mandatory peer-review for the additional of cached repositories. 
* implement remote logging solutions (heka->mozdef is an option, :dustin is evaluating others)
* implement the relevant items from the HIGH levels of Service and System security best practices:
  https://mana.mozilla.org/wiki/display/ITSECURITY/IT+Service+security+practices
  https://mana.mozilla.org/wiki/display/ITSECURITY/IT+System+security+practices
* review production service before opening it to build machines. test machines can access it prior to the review.

Comment 6

[Guillaume Destuynder [:kang] [this account is no longer active]]

RRA updated to new template: https://docs.google.com/a/mozilla.com/spreadsheets/d/1lH0LomN3m7pjNZMdS_7nN3Ja7nvO-wA4R1JoaE1PX4E/edit#gid=0

Comment 7

[Guillaume Destuynder [:kang] [this account is no longer active]]

Verified logs get into papertrail as a remote logging solution. (https://papertrailapp.com/groups/1264174)
Also, the proxxies seem in use right now according to logs.

Compliance, see http://mozdef1.private.scl3.mozilla.com:9090/#dashboard/temp/Cey8dg0gSgmXB8-5hVWdOw for example. Currently, is seems far from the compliance targets.
Puppet appears to be running.

Adding dustin as needinfo on the compliance items, since he probably has similar tracking bugs.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Hm, of the syslow stuff, I see
 syslowtime1 -- known
 syslowauth3 -- appears not to account for a *locked* root account
 syslowaudit1 -- known

Where would you like a bug for the middle one?  I'm not sure how to represent "or" in mig-system-compliance templates, or I'd just file a PR.

Comment 9

[Julien Vehent]

I have updated the check to include "*" and "!" as compliant root passwords, since they are used to disable password logging to the account.

Comment 10

[Julien Vehent]

Dashboard updated with fresh results: http://mozdef1.private.scl3.mozilla.com:9090/#dashboard/temp/Qplcl2TnTGeYFQMDRV5pLw

Comment 11

[Guillaume Destuynder [:kang] [this account is no longer active]]

Had a chat with :catlee and the main thing we care about is verifying the files transferred are legitmate/not modified (as per RRA).
Right now we decided the best way to do this is to have them signed at the distribution points and the clients to verify the signatures, since it works regardless of the transport. (see bug 1142797 for this)

The compliance remediation is in progress separately, since the host is puppet-managed. If comment 9 and 10 doesn't work  for you :dustin, please open a separate bug.

And therefore, closing the review