753588 - Setup Zeus for symbolapi.mozilla.org

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Vladan Djeric (:vladan)]

I would like TCP port 8000 opened to the public on the VM described in bug 721743.

The symbolication server running on the VM has passed security review & privacy review: bug 744126 and http://mzl.la/Jb31p0

Comment 1

[Benjamin Smedberg]

Port 8000, not port 80?

Comment 2

[Benoit Girard (:BenWa)]

I don't see any benefit to using a privileged port? We can point the add-on and scripts to that port. Maybe router restrictions?

Comment 3

[Shyam Mani [:fox2mike]]

What is going to be on port 8000? Is this the eventual symbolapi.mozilla.org? 

Also, we don't just open up ports, most of the VMs in scl3 (the datacenter this one is in) are fronted by the load balancer, which means we can get you a public IP on the load balancer and have requests come in there (80 is more standard than 8000, so I'm with bsmedberg on that) which will then be passed on to your VM on whatever port you need.

So once you have all that clarified, one of us can set this up for you :)

Comment 4

[Benjamin Smedberg]

This is the right-now symbolapi.mozilla.org.

Comment 5

[Shyam Mani [:fox2mike]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #4)
> This is the right-now symbolapi.mozilla.org.

What?

fox2mike@woodpecker ~ $ host symbolapi.mozilla.org
Host symbolapi.mozilla.org not found: 3(NXDOMAIN)

That domain doesn't exist, internally or externally.

Comment 6

[Benjamin Smedberg]

What I mean is: this bug is about deploying that code to production. So it's whatever we need to do in terms of hostnames and port opening and load balancer (why?) to make that happen.

Comment 7

[Benoit Girard (:BenWa)]

Agreed, any configuration will do, just let us know what to reconfigure the symbolication server process on 'breakpad-symbolapi1.dmz.phx1.mozilla.com' to.

Comment 8

[Vladan Djeric (:vladan)]

It would be ideal for symbolapi.mozilla.org:80 to point to VM's port 8000. 

Also, I think VM's port 80 is currently accessible from other machines on the subnet (e.g. from the jumphost). We should also close that port.

Comment 9

[Shyam Mani [:fox2mike]]

Sure. Do you need SSL? or this going to be http only? What server is listening on 8000? (so I can configure health checks accordingly)

Comment 10

[Vladan Djeric (:vladan)]

(In reply to Shyam Mani [:fox2mike] from comment #9)
> Sure. Do you need SSL? or this going to be http only?

No SSL needed.

> What server is listening on 8000? (so I can configure health checks accordingly)

It's the python script hosted here: https://github.com/vdjeric/Snappy-Symbolication-Server/

I'm not sure what kind of setup is typically used for our Web-facing servers. I've been starting this script from my home directory with "nohup python symbolicationWebService.py sample.conf > symbolServer.log", but we should probably put it in its own directory somewhere and run it as a different low-privilege user. and maybe add a start script to /etc/init.d?

Comment 11

[Corey Shields [:cshields]]

I'm concerned that this dev VM has snuck its way into a production service.  Usually opsec reviews are done with the idea of running in our production stack (which includes certain restrictions and security monitors).

Before we start pointing production dns and ports to places that we don't typically do I'd like an opsec nod.

The proper alternative here would be to set this up on one of our webhosting clusters.

Comment 12

[Benjamin Smedberg]

I am very confused: the original purpose of this VM was to be a place to host this service; we set it up privately at first to get the sec and privacy review, and then we'd open up the port when that was done. If there are different/additional steps, it would have been nice to know about them a lot sooner. This is a whole lot of friction for a low-volume developer service!

Comment 13

[Joe Stevensen [:joe]]

Benjamin, 

I'm very sorry if this response came as news to you, but Corey is correct. While a security and privacy review of a web application was done, it appears that you are running the application on your own system (or VM) that is not managed by the Infra team. When you do this, we're going down a path of taking significant and unnecessary risks, because much of the security controls, maintenance and security monitoring that we have worked hard to implement into the Mozilla infrastructure isn't getting applied to your system.

It is for this reason that we discourage developers from managing their own systems in production environments. These very systems are the ones that carry the highest security risk to Mozilla and have been our most commonly compromised systems.

Corey's recommendation is spot on. We would like you to allow the Infra systems team to set this web app up on our webhosting clusters. There, you get all the operations monitoring, maintenance, availability, and security that we've worked hard to implement.

Please let me know if you have any questions.

Comment 14

[Corey Shields [:cshields]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #12)
> I am very confused: the original purpose of this VM was to be a place to
> host this service; we set it up privately at first to get the sec and
> privacy review, and then we'd open up the port when that was done. If there
> are different/additional steps, it would have been nice to know about them a
> lot sooner. This is a whole lot of friction for a low-volume developer
> service!

Sorry for the confusion..  Our use of "developer" service and yours can be quite different.  We consider developer resources to be testbeds as a step to prod.  This is often where opsec reviews are done.  You are talking about a "developer" service to serve developers as a production-level service.

In the original vm request this was "setting up a new development VM".  So the confusion here is that on our side "development != production" and does not follow the same security guidelines.  We do not have a production environment for this yet.

Comment 15

[Benoit Girard (:BenWa)]

Any ETA on rolling this out? This is holding up important data gathering for Project Snappy.

Comment 16

[Corey Shields [:cshields]]

Sorry, accidentally sent this over to cturra while I was working on it.  All done, symbolapi.mozilla.org is live on port 80.

Comment 17

[Benoit Girard (:BenWa)]

Beautiful! Profiling now works on all platforms (win, osx, linux, fennec)! Thanks for all your help. The server is very fast. We're able to get process a sample in about a second or so which is much faster then most profilers.

I'll write up tutorial, documentation and fixes up a few usability feature this week. I'll announce this feature in the developer meeting next week.