Whitelist goldiewilson-onepointtwentyone-1.c.influxdb.com on jenkins1.qa.scl3.mozilla.com

RESOLVED FIXED

Status

Infrastructure & Operations
NetOps
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: Eli, Assigned: dcurado)

Tracking

other
Bug Flags:
sec-review +

Details

(Reporter)

Description

3 years ago
On Jenkins, Raptor makes requests to goldiewilson-onepointtwentyone-1.c.influxdb.com in order to store performance data for the Raptor dashboards [1]. When the Raptor tests were run on Jenkins slaves, this wasn't an issue, but since moving the test to the master box, Raptor can no longer make requests to this endpoint. Come to find out, all requests are supposed to be proxied through a proxy server, but Node.js does not have native proxy server support, and it is very difficult to proxy network connections that are not owned by application code, which is the case for Raptor's requests.

We need to whitelist goldiewilson-onepointtwentyone-1.c.influxdb.com, our InfluxDB server, on jenkins1.qa.scl3.mozilla.com so we can get performance metrics reporting again.

[1] http://raptor.mozilla.org
Flags: sec-review?(mpurzynski)
(Assignee)

Comment 1

3 years ago
Hi Eli,

I need a little help with translation.

Is the case that:
 - You used to have hosts (jenkins slaves) which were able to open connections to
   goldiewilson-onepointtwentyone-1.c.influxdb.com
 - then you stopped using those hosts (jenkins slaves) and instead starting using
   a different host (the master box)
 - and the master box can not open connections to goldiewilson-onepointtwentyone-1.c.influxdb.com

Did I get that right?
If so, then the source is jenkins1.qa.scl3.mozilla.com, the destination is goldiewilson-onepointtwentyone-1.c.influxdb.com.  What protocol is jenkins1.qa.scl3 using to send performance data?

Thanks -- Dave
Assignee: network-operations → dcurado
Status: NEW → ASSIGNED
Flags: needinfo?(eperelman)
(Reporter)

Comment 2

3 years ago
Yes Dave, that is correct. We currently only communicate with the server via HTTP via port 8086.
Flags: needinfo?(eperelman)
(Assignee)

Comment 3

3 years ago
OK, since this policy was previously in place, and ergo blessed by opsec, and only the source IP
is changing internally at Mozilla, I believe the policy is blessed by the transitive property.

I put the the following security policy in place:

 From zone: qa, To zone: untrust
  Source addresses:
    jenkins1: 10.22.73.155/32
  Destination addresses:
    goldiewilson-onepointtwentyone-1.c.influxdb.com: 104.131.136.49/32
  Application: tcp-8086
    IP protocol: tcp, ALG: 0, Inactivity timeout: 1800
      Source port range: [0-0] 
      Destination port range: [8086-8086]

Hopefully opsec will be OK with this, and if not, I apologize in advance.
Please let me know if you (Eli) have any problems?
Thanks -- Dave
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 4

3 years ago
Everything looks to be running correctly, thanks!
Flags: sec-review?(mpurzynski) → sec-review+
(Reporter)

Comment 5

3 years ago
I just received some logs from this job with markup indicating that the proxy denied us access to this server again. Here is the rendered markup:

https://jsfiddle.net/j5w6cbz8/embedded/result/

Did we someone lose this policy?
Flags: needinfo?(dcurado)
(Assignee)

Comment 6

3 years ago
The policy is still in place on fw1.scl3:

dcurado@fw1.ops.scl3.mozilla.net> ... to-zone untrust policy-name jenkins-goldiewilson--8086 detail                         
node0:
--------------------------------------------------------------------------
Policy: jenkins-goldiewilson--8086, action-type: permit, State: enabled, Index: 37, Scope Policy: 0
  Policy Type: Configured
  Sequence number: 10
  From zone: qa, To zone: untrust
  Source addresses:
    jenkins1: 10.22.73.155/32
  Destination addresses:
    goldiewilson-onepointtwentyone-1.c.influxdb.com: 104.131.136.49/32
  Application: tcp-8086
    IP protocol: tcp, ALG: 0, Inactivity timeout: 1800
      Source port range: [0-0] 
      Destination port range: [8086-8086]
  Per policy TCP Options: SYN check: No, SEQ check: No


Does jenkins1 still have the IP address of 10.22.73.155?
If this is a linux/unix OS, can you try: nc -vz 104.131.136.49 8086

After that, I'd say you may need to contact the folks on the other end?
Status: RESOLVED → REOPENED
Flags: needinfo?(dcurado)
Resolution: FIXED → ---
(Reporter)

Comment 7

3 years ago
nc -vz 104.131.136.49 8086
found 0 associations
found 1 connections:
     1:	flags=82<CONNECTED,PREFERRED>
	outif en0
	src 192.168.0.102 port 58722
	dst 104.131.136.49 port 8086
	rank info not available
	TCP aux info available

Connection to 104.131.136.49 port 8086 [tcp/*] succeeded!

--

nslookup jenkins1.qa.scl3.mozilla.com
Server:		10.22.72.136
Address:	10.22.72.136#53

Non-authoritative answer:
Name:	jenkins1.qa.scl3.mozilla.com
Address: 10.22.73.155
(Reporter)

Comment 8

3 years ago
Looks like everything is kosher permissions-wise. I'll do some digging on the other end.
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 9

3 years ago
Dave, would you mind running the following curl on that Jenkins box?

curl -G "http://goldiewilson-onepointtwentyone-1.c.influxdb.com:8086/db/raptor/series?u=heroku&p=heroku&q=list%20series"

I am able to get data from that query locally, but the Jenkins machine is still showing a proxy error.
Flags: needinfo?(dcurado)
(Assignee)

Comment 10

3 years ago
[dcurado@jenkins1.qa.scl3 ~]$ nc -vz goldiewilson-onepointtwentyone-1.c.influxdb.com 8086
Connection to goldiewilson-onepointtwentyone-1.c.influxdb.com 8086 port [tcp/d-s-n] succeeded!

Eli -- at layer 3 and layer 4 we're allowing your packets through.

I can not speak to your access to /db/raptor/series? etc. etc. etc.

Please contact the other side and discuss with them?
Thanks.
Flags: needinfo?(dcurado)
You need to log in before you can comment on or make changes to this bug.