1224318 - Can't access app listening on port 3003 on dashboard1.metrics.scl3.mozilla.com

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Ali Almossawi]

Node is running on port 3003 on dashboard1, yet connections to that port time out. ericz said to file a bug.

https://metrics.mozilla.com:3003/deps/filename=accessible:jsat:EventManager.jsm

Expected result: http://almossawi.com:3003/deps/filename=accessible:jsat:EventManager.jsm

Comment 1

[Dave Curado :dcurado]

Hello -- you need to let me know where you are trying to access this resource from.
Can you give the IP address from which you are coming from?

Thanks,

Comment 2

[Ali Almossawi]

I'm accessing it from 10.251.28.147 (SF, behind VPN), though this endpoint is meant to be public.

Comment 3

[Dave Curado :dcurado]

OK, there is nothing on the network blocking you from reaching that resource.
Can you verify that the service is running on port 3003/tcp on metrics.mozilla.com?
and/or that there aren't some host based controls preventing you from accessing it?

Thanks.

Comment 4

[Ali Almossawi]

netstat -anp |grep 3003
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:3003                0.0.0.0:*                   LISTEN      27822/node

I'm not sure about the latter question. Eric?

Comment 5

[Dave Curado :dcurado]

Also note that this appears to be a ZLB node.
Make sure that the ZLB is configured to handle port 3003?

Thanks.

Comment 6

[Eric Ziegenhorn :ericz]

dashboard1.metrics.scl3 is not a zlb, but if this is going to be a public facing website we should probably front it with Zeus.  Can you explain what you're running here Ali?

Comment 7

[Ali Almossawi]

This endpoint: http://almossawi.com:3003/deps/filename=accessible:jsat:EventManager.jsm

It's experimental work for Platform Engineering that returns the set of dependencies for an arbitrary file in the Firefox codebase. It would be accessible from here: https://metrics.mozilla.com/code-quality/

And the code for it is here: https://github.com/mozilla/firefox-code-quality/tree/master/deps

Comment 8

[Ali Almossawi]

I should point out that in the near term, only a few people are likely to access this. The decision to share the link more broadly would be contingent on how useful we find the work to be in the coming months.

Comment 9

[Eric Ziegenhorn :ericz]

Caught up a bit on IRC, this is accessible via VPN at http://dashboard1.metrics.scl3.mozilla.com:3003/deps/filename=accessible:jsat:EventManager.jsm

To get it accessible at metrics.mozilla.com, we'll have to add a Zeus VIP that forward to port 3003.

Comment 10

[Shyam Mani [:fox2mike]]

Not netops, moving.

Comment 11

[Eric Ziegenhorn :ericz]

April, can you please look at this proposed internet-facing service from an opsec perspective?  It's proposed to make the nodejs app on dashboard1.metrics.scl3.mozilla.com:3003 available to the public via Zeus.

Comment 12

[April King [:April]]

Sure, that shouldn't be a problem.  Has there been an RRA of this service completed?

Can you give me a broad level view of how sensitive this data is, and what it will be used for?

Thanks!

Comment 13

[Ali Almossawi]

Please refer to https://bugzilla.mozilla.org/show_bug.cgi?id=1224318#c7

It's all public data.

Thank you.

Comment 14

[Eric Ziegenhorn :ericz]

No RRA that I know of.  Ali is authoritative on the data and app.

Comment 15

[Ali Almossawi]

RRA filed: https://bugzilla.mozilla.org/show_bug.cgi?id=1225627

Comment 16

[Ali Almossawi]

April, is there anything else that I need to do in order to get the bug moving? Thank you.

Comment 17

[April King [:April]]

Hmm, sorry that the RRA hasn't been initiated.  I'm still okay with having it be publicly accessible, although I'm not the one who would set up the zlb rules.

Kang, can you setup an RRA with them soon?  Thanks!

Comment 18

[Guillaume Destuynder [:kang] [this account is no longer active]]

our workflow was broken hence the delay - see https://bugzilla.mozilla.org/show_bug.cgi?id=1225627#c1 for the RRA and thanks for the needinfo!

Comment 19

[Guillaume Destuynder [:kang] [this account is no longer active]]

The RRA is completed (Risk Record at https://bugzilla.mozilla.org/show_bug.cgi?id=1239502 - access needed to see this bug)
We have assessed both code quality (the dashboard) and the aforementioned NodeJS service above.
As per RR the risk is LOW currently and it seems reasonable to host NodeJS, however there are a couple of recommendations:

- specially if publicly accessible, it should go behind the load balancer (LBL) and the load balancer should provide TLS termination
- if no service with higher than LOW risk are hosted on the dashboard1 server, it's reasonable to deploy the NodeJS service there as well as they belong to the same project. Otherwise, these services should be separated (i.e. must not host nodejs service on dashboard1). In fact, would consider moving the rest of code quality to it's own server if that's the case.

Hope this help!

Comment 20

[Eric Ziegenhorn :ericz]

Zeus VIP setup at metrics.mozilla.com:3003 for now, app is unresponsive on the server side as far as I can tell.

Comment 21

[Shyam Mani [:fox2mike]]

Time out, please re-open if there's anything we can do.