1133956 - Access to balrog outside of vpn

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Description

[James Lal [:lightsofapollo]]

(I am not sure exactly what details need to be added here so please ping me with more questions).

We have some new infrastructure (http://docs.taskcluster.net/) one of the things we would like to do is update balrog configs for b2g builds. Currently this infra does not exist inside of mozilla vpn infrastructure so routing to balrog is impossible. I am looking for some kind of workaround so we can post our builds to our limited b2g audience.

Comment 1

[[e-mail] (:bhearsum)]

I spoke with James about this a bit today. The basic issue here is that AUS4 (Balrog) was initially deployed without considering that systems outside of our network may need to talk to it. Taskcluster machines live in AWS and do not have any contact with our internal network, so they cannot route to aus4-admin.mozilla.org.

I have a few potentially naive ideas of how we might accomplish this:
1) Give Taskcluster machines a tunnel to Mozilla's internal network. If I understand things correctly, this _should_ mean that no changes would be necessary to the web side. I'm not sure who the best person to talk to is about this.
2) Move aus4-admin.mozilla.org to a public IP (stage and dev are already public).
3) Create a public proxy to aus4-admin.mozilla.org, send Taskcluster traffic through it.

The original sec review for the Balrog admin app was premised on the app being private, so we may need to clear options 2 or 3 through app or opsec.

I'm not vested in any particular solution, I just hope we can find something that avoids James needing to run his own Balrog instances.

Comment 2

[:Atoll]

Adding sec-review? for options 2/3, making aus4-admin.m.o publicly accessible (in some fashion).

Comment 3

[Guillaume Destuynder [:kang] [this account is no longer active]]

We updated the Balrog RRA at https://docs.google.com/a/mozilla.com/spreadsheets/d/1Rya5ObK51nAYGd_kwFoiH2zh_KD6edKF9w4S_bdzQbg/edit#gid=0

While it's mainly public data we have a maximum risk impact if the data is modified (which is what the admin panel/API allows) therefore we want to be careful with how the service is exposed and accessed.

Currently its using LDAP via basic auth over TLS and is restricted to the internal network. The app then manages a list of ACLs per user. The code being this has been reviewed (bug 832462). The LDAP auth has a few issues such as:
- exposing clear text passwords on the server instance
- not using 2FA
- not having a reliable and centralized auditing mechanism (we audit LDAP logins but don't get the corresponding service logs)
- no API keys or tokens support
- No central ACL management (only authentication is centralized)

I would recommend using bhearsum's solution 1 (tunnel) until we work on the authentication side (which has been flagged as a recommendation in the RRA as well).

I had a quick chat with :jonasfj on #taskcluster as well to check that using some kind of tunnel or VPN is possible/a good tradeoff.

James, :atoll, would that work for you?
Otherwise, we can schedule some time to discuss (either here, IRC or Vidyo/FxHello).

Comment 4

[:Atoll]

I don't see any other option at this time, so it works for me insofar as it works at all, if we choose to do it.

I would propose a non-human LDAP account with just enough Mozilla VPN / SSH jumphost access to reach aus4-admin ONLY, and nothing else, so that the taskcluster instance responsible for making aus4 changes can tunnel into our network for the purpose.
 
:kang, presumably that would be okay if we can work out how to do it on the LDAP side?

Comment 5

[Guillaume Destuynder [:kang] [this account is no longer active]]

r+ for comment 4

Comment 6

[James Lal [:lightsofapollo]]

This works for me... Now who should I talk to to get the credentials? This should be fairly easy from there.

Comment 7

[:Atoll]

(In reply to James Lal [:lightsofapollo] from comment #6)
> This works for me... Now who should I talk to to get the credentials? This
> should be fairly easy from there.

I think the next step here is to file a bug in the "Infrastructure: LDAP" component requesting a "non-human LDAP account" with a single VPN ACL (https to whatever the *internal* DNS hostname for aus4-admin is).

Comment 8

[:Atoll]

(In reply to Richard Soderberg [:atoll] from comment #7)
> (In reply to James Lal [:lightsofapollo] from comment #6)
> > This works for me... Now who should I talk to to get the credentials? This
> > should be fairly easy from there.
> 
> I think the next step here is to file a bug in the "Infrastructure: LDAP"
> component requesting a "non-human LDAP account" with a single VPN ACL (https
> to whatever the *internal* DNS hostname for aus4-admin is).

This has been done, see bug 1144315 for details.

Comment 9

[:Atoll]

So, with the necessary VPN access provided for the solution we all agreed on above, there's nothing else for Webops to do here. Closing this as FIXED but feel free to reopen if you come across further requirements that we can help with.