Closed Bug 738389 Opened 12 years ago Closed 11 years ago

[aus4-dev] aus4-admin.allizom.org should not be publicly accessible

Categories

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task, P4)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Product Delivery
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: bhearsum, Unassigned)

References

Details

(Whiteboard: [triaged 20120831][waiting][744262]) 

The admin interface for AUS4 is not meant for public use, and should be protected behind a VPN. It does need to be accessible to the Build network, and eventually to some people not in RelEng.

bounceradmin.mozilla.com is setup similarly, where a bunch of people have accounts (IT, RelEng, maybe others), and the build slaves are also able to access its API.
It looks like there's an aus4-admin-dev.allizom.org too, which may or may not be the same thing?
Assignee: server-ops → bburton
Status: NEW → ASSIGNED
aus4-admin.allizom.org is nothing that is live yet

aus4-admin-dev.allizom.org is currently mostly setup, per bug 719581, and that is waiting on bug 734230

I'll see what's involved in making this locked down like bounceradmin is
Summary: aus4-admin.allizom.org should not be publicly accessible → [aus4-dev] aus4-admin.allizom.org should not be publicly accessible
Ok, so this our "standard" allow/deny for VPN access

    Order deny,allow
    Deny from all
    Allow from localhost
    Allow from 127.0.0.1
    Allow from 10.2.74.138 # SSL VPN only
    Allow from ssl1.dmz.sjc1.mozilla.net # SSL VPN only
    Allow from 10.8.32.5

What additional networks are needed for the "build network" to access this?
Assignee: bburton → server-ops
Component: Server Operations → Server Operations: Web Operations
QA Contact: phong → cshields
Assignee: server-ops → bburton
 10.250.48.0/22
 10.2.90.0/23
 10.2.48.0/24
 10.2.71.0/24
 10.12.48.0/21
 10.26.0.0./16
Ok, I pushed to puppet with the following

+    Order deny,allow
+    Deny from all
+    Allow from localhost
+    Allow from 127.0.0.1
+    Allow from 10.2.74.138 # SSL VPN only
+    Allow from ssl1.dmz.sjc1.mozilla.net # SSL VPN only
+    Allow from 10.8.32.5
+    Allow from 10.250.48.0/22
+    Allow from 10.2.90.0/23
+    Allow from 10.2.48.0/24
+    Allow from 10.2.71.0/24
+    Allow from 10.12.48.0/21
+    Allow from 10.26.0.0./16

Let me know how it looks
(In reply to Brandon Burton [:solarce] from comment #5)
> Ok, I pushed to puppet with the following
> 
> +    Order deny,allow
> +    Deny from all
> +    Allow from localhost
> +    Allow from 127.0.0.1
> +    Allow from 10.2.74.138 # SSL VPN only
> +    Allow from ssl1.dmz.sjc1.mozilla.net # SSL VPN only
> +    Allow from 10.8.32.5
> +    Allow from 10.250.48.0/22
> +    Allow from 10.2.90.0/23
> +    Allow from 10.2.48.0/24
> +    Allow from 10.2.71.0/24
> +    Allow from 10.12.48.0/21
> +    Allow from 10.26.0.0./16
> 
> Let me know how it looks

Unfortunately, I'm getting ISA 500 Service Unavailable from the admin app now.
Apache didn't like a typo on Allow from 10.26.0.0./16 and so Zeus was returning a 500.

Zeus now returns a 403 and access from specific networks looks to be working

I get a 401 when I test with curl

bburton@andesite ~$ curl -v -H "Host: aus4-admin-dev.allizom.org" http://aus1.dev.seamicro.phx1.mozilla.com:81/           ‹1.9.2-p290›
* About to connect() to aus1.dev.seamicro.phx1.mozilla.com port 81 (#0)
*   Trying 10.8.33.19... connected
* Connected to aus1.dev.seamicro.phx1.mozilla.com (10.8.33.19) port 81 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Accept: */*
> Host: aus4-admin-dev.allizom.org
> 
< HTTP/1.1 401 Authorization Required
< Date: Wed, 04 Apr 2012 06:55:58 GMT
< Server: Apache
< X-Backend-Server: node267
< WWW-Authenticate: Basic realm="Mozilla Corporation - LDAP Login"
< Content-Length: 401
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Authorization Required</title>
</head><body>
<h1>Authorization Required</h1>
<p>This server could not verify that you
are authorized to access the document
requested.  Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
</body></html>
* Connection #0 to host aus1.dev.seamicro.phx1.mozilla.com left intact
* Closing connection #0

How's it look for you?
(In reply to Brandon Burton [:solarce] from comment #7)
> Apache didn't like a typo on Allow from 10.26.0.0./16 and so Zeus was
> returning a 500.
> 
> Zeus now returns a 403 and access from specific networks looks to be working
> 
> I get a 401 when I test with curl
> How's it look for you?

https://aus4-dev.allizom.org looks OK now.

aus4-admin-dev is serving pure 403s though - both to my manual tests as well as to the client script that some build machines are running - and they look to be coming from Apache rather than the application. I don't even get an authorization prompt before getting the 403.
What IPs are the requests coming from? 

You should expect a 403 if the system is not in the allow list for the vhost now, you won't even get a 401 if you're not in that list first.
I _think_ my source IP is 10.26.20.18 while connected to build-vpn, but I'm not 100% sure about that.....
Are you still having issues with this? If so, please ping me on irc tomorrow and let's debug in real time :)
Per irc discussion, I've removed the allow/deny block for now and will be opening bugs to get an internal only ZLB and appropriate netflows.
Thanks! Can I get a cc on bug 744262?
Sorry for the delay, I've gotten caught up in a number of equally important bugs in the last couple of weeks and pulled into dealing with some tasks that weren't part of bugs, but post SJC1 cleanup/improvements that suddenly were needed more urgently. 

Unfortunately this has resulted in my losing state on this bug. I'm triaging this and other bugs today and will be asking another member of the webops team to try and make some forward progress on this bug this week, as I'll be on PTO getting LASIK tomorrow, Thursday, and potentially Friday.

Please ping me on irc if you have any questions for me specifically.
Whiteboard: [pending 744262]
Renormalizing priority levels... P4 is "normal" now.
Assignee: bburton → server-ops-webops
Priority: -- → P4
Whiteboard: [pending 744262] → [triaged 20120831][waiting][webops]
Whiteboard: [triaged 20120831][waiting][webops] → [triaged 20120831][waiting][744262]
Finally getting this underway, filed https://bugzilla.mozilla.org/show_bug.cgi?id=823318 for flows
Assignee: server-ops-webops → bburton
Blocks: 832461
Component: Server Operations: Web Operations → WebOps: Product Delivery
Product: mozilla.org → Infrastructure & Operations
Assignee: bburton → server-ops-webops
Perhaps removing the public IP would be useful first step ?
This is invalid as it refers to the old, deprecated dev nodes. Over in https://bugzilla.mozilla.org/show_bug.cgi?id=832460#c5 I OK'ed dev not being internal like stage and prod are. We can revisit that later if we do think it's an issue, though.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → WONTFIX
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.