1365662 - Restrict access to OrangeFactor to users with LDAP credentials

Status: RESOLVED FIXED

(Tree Management Graveyard :: OrangeFactor, enhancement, P1)

Description

[Ed Morley [:emorley]]

To reduce exposure to the issues in bug 1356410, it has been proposed that we put OrangeFactor behind auth and limit access to staff/NDA contributors (bug 1356410 comment 27).

Of note, the OrangeFactor API is consumed by the Bugzilla OrangeFactor extension, however:

(In reply to Ed Morley [:emorley] from bug 1356410 comment #29)
> * The "link to orangefactor" functionality (which I think is the most useful
> part) will still work
> (https://github.com/mozilla-bteam/bmo/blob/
> 20c1dbebae25818af68e68b77b8e33b76720c3c4/extensions/OrangeFactor/template/en/
> default/hook/bug_modal/edit-details_rhs.html.tmpl#L22)
> * The "display graph / count" functionality:
>  - fails gracefully
> (https://github.com/mozilla-bteam/bmo/blob/
> 20c1dbebae25818af68e68b77b8e33b76720c3c4/extensions/OrangeFactor/web/js/
> orange_factor.js#L84-L86)
>  - will presumably still work so long as users have authenticated via the
> OrangeFactor site first
> * the usage is very low (off by default, has to be enabled in Bugzilla
> prefs) - I'm guessing single digit users who actually use it
> 
> ...as such IMO it needn't block.

In addition I'd forgotten about the submissions Treeherder makes to Elasticsearch via Hawk-authenticated POSTs to:
https://brasstacks.mozilla.com/orangefactor/api/saveclassification

...though given that's behind Hawk auth (and we can add additional payload validation there), I believe it would be fine to exclude that endpoint from SSO auth.

As far as implementation:

(In reply to Eric Ziegenhorn :ericz from bug 1356410 comment #33)
> Kang, orangefactor is served with nginx on brasstacks1.dmz.scl3 and webops
> has not implemented SSO with nginx yet -- do you have a preferred SAML
> authenticator for nginx and could you help us estimate the level of effort
> here as compared to say mod_mellon with Okta?

(In reply to Guillaume Destuynder [:kang] (NEEDINFO to ensure replies) from bug 1356410 comment #34)
> @:ericz https://testrp.security.allizom.org/ has a demo of the nginx access
> proxy for SSO which is the recommended solution. It uses OIDC and cares for
> session management, etc. on your behalf. The nginx side setup is documented
> with guide and config examples at
> https://github.com/mozilla-iam/testrp.security.allizom.org/tree/master/webserver_configurations/OpenID_Connect/Nginx
> 
> Credentials can be requested via
> https://mana.mozilla.org/wiki/display/SECURITY/SSO+Request+Form (theres a
> link to request them in there together with some docs)
> 
> Feel free to ping me directly if you run into any issue or have questions
> though!

Comment 1

[Ed Morley [:emorley]]

Another complication:

There are scripts that run on brasstacks that connect to the API that we'll need to route around the SSO (presumably by using multiple nginx location blocks).


What I've dug up about brasstacks so far:

RHEL 6.9
nginx/1.10.2
Python 2.6.6

The nginx configs in /etc/nginx/* were a bit of a mess with backup versions of files around etc, so I gzipped the folder and left a backup at /root/nginx-backup.tar.gz, then pruned unused content for clarity.

/etc/nginx/nginx.conf seems to be entirely the stock config, and includes /etc/nginx/conf.d/*.conf

The only config there is /etc/nginx/conf.d/orangefactor.conf (I'll attach shortly), which includes /etc/nginx/default_locations/orangefactor.conf.

The locations file does a `fastcgi_pass 127.0.0.1:9500`.

The backend service is run by /etc/init.d/orangefactor (monitored/managed by monit), which uses /usr/bin/spawn-fcgi to run woo_server.fcgi with the OrangeFactor venv Python:
https://hg.mozilla.org/automation/orangefactor/file/tip/server/woo_server.fcgi

Comment 2

[Ed Morley [:emorley]]

Attachment: Existing /etc/nginx/conf.d/orangefactor.conf

Comment 3

[Ed Morley [:emorley]]

Attachment: Existing /etc/nginx/default_locations/orangefactor.conf

Comment 4

[Ed Morley [:emorley]]

We need to switch from stock nginx to openresty (nginx variant with lua support), and openresty's configs live in a different location etc, so the existing configs will need migrating before we get to setting the SSO parts up.

As such, the plan is:
1) Request client credentials using process at https://mana.mozilla.org/wiki/display/SECURITY/SSO+Request+Form
2) Install openresty, luarocks and lua-resty-openidc
3) Create new configs under /usr/local/openresty/... based on the existing nginx ones (also find a generic location for SSL keys rather than their existing nginx-specific home, move the static html directory somewhere more sensible etc)
4) Test by modifying the configs to listen on different ports, starting openresty and making requests to localhost
5) If working, stop openresty, switch ports back to expected values, stop nginx and start openresty
6) Ensure openresty run at boot and nginx not
7) Add a new server block that listens on a different port, that I can use to test adding the lua-resty-openidc specific parts and ensuring we can localhost and the Hawk-protected /api/saveclassification page
8) When SSO client credentials received, add them to the config file
9) If all looks good, apply the SSO config to the main port 443 server block
10) Notify about site now requiring SSO

Comment 5

[Ed Morley [:emorley]]

> 1) Request client credentials using process at
> https://mana.mozilla.org/wiki/display/SECURITY/SSO+Request+Form

Opened:
https://mozilla.service-now.com/sp?id=ticket&table=sc_req_item&sys_id=7ce078b113beba0087bcd0528144b071

> 2) Install openresty, luarocks and lua-resty-openidc

[[e-mail]3 ~]# yum-config-manager --add-repo https://openresty.org/yum/rhel-6/OpenR esty.repo
...
repo saved to /etc/yum.repos.d/OpenResty.repo

[[e-mail]3 ~]# yum install epel-release
...
Package epel-release-6-8.noarch already installed and latest version

[[e-mail]3 ~]# yum install openresty
...
Installed:
  openresty.x86_64 0:1.11.2.3-9.el6

Dependency Installed:
  openresty-openssl.x86_64 0:1.0.2k-1.el6             openresty-pcre.x86_64 0:8.40-1.el6
  openresty-zlib.x86_64 0:1.2.11-1.el6

[[e-mail]3 ~]# openresty -V
nginx version: openresty/1.11.2.3
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC)
built with OpenSSL 1.0.2k  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx <SNIP>

[[e-mail]3 ~]# yum install luarocks
...
Installed:
  luarocks.noarch 0:2.1.2-1.el6

[[e-mail]3 ~]# yum install luarocks openssl-devel lua-devel
...
Installed:
  lua-devel.x86_64 0:5.1.4-4.1.el6

[[e-mail]3 ~]# luarocks install lua-resty-openidc
...
lua-resty-openidc 1.3.1-1 is now built and installed in /usr (license: Apache 2.0)

[[e-mail]3 ~]# luarocks path
export LUA_PATH='/usr/share/lua/5.1/?.lua;/usr/share/lua/5.1/?/init.lua;/root/.luarocks/share/lua/5.1/?.lua;/root/.luarocks/share/lua/5.1/?/init.lua;./?.lua;/usr/lib64/lua/5.1/?.lua;/usr/lib64/lua/5.1/?/init.lua'
export LUA_CPATH='/usr/lib/lua/5.1/?.so;/root/.luarocks/lib/lua/5.1/?.so;./?.so;/usr/lib64/lua/5.1/?.so;/usr/lib64/lua/5.1/loadall.so'

Comment 6

[Ed Morley [:emorley]]

Steps 1-7 from comment 4 are now complete. 

In addition, I have:
* Fixed the issues with the port 80 server block, that were preventing it from being used by cron scripts locally (required so they can skip SSO later).
* Added and tested the SSO exemptions for localhost and /api/saveclassification
* Updated woo_cron.conf so the cron jobs use localhost:80 not brasstacks.mozilla.com:443, so they can skip SSO.
* Made openidc_layer.lua load globally (so I can check the LUA libs are loading fine), but return early unless accessing https://brasstacks.mozilla.com/login_test or the callback/login URLs.
* Updated the TLS configuration with the latest changes from the Mozilla config generator [1]. Notably:
  - HTTP2 is now enabled and working [2].
  - OCSP stapling is now enabled and working [3].
* Reworked the OrangeFactor static assets location block, so it serves them from the source html directory rather than having to sync them across after each deploy to the stock nginx html directory.
* Raised the nginx error_log level from error to warn to make the logs more useful.
* Cleared out a number of leftovers from autolog/mozmill-dashboard/other services that used to run on the box, to make it easier to see what's left.

Left to do (revised plan from that in comment 4):
8) Wait for Auth0 credentials
9) Once received, update openidc_config.lua and `sudo service openresty reload`.
10) Test SSO works using https://brasstacks.mozilla.com/login_test and tweak as needed.
11) If working, remove conditionals in openidc_layer.lua that return early for anything but the test pages, and `sudo service openresty reload`.
12) Check the exemptions correctly don't hit SSO, using:
  - `curl -I http://localhost/orangefactor/api/count` (from brasstacks)
  - `curl -IX POST https://brasstacks.mozilla.com/orangefactor/api/saveclassification` (expected to HTTP 401)
13) Check SSO works for:
  - https://brasstacks.mozilla.com/
  - https://brasstacks.mozilla.com/orangefactor/
  - https://brasstacks.mozilla.com/orangefactor/api/count
14) Check Treeherder submissions not reporting errors on New Relic.
15) Announce that SSO auth now required for OrangeFactor.
16) Move or rename old nginx configuration directory since it should in theory not be used by anything.
17) Check the next round of cron jobs after SSO enabling ran ok.
18) Attach the config files (minus openidc_config.lua secrets) to this bug or else commit them to the orangefactor repo.
19) After X amount of time, remove the old nginx config directory and uninstall nginx (to avoid confusion when people looking at the box in the future).

I'm also going to be collating docs rough edges and filing issues to help improve:
https://github.com/mozilla-iam/testrp.security.allizom.org/blob/master/webserver_configurations/OpenID_Connect/Nginx

[1] https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=nginx-1.11.2&openssl=1.0.1e&hsts=yes&profile=modern
[2] https://tools.keycdn.com/http2-test?url=https%3A%2F%2Fbrasstacks.mozilla.com%2Forangefactor%2F
[3] https://www.ssllabs.com/ssltest/analyze.html?d=brasstacks.mozilla.com&hideResults=on

Comment 7

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #5)
> > 1) Request client credentials using process at
> > https://mana.mozilla.org/wiki/display/SECURITY/SSO+Request+Form
> 
> Opened:
> https://mozilla.service-now.com/sp?id=ticket&table=sc_req_item&sys_id=7ce078b113beba0087bcd0528144b071

Justin, I don't suppose you know when you might be able to look at this Auth0 credential request ticket? :-)

Comment 8

[Ed Morley [:emorley]]

The SSO credentials were issued Friday evening (26th) - I've now updated the config on brasstacks to use them.

The test page worked well for me and also for non-staff contributor LDAP, so I've now enabled SSO site-wide, using steps 9-16 from comment 6, and emailed the mailing lists:
https://groups.google.com/forum/#!topic/mozilla.dev.tree-management/cIc63kVSN5M

Comment 9

[Ed Morley [:emorley]]

OrangeFactor has never set its own Cache-Control headers on static assets including the index page, so I've had to add one to ensure that when the SSO expires the browser still gets the redirect on the index page, rather than just seeing it only on the API XHRs.

Comment 10

[Ed Morley [:emorley]]

Attachment: New Openresty SSO configuration

https://hg.mozilla.org/automation/orangefactor/rev/dac579598e3de81e0b28dff3ae1ed53f8710f33a

Comment 11

[Ed Morley [:emorley]]

Further tweaks:
* yesterday I reduced caching from 60s to zero (since it broke the callback):
  https://hg.mozilla.org/automation/orangefactor/rev/244494578466496a111a3459295c904bfa50c6ef
* I've adapted the BMO OrangeFactor integration to work with the SSO in bug 1368739.
* Bug 1369481 has fixed an omission in the SSO example configs that meant the session cookie was 1 hour and so the refresh process didn't work, with returning visitors being treated as new users.
* I've performed the old nginx install cleanup/remaining steps from comment 6.

I've issues against the example SSO configs for some potential improvements:
https://github.com/mozilla-iam/testrp.security.allizom.org/issues/12
https://github.com/mozilla-iam/testrp.security.allizom.org/issues/13
https://github.com/mozilla-iam/testrp.security.allizom.org/issues/19
https://github.com/mozilla-iam/testrp.security.allizom.org/issues/20
https://github.com/mozilla-iam/testrp.security.allizom.org/issues/21
https://github.com/mozilla-iam/testrp.security.allizom.org/issues/22
https://github.com/mozilla-iam/testrp.security.allizom.org/issues/23

With that, I think we're done :-)

Comment 12

[Ed Morley [:emorley]]

For anyone wanting to access OrangeFactor's API in an automated manner, I'd recommend either:
* Using https://pypi.python.org/pypi/browsercookie/ to fetch the browser SSO cookie for brasstacks.mozilla.com (NB: the automation will need to use the same User Agent as the browser for the session to work)
* Pulling data from ActiveData (https://activedata.allizom.org/ ; the `orange_factor` index) or Treeherder's API instead (https://treeherder.mozilla.org/docs/#!/project/project_bug_job_map_list ; though at present the API cannot retrieve by bug, just by job_id).