832461 - create production instance for balrog/aus4

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Comment 1

[[e-mail] (:bhearsum)]

(This is a placeholder to create a production instance for this project.)

Please create a production instance for {Website Name}.

== Date Needed By ==
* No hard deadline, sometime in the next month would be nice though.

== Language ==
* Language: Python
* Framework: Flask

== URL ==
* Production URL: https://aus4.mozilla.org, https://aus4-admin.mozilla.org

== HTTP Auth ==
* HTTP Auth is required to access the admin site.
* Please allow the same set of users access to this site as can access the dev site (aus4-admin-dev.allizom.org), with two exceptions:
** stage-ffxbld should NOT have access
** ffxbld should have access

== HTTP / HTTPS ==
* HTTPS only

== VPN ==
* aus4-admin.mozilla.org should only be accessible via mpt-vpn

== Code Repository ==
* Github Repo: https://github.com/mozilla/balrog/

== Database ==
* This site requires a MySQL database. Both applications share a single database.

== Settings ==
* Settings are read from /etc/aus/admin.ini (aus4-admin.mozilla.org) and /etc/aus/balrog.ini (aus4.mozilla.org)
* admin.ini:
** Requires rw dburi
** sentry_dsn needs to be set to an URL with a secret in it. How should I send that to you?
* balrog.ini
** Requires _ro_ dburi
** sentry_dsn needs to be set similar to the admin one
** specialforcehosts should be set to http://download.mozilla.org

== Cron ==
* No automatic updates of the app, please. Ideally, we should be able to do our own pushes with Chief or similar, but manual pushes are also fine for now.


We also require some binary python modules to be installed, specifically:
* MySQL-python 1.2.3
* Jinja2 2.5.5
* SQLAlchemy 0.7.1
* simplejson 2.0.9

For reference, here's the bugs that were involved in setting up the dev environment:
https://bugzilla.mozilla.org/show_bug.cgi?id=719581
https://bugzilla.mozilla.org/show_bug.cgi?id=685169

Comment 2

[Chris Turra [:cturra]]

moving this into the product delivery component so it's on the correct radar. i will run with this now. stay tuned for more details :)

Comment 3

[Hal Wine [:hwine] use NI!]

Additional requirement -- in the production environment, the "balrog admin app" needs to be deployed on a physically separate node per :psiinon. Per discussion with :jakem we believe we can use the adm node for that.

This will impact which hosts have access to the db r/w vip, and which can only access the r/o vip.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Similar to aus3.m.o, please purchase two SSL certificates for aus4.mozilla.org, issued by completely separate providers, so that we can continue to provide updates in the event of a compromise. Firefox does a kind of cert pinning, where the commonName and issuerName of the cert must be pref-configured values (see app.update.certs.* in about:config). Bug 913971 tracks adding the aus4 certs to the app.

Comment 5

[[e-mail] (:bhearsum)]

Chris, here's the query we need to run to set-up an initial admin user that I mentioned:
INSERT INTO permissions VALUES ('admin', '[e-mail]', NULL, 1)

Comment 6

[Chris Turra [:cturra]]

done.

mysql> INSERT INTO permissions VALUES ('admin', '[e-mail]', NULL, 1);
Query OK, 1 row affected (0.00 sec)

mysql> select * from permissions\G
*************************** 1. row ***************************
  permission: admin
    username: [e-mail]
     options: NULL
data_version: 1
1 row in set (0.00 sec)

Comment 7

[[e-mail] (:bhearsum)]

Chris, I know we talked earlier today and yo usaid that the only thing left to do was the ACLs, but I just realized that Nick's comment about SSL certificates hasn't been specifically addressed yet. Where do we stand here?

(In reply to Nick Thomas [:nthomas] from comment #4)
> Similar to aus3.m.o, please purchase two SSL certificates for
> aus4.mozilla.org, issued by completely separate providers, so that we can
> continue to provide updates in the event of a compromise. Firefox does a
> kind of cert pinning, where the commonName and issuerName of the cert must
> be pref-configured values (see app.update.certs.* in about:config). Bug
> 913971 tracks adding the aus4 certs to the app.

Comment 8

[Chris Turra [:cturra]]

:bhearsum - i will be purchasing a ssl certificate for aus4.mozilla.org today through digi cert. any additional ones we want can and will be done, but are probably not necessary for the initial launch. we can go ahead and have another issued through Thawte, just like we did a couple weeks back with aus3, for this secondary signed certificate.

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Chris Turra [:cturra] from comment #8)
> :bhearsum - i will be purchasing a ssl certificate for aus4.mozilla.org
> today through digi cert. any additional ones we want can and will be done,
> but are probably not necessary for the initial launch. we can go ahead and
> have another issued through Thawte, just like we did a couple weeks back
> with aus3, for this secondary signed certificate.

Sounds good. I filed bug 919746 so we don't lose track.

Comment 10

[Chris Turra [:cturra]]

ssl certificate from digicert now in place for aus4.mozilla.org..

$ openssl s_client -connect aus4.mozilla.org:443
CONNECTED(00000003)
depth=1 /C=US/O=DigiCert Inc/CN=DigiCert Secure Server CA
...
---
Certificate chain
 0 s:/C=US/ST=CA/L=Mountain View/O=Mozilla Foundation/CN=aus4.mozilla.org
   i:/C=US/O=DigiCert Inc/CN=DigiCert Secure Server CA
 1 s:/C=US/O=DigiCert Inc/CN=DigiCert Secure Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
  ...


additionally, as mentioned on irc, i have used the (internal) mozilla certificate authority to sign the ssl certificate on aus4-admin.mozilla.org since it will be an internal only service (and saves up a couple hundred bucks).

Comment 11

[[e-mail] (:bhearsum)]

Chris, with bug 920115 resolved I'm able to route to this fine now. It looks like the issuer chain is partly missing though:
aus4-admin.mozilla.org uses an invalid security certificate.
The certificate is not trusted because no issuer chain was provided.
(Error code: sec_error_unknown_issuer)

The app appears functional if I click through that error, though. So close!

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #11)
> Chris, with bug 920115 resolved I'm able to route to this fine now. It looks
> like the issuer chain is partly missing though:
> aus4-admin.mozilla.org uses an invalid security certificate.
> The certificate is not trusted because no issuer chain was provided.
> (Error code: sec_error_unknown_issuer)
> 
> The app appears functional if I click through that error, though. So close!

Nevermind - I just realized that this is because we're using the Mozilla Root here.

Comment 13

[Chris Turra [:cturra]]

closing this off now that we're in production with aus4 for nightly and all the monitoring is in place for this new service \o/

Comment 14

[[e-mail] (:bhearsum)]

(In reply to Chris Turra [:cturra] from comment #13)
> closing this off now that we're in production with aus4 for nightly and all
> the monitoring is in place for this new service \o/

\o/!