Closed Bug 1264801 Opened 8 years ago Closed 8 years ago

Create a proxy for aus3.mozilla.org => aus5.mozilla.org

Categories

(Release Engineering Graveyard :: Applications: Balrog (backend), defect)

Product:
Release Engineering Graveyard
Component:
Applications: Balrog (backend)
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mostlygeek, Assigned: mostlygeek)

References

Details

aus3.mozilla.org serves a Thawte TLS certificate. Older versions of Firefox have this cert pinned so we need to maintain this endpoint. 

For the migration we will 

- create an ELB service the thawte certificate
- create an nginx proxy that sends all traffic to https://aus5.mozilla.org/<request path>

This proxy will essentially be stateless. Its only purpose is to serve the thawte cert until it expires and we can hopefully remove the service.
This proxy has been set up at: https://balrog-proxy.r53-2.services.mozilla.com. It serves the thawte certificate.

Testing with curl: 

$ curl -v --resolve aus3.mozilla.org:$(dig balrog-proxy.r53-2.services.mozilla.com +short | head -n 1):443  https://aus3.mozilla.org/update/3/Firefox/46.0/20160316065941/Linux_x86-gcc3/zh-TW/beta/default/default/default/update.xml


Certs:

== BALROG PROXY ==

$ openssl s_client -connect balrog-proxy.r53-2.services.mozilla.com:443 | openssl x509 -text -noout
depth=1 /C=US/O=Thawte, Inc./CN=Thawte SSL CA
verify error:num=20:unable to get local issuer certificate
verify return:0
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            14:6a:ab:c3:52:09:8c:4d:51:7b:fa:1b:aa:21:2c:6a
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=Thawte, Inc., CN=Thawte SSL CA
        Validity
            Not Before: Sep  9 00:00:00 2013 GMT
            Not After : Sep  8 23:59:59 2017 GMT
        Subject: C=US, ST=California, L=Mountain View, O=Mozilla Foundation, OU=Automatic Update System, CN=aus3.mozilla.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    00:cd:79:32:13:3c:56:4a:f4:29:3a:fd:49:f9:2c:
                    3e:a3:c5:bc:8a:f5:a5:74:5d:78:a8:7f:7b:40:28:
                    6b:35:ef:a0:c1:b8:5b:c1:16:e6:75:9f:9b:c1:0e:
                    70:c1:c7:a8:83:2c:65:4e:28:d6:70:27:80:39:66:
                    61:63:66:9b:94:61:37:33:e5:6c:14:1c:af:8d:76:
                    15:c4:78:4f:8a:35:3c:35:09:e0:68:bd:98:f4:7e:
                    eb:d1:68:c6:f8:50:a1:9e:37:e0:9a:5d:75:6d:52:
                    56:ea:8f:c5:47:77:48:ed:63:cb:0c:bf:88:a4:f9:
                    66:6f:96:6f:33:8a:1a:e9:32:42:2c:bd:3c:77:3f:
                    42:56:59:61:61:99:7d:9c:9f:51:40:58:c0:c1:f6:
                    ab:82:16:66:7f:30:da:35:91:9c:71:34:e3:7a:07:
                    bc:b1:de:01:b3:c1:d7:85:88:b1:2e:24:6e:e6:8b:
                    b0:c5:a7:29:e2:65:3d:34:a8:64:73:3e:38:ef:3e:
                    d3:de:81:bf:ff:a3:b7:61:c0:46:e2:7d:12:87:f0:
                    d9:ae:11:24:8b:63:4e:bf:aa:27:00:a9:39:12:4b:
                    2d:29:1a:81:ac:88:de:84:cf:90:17:9d:40:ee:7d:
                    37:f9:18:fd:1a:7b:6c:f4:97:04:19:f3:d6:21:91:
                    1e:dd
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:aus3.mozilla.org
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Certificate Policies:
                Policy: 2.16.840.1.113733.1.7.54
                  CPS: https://www.thawte.com/cps/

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Authority Key Identifier:
                keyid:A7:A2:83:BB:34:45:40:3D:FC:D5:30:4F:12:B9:3E:A1:01:9F:F6:DB

            X509v3 CRL Distribution Points:
                URI:http://svr-ov-crl.thawte.com/ThawteOV.crl

            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            Authority Information Access:
                OCSP - URI:http://ocsp.thawte.com
                CA Issuers - URI:http://svr-ov-aia.thawte.com/ThawteOV.cer

    Signature Algorithm: sha1WithRSAEncryption
        0f:5f:22:04:f9:f4:1b:fa:99:7f:2a:57:06:26:4b:6d:4d:bf:
        90:5b:16:68:b3:5a:86:64:ba:41:dd:87:41:42:df:8a:1c:37:
        76:fb:25:47:34:30:2c:13:42:1f:69:07:d0:8a:8d:f9:b6:6c:
        5d:13:4c:a9:51:55:11:97:28:3f:ef:8f:c0:dc:64:3b:97:cb:
        1e:19:e5:49:48:d8:d5:52:2b:15:58:40:c4:d7:cf:33:3d:af:
        ce:39:3e:41:d7:8d:e3:5e:c6:db:0c:04:53:2b:fb:d0:45:52:
        e6:a0:23:9a:83:cf:bb:e8:7b:d5:58:b3:bb:01:5f:cc:27:78:
        07:3e:8a:64:55:82:1f:24:f3:c3:f6:04:f4:a6:53:60:d0:7c:
        5f:c8:b0:45:80:d9:b5:39:17:84:7b:d7:38:cd:a1:68:02:32:
        e0:35:7f:c1:c9:bc:2a:f4:ab:56:70:8f:7e:f7:24:e7:1a:90:
        22:34:97:7b:4b:cb:6b:26:9d:98:d8:9b:1b:ee:c0:ff:ae:b7:
        41:6f:46:05:cf:3f:98:6e:79:e1:69:cd:42:8c:47:c0:78:ee:
        50:fa:b7:fa:f0:4f:a5:02:12:b2:34:a6:6c:96:2a:7e:31:41:
        ea:21:d5:0e:a9:1a:c3:f6:c6:f4:f3:91:1b:9a:10:a6:fd:5a:
        1f:7a:db:8c

== AUS3.MOZILLA.ORG ==
$ openssl s_client -connect aus3.mozilla.org:443 | openssl x509 -text -noout
depth=2 /C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
verify error:num=20:unable to get local issuer certificate
verify return:0
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            14:6a:ab:c3:52:09:8c:4d:51:7b:fa:1b:aa:21:2c:6a
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=Thawte, Inc., CN=Thawte SSL CA
        Validity
            Not Before: Sep  9 00:00:00 2013 GMT
            Not After : Sep  8 23:59:59 2017 GMT
        Subject: C=US, ST=California, L=Mountain View, O=Mozilla Foundation, OU=Automatic Update System, CN=aus3.mozilla.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    00:cd:79:32:13:3c:56:4a:f4:29:3a:fd:49:f9:2c:
                    3e:a3:c5:bc:8a:f5:a5:74:5d:78:a8:7f:7b:40:28:
                    6b:35:ef:a0:c1:b8:5b:c1:16:e6:75:9f:9b:c1:0e:
                    70:c1:c7:a8:83:2c:65:4e:28:d6:70:27:80:39:66:
                    61:63:66:9b:94:61:37:33:e5:6c:14:1c:af:8d:76:
                    15:c4:78:4f:8a:35:3c:35:09:e0:68:bd:98:f4:7e:
                    eb:d1:68:c6:f8:50:a1:9e:37:e0:9a:5d:75:6d:52:
                    56:ea:8f:c5:47:77:48:ed:63:cb:0c:bf:88:a4:f9:
                    66:6f:96:6f:33:8a:1a:e9:32:42:2c:bd:3c:77:3f:
                    42:56:59:61:61:99:7d:9c:9f:51:40:58:c0:c1:f6:
                    ab:82:16:66:7f:30:da:35:91:9c:71:34:e3:7a:07:
                    bc:b1:de:01:b3:c1:d7:85:88:b1:2e:24:6e:e6:8b:
                    b0:c5:a7:29:e2:65:3d:34:a8:64:73:3e:38:ef:3e:
                    d3:de:81:bf:ff:a3:b7:61:c0:46:e2:7d:12:87:f0:
                    d9:ae:11:24:8b:63:4e:bf:aa:27:00:a9:39:12:4b:
                    2d:29:1a:81:ac:88:de:84:cf:90:17:9d:40:ee:7d:
                    37:f9:18:fd:1a:7b:6c:f4:97:04:19:f3:d6:21:91:
                    1e:dd
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:aus3.mozilla.org
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Certificate Policies:
                Policy: 2.16.840.1.113733.1.7.54
                  CPS: https://www.thawte.com/cps/

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Authority Key Identifier:
                keyid:A7:A2:83:BB:34:45:40:3D:FC:D5:30:4F:12:B9:3E:A1:01:9F:F6:DB

            X509v3 CRL Distribution Points:
                URI:http://svr-ov-crl.thawte.com/ThawteOV.crl

            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            Authority Information Access:
                OCSP - URI:http://ocsp.thawte.com
                CA Issuers - URI:http://svr-ov-aia.thawte.com/ThawteOV.cer

    Signature Algorithm: sha1WithRSAEncryption
        0f:5f:22:04:f9:f4:1b:fa:99:7f:2a:57:06:26:4b:6d:4d:bf:
        90:5b:16:68:b3:5a:86:64:ba:41:dd:87:41:42:df:8a:1c:37:
        76:fb:25:47:34:30:2c:13:42:1f:69:07:d0:8a:8d:f9:b6:6c:
        5d:13:4c:a9:51:55:11:97:28:3f:ef:8f:c0:dc:64:3b:97:cb:
        1e:19:e5:49:48:d8:d5:52:2b:15:58:40:c4:d7:cf:33:3d:af:
        ce:39:3e:41:d7:8d:e3:5e:c6:db:0c:04:53:2b:fb:d0:45:52:
        e6:a0:23:9a:83:cf:bb:e8:7b:d5:58:b3:bb:01:5f:cc:27:78:
        07:3e:8a:64:55:82:1f:24:f3:c3:f6:04:f4:a6:53:60:d0:7c:
        5f:c8:b0:45:80:d9:b5:39:17:84:7b:d7:38:cd:a1:68:02:32:
        e0:35:7f:c1:c9:bc:2a:f4:ab:56:70:8f:7e:f7:24:e7:1a:90:
        22:34:97:7b:4b:cb:6b:26:9d:98:d8:9b:1b:ee:c0:ff:ae:b7:
        41:6f:46:05:cf:3f:98:6e:79:e1:69:cd:42:8c:47:c0:78:ee:
        50:fa:b7:fa:f0:4f:a5:02:12:b2:34:a6:6c:96:2a:7e:31:41:
        ea:21:d5:0e:a9:1a:c3:f6:c6:f4:f3:91:1b:9a:10:a6:fd:5a:
        1f:7a:db:8c
:ulfr 

Could you give me an R? on the certs in comment #2. They appear to match exactly so I think we're good and older FF's won't have any issues.
Flags: needinfo?(jvehent)
The new services matches exactly the cert currently served by aus3.mozilla.org, so r+

$ openssl s_client -connect balrog-proxy.r53-2.services.mozilla.com:443 2>/dev/null <<<Q | openssl x509 -outform der|sha256sum 
655713c0267b80027773260b7fd46c5ca8ba255b7d401165e9028ae81b868d08  -

$ openssl s_client -connect aus3.mozilla.org:443 2>/dev/null <<<Q | openssl x509 -outform der | sha256sum
655713c0267b80027773260b7fd46c5ca8ba255b7d401165e9028ae81b868d08  -
Flags: needinfo?(jvehent)
Nice CLI trick!!

Since you confirmed[1] that the ELB security policy 2015-05 is OK for FF back to v1 we just need to schedule a time to flip the DNS over. 


[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1248760#c12
:bhearsum 

Could you do a quick QA and R?
Also do you who in IT should be notified that we're moving the service?
Flags: needinfo?(bhearsum)
(In reply to Benson Wong [:mostlygeek] from comment #5)
> :bhearsum 
> 
> Could you do a quick QA and R?

My quick test was fine; I'm running a script to compare a lot more URLs to be sure. I'll report back when it's done.

> Also do you who in IT should be notified that we're moving the service?

Jake or C. are probably the best choice.
Flags: needinfo?(bhearsum)
I did a set of 4,000 URLs and every one got the same response body from the proxy as the real aus3. Looks totally fine to me.
Thanks for the testing Ben! 

We will repoint aus3.mozilla.org to the new proxy this week.
After getting the initial load of production data I made some changes to the stack. Specifically, nginx now ignores the Cache-Control header from aus5. It will cache *all* responses for 60 seconds before updating. The cache hit rate appears to be about 56%. 

The current prod cluster is 3x m3.medium sitting at about 15% CPU usage on each. Everything is looking pretty solid.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: Release Engineering → Release Engineering Graveyard
You need to log in before you can comment on or make changes to this bug.