1138797 - Acquire SSL certs for *.phish-error.mozilla.com and *.malware-error.mozilla.com

Status: VERIFIED FIXED

(Infrastructure & Operations :: SSL Certificates, task)

Description

[Reed Loden [:reed]]

Similar to the existing SSL cert (used under SNI) for *.phish-report.mozilla.com, need SSL certs for *.phish-error.mozilla.com and *.malware-error.mozilla.com.

This blocks the patch in bug 1109475.

Comment 1

[Justin Dolske [:Dolske]]

And, hmm, we also need to update the server redirects. Currently they just 302 to the non-SSL Google endpoint.


For example...

https://en-US.phish-report.mozilla.com/?hl=en-US&url=http://reedloden.com

Yields:

302 Found
Location:"http://www.google.com/safebrowsing/report_phish/?tpl=mozilla&hl=en-US&url=http://reedloden.com"

Which in turn yields:

301 Moved Permanently
Location:"https://www.google.com/safebrowsing/report_phish/?tpl=mozilla&hl=en-US&url=http://reedloden.com"


I suppose we could just put the final Google URL directly into Firefox, and skip the server redirects entirely. I'm not sure why we bother going through a Mozilla intermediary... I did a little code digging, and this dates back to bug 344063 and bug 348878 (but the bugs are vague on why). I _suspect_ the reason is that since this was before the rapid-release cycle (Firefox 2, baby!), a server redirect gave us the capability to quickly change where these pages went (eg if we needed to change providers, take hosting away from Google, or simply change what Google URL loads). These days we can just push out a hotfix or wait a few weeks for the next regular release.

Doesn't really matter, since I'd expect these URLs to only rarely be loaded by users anyway. So probably simplest to just fix the server certs and redirects.

Comment 2

[Justin Dolske [:Dolske]]

(In reply to Justin Dolske [:Dolske] from comment #1)

> Doesn't really matter, since I'd expect these URLs to only rarely be loaded
> by users anyway. So probably simplest to just fix the server certs and
> redirects.

Checked with Gavin, and he'd prefer these to remain redirects via mozilla.com.

Comment 3

[:Atoll]

Setting sec-review? to get opsec signoff on the wildcard certificate.

Comment 4

[:Atoll]

certificateS, sorry.

Comment 5

[Gian-Carlo Pascutto [:gcp]]

We'd like to have the ability to support multiple providers so yes redirects are preferable.

Comment 6

[:Atoll]

SHA2 certificates requested, #00655344 and #00655345.

Comment 7

[:Atoll]

Received the certificates. Uploaded them to Zeus along with their intermediate. Added their SNI bindings to the static - https VIP.

The next step here is to update the pp-web backends serving phish-error and malware-error to understand that they're available over HTTPS as well, as right now they're HTTP-only.

Once that's done and verified working, we would alter the DNS for phish-error and malware-error from CNAME static-non-ssl to CNAME static, so that the new SSL sites are available.

Comment 8

[:Atoll]

I observed *.malware-report.mozilla.com in the HTTP redirect configs, do we need to order an SSL cert for that as well?

The following patch should be sufficient to enable SSL on these VIPs; waiting for an answer the prior question and team review before I proceed.

--- a/files/generic/httpd-static/etc-httpd/conf/domains/virtual-redirect.mozilla.org.conf
+++ b/files/generic/httpd-static/etc-httpd/conf/domains/virtual-redirect.mozilla.org.conf

 # phish-error.mozilla.com
-<VirtualHost *:80>
+<VirtualHost *:80 *:81>

 # malware-error.mozilla.com
-<VirtualHost *:80>
+<VirtualHost *:80 *:81>

Comment 9

[:Gavin Sharp [email: [e-mail]]]

(In reply to Richard Soderberg [:atoll] from comment #8)
> I observed *.malware-report.mozilla.com in the HTTP redirect configs, do we
> need to order an SSL cert for that as well?

I think the idea is that we'll get rid of the in-product references to that host with the patch in bug 1109475. But we may need to maintain compat with old versions anyhow, and so might want to add it anyways. ni?dolske to confirm.

Comment 10

[:Atoll]

Alright. These carry a relatively high cost for SSL certs (the 3 wildcards add up to around ~14 normal certs), so if we don't absolutely need the older one, it'd be nice to avoid paying for it.

Comment 11

[Justin Dolske [:Dolske]]

Old Firefox versions (39 and earlier, assuming this lands for Firefox 40 starting next week) will still be making HTTP (not HTTPS) requests to the old URLs. Which is a bunch of %LOCALE%.[phish|malware]-[report|error].mozilla.com hosts.

So we should keep the existing non-SSL redirects, probably for quite a while (until our long-tail of slow-to-update users is so small that we don't care).

Comment 12

[:Atoll]

Alright, so I'll *enable* HTTPS support on phish-report, phish-error, malware-error but NOT malware-report, which is expected to have no impact on the plain HTTP redirects.

Comment 13

[Justin Dolske [:Dolske]]

Yep.

In fact, upon further reflection, I believe *.malware-report.mozilla.com could be turned off entirely. It seems to have never been used, as we never actually implemented a way for users to report malware from within the browser. My patch removes that deadwood entirely.

Comment 14

[:Atoll]

It costs us nothing to keep it alive and running, since it's on a generic redirect cluster and has no HTTPS certificate. But we can also decommission it entirely if you'd prefer - just file a separate bug for that, and indicate whether any coordination is needed for ESR or whatever. (The decom would make the hostname stop resolving, so any failures would be immediate rather than timeouts.)

Comment 15

[:Atoll]

I used a slightly more complex patch, which adds a new SSL vserver for phish-error and malware-error, that redirects to HTTPS targets instead. This way, nothing changes with the existing phish-error, malware-error VIPs (and with phish-report, either, which already has an existing HTTPS VIP) so I can verify this works properly - and coordinate with y'all - before continuing with the HTTPS-ification of the redirect targets of phish-report.

+<VirtualHost *:81>
+    ServerName phish-error.mozilla.com
+    ServerAlias *.phish-error.mozilla.com
+    RewriteEngine On
+    RewriteCond %{QUERY_STRING} ^hl=(.+)&url=(.+)
+    RewriteRule ^/$ https://www.google.com/safebrowsing/report_error/?tpl=mozilla&hl=%1&url=%2 [R,NE]
+    RewriteRule ^/.*$ https://www.mozilla.org/ [R]
+</VirtualHost>

+<VirtualHost *:81>
+    ServerName malware-error.mozilla.com
+    ServerAlias *.malware-error.mozilla.com
+    RewriteEngine On
+    RewriteCond %{QUERY_STRING} ^hl=(.+)
+    RewriteRule ^/$ https://www.stopbadware.org/firefox [R,NE]
+    RewriteRule ^/.*$ https://www.mozilla.org/ [R]
+</VirtualHost>

Comment 16

[Justin Dolske [:Dolske]]

Any update here?

Comment 17

[:Atoll]

Are these sites in use in production Firefox today? If the above patch has any impact, what issues will users encounter as a result of (for instance) an outage or a broken redirect?

I can ship this any time, theoretically, but I need guidance on what the risks are if there's any errors (since we do not have a staging site for these redirects).

Normally we would just ship redirector changes but in this case, since these URLs are baked into Firefox itself, I'm trying to take extra care as it's not (yet) apparent what is at stake here.

Comment 18

[Gian-Carlo Pascutto [:gcp]]

(In reply to Richard Soderberg [:atoll] from comment #17)
> Are these sites in use in production Firefox today?

Yes:
https://dxr.mozilla.org/mozilla-central/source/browser/app/profile/firefox.js#983

> If the above patch has any impact, what issues will users encounter as a result of (for instance)
> an outage or a broken redirect?

They will not be able to report phishing/malware sites that are false positives or false negatives. I would not consider that too critical.

Comment 19

[Justin Dolske [:Dolske]]

I think it should be fairly trivial for us to coordinate and do a quick manual test when this goes live (to make sure nothing broke). Since these URLs are only used when a user explicitly goes to report a phishing site / mistaken site (which I'd expect to be a fairly rare operation), even if the patch is utterly broken it would have negligible effect on users until it's rolled back.

LMK when you might be able to deploy this? Should only take a few minutes to verify.

Comment 20

[Justin Dolske [:Dolske]]

Attachment: Firefox testplan

I wrote up a testplan on how and what to verify in Firefox to make sure these changes don't break anything.

Comment 21

[Justin Dolske [:Dolske]]

Richard: can we do this in the next few days?

Comment 22

[:Atoll]

Sure, let's do this tomorrow. I have one deploy set up for 10:45am and that will only take ~30min, so we can do this at 11:30am. If that's not okay, suggest a time and we'll do that instead? I primarily am looking for help testing that things all still work after the changes are complete.

Comment 23

[:Atoll]

Updated the DNS entries for phish-error and malware-error from CNAME static-non-ssl to CNAME static.zlb.phx.mozilla.net, so that we can enable them as https endpoints.

Comment 24

[:Atoll]

And committed the http/https split support into the configs. Note that the https endpoints for *.phish/malware on our side will redirect to https://whatever, rather than the old http://whatever as served from the http endpoints.

:dolske is verifying, results are good with curl, so I'm tentatively closing this as fixed and we'll see.

Comment 25

[Justin Dolske [:Dolske]]

All good from my side: current Release channel Firefox works fine, as does my updated patch in bug 1109475.