1251768 - IT Request: SSL cert for Mozilla Switzerland (Michael Kohler [:mkohler])

Status: RESOLVED INVALID

(Infrastructure & Operations :: SSL Certificates, task)

Description

[Michael Kohler [:mkohler]]

Name:
Michael Kohler [:mkohler]

Mozillians.org Profile:
https://mozillians.org/en-US/u/mkohler/

Reps Profile:
https://reps.mozilla.org/u/michaelkohler/

Community Name:
Mozilla Switzerland

:: SSL

Domain Name:
mozilla.ch


Comments:
We're currently moving mozilla.ch to the Community Ops PaaS and we will need SSL for our website. This is the last point (apart from the DNS A record) that needs to be solved before we can finally move :)

Comment 1

[Michael Kohler [:mkohler]]

The domain is already owned by Mozilla.

Comment 2

[Tanner Filip [:tanner]]

Let's Encrypt isn't an option here, so we'll have to go through DigiCert.

Comment 3

[Tanner Filip [:tanner]]

pierros, r?

Comment 4

[Michael Kohler [:mkohler]]

A little bit of background regarding letsencrypt:

23:16:24 <~mkohler> An unexpected error occurred:
23:16:24 <~mkohler> The request message was malformed :: Error creating new authz :: Name is blacklisted
23:16:24 <~mkohler> Please see the logfiles in /var/log/letsencrypt for more details.


from the #letsencrypt IRC channel:

23:34:47 <mkohler> hi there, I just got "The request message was malformed :: Error creating new authz :: Name is blacklisted" .. where can i see the list of blacklisted names?
23:36:25 <+jsha> Unfortunately it's not public at the moment. It's a list of about 200 "high risk" names, mostly culled from the Alexa top 1k. Do you mind sharing your domain?
23:36:35 <mkohler> mozilla.ch
23:36:46 <mkohler> I can understand the high-risk part, but well, now I've run into it *g*
23:37:32 <+jsha> Yep, mozilla.<tld>'s definitely on there.
23:37:41 <kerio> how does mozilla get a cert then
23:38:08 <mkohler> well, I don't think we have ever used letsencrypt for our mozilla.tld websites tbh
23:38:13 <+jsha> Well, currently they don't get certs from Let's Encrypt. We're still talking through our policy on what to do for high risk domains that want to get a cert.

Comment 5

[Pierros Papadeas [:pierros]]

OK this is funny :) Thanks Michael for digging into this.

@Shyam do we have any idea on how we can work around this? We expect most (in not all) of our mozilla community sites to bump into this soon.

Comment 6

[Shyam Mani [:fox2mike]]

(In reply to Pierros Papadeas [:pierros] from comment #5)
> OK this is funny :) Thanks Michael for digging into this.
> 
> @Shyam do we have any idea on how we can work around this? We expect most
> (in not all) of our mozilla community sites to bump into this soon.

Good point. Let's see if ekr has an answer or an idea here. 

Eric, seems like there's a high risk domain block, which prevents Mozilla community sites from getting LE certs. Any timeline/ideas on how we can proceed here? As Pierros rightly calls this out in comment #5, we'll run into this with a lot of our other community sites. 

Thanks!

Comment 7

[Eric Rescorla (:ekr)]

Barnes?

Comment 8

[Richard Barnes [:rbarnes]]

I think Let's Encrypt has a way to whitelist names that would otherwise be blacklisted.  Hopefully Josh can comment on the current state of the art.

Comment 9

[Josh Aas]

Let's Encrypt has two blacklists. One blocks all subdomains of a domain, the other only blocks specific domains. Right now mozilla.com/org are on the one blocking all subdomains.

We can move Mozilla to the one blocking only specific domains if you want. I'll need a list of Mozilla domains you want to remain on the specific blacklist (top-level sites, addons?), and I encourage you to deploy CAA records before we make any change like this.

Comment 10

[Josh Aas]

Let's take this conversation to email because we'll have to gather some info per Let's Encrypt policy before making any changes.

Comment 11

[Shyam Mani [:fox2mike]]

(In reply to Josh Aas from comment #10)
> Let's take this conversation to email because we'll have to gather some info
> per Let's Encrypt policy before making any changes.

Josh,

Sure. But this domain in question is mozilla.ch (not mozilla.org or .com, which should continue to live in the blacklist for now). Who else should we include in the email discussion?

Thanks!

Comment 12

[Josh Aas]

Ah, sorry I missed that before. All domains on our blocklist are split into two parts - permute and no-permute. Anything on the permute list gets permuted across all TLDs. So mozilla.com turns into almost every TLD variation of mozilla(.*). We can remove mozilla to the no-permute list but we'll need a list of actual Mozilla TLDs for the no-permute list. For example:

mozilla.com
mozilla.org
mozilla.jp
... any more?

Comment 13

[John Giannelos [:nemo-yiannis]]

Hey all!
Is there any update regarding white-listing the domains of mozilla community-sites in let's encrypt?

Comment 14

[Pierros Papadeas [:pierros]]

Shyam can you provide the list that Josh wants, so we can move forward?

Comment 15

[:Atoll]

We are probably going to ask to prohibit a short list of 'mozilla.org' and 'prohibited.mozilla.org' WITHOUT prohibiting other 'subdomain.mozilla.org' or 'mozilla.org.tld' entries.

We intend to issue Let's Encrypt certificates for some fraction (for instance, comment 11) of the top-level "mozilla.xyz" domains in the list requested by Josh in comment 12, which significantly complicates providing the requested list of domains.

So whatever list we provide here will almost certainly be an explicit, no-permute, list of FQDNs for which we do not wish to issue Let's Encrypt certificates. It's not immediately apparent to me what that means in terms of "permute or no-permute" and so on, so I wanted to comment in plain speak first.

Comment 16

[April King [:April]]

Do folks want me to make the decision on what sites stay on the blacklist?  I would say that pretty much anything we have a EV cert for is probably a reasonable place to start.  AMO, BMO, SUMO, Bedrock, the usual suspects.

We have three ways we can probably approach this:

a) Prune the secret Let's Encrypt blacklist down to just the domains we don't want them to issue for
b) Keep that list in place and use CAA records to whitelist sites that LE can issue a cert for <-- josh, does this work?
c) Remove all of the Mozilla domains from the blacklist and add CAA records for all of our domains (blacklisting the sensitive domains, whitelist DigiCert/LE for everything else)

Comment 17

[:Atoll]

As noted in bug 882128, our DNS hosting provider does not support CAA records, so until they do, we aren't permitted to use them for these purposes.

Comment 18

[Eric Ziegenhorn :ericz]

So since we can't do CAA records yet that leaves only April's option a:

  Prune the secret Let's Encrypt blacklist down to just the domains we don't want them to issue for

April, can you come up with such a list?

Comment 19

[April King [:April]]

I'm discussing with my colleagues, and I'll get back to you on this asap.

Comment 20

[April King [:April]]

Okay, here is my initial blacklist:

Including domain and subdomains:
aus*.mozilla.org
addons.mozilla.org
addons.mozilla.net
bugzilla.org
bmoattachments.org
cdn.mozilla.org
cdn.mozilla.net
firefox.com
getfirefox.com
services.mozilla.com

Domain only:
mozilla.org
www.mozilla.org

But I'd like to get a quick r+ on this, at least from ulfr (Cloud Services), seceng (rbarnes?), and webops (atoll).  It's all the truly high-risk sites I could think of, plus all the sites pinned inside Firefox.  This should also be the list of sites that we should definitely get CAA records on, as soon as Akamai makes it possible.

Sorry for all the needinfos!

Comment 21

[:Atoll]

:jgmize, for bedrock (productdelivery?)

Comment 22

[Guillaume Destuynder [:kang] [this account is no longer active]]

> Domain only:
> mozilla.org
> www.mozilla.org

mozilla.com
www.mozilla.com

are +- equivalent

Comment 23

[April King [:April]]

Right, thanks for catching that.  :)

Comment 24

[Daniel Veditz [:dveditz]]

What about wikimo, SUMO, MDN? user-supplied content but with editorial oversight. People tend to trust the answers they find there (especially "old" ones since they figure we'll remove outright bad stuff once we catch it) and we wouldn't want fakes.

I really hate specific listing of protected domains; we're going to miss something or, more likely, forget to update the list when we invent some new and important site. Is there really no way to say "here are our known-good community sites" and keep the rest blacklisted? That way people can't add new ones (legit or fake) without having to go through us to tell LE to update their list.

Comment 25

[April King [:April]]

That seems totally reasonable to me.  So this is the current blacklist, as it stands:

Including subdomains:
aus*.mozilla.org
addons.mozilla.org
addons.mozilla.net
bugzilla.org
bmoattachments.org
cdn.mozilla.org
cdn.mozilla.net
firefox.com
getfirefox.com
services.mozilla.com

Domain only:
mozilla.org
www.mozilla.org
mozilla.com
www.mozilla.com
wiki.mozilla.org
support.mozilla.org
developer.mozilla.org
irc.mozilla.org

I agree that having a whitelist is better, but it's awkward to manage with the current system. I personally want people to get TLS certificates without having to go through the whole whitelisting process as it's too frequent on occurrence.

Comment 26

[:Atoll]

aus* expands to aus/aus2/aus3/aus4/aus5, but we should preemptively declare aus6/7/8/9 here which should last us a few years.

Should 'download.mozilla.org' be in this list? Are all Firefox-pinned domains in this list?

Comment 27

[April King [:April]]

:atoll: I can add it, but it's just a redirect to bedrock, and I don't see it used much at all.

And yes, all of the pinned sites are in this list.

Comment 28

[April King [:April]]

Updated list:

Including subdomains:
aus.mozilla.org
aus2.mozilla.org
aus3.mozilla.org
aus4.mozilla.org
aus5.mozilla.org
aus6.mozilla.org
aus7.mozilla.org
aus8.mozilla.org
aus9.mozilla.org
addons.mozilla.org
addons.mozilla.net
bugzilla.mozilla.org
bmoattachments.org
cdn.mozilla.org
cdn.mozilla.net
download.mozilla.org
firefox.com
getfirefox.com
services.mozilla.com

Domain only:
mozilla.org
www.mozilla.org
mozilla.com
www.mozilla.com
wiki.mozilla.org
support.mozilla.org
developer.mozilla.org
irc.mozilla.org

Barring any objections in the next few days, I'm going to say this is the final list.

Comment 29

[Kendall Libby [:fubar] (he/him)]

/me butts in

Please add hg.mozilla.org.

Comment 30

[April King [:April]]

Including subdomains:
aus.mozilla.org
aus2.mozilla.org
aus3.mozilla.org
aus4.mozilla.org
aus5.mozilla.org
aus6.mozilla.org
aus7.mozilla.org
aus8.mozilla.org
aus9.mozilla.org
addons.mozilla.org
addons.mozilla.net
bugzilla.mozilla.org
bmoattachments.org
cdn.mozilla.org
cdn.mozilla.net
download.mozilla.org
firefox.com
getfirefox.com
services.mozilla.com

Domain only:
mozilla.org
www.mozilla.org
mozilla.com
www.mozilla.com
wiki.mozilla.org
support.mozilla.org
developer.mozilla.org
irc.mozilla.org
hg.mozilla.org

Comment 31

[Richard Barnes [:rbarnes]]

(In reply to April King [:April] from comment #30)

r+

Comment 32

[Josh Mize [:jgmize]]

(In reply to April King [:April] from comment #30)

lgtm

Comment 33

[Julien Vehent]

(In reply to April King [:April] from comment #30)

r+

Comment 34

[Gene Wood [:gene]]

As April is out, she's asked me to represent on the infosec side on this.

Josh, now that we have a final blacklist that's been signed off on ( Comment 30 ), would you be able to get it loaded into letsnecrypt?

Comment 35

[Gene Wood [:gene]]

:jcj April also said you might be able to help get Comment 30 updated in the letsencrypt blacklist?

Comment 36

[J.C. Jones [:jcj] (he/him)]

To be clear, this will remove "mozilla.*" from Let's Encrypt's blacklist, but will retain everything in comment #30. So mozilla.info, etc., will be issueable.

Assuming that's the desired outcome, I've put together a patch for the Let's Encrypt folks. I'll let you know when it applies, probably in 10 days or so.

Comment 37

[J.C. Jones [:jcj] (he/him)]

Let me add to that: I'm not clear whether Let's Encrypt has completed its diligence on the blacklist removal, so while I've created the patch, I have no visibility into its application date. 10 days would be reasonable if all the policy requirements were already met, and I realize from reading the rest of the thread that /may/ not be true.

Comment 38

[Gene Wood [:gene]]

:jcj Can you let me know the status on the blacklist update in Let's Encrypt (if you know) or if not can you point me at who would know if the update has been made and if not when it will?

I'm also not sure from your comments if the 10 day timer started 11 days ago or if you're waiting on anything else?

Comment 39

[Gene Wood [:gene]]

I've tested and confirmed that letsencrypt still forbids by policy creating certs for, in this case a name like foo.security.mozilla.org

Comment 40

[J.C. Jones [:jcj] (he/him)]

I've reached out again to ascertain status on this relative to ISRG's policies.

Note, Gene, that after adoption *.mozilla.org, *.mozilla.com, *.mozilla.net *.firefox.com, *.getfirefox.com, and *.bmoattachments.org will continue to be blacklisted, as the CA software blacklists entire eTLD+1's. The original poster's mozilla.ch will be permitted, though. 

There's a bug open against the CA software to permit finer-grained blacklisting, and it has a patch, but I can't predict when it will go operational. https://github.com/letsencrypt/boulder/issues/1233

Comment 41

[J.C. Jones [:jcj] (he/him)]

[Clearing needinfo for Josh Aas]

We need a Mozilla attorney to produce a letter, on letterhead, asking for these changes, and send it to the ISRG/Let's Encrypt security@ email address. ISRG will then verify the attorney really sent it, and the change will take effect.

I don't have any contacts in Moz-Legal; can someone else on this thread take that action?

Comment 42

[Eric Rescorla (:ekr)]

Adding Mozilla from Legal

Comment 43

[Eric Rescorla (:ekr)]

I mean Mika

Comment 44

[Gene Wood [:gene]]

:jcj

> Note, Gene, that after adoption *.mozilla.org, *.mozilla.com, *.mozilla.net *.firefox.com, *.getfirefox.com, and *.bmoattachments.org will continue to be blacklisted, as the CA software blacklists entire eTLD+1's. The original poster's mozilla.ch will be permitted, though. 

I think :April's Comment 30 may have been misunderstood then. Comment 30 lays out two sets of domains to be blacklisted. The first set of domains to be blacklisted include subdomains. You'll see in that list is for example "firefox.com". So yes, as you stated, "*.firefox.com ... will continue to be blacklisted".

The second list in Comment 30 is a list of domains only. This means that the blacklisting should only apply to that domain name, not child domains. For example though we do want "mozilla.org" do be blacklisted we do *not* want "*.mozilla.org" to be blacklisted.

If you're saying that there is a technological limitation on the Let's Encrypt side from blacklisting for example "download.mozilla.org" but not "*.mozilla.org" then we will have to reconsider out list as it was made without knowledge that the only domains that Let's Encrypt blacklists are eTLDs+1.

Comment 45

[Gene Wood [:gene]]

Mika,
   Lemme give you an introduction to what's going on here, but also say that we should hold for the moment on the request in Comment 41 for a letter to Let's Encrypt from Mozilla legal as there may be a few more technical issues we need to workout first.

To bring you up to speed on what this is :

Currently, Let's Encrypt doesn't issue certificates for any domain names in a suite of Mozilla related zones (e.g. mozilla.com, mozilla.org, etc). They do this as a safety measure to prevent an unauthorized person that was able to get control of either one of our webservers or one of our DNS zones from being able to get Let's Encrypt to issue them a certificate for that DNS name.

Let's Encrypt accomplishes this by putting domains of ours in an internal blacklist of theirs so that any requests for certificates for those names are rejected.

We are asking them to relax this blacklisting so that we can still ensure that Let's Encrypt never issues a certificate for domains that we indeed do want to have blacklisted, but that other domains which we control are able to use the Let's Encrypt service.

An example to make this more concrete is that, though we want to continue to blacklist any ceritifcates from being generated for "mozilla.com" we don't want to blacklist "foo.mozilla.com"

Comment 30 above shows the list of what we want to retain as blacklisted.

:jcj lays out in Comment 41 what Let's Encrypt needs from us to complete this request which I believe is where you come in.

Comment 46

[J.C. Jones [:jcj] (he/him)]

(In reply to Gene Wood [:gene] from comment #44)
> If you're saying that there is a technological limitation on the Let's
> Encrypt side from blacklisting for example "download.mozilla.org" but not
> "*.mozilla.org" then we will have to reconsider out list as it was made
> without knowledge that the only domains that Let's Encrypt blacklists are
> eTLDs+1.

Alright. It is the case that Let's Encrypt can only blacklist eTLD+1s today, but that should change soon. I propose we do this in two passes to un-block the original request.

Step 1: Ask (on letterhead) for ISRG/Let's Encrypt to stop blacklisting "Mozilla" across all eTLDs, reducing their blacklist to "mozilla.org", "mozilla.com", and - if you like - adding "firefox.com", "getfirefox.com", and "bmoattachments.org". 

Step 2: Once Let's Encrypt supports finer-grained blacklists (~ June 2016), write a more tailored blacklist/whitelist and submit that update to Let's Encrypt.

Comment 47

[:Atoll]

Comment 46 is acceptable to Webops, and would address the majority of our outstanding LE requests until the future June support is introduced.

Comment 48

[Julien Vehent]

Or: get rid of the blacklisting of Mozilla domains entirely.

After all, we're not preventing other CAs from issuing certificates for our properties today, so a blacklist in LE is not making us a lot more secure than if it wasn't there.

We already protect sensitive properties with key pining, and should continue to do so.

Just my 2c

Comment 49

[:Atoll]

Comment 48 is also acceptable to Webops :)

Comment 50

[J.C. Jones [:jcj] (he/him)]

Yup, either way is fine, and you can always request changes so feel free to start conservative. Most of these requests are two sentences and a bullet list, or a link to a list on the web somewhere.

Comment 51

[Richard Barnes [:rbarnes]]

(In reply to Julien Vehent [:ulfr] from comment #48)
> Or: get rid of the blacklisting of Mozilla domains entirely.
> 
> After all, we're not preventing other CAs from issuing certificates for our
> properties today, so a blacklist in LE is not making us a lot more secure
> than if it wasn't there.
> 
> We already protect sensitive properties with key pining, and should continue
> to do so.

You make a fair point, but it seems like it doesn't *hurt* to have these domains blacklisted with LE.  Defense in depth.

I don't feel all that strongly one way or the other.  Just being conservative.

Comment 52

[Gene Wood [:gene]]

Ok, I talked to :jeff (manager of Infosec) and he's asked that we hold on lifting the blacklist until we've established monitoring of the letsencrypt issued certs through the Certificate Transparency logs ( https://www.certificate-transparency.org/ ).

So I think this puts us back at where we were with Comment 2

Webops can you help the requester get a cert setup through Digicert?

Here's a ticket on the monitoring work : https://bugzilla.mozilla.org/show_bug.cgi?id=1270657

Once that's done we pick this back up, remove our domains from the blacklist and rely on monitoring to alert us to anything unexpected in regards to certs being issued by letsencrypt (or any transparent CA).

Comment 53

[John Giannelos [:nemo-yiannis]]

(In reply to Gene Wood [:gene] from comment #52)
> So I think this puts us back at where we were with Comment 2
> 
> Webops can you help the requester get a cert setup through Digicert?

Let's coordinate this because we need a SAN for multiple mozilla community sites instead. We need to draft a list with the first batch of sites that are going to be migrated. Ideally by the time we are ready to migrate more sites, Let's Encrypt blacklisting issue will be resolved so we can issue certs without the need to go through Digicert.

Comment 54

[Elvin Lee [:ellee] (no longer with Mozilla)]

Subbing myself in for Mika from Legal. Thanks, :gene and :jcj, for the helpful summaries. From comment 52 it sounds like this is back on hold for the moment. If the request is needed again, can someone from the appropriate team (not sure if it would be webops or infosec) open a Legal bug for the requested letter (Product: Legal, Component: Trademark) so we can track that work?

Comment 55

[Gene Wood [:gene]]

:ellee Thanks for the clarification, I've created Bug 1271005 to trigger us contacting Legal once we've completed our work related to monitoring.

Comment 56

[Eric Ziegenhorn :ericz]

I've submitted a request for the cert through DigiCert.  It is pending domain validation so a few addresses such as [e-mail] should receive a confirmation email from DigiCert.  Once you confirm the domain via the instructions in that email Michael, we should be good to issue the cert.  Thanks for your patience!

Comment 57

[Michael Kohler [:mkohler]]

(In reply to Eric Ziegenhorn :ericz from comment #56)
> I've submitted a request for the cert through DigiCert.  It is pending
> domain validation so a few addresses such as [e-mail] should
> receive a confirmation email from DigiCert.  Once you confirm the domain via
> the instructions in that email Michael, we should be good to issue the cert.
> Thanks for your patience!

We currently don't have any email addresses set up on this domain (at least none that the community would control). DNS seems to suggest the same.

; <<>> DiG 9.8.3-P1 <<>> -t ANY mozilla.ch
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3544
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;mozilla.ch.			IN	ANY

;; ANSWER SECTION:
mozilla.ch.		180	IN	A	46.101.237.42
mozilla.ch.		180	IN	NS	ns1-240.akam.net.
mozilla.ch.		3600	IN	SOA	ns.mozilla.org. sysadmins.mozilla.org. 2016031400 180 180 1209600 180
mozilla.ch.		180	IN	NS	ns5-65.akam.net.
mozilla.ch.		180	IN	NS	ns4-64.akam.net.
mozilla.ch.		180	IN	NS	ns7-66.akam.net.

Comment 58

[tad]

Easiest option is probably for us to create that email address.

Webops,

If you can create the following DNS records, I can create that email address.


Blank or @	MX	1	ASPMX.L.GOOGLE.COM
Blank or @	MX	5	ALT1.ASPMX.L.GOOGLE.COM
Blank or @	MX	5	ALT2.ASPMX.L.GOOGLE.COM
Blank or @	MX	10	ALT3.ASPMX.L.GOOGLE.COM
Blank or @	MX	10	ALT4.ASPMX.L.GOOGLE.COM
Blank or @      TXT             google-site-verification=-rq8BHzpkqv9xZb-Jydkqsi0AaZasOzxoGwmk0Htq_U

Comment 59

[tad]

(hold fire on that, there may be a change of plan)

Comment 60

[Yousef Alam [:yalam96] (use NEEDINFO)]

We're opened a request for a SAN cert for multiple sites so I'm going to close this bug. Hopefully in the near future all of the community sites can be secured with Let's Encrypt instead.

Comment 61

[Gene Wood [:gene]]

I've written up a summary page that gives the background to all of this : Certificate Authority Authorization CAA including Let's Encrypt