1399926 - obtain a valid ssl cert for the internal pypi mirror

Status: RESOLVED FIXED

(Infrastructure & Operations :: SSL Certificates, task)

Description

[Amy Rich [:arr] [:arich]]

Having an SSL cert that doesn't have a public root CA causes us a bunch of problems on the pypi mirror. We should just purchase a valid cert to install on the machine so that we don't have to try to hack the cert chain into every machine and application that uses the mirror.

Comment 1

[Aki Sasaki (not active)]

We're hitting a bunch of issues with pip installs. My current thoughts are here [1].
This bug is a blocker to updating our old python 2.7.3 installs, which is probably one of the first steps we should take.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=885780#c23

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I believe this bug was about the self-signed cert used for http://repos

http://pypi.{pub,pvt}.build.mozilla.org don't have SSL certs at all (they share an IP with secure.pub.b.m.o, so that's the cert you get if you try).

Comment 3

[Aki Sasaki (not active)]

This bug blocks bug 1390946, which references pypi.pvt.build.mozilla.org, so I don't think it's about http://repos.

And that's correct that pypi.{pub,pvt}.b.m.o shares its ssl cert with secure.pub.b.m.o, and that cert doesn't cover the cnames we use. We need a valid cert that does.

Comment 4

[Kendall Libby [:fubar] (he/him)]

Webops manages the certs for all of the things on the relengweb cluster, so I'd open a bug with them and ask them to update the cert to add them (presumably it's already a SAN cert so it's relatively easy to do).

Comment 5

[Aki Sasaki (not active)]

Hi WebOps,

Is it possible to extend secure.pub.b.m.o's ssl cert to support the cnames pypi.pub.build.mozilla.org and pypi.pvt.build.mozilla.org ? Looks like it's currently a digicert cert.

Comment 6

[Scott Idler [:sidler] [:escote]]

><(((º> autocert create san.build.mozilla.org -o c -b 1399926 --sans secure.pub.build.mozilla.org pypi.pub.build.mozilla.org pypi.pvt.build.mozilla.org -v2
certs:
- san.build.mozilla.org@e2d8d81e:
    authority:
      digicert:
        order_id: 2777140
    bug: '1399926'
    common_name: san.build.mozilla.org
    destinations: {}
    expiry: Wed, 10 Apr 2019 00:00:00 GMT
    modhash: e2d8d81ebd645cb58d8de5e41809635f
    sans:
    - secure.pub.build.mozilla.org
    - pypi.pub.build.mozilla.org
    - pypi.pvt.build.mozilla.org
    tardata:
      [e-mail]:
        [e-mail]: CRT
        [e-mail]: CSR
        [e-mail]: KEY
    timestamp: Mon, 02 Apr 2018 16:02:06 GMT

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I believe some of the clients accessing this site do not support SAN.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

(that's worth verifying, of course, but we use some ancient pythons on Windows)

Comment 9

[Scott Idler [:sidler] [:escote]]

Aki, Dustin
I have made a SAN cert and was about to install it.  Please make sure this is what you want.  The bug says create a san cert, and I have.  Dustin is saying it may not work.  Please verify that and let me know ASAP.

Thanks,
Scott

Comment 10

[Aki Sasaki (not active)]

What do we hit when we go to http://pypi.pub.build.mozilla.org/pub/ ?
Afaict it's a web front end to something. I'd like https://pypi.pub.build.mozilla.org/pub/ to go to the same root, but have the CNAME included in the ssl cert. If the above SAN cert does that, I'm not sure we need to support SAN; we'd just be using https.

Comment 11

[Aki Sasaki (not active)]

Ah, Subject Alternative Name, not Storage Area Network.

Status quo is pointing at http://pypi.{pub,pvt}.build.mozilla.org with old python 2.7.3 or so.
The new hotness will be pointing at https://pypi.{pub,pvt}.build.mozilla.org with 2.7.12 (2.7.14 at some point) or 3.5.2 (3.6.5 soon). Since we can leave the old clients pointing at the http endpoint, I'm hoping we're good with the SAN cert.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, good point, I forgot they were only using http.  Sorry for the false alarm!

Comment 13

[Scott Idler [:sidler] [:escote]]

To be clear this will be a change to a san cert named san.build.mozilla.org which will have 3 SANs for secure.pub.build.mozilla.org, pypi.pub.build.mozilla.org, pypi.pvt.build.mozilla.org.  The old cert for secure.pub.build.mozilla.org will be replaced by this new cert and all clients for all three will have to be able handle that.  If I can get the sign off on this, I will make the change today.

Thanks,
Scott Idler

Comment 14

[Aki Sasaki (not active)]

Are you able to roll back quickly? If so, let's do it today or tomorrow (please ping me beforehand on irc or slack).
I believe we'll be ok.

Comment 15

[Aki Sasaki (not active)]

Post-cert install, I edited my /etc/hosts to point pypi.python.org to 127.0.0.1, then ran this in a py27 virtualenv (python 2.7.14, pip 10.0.0b2):

+ pip install --timeout 120 --no-index --find-links https://pypi.pvt.build.mozilla.org/pub --find-links https://pypi.pub.build.mozilla.org/pub jsonschema==2.5.1
Looking in links: https://pypi.pvt.build.mozilla.org/pub, https://pypi.pub.build.mozilla.org/pub
Collecting jsonschema==2.5.1
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x104b4add0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x104b4af90>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x104b75110>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x104b75250>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x104b75390>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Downloading https://pypi.pub.build.mozilla.org/pub/jsonschema-2.5.1-py2.py3-none-any.whl
Collecting functools32; python_version == "2.7" (from jsonschema==2.5.1)
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x105af0e90>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x105af0fd0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x105b0c590>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x105b0c4d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x105b0c1d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Downloading https://pypi.pub.build.mozilla.org/pub/functools32-3.2.3-2.tar.gz
Building wheels for collected packages: functools32
  Running setup.py bdist_wheel for functools32 ... done
  Stored in directory: /Users/asasaki/Library/Caches/pip/wheels/58/14/4c/707c05ed77f528c717db124c374fa788560a0cb60491a6d1f8
Successfully built functools32
Installing collected packages: functools32, jsonschema
Successfully installed functools32-3.2.3.post2 jsonschema-2.5.1

Comment 16

[Scott Idler [:sidler] [:escote]]

ericz helped me with this.  We created a new apache virtualhost for handling https on port 81 for the pypi.pub but the pypi.pvt required this is well on port 83 per the releng MOTD.  We also uploaded the new san cert to both scl3-{int,ext} and had to create a new virtualserver and virtual ip group on scl3-int for the pypi.pvt.  We then associated the san.build.mozilla.org cert with all three urls.  We verified with aki that all is working.

Comment 17

[Chris AtLee [:catlee]]

possible fallout in bug 1451432?

[task 2018-04-04T16:28:09.166Z] 16:28:09     INFO -    Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x7fc3b1b7e4d0>, 'Connection to pypi.pub.build.mozilla.org timed out. (connect timeout=120.0)')': /pub

Comment 18

[Aki Sasaki (not active)]

Possible. I'd guess it's a hiccup around bouncing the web daemon rather, or we'd see widespread burning?