obtain a valid ssl cert for the internal pypi mirror

RESOLVED FIXED

Status

RESOLVED FIXED
a year ago
11 months ago

People

(Reporter: arich, Assigned: sidler)

Tracking

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/6421])

(Reporter)

Description

a year ago
Having an SSL cert that doesn't have a public root CA causes us a bunch of problems on the pypi mirror. We should just purchase a valid cert to install on the machine so that we don't have to try to hack the cert chain into every machine and application that uses the mirror.

Comment 1

11 months ago
We're hitting a bunch of issues with pip installs. My current thoughts are here [1].
This bug is a blocker to updating our old python 2.7.3 installs, which is probably one of the first steps we should take.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=885780#c23
I believe this bug was about the self-signed cert used for http://repos

http://pypi.{pub,pvt}.build.mozilla.org don't have SSL certs at all (they share an IP with secure.pub.b.m.o, so that's the cert you get if you try).

Comment 3

11 months ago
This bug blocks bug 1390946, which references pypi.pvt.build.mozilla.org, so I don't think it's about http://repos.

And that's correct that pypi.{pub,pvt}.b.m.o shares its ssl cert with secure.pub.b.m.o, and that cert doesn't cover the cnames we use. We need a valid cert that does.
Webops manages the certs for all of the things on the relengweb cluster, so I'd open a bug with them and ask them to update the cert to add them (presumably it's already a SAN cert so it's relatively easy to do).

Comment 5

11 months ago
Hi WebOps,

Is it possible to extend secure.pub.b.m.o's ssl cert to support the cnames pypi.pub.build.mozilla.org and pypi.pvt.build.mozilla.org ? Looks like it's currently a digicert cert.
Assignee: relops → server-ops-webops
Component: RelOps → WebOps: SSL and Domain Names

Updated

11 months ago
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/6421]

Updated

11 months ago
Assignee: server-ops-webops → sidler
(Assignee)

Comment 6

11 months ago
><(((º> autocert create san.build.mozilla.org -o c -b 1399926 --sans secure.pub.build.mozilla.org pypi.pub.build.mozilla.org pypi.pvt.build.mozilla.org -v2
certs:
- san.build.mozilla.org@e2d8d81e:
    authority:
      digicert:
        order_id: 2777140
    bug: '1399926'
    common_name: san.build.mozilla.org
    destinations: {}
    expiry: Wed, 10 Apr 2019 00:00:00 GMT
    modhash: e2d8d81ebd645cb58d8de5e41809635f
    sans:
    - secure.pub.build.mozilla.org
    - pypi.pub.build.mozilla.org
    - pypi.pvt.build.mozilla.org
    tardata:
      san.build.mozilla.org@e2d8d81e.tar.gz:
        san.build.mozilla.org@e2d8d81e.crt: CRT
        san.build.mozilla.org@e2d8d81e.csr: CSR
        san.build.mozilla.org@e2d8d81e.key: KEY
    timestamp: Mon, 02 Apr 2018 16:02:06 GMT
I believe some of the clients accessing this site do not support SAN.
(that's worth verifying, of course, but we use some ancient pythons on Windows)
(Assignee)

Comment 9

11 months ago
Aki, Dustin
I have made a SAN cert and was about to install it.  Please make sure this is what you want.  The bug says create a san cert, and I have.  Dustin is saying it may not work.  Please verify that and let me know ASAP.

Thanks,
Scott
Flags: needinfo?(dustin)
Flags: needinfo?(aki)
Flags: needinfo?(dustin)

Comment 10

11 months ago
What do we hit when we go to http://pypi.pub.build.mozilla.org/pub/ ?
Afaict it's a web front end to something. I'd like https://pypi.pub.build.mozilla.org/pub/ to go to the same root, but have the CNAME included in the ssl cert. If the above SAN cert does that, I'm not sure we need to support SAN; we'd just be using https.
Flags: needinfo?(aki)

Comment 11

11 months ago
Ah, Subject Alternative Name, not Storage Area Network.

Status quo is pointing at http://pypi.{pub,pvt}.build.mozilla.org with old python 2.7.3 or so.
The new hotness will be pointing at https://pypi.{pub,pvt}.build.mozilla.org with 2.7.12 (2.7.14 at some point) or 3.5.2 (3.6.5 soon). Since we can leave the old clients pointing at the http endpoint, I'm hoping we're good with the SAN cert.
Ah, good point, I forgot they were only using http.  Sorry for the false alarm!
(Assignee)

Comment 13

11 months ago
To be clear this will be a change to a san cert named san.build.mozilla.org which will have 3 SANs for secure.pub.build.mozilla.org, pypi.pub.build.mozilla.org, pypi.pvt.build.mozilla.org.  The old cert for secure.pub.build.mozilla.org will be replaced by this new cert and all clients for all three will have to be able handle that.  If I can get the sign off on this, I will make the change today.

Thanks,
Scott Idler
Flags: needinfo?(dustin)
Flags: needinfo?(aki)
Flags: needinfo?(dustin)

Comment 14

11 months ago
Are you able to roll back quickly? If so, let's do it today or tomorrow (please ping me beforehand on irc or slack).
I believe we'll be ok.
Flags: needinfo?(aki)

Comment 15

11 months ago
Post-cert install, I edited my /etc/hosts to point pypi.python.org to 127.0.0.1, then ran this in a py27 virtualenv (python 2.7.14, pip 10.0.0b2):

+ pip install --timeout 120 --no-index --find-links https://pypi.pvt.build.mozilla.org/pub --find-links https://pypi.pub.build.mozilla.org/pub jsonschema==2.5.1
Looking in links: https://pypi.pvt.build.mozilla.org/pub, https://pypi.pub.build.mozilla.org/pub
Collecting jsonschema==2.5.1
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x104b4add0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x104b4af90>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x104b75110>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x104b75250>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x104b75390>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Downloading https://pypi.pub.build.mozilla.org/pub/jsonschema-2.5.1-py2.py3-none-any.whl
Collecting functools32; python_version == "2.7" (from jsonschema==2.5.1)
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x105af0e90>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x105af0fd0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x105b0c590>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x105b0c4d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x105b0c1d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
  Downloading https://pypi.pub.build.mozilla.org/pub/functools32-3.2.3-2.tar.gz
Building wheels for collected packages: functools32
  Running setup.py bdist_wheel for functools32 ... done
  Stored in directory: /Users/asasaki/Library/Caches/pip/wheels/58/14/4c/707c05ed77f528c717db124c374fa788560a0cb60491a6d1f8
Successfully built functools32
Installing collected packages: functools32, jsonschema
Successfully installed functools32-3.2.3.post2 jsonschema-2.5.1
(Assignee)

Comment 16

11 months ago
ericz helped me with this.  We created a new apache virtualhost for handling https on port 81 for the pypi.pub but the pypi.pvt required this is well on port 83 per the releng MOTD.  We also uploaded the new san cert to both scl3-{int,ext} and had to create a new virtualserver and virtual ip group on scl3-int for the pypi.pvt.  We then associated the san.build.mozilla.org cert with all three urls.  We verified with aki that all is working.
Status: NEW → RESOLVED
Last Resolved: 11 months ago
Resolution: --- → FIXED
possible fallout in bug 1451432?

[task 2018-04-04T16:28:09.166Z] 16:28:09     INFO -    Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x7fc3b1b7e4d0>, 'Connection to pypi.pub.build.mozilla.org timed out. (connect timeout=120.0)')': /pub

Comment 18

11 months ago
Possible. I'd guess it's a hiccup around bouncing the web daemon rather, or we'd see widespread burning?
You need to log in before you can comment on or make changes to this bug.