952881 - Deploy bugzfeed (WebSocket server for the Bugzilla Change Notification System) to Heroku

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[Mark Côté [:mcote]]

Bugzfeed is the WebSockets application that serves data from the Bugzilla Change Notification System.

It is available from pypi under the package name "bugzfeed".  It is a Tornado application, installed as "bugzfeed-server".  Source is at https://github.com/mozilla/bugzfeed.  The default configuration should work, although you will probably want to specify the port via --port=$PORT or in a config file (see example in the source).

I have no idea how many users it will have and hence no idea about system requirements.  It is, however, designed to be scalable; since it creates an independent consumer on pulse, it can be run on multiple machines with a load balancer.

Comment 1

[Chris Turra [:cturra]]

Mark - before we can commit to setting this up anywhere, we're going to need bugzfeed to go through a security review. i am marking this bug for a review.

Comment 2

[Mark Goodwin [:mgoodwin]]

Clearing secreview flag as I've filed a separate secreview bug.

Comment 3

[Julien Vehent]

We'll have an opsec review. :cturra, do you know where this would live? afaik, we don't have websocket support behind the ZLBs at the moment.

Comment 4

[Chris Turra [:cturra]]

that's a great question - i don't know. this might be a good case to spend some time sorting out how we want to deal with websockets behind zeus tho.

Comment 5

[Brandon Burton [:solarce]]

(In reply to Chris Turra [:cturra] from comment #4)
> that's a great question - i don't know. this might be a good case to spend
> some time sorting out how we want to deal with websockets behind zeus tho.

We could do a direct NAT and SSL termination in Apache, like we do with people.mo

Comment 6

[Mark Côté [:mcote]]

FYI the security reviews were completed last week.  What kind of an ETA could I expect on this?  Note that it's related to an IT goal (Bugzilla performance) indirectly, in that, when it is widely adopted, it should cut down on the number of clients constantly polling BMO for updates.

Comment 7

[Mark Côté [:mcote]]

For now, at least, I have deployed bugzfeed on AWS at ws://bugzfeed.mozilla.org/.

Comment 8

[Mark Côté [:mcote]]

Going to reopen this, since we do want this deployed on Mozilla infrastructure at some point.

Comment 9

[Laura Thomson :laura]

Mark: can you please file a new bug for it when that future point arrives? Thanks!

Comment 10

[Mark Côté [:mcote]]

Anytime you're ready. :) Some groups want apps that rely on it, so it should have some sort of proper admin.

Comment 11

[:glob ✱]

ping?

Comment 12

[Julien Vehent]

I just had a chat with :glob about this. The current, temporary, bugzfeed setup is bad and we need a proper production deployment ASAP. see 1079596

Bugzfeed needs TLS and Websocket support. Which means a TCP load balancer that handles the TLS termination but does not touch the content (no HTTP).

I see 3 options, sorted by preference:
1. Use ZLB to terminate TLS and source-ip load balancing to VMs running bugzfeed
2. Use ELB in AWS to terminate TLS and source-ip load balancing to AWS instances running bugzfeed
3. NAT an IP to a single Apache server and terminate TLS there

Can someone in WebOps or Infra pick a preferred solution and move this bug forward? If solution 2 is chosen, let's run this in IT AWS, and I'll help with the ELB setup (we have scripts).

Comment 13

[Kendall Libby [:fubar] (he/him)]

BMO infra is handled by Dev Services (e.g. me). :glob appears to have handled the SSLv3 issue in 1079596, so how ASAP is ASAP? I haven't forgotten about this, but I can only juggle so many chainsaws at once. If WebOps has cycles, then feel free to use them.

Comment 14

[Mark Côté [:mcote]]

By the end of the quarter would be nice.  We want to integrate bugzfeed into BMO, but we want to wait until it's on our dev-service-maintained infrastructure to do that.

Comment 15

[:glob ✱]

due to ssl issues i have turned off the ssl endpoint for our "temporary" bugzfeed server (see bug 1079596).

we're now looking at almost a full year from the time when we were good to go (sec-review was completed on 2014-02-04).

Comment 16

[Mark Côté [:mcote]]

I'm going to look into moving this to Heroku soon.

Comment 17

[Mark Côté [:mcote]]

I have finally set up Bugzfeed on Heroku.  I'm just waiting on DNS changeover (bug 1222882), but it is currently accessible via ws://bugzfeed.herokuapp.com/ and (with SSL!) wss://bugzfeed.herokuapp.com/.  SSL via wss://bugzfeed.mozilla.org/ will be available after the DNS update.

You can see an example app, currently using wss://bugzfeed.herokuapp.com/, at https://people.mozilla.org/~mcote/bugzfeed_example/ (also available in the Bugzfeed source on GitHub).

Comment 18

[Kohei]

Will wss://bugzfeed-dev.allizom.org (for https://bugzilla-dev.allizom.org) be also available as before?

Comment 19

[Mark Côté [:mcote]]

Good point.  I'll incorporate something into the production bugzfeed app, since it doesn't change enough to warrant a dev instance.  So it'll probably be something like wss://bugzfeed.mozilla.org/dev/.

Comment 20

[Julien Vehent]

Nice! SSL conf looks good, but if you could just disable RC4-SHA, it's be just perfect.

$ ./cipherscan bugzfeed.herokuapp.com
........................
Target: bugzfeed.herokuapp.com:443

prio  ciphersuite                  protocols              pfs                 curves
1     ECDHE-RSA-AES128-GCM-SHA256  TLSv1.2                ECDH,P-256,256bits  prime256v1
2     ECDHE-RSA-AES128-SHA256      TLSv1.2                ECDH,P-256,256bits  prime256v1
3     ECDHE-RSA-AES128-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
4     DHE-RSA-AES128-SHA           TLSv1,TLSv1.1,TLSv1.2  DH,1024bits         None
5     ECDHE-RSA-AES256-GCM-SHA384  TLSv1.2                ECDH,P-256,256bits  prime256v1
6     ECDHE-RSA-AES256-SHA384      TLSv1.2                ECDH,P-256,256bits  prime256v1
7     ECDHE-RSA-AES256-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
8     AES128-GCM-SHA256            TLSv1.2                None                None
9     AES128-SHA256                TLSv1.2                None                None
10    AES128-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
11    AES256-GCM-SHA384            TLSv1.2                None                None
12    AES256-SHA256                TLSv1.2                None                None
13    AES256-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
14    ECDHE-RSA-RC4-SHA            TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
15    RC4-SHA                      TLSv1,TLSv1.1,TLSv1.2  None                None

Certificate: trusted, 2048 bits, sha256WithRSAEncryption signature
TLS ticket lifetime hint: 300
OCSP stapling: not supported
Cipher ordering: server
Curves ordering: server - fallback: no
Server supports secure renegotiation
Server supported compression methods: NONE
TLS Tolerance: yes

Comment 21

[Mark Côté [:mcote]]

Hm, I don't think I can change anything in Heroku's SSL config, unfortunately.

Comment 22

[Mark Côté [:mcote]]

I put in support for bugzilla-dev today.  You can now access wss://bugzfeed.herokuapp.com/dev/ for updates to bugs on bugzilla-dev.

This, and the general Heroku work, are still in (and deployed from) separate branches, but I'll merge it in as soon as we switch away from the old host.

Comment 23

[Mark Côté [:mcote]]

The example at https://people.mozilla.org/~mcote/bugzfeed_example/ has also been updated so you can (optionally) subscribe to bugzilla-dev.

Comment 24

[Kohei]

I have already re-enabled the Bugzfeed support in my BzDeck app! Will update the endpoint URL once the DNS is updated.

Comment 25

[Kohei]

The DNS record has been updated but I get the Connection Insecure message at https://bugzfeed.mozilla.org/ due to the cert mismatch. Also, Heroku says No such app.

Comment 26

[Mark Côté [:mcote]]

Yeah, something is not right here.  Not only is the cert not set up properly, but I can't even get the non-SSL version to work. ws://bugzfeed.herokuapp.com/ is fine, but ws://bugzfeed.mozilla.org/ isn't.  Hm.

Comment 27

[Kohei]

Now it works for me :)

Comment 28

[Mark Côté [:mcote]]

\o/

Comment 29

[Mark Côté [:mcote]]

ulfr: I'm assuming, from your lack of reply, that leaving RC4-SHA is not ideal but acceptable.

Comment 30

[Mark Côté [:mcote]]

Moving this conversation to the original security-related bug, bug 1079596.