821295 - Create Dev Instance for 'Witness'

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P3)

Description

[Gervase Markham [:gerv]]

Please create a stage instance for Witness.

== Date Needed By ==
* 2012-12-31 ideally

== Language ==
* Language: Python
* Framework: Django
* Base: Playdoh

== URL ==
* Production URL: https://witness.mozilla.org/
* Stage URL: https://witness.allizom.org/

== HTTP Auth ==

* HTTP Auth is not required to access this site.
 
== HTTP / HTTPS ==

* HTTPS is required this site.

== Code Repository ==
* Github Repo: https://github.com/khchen428/witness
** Setup Github to auto-update every 5 minutes.

* SVN Localization Repo: N/A (no l10n yet)
** Setup SVN to auto-update every 15 minutes.

== Database ==

* This site requires a MySQL database.

== Other Services ==

None.

== Settings ==

* cp settings_local.py-dist settings_local.py
* Update settings values accordingly.
* Set DEBUG to False in settings_local.py.

== Cron ==

* bin/update_site.py -e stage
** frequency?

== Tracebacks ==
* Please configure Django tracebacks to be sent to:
** [e-mail], [e-mail]

Comment 2

[Jake Maul [:jakem]]

Typically we'll have a -dev instance that auto-updates as described above, and stage/prod will be manually updated. The purpose of stage in this design is to provide a stable testing platform that 1) doesn't block dev, and 2) matches prod closely, so that *deployments* are effectively tested as well.

That update_site.py script is unfortunately not Commander-aware so we can't drop it straight into Chief for you to be able to do your own deploys, but it looks suitable for wrapping in a shell script for standard IT/WebOps deploys. If you're interested in self-managed deploys (which I'm guessing you are, since you wrote update_site.py), we can provide examples on how to do this.


This seems like a good candidate for the generic cluster in phx1. Until sec-review is completed, there will have to be an HTTP Basic Auth layer in front of it... but we can strip that out once they give an approval.


Thank you for the detailed request... it helps a lot. :)

Comment 3

[Gervase Markham [:gerv]]

All of that sounds fine to me, but I'm not on the Witness dev team :-)

Gerv

Comment 4

[joech]

The -dev instance and deployment shell script strategy sound reasonable to me.  I am not familiar with the phx1 cluster or WebOps procedures.  How does one deploy to the cluster and add the Auth Layer to the instance?

Thanks!
Joe

Comment 5

[Brandon Burton [:solarce]]

Since this will be the first instance of the application, it will be the -dev instance and accessible from witness-dev.allizom.org when created

At this point all the steps will be done by webops, we should be able to give an ETA after our Thursday meeting

Comment 6

[Gervase Markham [:gerv]]

:solarce: the privacy review has been completed; I think the next thing is a dev instance, and after that the security review (bug 821289). Are you able to say when we can get a deployment of this up and running?

Gerv

Comment 7

[Brandon Burton [:solarce]]

Since this bug was oepned we've launched https://mana.mozilla.org/wiki/display/websites/paas.allizom.org


I would suggest you set it up on paas.allizom.org to provide something for the security review for now  

I've added it to this week's webops meeting to get an ETA on setup on a cluster

Comment 8

[Gervase Markham [:gerv]]

:solarce: that sounds like a fine idea. The only problem is that access to PAAS requires an LDAP login, and neither of the Witness developers (who are best placed to set it up) have one. What's the right way to solve that?

Gerv

Comment 9

[Brandon Burton [:solarce]]

(In reply to Gervase Markham [:gerv] from comment #8)
> :solarce: that sounds like a fine idea. The only problem is that access to
> PAAS requires an LDAP login, and neither of the Witness developers (who are
> best placed to set it up) have one. What's the right way to solve that?
> 
> Gerv

Ah, then at this time they wouldn't be able to use paas.allizom.org

Comment 10

[Gervase Markham [:gerv]]

solarce: OK; so can you discuss in your meeting what the best way forward is to get a dev instance up?

Thanks,

Gerv

Comment 11

[Brandon Burton [:solarce]]

(In reply to Gervase Markham [:gerv] from comment #10)
> solarce: OK; so can you discuss in your meeting what the best way forward is
> to get a dev instance up?
> 
> Thanks,
> 
> Gerv

A question about the traceback part of things, would there be any possible PII included in data this app collects, that could be sent as part of a traceback?

Comment 12

[Gervase Markham [:gerv]]

Yes, this app collects names and addresses in some circumstances. As to whether it could make it into a traceback, I defer to those with more knowledge of Django.

Gerv

Comment 13

[Brandon Burton [:solarce]]

(In reply to Gervase Markham [:gerv] from comment #12)
> Yes, this app collects names and addresses in some circumstances. As to
> whether it could make it into a traceback, I defer to those with more
> knowledge of Django.
> 
> Gerv

If it's collecting PII and there is any chance that data could be included in an exception then we can't do tracebacks by email to non @mozilla.com email addresses

What'd be ideal is if we can skip email tracebacks all together and just use https://mana.mozilla.org/wiki/display/websites/errormill.mozilla.org via http://raven.readthedocs.org/en/latest/config/django.html

Comment 14

[joech]

The app collects the user's email, IP, name, and physical address (optional) for all the signatures.  I don't have access to the https://mana.mozilla.org/wiki/display/websites/errormill.mozilla.org link as it requires LDAP.  How does one use the errormill? I can't seem to find information about it online.

Comment 15

[Brandon Burton [:solarce]]

(In reply to joech from comment #14)
> The app collects the user's email, IP, name, and physical address (optional)
> for all the signatures.  I don't have access to the
> https://mana.mozilla.org/wiki/display/websites/errormill.mozilla.org link as
> it requires LDAP.  How does one use the errormill? I can't seem to find
> information about it online.

It's in instance of https://sentry.readthedocs.org/en/latest/, so you use the Raven client, see http://raven.readthedocs.org/en/latest/config/django.html

What's the overall plan/expectation of this app? Is it something that's meant to be put into widespread use for moco/mofo/the community?

Comment 16

[joech]

OK I will look into how to set up Raven for this client.

I'm not familiar with the acronyms moco&mofo.  The users of this app will be Mozilla contributors but I'd expect the service would be hosted by Mozilla Foundation itself to track the signatures.

Comment 17

[Brandon Burton [:solarce]]

(In reply to joech from comment #16)
> OK I will look into how to set up Raven for this client.
> 
> I'm not familiar with the acronyms moco&mofo.  The users of this app will be
> Mozilla contributors but I'd expect the service would be hosted by Mozilla
> Foundation itself to track the signatures.

Sorry, MoCo == Mozilla Corporation, MoFo == Mozilla Foundation

Were you hoping to have this live under a mozilla.org domain, e.g. witness.mozilla.org?

I want to understand the scope of things so I can determine the best option for where to host this.

Comment 18

[joech]

witness.mozilla.org sounds good to me.   Gerv - what do you think?

Comment 19

[Gervase Markham [:gerv]]

Yes, witness.mozilla.org seems like the right place. The idea is that many community members will be using it to sign stuff, and a few to add things to be signed.

Gerv

Comment 20

[joech]

Hi Brandon,

Just wanted to follow-up on this ticket.  Witness.mozilla.org makes sense to us.  Is there anything I can do to help?

Comment 21

[Gervase Markham [:gerv]]

solarce: ping? Is there anything else you need from us?

Gerv

Comment 22

[Gervase Markham [:gerv]]

solarce: it's been 3 months now. Is there something blocking that we can help with?

Gerv

Comment 23

[Brandon Burton [:solarce]]

(In reply to Gervase Markham [:gerv] from comment #22)
> solarce: it's been 3 months now. Is there something blocking that we can
> help with?
> 
> Gerv

Please discuss having this worked on with :jakem

Comment 24

[Eric Zounes [:zounese]]

I'll start work on the dev environment tomorrow morning. Once I get it rolling I'll report back with updates.

Comment 25

[Chris Turra [:cturra]]

:gerv - Eric has made progress getting witness setup in our generic-dev environment, but has run into a road block. the application does not include a .wsgi file, which is required for our environment (apache+mod_wsgi). 

to complete this deployment, we will need you to add wsgi support to the application.

Comment 26

[Gervase Markham [:gerv]]

Joe, Zach: sorry this is taking so long. Are you able to add WSGI support to Witness?

Gerv

Comment 27

[Jake Maul [:jakem]]

Another note on this...

One of the things I see that will be a hold-up is that this is not a normal Playdoh app. There is no "vendor" directory, and it's expecting to "pip install" many things from a requirements.txt file. This may be why there isn't already a wsgi file.

This isn't a big deal in Stackato, because we can rely on the requirements files. However, on our bare-metal clusters (like generic, where we've started to deploy this), we're not set up to do this... we rely on /vendor/ to have all of the native Python dependencies, and then system libraries for compiled dependencies.

In other words, we're geared to do a particular type of app fast (relatively speaking), and this one doesn't quite fit that mold. :)

We can solve this in a couple ways, and we're actively pursuing some of these:


1) Get community LDAP accounts for the 2 developers, so they can use Stackato, and thus not have to change the app too much. This is still blocked on production-ready Stackato, which is blocked on sec-review, but it more or less immediately unblocks dev work.

2) Attempt to deploy from a virtualenv. We don't have any processes or best practices around this currently, so it will take longer. It's been on our wishlist for a while to figure out a good way to do this. We haven't allocated time to it due to the long-impending Stackato system.

3) Spin up dedicated nodes (VMs) for this app. I'm not a fan of this, because we already have far too many "one-off" things in production, and factory-izing IT is a big priority these days.


We're currently focusing on #1, with a fallback plan of #2.

If there's any chance of converting this app to the standard Playdoh/funfactory style (ex: https://github.com/mozilla/dragnet), that would help us a lot. However I understand this could be a rather large change, so I'm merely bringing it up as an option. If it's infeasible, I understand. :)

Comment 28

[Gervase Markham [:gerv]]

jakem: is the "standard style" documented? I mean, we asked Joe and Zach (and their predecessor) to build on playdoh as we thought that would make it easy for IT to deploy. If there's more to that than "build on playdoh", then it would be good to have docs of what that "more" is...

Gerv

Comment 29

[Jake Maul [:jakem]]

There are docs, maintained by webdev. They appear to be pretty complete, too. Yay! :)

The repo is here: https://github.com/mozilla/playdoh-lib/
The readthedocs page is here: http://playdoh.readthedocs.org/en/latest/

There's a good bit of info in there. Especially relevant in this case would be:

http://playdoh.readthedocs.org/en/latest/getting-started/installation.html
http://playdoh.readthedocs.org/en/latest/packages.html
http://playdoh.readthedocs.org/en/latest/operations.html

Any questions on them should most likely be directed to webdev (most of whom live in #webdev), as they're much more familiar with *developing* playdoh apps, whereas we simply *deploy* them.

Comment 30

[joech]

Thanks Jake!  I'll take a look at the wsgi docs and create the files.  As Gerv mentioned, we're not too familiar with deploying at Mozilla, so chances are we'll have questions for #webdev or for you.

Thanks,
Joe

Comment 31

[Brandon Burton [:solarce]]

Hello,

Is IT managed hosting for this still required?

If so, then having it be hosted on Stackato is the way to proceed, to start that, we'll need whomever is working on the application and will want to push updates to the site to obtain community LDAP accounts

To obtain community LDAP accounts follow the steps on http://www.mozilla.org/hacking/committer/ but in #2's bug, note that you need community LDAP for accessing resources, not for committing to repositories, but the rest of the process should be the same

Once your LDAP accounts are working, update this bug and we can walk through how to use Stackato

Comment 32

[joech]

Thanks :solarce for the link.  I'll start the process for an LDAP account and let you know once it's working.

Comment 33

[Gervase Markham [:gerv]]

I'm afraid time and inertia have left the Witness project behind :-( A big thankyou to Joe and everyone else who worked on it, but I just can't see us wading through the process to get this up and running. :-(

Gerv