504377 - Stage Mozilla CRM (mozillians.org) for QA and security auditing

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Jay Patel [:jay]]

Filing a bug to track the migration of the CRM from Trellon servers to Mozilla servers.  

There are specific requirements and configuration needs that we'll have to work through with them, so wanted to track all of that information here until we have a live site staged.

After discussing this with webdev and IT, it was recommended we have the site on our servers before we move forward with QA and security auditing... so hopefully we can have something up once the code is complete.

I have reached out to Trellon for a development update and will let folks know when the code is in SVN and I have the DB dumps attached to this bug.

Thanks.

Comment 1

[Jay Patel [:jay]]

Here is what Trellon sent in terms of the server config:

Web server
Recommended: Apache 2.x

Drupal will work on Apache 1.3 or Apache 2.x hosted on UNIX/Linux, OS X, or Windows. In the case of this project, the entire stack will run on Linux.

Use the Apache 'mod_rewrite' extension to allow for clean URLs.

The total file size of the Drupal installation will depend on site configuration, but Drupal core files alone will take up approximately 2 to 3 MB uncompressed. The exact size depends on the version of Drupal installed.


PHP

Recommended: PHP 5.2 or higher

PHP memory of 64 MB or higher

The PHP extension for MySQL must be installed and enabled. Drupal's currently supported database connectors are: mysql (the original MySQL extension), mysqli (an improved connector for newer MySQL installations), and pgsql (for PostgreSQL).

PHP XML extension (for blogapi, drupal, and ping modules). This extension is enabled by default in a standard PHP installation; the Windows version of PHP has built-in support for this extension.

An image library for PHP such as the GD library is needed for image manipulation (resizing user pictures, image and imagecache modules). GD is included with PHP 4.3 and higher and enabled by default. ImageMagick is also supported for basic image manipulations in Drupal core but there is much less support from contribute modules.

PHP requires the following configuration directives for Drupal to work (only directives that differ from the default php.ini-dist / php.ini-recommended):

–      register_globals: off; this is the default value, but some hosts have it enabled

–      session.save_handler: user

–      error_reporting set to E_ALL & ~E_NOTICE. Work is ongoing to change this to E_ALL for Drupal 6.

–      safe_mode: off. Safe mode may interfere with file and image uploads.

–      In addition, we recommend the following setting: session.cache_limiter: nocache

Note that setting PHP configuration options from .htaccess only works under the following conditions:

–      With Apache (or a compatible web server)

–      If the .htaccess file is actually read, i.e. AllowOverride is not None

–      If PHP is installed as an Apache module

Database server

Recommended: MySQL 5.x

These rights are sufficient to run Drupal core 6.x:

–      SELECT

–      INSERT

–      UPDATE

–      DELETE

–      CREATE

–      DROP

–      INDEX

–      ALTER

 Some contributed modules additionally require the following rights:

–      CREATE TEMPORARY TABLES

Our system engineer, Kevin Hemenway also would recommend adding the php-mbstring extension to the server. Mbstring is something Drupal will complain about if it's missing (it has fallback, but really wants mbstring).

Comment 2

[Jay Patel [:jay]]

Trellon's answers to Morgamic's security-related questions:

The other security items that were raised are answered below by Kevin Hemenway, our systems architect;

    * Do logins and sessions support https

We don't have a definitive answer to this and are doing a bit more research. They've worked fine with some of our other clients and we've done numerous other things with SSL and Drupal. However, our systems engineer is not 100%, and thus, can't say "Definitively yes", from a security standpoint.

    * Are there uploads?  Do upload directories whitelist filetypes?

I don't know if the Mozilla site has any upload functionality. However, Drupal does protect its 'files' directories by creating an .htaccess file in it (this presumes AllowOverride All is working, which they know about already in their server configuration e-mail). This .htaccess file specifically sets a non-existent handler for all files, which effectively stops any interpretation of uploaded files:

 SetHandler Drupal_Security_Do_Not_Remove_
See_SA_2006_006
 Options None
 Options +FollowSymLinks

Drupal will also apply additional .txt extensions to known executable extensions. However, depending on your Apache configuration, a file named whee.php.txt would still be executable - that's where the SetHandler above comes in.

 http://api.drupal.org/api/function/file_save_upload/6

Note, however, that bad code (or external code that is not Drupal specific, like CiviCRM) could circumvent these measures by either a) not using Drupal's existing 'files' directory to save files, or b) not using existing Drupal functionality like file_save_upload().

    There was also a question about where the private/sensitive data will live.  Will everything be in one DB?  Is it possible to separate out the private information to a DB that is protected from hacking from the public Drupal interface?  Will it be possible to put the CiviCRM urls?

I'd have to know what they consider public and private information. However, all manner of multiple-database configuration I've seen inside Drupal requires write-access to the databases. Segmenting out the different tables to different databases wouldn't improve security.

    (/crm/*) behind a VPN/firewall so that we have additional security for accessing that part of the site (aside from the standard Admin privilege/permissions)?

This would be possible using Apache's <Location> handler,
I believe. Something like (pseudo-code untested, etc.):

 <Location /civicrm>
  Order deny,allow
  Deny from all
  Allow from vpn.ip.addr.ess
 </Location>

However, this would only affect that path, of course - if CiviCRM profiles are hooked into Drupal user profiles, then users would be able to write to the CiviCRM database from their own /user/1234 page (which is entirely Normal, but just throwing it out there).

Comment 3

[Jay Patel [:jay]]

Initial QA is happening on the Trellon servers and we have a few theme issues to work out.  We're hoping to have the code ready in a couple of weeks.  

I'm shooting to do the migration the week of August 10.  I will be out that week on PTO, but hopefully IT can get things up with the help from Trellon and we can sync up when I'm back on August 17.

Comment 4

[Jeremy Orem [:oremj]]

Status update?

Comment 5

[Jay Patel [:jay]]

oremj: We're at least a week away from code complete (theme issues still being resolved)... but the Trellon dev team wants to meet with you (and any other IT folks that will be getting this setup on our end) to go over the requirements and performance tool options.

Please let me know what time you are available to meet with them next week.  Early morning to afternoon is ideal since they're on the East coast.

Comment 6

[Jeremy Orem [:oremj]]

I am available all day Friday. It'd be best if they could write up the requirements ahead of time.

Comment 7

[Jay Patel [:jay]]

Jeremy: The basic requirements are in comment #1, but please get in touch with Kevin Hemenway, [e-mail] to discuss any specific tweaks or configuration details.  I am told that Kevin isn't a big phone guy, but should be happy to go over anything you need on IRC, IM, or email.

Can you please touch base and get any questions answered that you need to get the production environment setup.  Thanks!

Comment 8

[Jay Patel [:jay]]

I just got out of a meeting with Trellon and it looks like we'll have a snapshot ready on SVN by tonight and I will get an email with a link to the DB that we'll need to get the site staged.

Oremj: I was told you had been in touch with Kevin and that you might have enough info to get things setup once we have code ready for checkout on SVN and the DB file(s).  Will you be ready to stage the site tomorrow?

Comment 9

[Jeremy Orem [:oremj]]

I e-mailed Kevin this morning.

Comment 10

[Jay Patel [:jay]]

The code is ready and Marc has provided the last few details we need to get this staged on our end.  Here are the notes from Marc:

All the contribution changes are checked into svn. I have added 2 database dumps - one for civicrm and one for drupal.

I have also included copies of settings.php and civicrm.settings.php. It is likely that the database username, password and database name will need to be changed. These both need to be dropped into <site_path>/sites/default.
This affects lines 93 and 94 in settings.php and lines 33 and 54 in civicrm.settings.php. In addition the following paths will have to be updated in civicrm.settings.php

    * line 98 - this is the path to where your civicrm module exists - this will be <site_path>/sites/all/modules/civicrm
    * line 99 - this is the path to where templates are complied - this will be <site_path>/sites/default/files/civicrm/templates_c/ (depending on permissions on the folder you may have to create subdirectories)
    * line 134 - this is the base URL and is the base path like it will be http://mozillians.org.

civicrm.settings.php - civicrm config file (6.4 KB) Trellon - Marc Ingram, 09/17/2009 12:19 PM

settings.php - drupal config file (16.6 KB) Trellon - Marc Ingram, 09/17/2009 12:19 PM

mozcrm_civicrm.stage.all.sql.gz - civicrm database dump (144.1 KB) Trellon - Marc Ingram, 09/17/2009 12:19 PM

mozcrm_drupal.stage.all.sql.gz - drupal database dump (1.9 MB) Trellon - Marc Ingram, 09/17/2009 12:19 PM

------

I will attach all of those files to this bug so you can grab them from here.  And the code should be ready to checkout from SVN at /projects/crm/trunk (http://viewvc.svn.mozilla.org/vc/projects/crm/trunk/).

Comment 11

[Jay Patel [:jay]]

Attachment: Drupal DB file

Comment 12

[Jay Patel [:jay]]

Attachment: CiviCRM DB file

Comment 13

[Jay Patel [:jay]]

Attachment: CiviCRM settings file

Comment 14

[Jay Patel [:jay]]

Attachment: Drupal settings file

Comment 15

[Jay Patel [:jay]]

Oremj:  Hopefully you have everything you need to get started.  Please let me know when you have an ETA for mozillians.org being staged and ready for testing.

If you have any questions or need help, please contact the Trellon folks (I will send you a separate email with the contacts).

FYI: I'm on PTO Fri 9/18 - Mon 9/21, but I hope things will be ready when I get back on Tue 9/22.  This is fairly urgent for the Mozilla Foundation folks so they can get the donations up for One Web Day, so please help us get things moving.

Thanks!

Comment 16

[matthew zeier [:mrz]]

jay, where are we with this?

Comment 17

[Jay Patel [:jay]]

mrz: Waiting for ETA from Trellon for code handoff #2.  We ended up doing QA on their servers and they have been fixing bugs the past few weeks.  They're almost done.  Should have an updated by Monday 11/2. 

It will be great to have IT on standby to do this next week.  Was going to chat with you tomorrow about that.

Comment 18

[Jay Patel [:jay]]

Nevermind... Rebecca from Trellon just emailed me and said they were done.

I will be uploading the DB files shortly and the code should be in SVN.

mrz: Can we get this staged by early next week at stage.mozillians.org (or wherever this should live for further testing and the security audit)?

Comment 19

[Jay Patel [:jay]]

Attachment: CiviCRM DB file

DB file for CiviCRM

Comment 20

[Jay Patel [:jay]]

Can't attache Drupal DB file.. I will email it to ya.
- Jay

Comment 21

[David Boswell]

I noticed the whiteboard says 'Waiting on code/SQL'.  Is there any other code we're waiting for Trellon to deliver?

Comment 22

[Jay Patel [:jay]]

Code should be in SVN at: projects/crm/trunk/

DB files were emailed to mrz and the config files are attached here.

As far as I know, we have handed off everything IT needs to get the CRM staged.  

mrz/oremj: Let us know if you need anything else.  Updated the whiteboard.

Comment 23

[Jay Patel [:jay]]

Comment on attachment 401278 [details]
Drupal DB file

This was the old file.  mrz should have a new version of the DB dump.

Comment 24

[Jeremy Orem [:oremj]]

Done: https://mozillians.stage.mozilla.com/

Admin username and password is admin:test that should probably be changed.

Comment 25

[Jay Patel [:jay]]

Thanks oremj! 

On first launch, I see this at the top of the page:

The selected file /tmp/file9zggb2 could not be uploaded, because the destination js/gmap_markers.js is not properly configured.

Any idea what this is about?  

I am going to get QA testing this ASAP and we'll start tracking new bugs in Bugzilla.

Comment 26

[Jay Patel [:jay]]

Oremj:  As we discussed, a list of bugs are compiled here: https://wiki.mozilla.org/Mozilla_CRM/QA#Initial_QA_on_http:.2F.2Fmozillians.stage.mozilla.com_.2811.2F16_-_11.2F18.5D

I'm guessing most of them are config related... so hopefully you can work through them to get images back and make pages load as expected.  If you have any questions, please add them to this bug and hopefully Marc from Trellon can help.

Thanks!

Comment 27

[Jay Patel [:jay]]

Jeremy:  Looks like we still have a couple of big issues:

1. the db login credentials issue (causing errors in a few places)
2. what we plan to do to support SOLR search (it will be great if we can support it, because alternatives seem to be difficult to implement now).  

Can we try to have these bugs fixed by next week?  I want to demo the site during the All-Hands and have people start populating the system with their profile info... 

It'll be great to be able to have everything working well enough to clean out some old data and start clean with new profiles when folks are in town.

Comment 28

[Jeremy Orem [:oremj]]

I've set this up for trellon @ 63.245.208.165. Accessible by going to https://mozillians.stage.mozilla.com/