1004624 - [tracking] d2g dev environment

Status: RESOLVED FIXED

(Cloud Services :: Operations: Marketplace, task)

Description

[Austin King [:ozten]]

We should have a kick-off meeting and discussion around d2g bug#1004611.

To start, we'll need a development environment (NodeJS, MySQL, usual suspects).

Comment 1

[Chris More [:cmore]]

Moving this into the right component for ozten.

Comment 2

[Austin King [:ozten]]

I want to schedule a kick off meeting to discuss.
Who is on point for Ops (or who should I invite)?

Feel free to respond in IRC or email outside of this bug.

Comment 3

[Jake Maul [:jakem]]

I'd go for srich, and all of myself, cturra, jd, gozer, or cyliang. Not sure who'll be point on helping it get off the ground, but likely one of us... we'll figure it out and all but 1-2 will decline. Does that wfy?

Comment 4

[Austin King [:ozten]]

Awesome, thanks!

Also, this is very similar to the APK Factory that oremj and jason deploy.

I'd like to keep the deployment as similar as possible to reduce work and increase consistency.

Comment 5

[Jeremy Orem [:oremj]]

It sounds like this is very similar to APK factory, which cloudops already hosts. We can probably take this as well.

:jakem, does that work for you? I don't want to steal the project if it's something you are already planning on.

Comment 6

[Jake Maul [:jakem]]

I have no particular attachment to it... today is the first day I'd heard about it.

My immediate thought was: this sounds like a reasonable candidate for the PaaS... that supports MySQL and Node.js out of the box. If that does indeed work for everyone, there may not be much work for either of our teams to do on it.

But in any case, I don't mind if you want to help Austin (et al) get started with either that or an environment similar to APK factory. :)

Comment 7

[Jake Maul [:jakem]]

Tentatively moving this to Cloud Services....

Comment 8

[Austin King [:ozten]]

Attachment: deployment_architecture.png

Adding architecture diagram.

For 1.0, we want a stand-alone "enterprise" self-hosted architecture.
This uses MySQL and local disk which are partners are more comfortable with than 3rd party hosting.

We could follow up with a "cloud" flavor, which if given AWS credentials, uses S3 and DynamoDB.

Comment 9

[Austin King [:ozten]]

The mozilla deployed version will be the Cloud stack.

We can't do local disk, because it's not fault tolerant on AWS. This means MySQL + local disk is out.

So we'll target S3 + DynamoDB.

I'll host a MySQL + local disk at http://d2gk.co, so we can test the "enterprise" use case separately.

Comment 10

[Austin King [:ozten]]

What are betafox dependencies?

https://github.com/mozilla/betafox/blob/master/docs/AWS.md documents packages required by Ubuntu 14.10.

Comment 11

[Austin King [:ozten]]

Checking in on dev progress.

Codebase: https://github.com/mozilla/betafox

DynamoDB / S3 Config:
awsAccessKeyId = 'SomeValue';
awsSecretAccessKey = 'SomeSecretValue';
dynamoReadCapacityUnits = 1;
dynamoWriteCapacityUnits = 1;
awsS3PublicBucket = 'betafox-assets';

(This is present in server/config/default.js, but commented out)

Comment 12

[Jeremy Orem [:oremj]]

Can we create a "dynamodb-table-prefix" option, defaulting to "betafox.", so prefix="dev.betafox." would result in table names like "dev.betafox.versions", etc?

Comment 13

[Austin King [:ozten]]

Fixed in 80dce39b1c05a05a32aac321300ea4748e5c4abd

If awsAccessKeyId is set, then dynamodbTablePrefix will prefix the tables created.

Example in commented out section of server/config/default.js

Thanks!

Comment 14

[Jeremy Orem [:oremj]]

Getting this now:
connect.limit() will be removed in connect 3.0
doing health check
WARNING: No d2g certificates found, creating ephemeral ones
WARNING: Generating DER file at /opt/betafox/betafox/server/config/secrets/betafox.der
WARNING: Generating Cert DB at /opt/betafox/betafox/server/config/secrets
WARNING: If you care about app compatibility, backup and secure these!
Creating Keypair  /opt/betafox/betafox/server/bin /opt/betafox/betafox/server/config/secrets /opt/betafox/betafox/server/config/secrets/betafox.der
/opt/betafox/betafox/server/bin/generate_cert.sh /opt/betafox/betafox/server/config/secrets /opt/betafox/betafox/server/config/secrets/betafox.der
STDOUT configCertsDir is /opt/betafox/betafox/server/config/secrets

STDOUT \n*** generate_phone_cert_db.sh /opt/betafox/betafox/server/config/secrets/betafox /opt/betafox/betafox/server/config/public
\n*** wiping temporary cert DB
\n*** create new temporary cert DB
/opt/betafox/betafox
\n*** add d2g cert to temporary cert DB
Neither cert8.db nor cert9.db were found in /opt/betafox/betafox/server/config/public/certdb.tmp

STDERR mkdir: cannot create directory `/opt/betafox/betafox/server/config/public': Permission denied
mkdir: cannot create directory `/opt/betafox/betafox/server/config/public': Permission denied
/opt/betafox/betafox/server/bin/generate_phone_cert_db.sh: line 34: [: -ne: unary operator expected

{ [Error: Command failed: mkdir: cannot create directory `/opt/betafox/betafox/server/config/public': Permission denied
mkdir: cannot create directory `/opt/betafox/betafox/server/config/public': Permission denied
/opt/betafox/betafox/server/bin/generate_phone_cert_db.sh: line 34: [: -ne: unary operator expected
] killed: false, code: 1, signal: null }
DONE doing health check
BetaFox... Listening on https://betafox.dev.mozaws.net

Comment 15

[Austin King [:ozten]]

Dev has Python 2.6.6. We actually need Python 2.7.x. I've tested with Python 2.7.6.

Updated README.md.

Comment 16

[Austin King [:ozten]]

I've found and fixed 3 issues that affected production mode and S3.

Please re-deploy https://betafox.dev.mozaws.net/

Comment 17

[Jeremy Orem [:oremj]]

deployed

Comment 18

[Austin King [:ozten]]

Can we set dev for auto-deploy off the master branch?

Comment 19

[Jeremy Orem [:oremj]]

Auto deploy is on.