1007762 - New openstack dbs for Releng

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Jake Watkins [:dividehex]]

Hi, We are going to be building out an openstack environment for releng and will need several Mysql databases; one for each service. Each DB should also have both a staging and production instance.

openstack_keystone
openstack_glance
openstack_nova
openstack_ironic
openstack_neutron
openstack_dash
openstack_cinder
openstack_swift
openstack_trove
openstack_heat
openstack_ceilometer

Thanks

Comment 1

[Brandon Johnson [:cyborgshadow]]

Hi Jake,

I'd be happy to setup these databases. In order to better serve you, I'd like to ask a few questions so we can find the best home for them. :)

Do you want these databases to be on the same server as any of your other current databases? If so, which one(s)?

How many users and what permissions will we need for these databases?

Can you give me a brief description of the service these databases will be behind?

What's the impact of loss of connectivity on these databases? (e.g. "I can still do my job but it'd be unpleasant", "tree closing", "It would halt work.", "no major impact", etc.)

What's the expected size and throughput of these databases? (Rough estimates are fine, we just want a traffic estimate).

Comment 2

[Jake Watkins [:dividehex]]

(In reply to Brandon Johnson [:cyborgshadow] from comment #1)
> Hi Jake,
> 
> I'd be happy to setup these databases. In order to better serve you, I'd
> like to ask a few questions so we can find the best home for them. :)
> 
> Do you want these databases to be on the same server as any of your other
> current databases? If so, which one(s)?

No.  There is no need for these DBs to share a server.  Each is independent to its own service and won't be access by anything other said service.

> How many users and what permissions will we need for these databases?

Each db will need one user for rw access with GRANT ALL PRIVILEGES.
 
> Can you give me a brief description of the service these databases will be
> behind?

c&p from dev docs

openstack_keystone - Keystone is an OpenStack project that provides Identity, Token, Catalog and Policy services for use specifically by projects in the OpenStack family

openstack_glance - The Glance project provides services for discovering, registering, and retrieving virtual machine images.

openstack_nova - Nova is the project name for OpenStack Compute, a cloud computing fabric controller, the main part of an IaaS system

openstack_ironic - Ironic is an Incubated OpenStack project which aims to provision bare metal (as opposed to virtual) machines by leveraging common technologies such as PXE boot and IPMI to cover a wide range of hardware, while supporting pluggable drivers to allow vendor-specific functionality to be added.

openstack_neutron - Neutron is an OpenStack project to provide “network connectivity as a service” between interface devices (e.g., vNICs) managed by other OpenStack services (e.g., nova).

openstack_dash - Horizon is the canonical implementation of OpenStack’s Dashboard, which provides a web based user interface to OpenStack services including Nova, Swift, Keystone, etc.

openstack_cinder - Cinder is an OpenStack project to provide “block storage as a service”.

openstack_swift - Swift is a highly available, distributed, eventually consistent object/blob store

openstack_trove - Trove is Database as a Service for OpenStack

openstack_heat - Heat is a service to orchestrate multiple composite cloud applications using the AWS CloudFormation template format, through both an OpenStack-native ReST API and a CloudFormation-compatible Query API.

openstack_ceilometer - The ceilometer project aims to deliver a unique point of contact for billing systems to acquire all of the measurements they need to establish customer billing, across all current OpenStack core components with work underway to support future OpenStack components

> What's the impact of loss of connectivity on these databases? (e.g. "I can
> still do my job but it'd be unpleasant", "tree closing", "It would halt
> work.", "no major impact", etc.)

For the staging environment, loss of connectivity would not have much on an impact to day to day work but would prevent testing changes to be pushed out to the production environment.  For this quarter (and probably next q), staging will be used to do most of the openstack development and testing so it will much more impacting in the beginning.  Just not tree closing.

For the production environment, this would be tree closing.  Even though each db is independent to a single service, loss of a single service would impact the entire openstack environment. eg, Loss of keystone would prevent all other services from auth.  Loss of glance would prevent image deployment.  etc.

> What's the expected size and throughput of these databases? (Rough estimates
> are fine, we just want a traffic estimate).

These might be difficult to estimate.  I really don't have enough insight into how each service uses its respective database but afaik, they are primarily for storing service state.  It would also be safe to assume the staging environment will be far smaller in size and traffic.  Maybe we could talk offline about finding estimates.

Also, the current plan is to build out the staging environment first.  This could give us a good idea of the metrics to expect when it comes time for replicating the design to a production environment providing we can stress test it enough to simulate what we could expect in production.

Comment 3

[Brandon Johnson [:cyborgshadow]]

The request for your stage servers has been sent in. The associated bug is linked. Per our irc conversation your fqdn's will be openstack[1-2].db.scl3.mozilla.com. 

If the apps are capable of load balancing I'll setup a zeus ticket once the virtual servers are live.

Comment 4

[Jake Watkins [:dividehex]]

(In reply to Brandon Johnson [:cyborgshadow] from comment #3)
> The request for your stage servers has been sent in. The associated bug is
> linked. Per our irc conversation your fqdn's will be
> openstack[1-2].db.scl3.mozilla.com. 
> 
> If the apps are capable of load balancing I'll setup a zeus ticket once the
> virtual servers are live.

Thanks Brandon!  And yes, zlb would be preferred for HA

Comment 5

[Brandon Johnson [:cyborgshadow]]

Hi Jake,

I'm working on these actively hoping to have them done by the end of the day.

Waiting on the zeus config now and when that's completed I have about an hour's worth of work to do on them before turning them over to you. Just wanted to let you know the status and that we're working on it. :)

Comment 6

[Jake Watkins [:dividehex]]

(In reply to Brandon Johnson [:cyborgshadow] from comment #5)
> Hi Jake,
> 
> I'm working on these actively hoping to have them done by the end of the day.
> 
> Waiting on the zeus config now and when that's completed I have about an
> hour's worth of work to do on them before turning them over to you. Just
> wanted to let you know the status and that we're working on it. :)

Hey Brandon, any word on this?  Is the zlb config complete?  If there is a bug for the zlb work, can you block this bug on it?
Thanks!

Comment 7

[Brandon Johnson [:cyborgshadow]]

ZLB config is complete. Dropping in my work now. :) Sorry for not catching your bug update yesterday. I'm in US EST so that was right before my day's end.

Comment 8

[Brandon Johnson [:cyborgshadow]]

Alright, so the servers are up and running. I've made the databases. Running grants now and giving you the passwords over IRC.

Comment 9

[Jake Watkins [:dividehex]]

Brandon, I'm have trouble connecting to the zlb VIPs.  Does zlb need to be configured to allow connections from *.admin.cloud.releng.scl3.mozilla.com?

[root@glance1 ~]# mysql -h openstack-ro-vip.db.scl3.mozilla.com -D openstack_keystone -u keystone_rw -p
Enter password:
ERROR 2003 (HY000): Can't connect to MySQL server on 'openstack-ro-vip.db.scl3.mozilla.com' (111)


[root@glance1 ~]# nc -zv openstack-ro-vip.db.scl3.mozilla.com 3306
nc: connect to openstack-ro-vip.db.scl3.mozilla.com port 3306 (tcp) failed: Connection refused
[root@glance1 ~]# nc -zv openstack-rw-vip.db.scl3.mozilla.com 3306
nc: connect to openstack-rw-vip.db.scl3.mozilla.com port 3306 (tcp) failed: Connection refused
[root@glance1 ~]# nc -zv openstack1.db.scl3.mozilla.com 3306
Connection to openstack1.db.scl3.mozilla.com 3306 port [tcp/mysql] succeeded!
[root@glance1 ~]# nc -zv openstack2.db.scl3.mozilla.com 3306
Connection to openstack2.db.scl3.mozilla.com 3306 port [tcp/mysql] succeeded!

Comment 10

[Jake Watkins [:dividehex]]

After some troubleshooting with :cyborgshadow and :dustin, it was determined that the rw and ro vservers in zlb weren't running.  Once those were started, I was able to connect.  I'll go ahead and close this now that I've verified the connections from the openstack admin hosts.