Closed Bug 599133 Opened 14 years ago Closed 14 years ago

Move node_config.json from a flat file to the available_nodes database

Categories

(Cloud Services :: Operations: Miscellaneous, task)

x86
macOS
task
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: telliott, Assigned: Atoll)

References

Details

Attachments

(1 file)

We should add a bunch of columns to available_nodes which would allow us to move all the db metadata currently contained in node_config into available nodes. At that point, we use scripts to generate the shards_constants files, etc

Note that this includes db passwords.
I believe this should be the structure of the table:

create table available_nodes (
node varchar(30) not null primary key,
cluster varchar(10) not null,
db_host varchar(30),
db_name varchar(10),
db_modulo int,
test_username varchar(10),
test_password varchar(10),
available_assignments int,
actives int,
max_capacity int not null,
quota int,
downed tinyint default 0,
backoff int default 0
)

We then have a cron that pulls all these records and sticks the values into memcache every minute. The server then grabs that out of memcache and overrides the default values defined in default config (thus, defining the default db_host to be localhost will cause 503s if the memcache record isn't available). I believe these are all the values where we would possibly want to override a site-wide config with the possible exception of db password, which should probably be handled separately.

This removes the need for shard_constants and lets us configure the nodes just by altering the database (through an internal api). It also gives us a fast way to down nodes/set backoffs.
hmm, it would be a good idea to add

assignments_per_period int

which would be the number of acconts to allow during the cron period. That way, the refresh accounts cron wouldn't need a config file - it would simply copy the value from assignments_per_period to available_assignments
Blocks: 594511
Need to add a block to node_config.json:

 "memcache": {
        "<cluster>": [<memcache ips or hosts>]
 },

cluster corresponds to the new field in the available_nodes table
Blocks: 605713
Blocks: 598800
Ignore the script, we'll need to do something on the client side rather than the server side. I'll hack something new up there.

At this point the changes (including assignments_per_period) look good, and we should do it on dev/stage and then prod. It should be entirely backwards-compatible, so we should pick a time and alter the table.
Assignee: telliott → rsoderberg
Component: Server: Other → Operations
QA Contact: other-server → operations
CREATE TABLE `available_nodes` (
  `node` varchar(255) NOT NULL default '',
  `cluster` varchar(10) NOT NULL,
  `db_host` varchar(30) default NULL,
  `db_name` varchar(10) default NULL,
  `db_modulo` int(11) default NULL,
  `test_username` varchar(10) default NULL,
  `test_password` varchar(10) default NULL,
  `available_assignments` int(11) default NULL,
  `assignments_per_period` int(11) default NULL,
  `actives` int(11) default NULL,
  `max_capacity` int(11) NOT NULL,
  `quota` int(11) default NULL,
  `downed` tinyint(4) default '0',
  `backoff` int(11) default '0',
  PRIMARY KEY  (`node`),
  UNIQUE KEY `db_idx` (`db_host`,`db_name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

INSERT INTO NEW_available_nodes (cluster, max_capacity, node, available_assignments, actives, downed) SELECT 'phx1' as cluster, 10000000 as max_capacity, node, ct AS available_assignments, actives, downed from available_nodes;

ALTER TABLE available_nodes rename OLD_available_nodes; alter table NEW_available_nodes RENAME available_nodes
Status: NEW → ASSIGNED
Blocks: 612450
some starting values for the current entries in the phonenix cluster:

cluster should be the same for everything in the cluster ('phx' or 'phx1' or something). It just represents a logical group to do operations on.

db_host and db_name are those values currently set in shard_constants for each node.

db_modulo is, I believe, 100 everywhere right now. It's the number of tables in each wbo db.

test_username and test_password are for when we do heartbeats, etc. Those don't need to be set right now.

available_assignments is set by the cron.

assignments_per_period is the number that used to be set by the cron. That was 100 per hour, so if you're running the cron hourly, set this to 100 for everything

actives is set by scripts

max_capacity is indeterminate and not something we're worried about yet, so set it to something high and we'll adjust as needed.

quota is whatever mconnor and rags settle on for quotas. It's in bytes.
What is max_capacity?
If the number of actives on the box exceeds max_capacty, we won't assign users to the box.
Above alter table run on weave-dev-adm01 and weave-stage-adm01.  Verifying that staging is ready for QA testing, results soon.
Severity: normal → major
Staging passed QA and this has been deployed to production as part of reg/sreg 1.3.101115.
Status: ASSIGNED → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: