694234 - sync: verify rseg=64 with SELECT COUNT(*) FROM information_schema.INNODB_RSEG; everywhere

Status: RESOLVED WONTFIX

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[:Atoll]

databases that were upgraded to percona should have SELECT COUNT(*) FROM information_schema.INNODB_RSEG; return 64 or 65.  if not, needs to be shutdown using:

SET GLOBAL innodb_fast_shutdown=0;
\q
mysqladmin shutdown
wait, ps awx | grep mysqld
/etc/init.d/mysql start

scl2 1..72, phx 1..60

Comment 1

[Pete Fritchman [:petef]]

prod DBs out of spec:

# host: SELECT COUNT(*) FROM information_schema.INNODB_RSEG;
sync1.db: 1
sync2.db: 1
sync3.db: 1
sync5.db: 1
sync6.db: 1
sync7.db: 1
sync8.db: 1
sync9.db: 1
sync10.db: 1
sync11.db: 1
sync15.db: 1
sync18.db: 1
sync19.db: 1
sync20.db: 1
sync21.db: 1
sync22.db: 1
sync24.db: 1
sync25.db: 1
sync27.db: 1
sync29.db: 1
sync30.db: 1
sync31.db: 1
sync32.db: 1
sync34.db: 1
sync37.db: 1
sync39.db: 1
sync40.db: 1
sync42.db: 1
sync45.db: 1
sync46.db: 1
sync47.db: 1
sync49.db: 1
sync50.db: 0
sync51.db: 1
sync52.db: 1
sync53.db: 1
sync54.db: 1
sync56.db: 1
sync57.db: 1
sync59.db: 1
sync60.db: 1
sync62.db: 1
sync63.db: 1
sync64.db: 1
sync65.db: 1
sync66.db: 1
sync67.db: 1
sync68.db: 1
sync69.db: 1
sync71.db: 1
sync72.db: 1
wp-db01: 1
wp-db02: 1
wp-db03: 1
wp-db04: 1
wp-db05: 1
wp-db06: 1
wp-db07: 1
wp-db08: 1
wp-db09: 1
wp-db10: 1
wp-db11: 1
wp-db12: 1
wp-db13: 1
wp-db14: 1
wp-db15: 1
wp-db16: 1
wp-db17: 1
wp-db18: 1
wp-db20: 1
wp-db21: 1
wp-db23: 1
wp-db26: 1
wp-db27: 1
wp-db28: 1
wp-db29: 1
wp-db30: 1
wp-db32: 1
wp-db33: 1
wp-db34: 1
wp-db35: 1
wp-db36: 1
wp-db38: 1
wp-db40: 1
wp-db41: 1
wp-db42: 1
wp-db43: 1
wp-db44: 1
wp-db45: 1
wp-db46: 1
wp-db47: 1
wp-db48: 1
wp-db49: 1
wp-db51: 1
wp-db53: 1
wp-db54: 1
wp-db55: 1
wp-db56: 1
wp-db57: 1
wp-db58: 1
wp-db59: 1
wp-db60: 1

before we go shutdown these databases, let's see if any of the other DB changes we're rolling out will require a restart too.

Comment 2

[Pete Fritchman [:petef]]

I ran "SET GLOBAL innodb_fast_shutdown=0;" on all of the hosts in Comment 1 so they'll magically pick this up on the next restart.

Comment 3

[:Atoll]

Note that next shutdown may be slow in scl2, which seems to take about 45 minutes longer than phx (1 minute).  If this is inconvenient, it is safe to kill -9 mysqld and restart it. The repair is typically 5 minutes and then it's restarted and ready for service.

Comment 4

[:Atoll]

Scheduled. https://intranet.mozilla.org/Services/Ops/ChangeWindow_20120124 and _20120131 so far.

Comment 5

[:Atoll]

as databases perish and are rebuilt, this is deployed automatically. it isn't an issue under daily load, and only crops up during db29 incidents. all new builds have it, and there's no pressing reason to ship it out for existing builds. wontfix'ing due to lack of end-user benefit.