Closed Bug 742049 Opened 12 years ago Closed 12 years ago

sync106.db.scl2.svc: mysql_uptime errors, users migrated

Categories

(Cloud Services :: Operations: Miscellaneous, task)

Product:
Cloud Services
Component:
Operations: Miscellaneous
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: Atoll, Assigned: jlaz)

Details

(Whiteboard: [Has new drives, rekicking, needs to be puppetized and /data rebuilt][qa-]) 

      No description provided.
Whiteboard: [qa-]
All drives replaced with refurbished drives.  Will need a rekick and /data rebuilt.
Whiteboard: [qa-] → [Has new drives, needs a rekick and /data rebuild][qa-]
users were not migrated properly:

scl2-sync1051 downed=1 migrating=0 capacity=2078/40000
scl2-sync1052 downed=1 migrating=0 capacity=2078/40000
scl2-sync1053 downed=1 migrating=0 capacity=2078/40000
scl2-sync1054 downed=1 migrating=0 capacity=2078/40000
scl2-sync1055 downed=1 migrating=0 capacity=2078/40000
scl2-sync1056 downed=1 migrating=0 capacity=2078/40000
scl2-sync1057 downed=1 migrating=0 capacity=2078/40000
scl2-sync1058 downed=1 migrating=0 capacity=2078/40000
scl2-sync1059 downed=1 migrating=0 capacity=2078/40000
scl2-sync1060 downed=1 migrating=0 capacity=2078/40000

to correct, I ran:

nodenuke setmigrate $(nodegrep -nw sync106.db)
SLA impact: 20780 users getting 503s, 2012-04-03 23:37 PDT -> 2012-04-11 08:41 PDT.  (found out because of bug 744399)
Swapped the failing drive, rekick is in progress, needs to be puppetized and have /data rebuilt
Whiteboard: [Has new drives, needs a rekick and /data rebuild][qa-] → [Has new drives, rekicking, needs to be puppetized and /data rebuilt][qa-]
Rekicked, puppetized, rebuilt /data and pushed back to prod.
Assignee: nobody → jlaz
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Component: Operations: Hardware → Operations
You need to log in before you can comment on or make changes to this bug.