sync106.db.scl2.svc: mysql_uptime errors, users migrated

RESOLVED FIXED

Status

Cloud Services
Operations
RESOLVED FIXED
6 years ago
2 years ago

People

(Reporter: atoll, Assigned: jlaz)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [Has new drives, rekicking, needs to be puppetized and /data rebuilt][qa-])

Comment hidden (empty)
Whiteboard: [qa-]
(Assignee)

Comment 1

6 years ago
All drives replaced with refurbished drives.  Will need a rekick and /data rebuilt.
Whiteboard: [qa-] → [Has new drives, needs a rekick and /data rebuild][qa-]
users were not migrated properly:

scl2-sync1051 downed=1 migrating=0 capacity=2078/40000
scl2-sync1052 downed=1 migrating=0 capacity=2078/40000
scl2-sync1053 downed=1 migrating=0 capacity=2078/40000
scl2-sync1054 downed=1 migrating=0 capacity=2078/40000
scl2-sync1055 downed=1 migrating=0 capacity=2078/40000
scl2-sync1056 downed=1 migrating=0 capacity=2078/40000
scl2-sync1057 downed=1 migrating=0 capacity=2078/40000
scl2-sync1058 downed=1 migrating=0 capacity=2078/40000
scl2-sync1059 downed=1 migrating=0 capacity=2078/40000
scl2-sync1060 downed=1 migrating=0 capacity=2078/40000

to correct, I ran:

nodenuke setmigrate $(nodegrep -nw sync106.db)
SLA impact: 20780 users getting 503s, 2012-04-03 23:37 PDT -> 2012-04-11 08:41 PDT.  (found out because of bug 744399)
(Assignee)

Comment 4

6 years ago
Swapped the failing drive, rekick is in progress, needs to be puppetized and have /data rebuilt
Whiteboard: [Has new drives, needs a rekick and /data rebuild][qa-] → [Has new drives, rekicking, needs to be puppetized and /data rebuilt][qa-]
(Assignee)

Comment 5

6 years ago
Rekicked, puppetized, rebuilt /data and pushed back to prod.
Assignee: nobody → jlaz
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Component: Operations: Hardware → Operations
Product: Cloud Services → Cloud Services
You need to log in before you can comment on or make changes to this bug.