Closed
Bug 614821
Opened 14 years ago
Closed 14 years ago
reboots 20101125
Categories
(mozilla.org Graveyard :: Server Operations, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: armenzg, Assigned: zandr)
References
Details
(Whiteboard: [needs SCL visit])
talos-r3-w7-012.build - unreachable talos-r3-w7-040.build - host unknown
Comment 1•14 years ago
|
||
linux-ix-slave12 -- down and unreachable via IPMI.
Comment 2•14 years ago
|
||
mw32-ix-slave22 - down and unreachable by ipmi
Comment 3•14 years ago
|
||
talos-r3-fed-012 talos-r3-fed-027 talos-r3-fed64-030 talos-r3-fed64-049 talos-r3-xp-012
Comment 4•14 years ago
|
||
talos-r3-fed64-053 talos-r3-fed64-047.build talos-r3-fed64-043.build talos-r3-fed64-013.build talos-r3-fed64-001.build talos-r3-fed-040.build talos-r3-fed-036.build talos-r3-fed-027.build talos-r3-fed-024.build talos-r3-fed-022.build
Comment 5•14 years ago
|
||
talos-r3-fed-030.build
Comment 6•14 years ago
|
||
talos-r3-xp-052.build
Comment 7•14 years ago
|
||
talos-r3-w7-009.build
Updated•14 years ago
|
Flags: colo-trip+
Whiteboard: [needs SCL visit]
Comment 8•14 years ago
|
||
talos-r3-fed-009.build talos-r3-w7-036.build
Comment 9•14 years ago
|
||
jlaz/jabba? Phong's outta town. jlaz, punt around as needed. Thanks guys!
Assignee: server-ops → jlazaro
Comment 10•14 years ago
|
||
talos-r3-w7-011.build talos-r3-fed-044.build talos-r3-fed-037.build Bumping severity because we've lost 20% of our Fedora 32-bit pool.
Severity: normal → critical
Comment 11•14 years ago
|
||
talos-r3-fed64-016.build Any ETA on these?
Comment 12•14 years ago
|
||
Copying from previous reboot bug. Needs re-image (mount errors): fed-012 fed-022 fed-024 fed-036 fed-040 fed64-53
Comment 13•14 years ago
|
||
Is there any way we can get these done today? The 32-bit Fedora wait times are getting really bad.
Severity: critical → blocker
Updated•14 years ago
|
Assignee: jlazaro → server-ops
Comment 14•14 years ago
|
||
It would be really nice to figure out the issue (hw clock issue) that causes the Fedora minis to need to be re-imaged on a regular basis. Re-imaging isn't as quick as rebooting.
Assignee: server-ops → zandr
Comment 15•14 years ago
|
||
What happened to cause so many Fedora hosts to need manual reboots? What happened to require so many reimages?
Assignee | ||
Comment 16•14 years ago
|
||
w7-11: rebooted fed-044: rebooted fed-037: rebooted fed-009: rebooted w7-009: rebooted xp-052: offline for power reasons fed-012: rebooted fed-027: rebooted fed64-030: on my desk in mv fed64-049: offline for power reasons xp-012: offline for power reasons fed64-053: offline for power reasons fed64-016: rebooted fed64-047: MIA, possibly in MV fed64-043: offline for power reasons fed64-013: rebooted fed64-001: reboooted fed-040: rebooted fed-036: rebooted fed-027: rebooted fed-024: rebooted fed-022: rebooted fed-030: rebooted w7-036: rebooted
Assignee | ||
Comment 17•14 years ago
|
||
fed64-053: offline for power reasons
Comment 18•14 years ago
|
||
(In reply to comment #16) Still can't ping: > fed-044: rebooted > fed-012: rebooted > fed64-016: rebooted > fed64-001: reboooted > fed-040: rebooted > fed-036: rebooted > fed-024: rebooted > fed-022: rebooted
Comment 19•14 years ago
|
||
(In reply to comment #16) These seem to be online and connected > w7-11: rebooted > fed-037: rebooted > fed-009: rebooted > fed64-013: rebooted > fed-027: rebooted Online, needs puppet cleanup: > fed-027: rebooted Also not pingable: > w7-036: rebooted > w7-009: rebooted > fed-030: rebooted
Assignee | ||
Comment 20•14 years ago
|
||
fed-012: reimaged fed-022: reimaged
Assignee | ||
Comment 21•14 years ago
|
||
fed-024: reimaged
Assignee | ||
Comment 22•14 years ago
|
||
fed-036: reimaged
Assignee | ||
Comment 23•14 years ago
|
||
fed-040: pulled. Has a CD stuck in the drive that it won't boot from.
Assignee | ||
Comment 24•14 years ago
|
||
>linux-ix-slave12 -- down and unreachable via IPMI.
>mw32-ix-slave22 - down and unreachable by ipmi
Bounced around 18:00PDT
Reporter | ||
Comment 25•14 years ago
|
||
Can we reboot these? talos-r3-w7-052.build 7d 7h 25m 50s talos-r3-w7-036.build 6d 22h 13m 44s talos-r3-w7-032.build 0d 18h 6m 56s talos-r3-w7-012.build 16d 14h 55m 21s talos-r3-w7-009.build 11d 15h 34m 56s talos-r3-w7-008.build 1d 14h 57m 51s That's ~10% of our win7 capacity.
Assignee | ||
Comment 26•14 years ago
|
||
(In reply to comment #25) > Can we reboot these? > > talos-r3-w7-052.build 7d 7h 25m 50s > talos-r3-w7-036.build 6d 22h 13m 44s > talos-r3-w7-032.build 0d 18h 6m 56s > talos-r3-w7-012.build 16d 14h 55m 21s > talos-r3-w7-009.build 11d 15h 34m 56s > talos-r3-w7-008.build 1d 14h 57m 51s > > That's ~10% of our win7 capacity. Will swing by scl1 on the way home tonight. -Z
Reporter | ||
Comment 27•14 years ago
|
||
It could wait until Monday/Tuesday as there is no pending jobs and at this time of the day people won't be pushing like mad people. Your call. Have a good weekend.
Assignee | ||
Comment 28•14 years ago
|
||
Given the allhands next week, I'm not certain I'll be able to get down there. It's not really out of my way tonight.
Assignee | ||
Comment 29•14 years ago
|
||
(In reply to comment #25) > Can we reboot these? > > talos-r3-w7-052.build 7d 7h 25m 50s > talos-r3-w7-036.build 6d 22h 13m 44s > talos-r3-w7-032.build 0d 18h 6m 56s > talos-r3-w7-012.build 16d 14h 55m 21s > talos-r3-w7-009.build 11d 15h 34m 56s > talos-r3-w7-008.build 1d 14h 57m 51s Rebooted, responding to ping, lots of ports open. Two of these we'd pulled power on, but I think we can get away with it for now.
Comment 30•14 years ago
|
||
This looks like the list of machines in Nagios that need rebooting; sorry for any dups. linux-ix-slave31.build.scl1 linux-ix-slave32.build.scl1 linux-ix-slave33.build.scl1 linux-ix-slave35.build.scl1 linux-ix-slave38.build.scl1 linux-ix-slave42.build.scl1 mv-moz2-linux-ix-slave05.build talos-r3-fed-012.build talos-r3-fed-029.build talos-r3-fed-033.build talos-r3-fed-036.build talos-r3-fed-038.build talos-r3-fed-041.build talos-r3-fed64-021.build talos-r3-fed64-027.build talos-r3-fed64-044.build talos-r3-fed64-055.build talos-r3-snow-004.build
Comment 31•14 years ago
|
||
All SCL machines got a reboot today, closing this. Filed bug 620041 to track remaining down minis.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Reporter | ||
Updated•14 years ago
|
Alias: reboots
Updated•9 years ago
|
Product: mozilla.org → mozilla.org Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•