Closed
Bug 936042
(t-w864-ix-092)
Opened 11 years ago
Closed 8 years ago
t-w864-ix-092 problem tracking
Categories
(Infrastructure & Operations Graveyard :: CIDuty, task, P3)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: jhopkins, Unassigned)
References
Details
(Whiteboard: [buildduty][buildslaves][capacity])
Attachments
(1 file)
73.52 KB,
image/png
|
Details |
burning jobs due to "can't clone scripts" error. Disabled in slavealloc.
Comment 1•11 years ago
|
||
Re-enabled because we're pretty sure this is rooted in an issue external to the machine that breaks DNS resolution until the machine reboots. bug 937279 has more.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Comment 2•9 years ago
|
||
I've seen multiple instances of "Access Denied" spew during xpcshell runs contributing to timeouts on this slave. Disabled. https://treeherder.mozilla.org/logviewer.html#?job_id=498896&repo=mozilla-aurora
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 3•9 years ago
|
||
Re-imaged and re-enabled.
Status: REOPENED → RESOLVED
Closed: 11 years ago → 9 years ago
Resolution: --- → FIXED
Comment 4•9 years ago
|
||
Too many failures either unique to this slave, or predominantly this slave (10 of the 25 instances of bug 1075419 have been on this). Disabled, needs diagnostics.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 8•9 years ago
|
||
Reenabled so it can burn more jobs and create more invalid test-failure bugs.
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
QA Contact: armenzg → bugspam.Callek
Resolution: --- → FIXED
Comment 9•9 years ago
|
||
Or not. Rebooted twice, it's not connecting to a master. Redisabled.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 10•9 years ago
|
||
Something was amiss. The host was up, but I couldn't connect via ssh or VNC. I've kicked off another re-image and we'll see if it come back. I'll file a diagnostics bug if it doesn't come back in a functional state.
Comment 11•9 years ago
|
||
This machine is currently stuck setting the screen resolution: http://imgur.com/miqzfw4
Comment 12•9 years ago
|
||
disabled as last few jobs are not looking good and comment 11 suggests it needs love. noticed this slave as slaveapi was trying to do something (shutdown?) with it before we rebooted slaveapi1 prod node.
Comment 13•9 years ago
|
||
This went rather well, wouldn't you say?
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Comment 14•9 years ago
|
||
Seems to be quite orangetastic according to slave health, including xpcshell 120min timeouts (when it normally runs ~20min)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 15•9 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1172763)
Comment 16•9 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1172990)
Comment 17•9 years ago
|
||
Reenabled with a new motherboard.
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Comment 18•9 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1193750)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 19•9 years ago
|
||
Running at 1024x768 per http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-inbound/sha512/a796f1591bd1aef17cefa5c4b6695ce1f664ad373735f1c42b794f2cea35f03d53b16f970ea495563aaa30009e48660fbc4060407185f0f2134ddfb85b5e7634
Comment 20•9 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1195648)
Comment 21•9 years ago
|
||
Reenabled, we'll see.
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Comment 22•9 years ago
|
||
Disabled. Just like the other Win8 slave that was reimaged this week, it can't make it through more than 1.5 jobs before disconnecting and requiring a reboot to send it back to work.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 23•9 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1204670)
Comment 24•9 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1205655)
Comment 25•9 years ago
|
||
hmm, the host is up and sshable. i am also able to log in to the host via IPMI with the releng password in the gpg file. vans-MacBook-Pro:~ vle$ ping t-w864-ix-092.wintest.releng.scl3.mozilla.com PING t-w864-ix-092.wintest.releng.scl3.mozilla.com (10.26.40.122): 56 data bytes 64 bytes from 10.26.40.122: icmp_seq=0 ttl=124 time=3.020 ms 64 bytes from 10.26.40.122: icmp_seq=1 ttl=124 time=4.563 ms ^C --- t-w864-ix-092.wintest.releng.scl3.mozilla.com ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 3.020/3.792/4.563/0.771 ms vans-MacBook-Pro:~ vle$ ssh !$ ssh t-w864-ix-092.wintest.releng.scl3.mozilla.com The authenticity of host 't-w864-ix-092.wintest.releng.scl3.mozilla.com (10.26.40.122)' can't be established. RSA key fingerprint is e3:01:f7:a3:a1:b6:17:d2:b4:ca:97:c5:3c:54:56:e1. Are you sure you want to continue connecting (yes/no)?
Comment 26•9 years ago
|
||
(In reply to Van Le [:van] from comment #25) > hmm, the host is up and sshable. i am also able to log in to the host via > IPMI with the releng password in the gpg file. It apparently didn't like you looking at it, though: 12 minutes after your comment it failed out of the job it was running, hasn't taken another, and won't reboot through slaverebooter. Curious thing about https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=test&type=t-w864-ix&name=t-w864-ix-092 - shouldn't all those "Attempting SSH reboot...Failed. Filed IT bug..." lines have an IPMI attempt after the failed ssh attempt?
Comment 27•9 years ago
|
||
looks like it failed the reimage. after checking and finding the host sshable and ipmi reachable, i gave it another reimage to see if it was the image that was causing issues - it was an issue previously, unsure if Q fixed it. i'll attach screen shot to this bug and the child bug in dcops queue for tracking as well.
Comment 28•9 years ago
|
||
Comment 29•9 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1212525)
Updated•9 years ago
|
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Comment 30•9 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1214457)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 31•9 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1214814)
Updated•9 years ago
|
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Comment 32•9 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1215830)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Updated•9 years ago
|
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Comment 33•9 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1217556)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 34•9 years ago
|
||
Disabled while it takes another trip back to iX.
Updated•9 years ago
|
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Comment 35•9 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1224024)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Updated•9 years ago
|
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Comment 36•9 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1225006)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Updated•9 years ago
|
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Comment 37•8 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1227166)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Updated•8 years ago
|
Status: REOPENED → RESOLVED
Closed: 9 years ago → 8 years ago
Resolution: --- → FIXED
Comment 38•8 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1228370)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Updated•8 years ago
|
Status: REOPENED → RESOLVED
Closed: 8 years ago → 8 years ago
Resolution: --- → FIXED
Comment 39•8 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1230979)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 40•8 years ago
|
||
(In reply to Phil Ringnalda (:philor) from comment #26) > Curious thing about > https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave. > html?class=test&type=t-w864-ix&name=t-w864-ix-092 - shouldn't all those > "Attempting SSH reboot...Failed. Filed IT bug..." lines have an IPMI attempt > after the failed ssh attempt? Still true, and... (In reply to Phil Ringnalda (:philor) from comment #17) > Reenabled with a new motherboard. isn't there something that has to be done after a motherboard replacement to get IPMI reboots working again?
Comment 41•8 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1232565)
Updated•8 years ago
|
Status: REOPENED → RESOLVED
Closed: 8 years ago → 8 years ago
Resolution: --- → FIXED
Comment 42•8 years ago
|
||
Disabled since bug 1230979 is eventually planning on giving it a new disk, and it's now unrebootable but that open unreachable bug prevents us from getting another one as it fails to reboot like it always fails to reboot.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 43•8 years ago
|
||
From what I can see the hdd have been replaced and the slave take jobs.
Status: REOPENED → RESOLVED
Closed: 8 years ago → 8 years ago
Resolution: --- → FIXED
Comment 44•8 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1242319)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 45•8 years ago
|
||
What's the worst that can happen?
Status: REOPENED → RESOLVED
Closed: 8 years ago → 8 years ago
Resolution: --- → FIXED
Comment 46•8 years ago
|
||
Attempting SSH reboot...Failed. Filed IT bug for reboot (bug 1253962)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Updated•8 years ago
|
Status: REOPENED → RESOLVED
Closed: 8 years ago → 8 years ago
Resolution: --- → FIXED
Updated•6 years ago
|
Product: Release Engineering → Infrastructure & Operations
Updated•4 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•