Closed
Bug 1039170
Opened 10 years ago
Closed 10 years ago
Integration Trees closed due Android tests fail Automation Error: Unable to reboot panda-xyz via Relay Board.
Categories
(Infrastructure & Operations Graveyard :: CIDuty, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: cbook, Unassigned)
References
()
Details
23:26:28 INFO - 07/15/2014 23:26:28: ERROR: Failed to powercycle 15 times, giving up
23:26:28 INFO - 07/15/2014 23:26:28: WARNING: Automation Error: Unable to reboot panda-0733 via Relay Board.
https://tbpl.mozilla.org/php/getParsedLog.php?id=43903939&tree=Mozilla-Inbound
seems a recent problem and hitting several pandas
|
Reporter | |
Comment 1•10 years ago
|
||
affected pandas it seems
panda-0700
panda-0847
panda-0758
panda-0853
panda-0727
panda-0733
panda-0752
panda-0699
panda-0041
panda-0800
panda-0733
panda-0772
panda-0710
panda-0040
panda-0808
|
||
Comment 2•10 years ago
|
||
So my first thoughts are:
*) This is probably related to the most recent panda move from sc1 -> scl3 which I think ran yesterday.
*) There could be problems with the relay board data in https://hg.mozilla.org/build/tools/file/tip/buildfarm/mobile/devices.json -> I can compare this to what we have in https://inventory.mozilla.org/en-US/#inventory
*) I will check if all these pandas are connected to the same panda relay, in case there is a problem with the relay (should not be foopy related - foopies are not involved in the rebooting via panda relay)
|
|
Reporter | |
Comment 3•10 years ago
|
||
reopened the trees since retrigger on other pandas came back green, will keep monitor the situation
|
|
||
Comment 4•10 years ago
|
||
Should be safe to disable these pandas in slavealloc until cause has been established
|
|
||
Comment 5•10 years ago
|
||
Looks like a lot of different relay boards are involved, so not related to a particular relay:
pmoore@Elisandra:~/git/tools/buildfarm/mobile master $ cat badpandas.list | while read panda; do line="$(sed -n "/${panda}/=" devices.json)"; sed -n ${line},$((line+4))p devices.json ; done
"panda-0700": {
"foopy": "foopy88",
"relayhost": "panda-relay-063.p8.releng.scl3.mozilla.com",
"relayid": "1:3"
},
"panda-0847": {
"foopy": "foopy99",
"relayhost": "panda-relay-076.p9.releng.scl3.mozilla.com",
"relayid": "2:3"
},
"panda-0758": {
"foopy": "foopy92",
"relayhost": "panda-relay-068.p8.releng.scl3.mozilla.com",
"relayid": "2:2"
},
"panda-0853": {
"foopy": "foopy100",
"relayhost": "panda-relay-077.p9.releng.scl3.mozilla.com",
"relayid": "1:2"
},
"panda-0727": {
"foopy": "foopy90",
"relayhost": "panda-relay-065.p8.releng.scl3.mozilla.com",
"relayid": "2:4"
},
"panda-0733": {
"foopy": "foopy90",
"relayhost": "panda-relay-066.p8.releng.scl3.mozilla.com",
"relayid": "1:3"
},
"panda-0752": {
"foopy": "foopy92",
"relayhost": "panda-relay-067.p8.releng.scl3.mozilla.com",
"relayid": "2:7"
},
"panda-0699": {
"foopy": "foopy88",
"relayhost": "panda-relay-063.p8.releng.scl3.mozilla.com",
"relayid": "1:2"
},
"panda-0041": {
"foopy": "foopy103",
"relayhost": "panda-relay-003.p10.releng.scl3.mozilla.com",
"relayid": "2:4"
},
"panda-0800": {
"foopy": "foopy96",
"relayhost": "panda-relay-072.p9.releng.scl3.mozilla.com",
"relayid": "1:4"
},
"panda-0733": {
"foopy": "foopy90",
"relayhost": "panda-relay-066.p8.releng.scl3.mozilla.com",
"relayid": "1:3"
},
"panda-0772": {
"foopy": "foopy93",
"relayhost": "panda-relay-069.p8.releng.scl3.mozilla.com",
"relayid": "2:5"
},
"panda-0710": {
"foopy": "foopy89",
"relayhost": "panda-relay-064.p8.releng.scl3.mozilla.com",
"relayid": "1:2"
},
"panda-0040": {
"foopy": "foopy103",
"relayhost": "panda-relay-003.p10.releng.scl3.mozilla.com",
"relayid": "2:3"
},
"panda-0808": {
"foopy": "foopy96",
"relayhost": "panda-relay-073.p9.releng.scl3.mozilla.com",
"relayid": "1:1"
},
|
|
||
Comment 6•10 years ago
|
||
|
|
||
Comment 7•10 years ago
|
||
Also releng part (e.g. updates to devices.json): https://bugzilla.mozilla.org/show_bug.cgi?id=1038014
|
|
||
Comment 8•10 years ago
|
||
I checked inventory against devices.json: the first half all match exactly:
panda-0700: https://inventory.mozilla.org/en-US/systems/show/8172/
panda-0847: https://inventory.mozilla.org/en-US/systems/show/8319/
panda-0758: https://inventory.mozilla.org/en-US/systems/show/8230/
panda-0853: https://inventory.mozilla.org/en-US/systems/show/8325/
panda-0727: https://inventory.mozilla.org/en-US/systems/show/8199/
panda-0733: https://inventory.mozilla.org/en-US/systems/show/8205/
panda-0752: https://inventory.mozilla.org/en-US/systems/show/8224/
panda-0699: https://inventory.mozilla.org/en-US/systems/show/8171/
I didn't check the second half, as this clearly isn't the problem.
In other words, "system.relay.0" in inventory matches the "relayhost" and "relayid" in devices.json for these pandas.
|
|
||
Comment 9•10 years ago
|
||
Can also ping a random relay host:
pmoore@Elisandra:~/git/mozilla master $ ping -c5 panda-relay-073.p9.releng.scl3.mozilla.com
PING panda-relay-073.p9.releng.scl3.mozilla.com (10.26.136.42): 56 data bytes
64 bytes from 10.26.136.42: icmp_seq=0 ttl=56 time=185.868 ms
64 bytes from 10.26.136.42: icmp_seq=1 ttl=56 time=185.788 ms
64 bytes from 10.26.136.42: icmp_seq=2 ttl=56 time=187.358 ms
64 bytes from 10.26.136.42: icmp_seq=3 ttl=56 time=185.341 ms
64 bytes from 10.26.136.42: icmp_seq=4 ttl=56 time=185.613 ms
--- panda-relay-073.p9.releng.scl3.mozilla.com ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 185.341/185.994/187.358/0.706 ms
|
|
||
Comment 10•10 years ago
|
||
Will find code for "Calling PDU powercycle" log lines, to see what it is doing...
|
|
||
Comment 11•10 years ago
|
||
Was able to pdu powercycle panda-700 manually:
[cltbld@foopy88.p8.releng.scl3.mozilla.com sut_lib]$ python
Python 2.6.6 (r266:84292, Dec 7 2011, 20:48:22)
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import relay
>>> relay.powercycle('panda-relay-063.p8.releng.scl3.mozilla.com', 1, 3)
True
>>>
|
|
Reporter | |
Comment 12•10 years ago
|
||
panda-0710
panda-0759
panda-0846
panda-0733
panda-0852
panda-0788
panda-0028
panda-0827
panda-0029
panda-0046
panda-0746
panda-0790
panda-0715
panda-0805
panda-0846
panda-0805
|
||
Comment 13•10 years ago
|
||
dustin: looks like the releng software (which bypasses mozpool) is sometimes having difficulty rebooting the pandas from move train 3. We didn't experience this with the other two move trains, so I don't know if there's something different about this one. Could you take a look and help diagnose, please?
|
|
||
Comment 14•10 years ago
|
||
OK I'll disable these train 3 pandas in slavealloc now:
panda-0698.p8.releng.scl3.mozilla.com
panda-0699.p8.releng.scl3.mozilla.com
panda-0700.p8.releng.scl3.mozilla.com
panda-0701.p8.releng.scl3.mozilla.com
panda-0702.p8.releng.scl3.mozilla.com
panda-0703.p8.releng.scl3.mozilla.com
panda-0704.p8.releng.scl3.mozilla.com
panda-0705.p8.releng.scl3.mozilla.com
panda-0706.p8.releng.scl3.mozilla.com
panda-0707.p8.releng.scl3.mozilla.com
panda-0708.p8.releng.scl3.mozilla.com
panda-0709.p8.releng.scl3.mozilla.com
panda-0710.p8.releng.scl3.mozilla.com
panda-0711.p8.releng.scl3.mozilla.com
panda-0712.p8.releng.scl3.mozilla.com
panda-0713.p8.releng.scl3.mozilla.com
panda-0714.p8.releng.scl3.mozilla.com
panda-0715.p8.releng.scl3.mozilla.com
panda-0716.p8.releng.scl3.mozilla.com
panda-0717.p8.releng.scl3.mozilla.com
panda-0718.p8.releng.scl3.mozilla.com
panda-0719.p8.releng.scl3.mozilla.com
panda-0721.p8.releng.scl3.mozilla.com
panda-0722.p8.releng.scl3.mozilla.com
panda-0723.p8.releng.scl3.mozilla.com
panda-0724.p8.releng.scl3.mozilla.com
panda-0725.p8.releng.scl3.mozilla.com
panda-0726.p8.releng.scl3.mozilla.com
panda-0727.p8.releng.scl3.mozilla.com
panda-0728.p8.releng.scl3.mozilla.com
panda-0729.p8.releng.scl3.mozilla.com
panda-0730.p8.releng.scl3.mozilla.com
panda-0731.p8.releng.scl3.mozilla.com
panda-0732.p8.releng.scl3.mozilla.com
panda-0733.p8.releng.scl3.mozilla.com
panda-0734.p8.releng.scl3.mozilla.com
panda-0735.p8.releng.scl3.mozilla.com
panda-0736.p8.releng.scl3.mozilla.com
panda-0737.p8.releng.scl3.mozilla.com
panda-0738.p8.releng.scl3.mozilla.com
panda-0739.p8.releng.scl3.mozilla.com
panda-0740.p8.releng.scl3.mozilla.com
panda-0741.p8.releng.scl3.mozilla.com
panda-0742.p8.releng.scl3.mozilla.com
panda-0743.p8.releng.scl3.mozilla.com
panda-0744.p8.releng.scl3.mozilla.com
panda-0745.p8.releng.scl3.mozilla.com
panda-0746.p8.releng.scl3.mozilla.com
panda-0747.p8.releng.scl3.mozilla.com
panda-0748.p8.releng.scl3.mozilla.com
panda-0749.p8.releng.scl3.mozilla.com
panda-0750.p8.releng.scl3.mozilla.com
panda-0751.p8.releng.scl3.mozilla.com
panda-0752.p8.releng.scl3.mozilla.com
panda-0753.p8.releng.scl3.mozilla.com
panda-0754.p8.releng.scl3.mozilla.com
panda-0755.p8.releng.scl3.mozilla.com
panda-0756.p8.releng.scl3.mozilla.com
panda-0757.p8.releng.scl3.mozilla.com
panda-0758.p8.releng.scl3.mozilla.com
panda-0759.p8.releng.scl3.mozilla.com
panda-0760.p8.releng.scl3.mozilla.com
panda-0761.p8.releng.scl3.mozilla.com
panda-0762.p8.releng.scl3.mozilla.com
panda-0763.p8.releng.scl3.mozilla.com
panda-0764.p8.releng.scl3.mozilla.com
panda-0765.p8.releng.scl3.mozilla.com
panda-0766.p8.releng.scl3.mozilla.com
panda-0767.p8.releng.scl3.mozilla.com
panda-0768.p8.releng.scl3.mozilla.com
panda-0769.p8.releng.scl3.mozilla.com
panda-0770.p8.releng.scl3.mozilla.com
panda-0771.p8.releng.scl3.mozilla.com
panda-0772.p8.releng.scl3.mozilla.com
panda-0773.p8.releng.scl3.mozilla.com
panda-0774.p8.releng.scl3.mozilla.com
panda-0775.p8.releng.scl3.mozilla.com
panda-0776.p8.releng.scl3.mozilla.com
panda-0777.p8.releng.scl3.mozilla.com
panda-0778.p8.releng.scl3.mozilla.com
panda-0779.p8.releng.scl3.mozilla.com
panda-0780.p8.releng.scl3.mozilla.com
panda-0782.p8.releng.scl3.mozilla.com
panda-0783.p8.releng.scl3.mozilla.com
panda-0784.p8.releng.scl3.mozilla.com
panda-0785.p8.releng.scl3.mozilla.com
panda-0885.p8.releng.scl3.mozilla.com
panda-0786.p9.releng.scl3.mozilla.com
panda-0787.p9.releng.scl3.mozilla.com
panda-0788.p9.releng.scl3.mozilla.com
panda-0789.p9.releng.scl3.mozilla.com
panda-0790.p9.releng.scl3.mozilla.com
panda-0791.p9.releng.scl3.mozilla.com
panda-0792.p9.releng.scl3.mozilla.com
panda-0794.p9.releng.scl3.mozilla.com
panda-0795.p9.releng.scl3.mozilla.com
panda-0796.p9.releng.scl3.mozilla.com
panda-0797.p9.releng.scl3.mozilla.com
panda-0798.p9.releng.scl3.mozilla.com
panda-0799.p9.releng.scl3.mozilla.com
panda-0800.p9.releng.scl3.mozilla.com
panda-0801.p9.releng.scl3.mozilla.com
panda-0802.p9.releng.scl3.mozilla.com
panda-0803.p9.releng.scl3.mozilla.com
panda-0804.p9.releng.scl3.mozilla.com
panda-0805.p9.releng.scl3.mozilla.com
panda-0806.p9.releng.scl3.mozilla.com
panda-0807.p9.releng.scl3.mozilla.com
panda-0808.p9.releng.scl3.mozilla.com
panda-0809.p9.releng.scl3.mozilla.com
panda-0810.p9.releng.scl3.mozilla.com
panda-0811.p9.releng.scl3.mozilla.com
panda-0812.p9.releng.scl3.mozilla.com
panda-0813.p9.releng.scl3.mozilla.com
panda-0814.p9.releng.scl3.mozilla.com
panda-0815.p9.releng.scl3.mozilla.com
panda-0816.p9.releng.scl3.mozilla.com
panda-0817.p9.releng.scl3.mozilla.com
panda-0818.p9.releng.scl3.mozilla.com
panda-0819.p9.releng.scl3.mozilla.com
panda-0820.p9.releng.scl3.mozilla.com
panda-0821.p9.releng.scl3.mozilla.com
panda-0822.p9.releng.scl3.mozilla.com
panda-0823.p9.releng.scl3.mozilla.com
panda-0824.p9.releng.scl3.mozilla.com
panda-0825.p9.releng.scl3.mozilla.com
panda-0826.p9.releng.scl3.mozilla.com
panda-0827.p9.releng.scl3.mozilla.com
panda-0828.p9.releng.scl3.mozilla.com
panda-0829.p9.releng.scl3.mozilla.com
panda-0830.p9.releng.scl3.mozilla.com
panda-0831.p9.releng.scl3.mozilla.com
panda-0832.p9.releng.scl3.mozilla.com
panda-0833.p9.releng.scl3.mozilla.com
panda-0834.p9.releng.scl3.mozilla.com
panda-0835.p9.releng.scl3.mozilla.com
panda-0836.p9.releng.scl3.mozilla.com
panda-0837.p9.releng.scl3.mozilla.com
panda-0838.p9.releng.scl3.mozilla.com
panda-0839.p9.releng.scl3.mozilla.com
panda-0840.p9.releng.scl3.mozilla.com
panda-0841.p9.releng.scl3.mozilla.com
panda-0842.p9.releng.scl3.mozilla.com
panda-0843.p9.releng.scl3.mozilla.com
panda-0844.p9.releng.scl3.mozilla.com
panda-0845.p9.releng.scl3.mozilla.com
panda-0846.p9.releng.scl3.mozilla.com
panda-0847.p9.releng.scl3.mozilla.com
panda-0848.p9.releng.scl3.mozilla.com
panda-0849.p9.releng.scl3.mozilla.com
panda-0850.p9.releng.scl3.mozilla.com
panda-0851.p9.releng.scl3.mozilla.com
panda-0852.p9.releng.scl3.mozilla.com
panda-0853.p9.releng.scl3.mozilla.com
panda-0854.p9.releng.scl3.mozilla.com
panda-0855.p9.releng.scl3.mozilla.com
panda-0856.p9.releng.scl3.mozilla.com
panda-0857.p9.releng.scl3.mozilla.com
panda-0858.p9.releng.scl3.mozilla.com
panda-0859.p9.releng.scl3.mozilla.com
panda-0860.p9.releng.scl3.mozilla.com
panda-0861.p9.releng.scl3.mozilla.com
panda-0862.p9.releng.scl3.mozilla.com
panda-0863.p9.releng.scl3.mozilla.com
panda-0864.p9.releng.scl3.mozilla.com
panda-0865.p9.releng.scl3.mozilla.com
panda-0866.p9.releng.scl3.mozilla.com
panda-0867.p9.releng.scl3.mozilla.com
panda-0868.p9.releng.scl3.mozilla.com
panda-0869.p9.releng.scl3.mozilla.com
panda-0870.p9.releng.scl3.mozilla.com
panda-0871.p9.releng.scl3.mozilla.com
panda-0872.p9.releng.scl3.mozilla.com
panda-0873.p9.releng.scl3.mozilla.com
panda-0886.p9.releng.scl3.mozilla.com
panda-0022.p10.releng.scl3.mozilla.com
panda-0023.p10.releng.scl3.mozilla.com
panda-0024.p10.releng.scl3.mozilla.com
panda-0025.p10.releng.scl3.mozilla.com
panda-0026.p10.releng.scl3.mozilla.com
panda-0027.p10.releng.scl3.mozilla.com
panda-0028.p10.releng.scl3.mozilla.com
panda-0029.p10.releng.scl3.mozilla.com
panda-0030.p10.releng.scl3.mozilla.com
panda-0031.p10.releng.scl3.mozilla.com
panda-0032.p10.releng.scl3.mozilla.com
panda-0034.p10.releng.scl3.mozilla.com
panda-0035.p10.releng.scl3.mozilla.com
panda-0037.p10.releng.scl3.mozilla.com
panda-0038.p10.releng.scl3.mozilla.com
panda-0039.p10.releng.scl3.mozilla.com
panda-0040.p10.releng.scl3.mozilla.com
panda-0041.p10.releng.scl3.mozilla.com
panda-0042.p10.releng.scl3.mozilla.com
panda-0043.p10.releng.scl3.mozilla.com
panda-0046.p10.releng.scl3.mozilla.com
panda-0047.p10.releng.scl3.mozilla.com
panda-0048.p10.releng.scl3.mozilla.com
panda-0049.p10.releng.scl3.mozilla.com
panda-0050.p10.releng.scl3.mozilla.com
panda-0051.p10.releng.scl3.mozilla.com
panda-0052.p10.releng.scl3.mozilla.com
panda-0053.p10.releng.scl3.mozilla.com
panda-0054.p10.releng.scl3.mozilla.com
panda-0055.p10.releng.scl3.mozilla.com
panda-0056.p10.releng.scl3.mozilla.com
panda-0058.p10.releng.scl3.mozilla.com
panda-0059.p10.releng.scl3.mozilla.com
panda-0060.p10.releng.scl3.mozilla.com
panda-0061.p10.releng.scl3.mozilla.com
panda-0062.p10.releng.scl3.mozilla.com
panda-0063.p10.releng.scl3.mozilla.com
panda-0064.p10.releng.scl3.mozilla.com
panda-0065.p10.releng.scl3.mozilla.com
panda-0066.p10.releng.scl3.mozilla.com
panda-0067.p10.releng.scl3.mozilla.com
panda-0068.p10.releng.scl3.mozilla.com
panda-0070.p10.releng.scl3.mozilla.com
panda-0071.p10.releng.scl3.mozilla.com
panda-0072.p10.releng.scl3.mozilla.com
panda-0073.p10.releng.scl3.mozilla.com
panda-0074.p10.releng.scl3.mozilla.com
panda-0075.p10.releng.scl3.mozilla.com
panda-0076.p10.releng.scl3.mozilla.com
panda-0077.p10.releng.scl3.mozilla.com
panda-0078.p10.releng.scl3.mozilla.com
panda-0079.p10.releng.scl3.mozilla.com
panda-0080.p10.releng.scl3.mozilla.com
panda-0874.p10.releng.scl3.mozilla.com
panda-0875.p10.releng.scl3.mozilla.com
panda-0876.p10.releng.scl3.mozilla.com
panda-0877.p10.releng.scl3.mozilla.com
panda-0878.p10.releng.scl3.mozilla.com
panda-0879.p10.releng.scl3.mozilla.com
panda-0880.p10.releng.scl3.mozilla.com
panda-0881.p10.releng.scl3.mozilla.com
panda-0882.p10.releng.scl3.mozilla.com
panda-0883.p10.releng.scl3.mozilla.com
panda-0884.p10.releng.scl3.mozilla.com
panda-0887.p10.releng.scl3.mozilla.com
panda-0888.p10.releng.scl3.mozilla.com
panda-0889.p10.releng.scl3.mozilla.com
panda-0890.p10.releng.scl3.mozilla.com
panda-0891.p10.releng.scl3.mozilla.com
panda-0892.p10.releng.scl3.mozilla.com
panda-0893.p10.releng.scl3.mozilla.com
panda-0894.p10.releng.scl3.mozilla.com
panda-0895.p10.releng.scl3.mozilla.com
panda-0896.p10.releng.scl3.mozilla.com
panda-0897.p10.releng.scl3.mozilla.com
panda-0898.p10.releng.scl3.mozilla.com
panda-0899.p10.releng.scl3.mozilla.com
panda-0900.p10.releng.scl3.mozilla.com
panda-0901.p10.releng.scl3.mozilla.com
panda-0902.p10.releng.scl3.mozilla.com
panda-0903.p10.releng.scl3.mozilla.com
panda-0904.p10.releng.scl3.mozilla.com
panda-0905.p10.releng.scl3.mozilla.com
panda-0906.p10.releng.scl3.mozilla.com
panda-0907.p10.releng.scl3.mozilla.com
panda-0908.p10.releng.scl3.mozilla.com
panda-0909.p10.releng.scl3.mozilla.com
|
|
||
Comment 15•10 years ago
|
||
|
|
||
Comment 16•10 years ago
|
||
I cannot see a way to specify a comment via this REST API, so I am having to do it without. I also don't see a way to query the current enabled/disabled status - but maybe I missed this.
Running:
cat naughtypandas | sed 's/\..*//' | while read panda; do curl -u "${MY_LDAP_USER}:${MY_LDAP_PASSWORD}" -dfoo=bar "https://secure.pub.build.mozilla.org/slaveapi/slaves/${panda}/actions/disable"; done
(where naughtypandas is a sorted copy and paste of the list of pandas above, with blank lines removed)
|
|
||
Comment 17•10 years ago
|
||
Dustin found problem! devices.json was out of date on the foopies - needed to reconfig. On foopies we had e.g. https://hg.mozilla.org/build/tools/file/408f66b7fd6e/buildfarm/mobile/devices.json
Kim is doing a reconfig now, this should fix it.
|
|
||
Comment 18•10 years ago
|
||
We split it so I updated foopies, Mim did rest of reconfig. manage_foopies.py hung, but seems all foopies were updated, except foopy101 which has known problems (bug 1038452):
(fabric)pmoore@Elisandra:~/git/tools master $ buildfarm/maintenance/manage_foopies.py -f buildfarm/mobile/devices.json -H all -j 16 show_revision
foopy115 82fb9d827b24
foopy114 82fb9d827b24
foopy105 82fb9d827b24
foopy107 82fb9d827b24
foopy102 82fb9d827b24
foopy104 82fb9d827b24
foopy100 82fb9d827b24
foopy103 82fb9d827b24
foopy106 82fb9d827b24
foopy108 82fb9d827b24
foopy110 82fb9d827b24
foopy111 82fb9d827b24
foopy113 82fb9d827b24
foopy109 82fb9d827b24
foopy112 82fb9d827b24
foopy117 82fb9d827b24
foopy123 82fb9d827b24
foopy116 82fb9d827b24
foopy120 82fb9d827b24
foopy119 82fb9d827b24
foopy122 82fb9d827b24
foopy121 82fb9d827b24
foopy124 82fb9d827b24
foopy125 82fb9d827b24
foopy126 82fb9d827b24
foopy28 82fb9d827b24
foopy27 82fb9d827b24
foopy128 82fb9d827b24
foopy29 82fb9d827b24
foopy30 82fb9d827b24
foopy31 82fb9d827b24
foopy32 82fb9d827b24
foopy39 82fb9d827b24
foopy43 82fb9d827b24
foopy41 82fb9d827b24
foopy42 82fb9d827b24
foopy44 82fb9d827b24
foopy47 82fb9d827b24
foopy45 82fb9d827b24
foopy46 82fb9d827b24
foopy40 82fb9d827b24
foopy48 82fb9d827b24
foopy49 82fb9d827b24
foopy50 82fb9d827b24
Failed to run show_revision on foopy101
Traceback (most recent call last):
File "buildfarm/maintenance/manage_foopies.py", line 68, in run_action_on_foopy
action_func(foopy)
File "/Users/pmoore/git/tools/buildfarm/maintenance/foopy_fabric.py", line 29, in show_revision
tools_rev = run('hg -R /builds/tools ident -i')
File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/network.py", line 535, in host_prompting_wrapper
return func(*args, **kwargs)
File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/operations.py", line 1017, in run
warn_only=warn_only, stdout=stdout, stderr=stderr, timeout=timeout)
File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/operations.py", line 891, in _run_command
channel=default_channel(), command=wrapped_command, pty=pty,
File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/state.py", line 380, in default_channel
chan = _open_session()
File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/state.py", line 372, in _open_session
return connections[env.host_string].get_transport().open_session()
File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/network.py", line 109, in __getitem__
self.connect(key)
File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/network.py", line 101, in connect
self[key] = connect(user, host, port, sock)
File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/network.py", line 461, in connect
raise NetworkError(msg, e)
NetworkError: Timed out trying to connect to foopy101.build.mozilla.org (tried 1 time)
foopy101 FAILED
foopy51 82fb9d827b24
foopy52 82fb9d827b24
foopy53 82fb9d827b24
foopy54 82fb9d827b24
foopy57 82fb9d827b24
foopy56 82fb9d827b24
foopy55 82fb9d827b24
foopy60 82fb9d827b24
foopy58 82fb9d827b24
foopy59 82fb9d827b24
foopy61 82fb9d827b24
foopy63 82fb9d827b24
foopy62 82fb9d827b24
foopy65 82fb9d827b24
foopy66 82fb9d827b24
foopy67 82fb9d827b24
foopy68 82fb9d827b24
foopy69 82fb9d827b24
foopy70 82fb9d827b24
foopy72 82fb9d827b24
foopy74 82fb9d827b24
foopy73 82fb9d827b24
foopy76 82fb9d827b24
foopy77 82fb9d827b24
foopy75 82fb9d827b24
foopy71 82fb9d827b24
foopy79 82fb9d827b24
foopy78 82fb9d827b24
foopy80 82fb9d827b24
foopy81 82fb9d827b24
foopy64 82fb9d827b24
foopy82 82fb9d827b24
foopy83 82fb9d827b24
foopy84 82fb9d827b24
foopy85 82fb9d827b24
foopy91 82fb9d827b24
foopy89 82fb9d827b24
foopy93 82fb9d827b24
foopy86 82fb9d827b24
foopy88 82fb9d827b24
foopy92 82fb9d827b24
foopy90 82fb9d827b24
foopy95 82fb9d827b24
foopy87 82fb9d827b24
foopy94 82fb9d827b24
foopy96 82fb9d827b24
foopy97 82fb9d827b24
foopy98 82fb9d827b24
foopy99 82fb9d827b24
|
|
||
Comment 19•10 years ago
|
||
s/Mim/Kim/ (of course!)
Will now re-enable train 3 pandas in slavealloc...
|
|
||
Comment 20•10 years ago
|
||
Running from cruncher to re-enable them all:
(tools_venv)[pmoore@cruncher.srv.releng.scl3 tools]$ cat naughtypandas | while read panda; do slavealloc enable ${panda}; done
(could not run locally as firewall seemed to block)
In a couple of minutes, all should be enabled again (together with any from train 3 that were previously intentionally disabled for another reason - hopefully not).
|
|
||
Updated•10 years ago
|
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
|
||
Updated•6 years ago
|
Product: Release Engineering → Infrastructure & Operations
|
|
||
Updated•5 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•