Closed Bug 1162576 Opened 9 years ago Closed 8 years ago

Problems with pandas after power work on May 6

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: selenamarie, Unassigned)

References

(
URL
)

Details

We've got quite a few pandas not coming back after the power outages last night/this morning. 

This is to track down and restart systems.
Depends on: 1162607
Component: Other → Buildduty
QA Contact: mshal → bugspam.Callek
We had a few stale lockfiles. Cleanup documented below: 

Script started on Thu 07 May 2015 10:50:23 AM PDT
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 102 108`; do echo foopy$i ; ssh root@foopy$i 'ps auxwww | grep buildbot | grep -v grep | wc -l ' ; done
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2P39 80[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2@102 108[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2P39 80[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2@102 108[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[Kfor i in `seq 102 108`; do echo foopy$i ; ssh root@foopy$i 'ps auxwww | grep buildbot | grep -v grep | wc -l ' ; done
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[Kfor i in `seq 39 44`; do echo foopy$i ; ssh root@foopy$i 'ps auxwww | grep buildbot | grep -v grep | wc -l ' ; done
foopy39
0
foopy40
0
foopy41
0
foopy42
0
foopy43
0
foopy44
0
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 39 44`; do echo foopy$i ; ssh root@foopy$i 'ps auxwww | grep buildbot | grep -v grep | wc -l ' ; done[C[C[C[C[C[C[C[C[2P[3P[2P[5P[3P[5P[2P[9P[5P[2P[7P[1P[1P[1P[1@c[1@d[C[1@ [1@/[1@b[1@u[1@i[1@l[1@d[1@s[1@;[C[1@ [1@l[1@s[C[1@ [1@|[C[1@ [1@g[1@r[1@e[1@p[C[1@ [1@p[1@a[1@n[1@d[1@a[1@s[1P[C[1@ [1@|[C[1@ [1@a[1@w[1@k[C[1@ [1P[1P[1P[1P[1@x[1@a[1@r[1@g[1@s[C[1@ [1@-[1@I[1@x[1P[1@X[1@X[C[1@ [1@l[1@s[C[1@ [1@X[1@X[1@/[1@w[1@a[1@t[1@c[1@h[1@e[1@r[1@.[1@l[1@o[1@g[1P[1P[1P[1@l[1@o[1@c[1@k
foopy39
panda-0082/watcher.lock
panda-0083/watcher.lock
panda-0084/watcher.lock
panda-0085/watcher.lock
panda-0086/watcher.lock
panda-0087/watcher.lock
panda-0088/watcher.lock
panda-0089/watcher.lock
panda-0090/watcher.lock
panda-0092/watcher.lock
panda-0093/watcher.lock
panda-0094/watcher.lock
panda-0610/watcher.lock
foopy40
panda-0096/watcher.lock
panda-0097/watcher.lock
panda-0098/watcher.lock
panda-0099/watcher.lock
panda-0100/watcher.lock
panda-0101/watcher.lock
panda-0102/watcher.lock
panda-0103/watcher.lock
panda-0104/watcher.lock
panda-0105/watcher.lock
panda-0106/watcher.lock
panda-0611/watcher.lock
panda-0612/watcher.lock
foopy41
panda-0108/watcher.lock
panda-0109/watcher.lock
panda-0110/watcher.lock
panda-0111/watcher.lock
panda-0112/watcher.lock
panda-0113/watcher.lock
panda-0114/watcher.lock
panda-0115/watcher.lock
panda-0116/watcher.lock
panda-0117/watcher.lock
panda-0118/watcher.lock
panda-0119/watcher.lock
panda-0120/watcher.lock
foopy42
panda-0121/watcher.lock
panda-0122/watcher.lock
panda-0123/watcher.lock
panda-0124/watcher.lock
panda-0125/watcher.lock
panda-0127/watcher.lock
panda-0128/watcher.lock
panda-0130/watcher.lock
panda-0131/watcher.lock
panda-0133/watcher.lock
panda-0613/watcher.lock
panda-0614/watcher.lock
foopy43
panda-0134/watcher.lock
panda-0135/watcher.lock
panda-0136/watcher.lock
panda-0137/watcher.lock
panda-0138/watcher.lock
panda-0139/watcher.lock
panda-0140/watcher.lock
panda-0141/watcher.lock
panda-0142/watcher.lock
panda-0143/watcher.lock
panda-0144/watcher.lock
panda-0145/watcher.lock
panda-0146/watcher.lock
foopy44
panda-0147/watcher.lock
panda-0148/watcher.lock
panda-0149/watcher.lock
panda-0150/watcher.lock
panda-0151/watcher.lock
panda-0152/watcher.lock
panda-0153/watcher.lock
panda-0154/watcher.lock
panda-0155/watcher.lock
panda-0156/watcher.lock
panda-0158/watcher.lock
panda-0159/watcher.lock
panda-0615/watcher.lock
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 39 44`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX ls XX/watcher.lock ' ; done[1P[1P[1@r[1@m[C[1@ [1@-[1@r[1@f
foopy39
foopy40
foopy41
foopy42
foopy43
foopy44
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 39 44`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock ' ; done[4Pls[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[11Pps auxwww | grep buildbot | grep -v grep | wc -l[C[C[C[C[C[C[C[C[C[11@cd /builds; ls | grep panda | xargs -IXX ls XX/watcher.lock[C[C[C[C[C[C[C[C[C[4@rm -rf[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[Kfor i in `seq 39 44`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock ' ; done[4Pls[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[11Pps auxwww | grep buildbot | grep -v grep | wc -l[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2@102 108[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2P39 80[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2@102 108[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2P39 44[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[1P[1P[1P[1P[1P[1@5[1@3[1@ [1@5[1@6
foopy53
0
foopy54
0
foopy55
0
foopy56
0
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 53 56`; do echo foopy$i ; ssh root@foopy$i 'ps auxwww | grep buildbot | grep -v grep | wc -l ' ; done
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[15@39 44`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock[C[C[C[C[C[C[C[C[C[4Pls[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[1P[1P[1P[1P[1P[1@5[1@3[1@ [1@5[1@6
foopy53
panda-0258/watcher.lock
panda-0259/watcher.lock
panda-0260/watcher.lock
panda-0261/watcher.lock
panda-0262/watcher.lock
panda-0263/watcher.lock
panda-0264/watcher.lock
panda-0265/watcher.lock
panda-0266/watcher.lock
panda-0267/watcher.lock
panda-0268/watcher.lock
panda-0269/watcher.lock
foopy54
panda-0270/watcher.lock
panda-0271/watcher.lock
panda-0272/watcher.lock
panda-0273/watcher.lock
panda-0274/watcher.lock
panda-0275/watcher.lock
panda-0276/watcher.lock
panda-0277/watcher.lock
panda-0278/watcher.lock
panda-0279/watcher.lock
panda-0280/watcher.lock
panda-0281/watcher.lock
foopy55
panda-0282/watcher.lock
panda-0283/watcher.lock
panda-0284/watcher.lock
panda-0285/watcher.lock
panda-0286/watcher.lock
panda-0287/watcher.lock
panda-0288/watcher.lock
panda-0289/watcher.lock
panda-0290/watcher.lock
panda-0291/watcher.lock
panda-0292/watcher.lock
panda-0293/watcher.lock
foopy56
panda-0295/watcher.lock
panda-0296/watcher.lock
panda-0297/watcher.lock
panda-0298/watcher.lock
panda-0299/watcher.lock
panda-0300/watcher.lock
panda-0301/watcher.lock
panda-0303/watcher.lock
panda-0304/watcher.lock
panda-0305/watcher.lock
panda-0620/watcher.lock
panda-0622/watcher.lock
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 53 56`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX ls XX/watcher.lock ' ; done[11Pps auxwww | grep buildbot | grep -v grep | wc -l[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[15@39 44`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[1P[1P[1P[1P[1P[1@5[1@3[1@ [1@5[1@6
foopy53
foopy54
foopy55
foopy56
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 53 56`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock ' ; done[4Pls[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[11Pps auxwww | grep buildbot | grep -v grep | wc -l[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[15@39 44`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[15P53 56`; do echo foopy$i ; ssh root@foopy$i 'ps auxwww | grep buildbot | grep -v grep | wc -l[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[15@39 44`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock[C[C[C[C[C[C[C[C[C[4Pls[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[11Pps auxwww | grep buildbot | grep -v grep | wc -l[C[C[C[C[C[C[C[C[C
foopy39
0
foopy40
0
foopy41
0
foopy42
0
foopy43
0
foopy44
0
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 39 44`; do echo foopy$i ; ssh root@foopy$i 'ps auxwww | grep buildbot | grep -v grep | wc -l ' ; done[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[1P[1P[1@5[1@8[C[C[C[1P[1P[1@5[1@9
foopy58
0
foopy59
0
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 58 59`; do echo foopy$i ; ssh root@foopy$i 'ps auxwww | grep buildbot | grep -v grep | wc -l ' ; done
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C39 44[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[15@53 56`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock[C[C[C[C[C[C[C[C[C[4Pls[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[1P[1P[1P[1P[1P[1@5[1@8[1@ [1@5[1@9
foopy58
panda-0320/watcher.lock
panda-0321/watcher.lock
panda-0322/watcher.lock
panda-0323/watcher.lock
panda-0324/watcher.lock
panda-0325/watcher.lock
panda-0326/watcher.lock
panda-0327/watcher.lock
panda-0328/watcher.lock
panda-0329/watcher.lock
panda-0331/watcher.lock
panda-0332/watcher.lock
panda-0623/watcher.lock
foopy59
panda-0333/watcher.lock
panda-0334/watcher.lock
panda-0335/watcher.lock
panda-0336/watcher.lock
panda-0338/watcher.lock
panda-0339/watcher.lock
panda-0340/watcher.lock
panda-0341/watcher.lock
panda-0342/watcher.lock
panda-0343/watcher.lock
panda-0345/watcher.lock
panda-0624/watcher.lock
panda-0625/watcher.lock
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 58 59`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX ls XX/watcher.lock ' ; done[11Pps auxwww | grep buildbot | grep -v grep | wc -l[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C39 44[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[15@53 56`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2P[3P[1@5[1@8[1@ [1@5[1@9
foopy58
foopy59
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ ^C
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ exit

Script done on Thu 07 May 2015 10:53:26 AM PDT
oops missed a few: 

Script started on Thu 07 May 2015 10:59:14 AM PDT
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 58 59`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock ' ; done[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[4Pls[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[1P[1P[1P[1P[1@=[1P[1@6[1P[1P[1@6[1@1[1@ [1@6[1@7^C
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 58 59`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock ' ; done[4Pls[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[11Pps auxwww | grep buildbot | grep -v grep | wc -l[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C39 44[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[1P[1P[1P[1P[1P[1@6[1@1[1@ [1@6[1@7
foopy61
0
foopy62
0
foopy63
0
foopy64
0
foopy65
0
foopy66
0
foopy67
0
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 61 67`; do echo foopy$i ; ssh root@foopy$i 'ps auxwww | grep buildbot | grep -v grep | wc -l ' ; done
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[15@58 59`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock[C[C[C[C[C[C[C[C[C[4Pls[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2P[3P[1@6[1@1[1@ [1@6[1@7
foopy61
panda-0358/watcher.lock
panda-0359/watcher.lock
panda-0360/watcher.lock
panda-0361/watcher.lock
panda-0362/watcher.lock
panda-0363/watcher.lock
panda-0364/watcher.lock
panda-0365/watcher.lock
panda-0366/watcher.lock
panda-0367/watcher.lock
panda-0369/watcher.lock
foopy62
panda-0371/watcher.lock
panda-0372/watcher.lock
panda-0374/watcher.lock
panda-0375/watcher.lock
panda-0376/watcher.lock
panda-0377/watcher.lock
panda-0378/watcher.lock
panda-0379/watcher.lock
panda-0380/watcher.lock
panda-0381/watcher.lock
panda-0626/watcher.lock
panda-0627/watcher.lock
foopy63
panda-0382/watcher.lock
panda-0383/watcher.lock
panda-0384/watcher.lock
panda-0385/watcher.lock
panda-0386/watcher.lock
panda-0387/watcher.lock
panda-0388/watcher.lock
panda-0389/watcher.lock
panda-0390/watcher.lock
panda-0391/watcher.lock
panda-0392/watcher.lock
panda-0393/watcher.lock
panda-0394/watcher.lock
foopy64
panda-0396/watcher.lock
panda-0397/watcher.lock
panda-0398/watcher.lock
panda-0399/watcher.lock
panda-0400/watcher.lock
panda-0401/watcher.lock
panda-0402/watcher.lock
panda-0403/watcher.lock
panda-0404/watcher.lock
panda-0405/watcher.lock
panda-0406/watcher.lock
panda-0407/watcher.lock
foopy65
panda-0408/watcher.lock
panda-0409/watcher.lock
panda-0410/watcher.lock
panda-0411/watcher.lock
panda-0412/watcher.lock
panda-0413/watcher.lock
panda-0414/watcher.lock
panda-0415/watcher.lock
panda-0416/watcher.lock
panda-0417/watcher.lock
panda-0418/watcher.lock
panda-0419/watcher.lock
panda-0420/watcher.lock
foopy66
panda-0421/watcher.lock
panda-0422/watcher.lock
panda-0423/watcher.lock
panda-0424/watcher.lock
panda-0425/watcher.lock
panda-0426/watcher.lock
panda-0427/watcher.lock
panda-0428/watcher.lock
panda-0429/watcher.lock
panda-0430/watcher.lock
panda-0431/watcher.lock
panda-0432/watcher.lock
panda-0433/watcher.lock
foopy67
panda-0434/watcher.lock
panda-0435/watcher.lock
panda-0436/watcher.lock
panda-0437/watcher.lock
panda-0438/watcher.lock
panda-0439/watcher.lock
panda-0440/watcher.lock
panda-0441/watcher.lock
panda-0442/watcher.lock
panda-0443/watcher.lock
panda-0445/watcher.lock
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 61 67`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX ls XX/watcher.lock ' ; done[11Pps auxwww | grep buildbot | grep -v grep | wc -l[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[15@58 59`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2P[3P[1@6[1@1[1@ [1@6[1@7
foopy61
foopy62
foopy63
foopy64
foopy65
foopy66
foopy67
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 61 67`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock ' ; done[4Pls[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[11Pps auxwww | grep buildbot | grep -v grep | wc -l[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2P[3P[1@7[1@0[1@ [1@7[1@3
foopy70
0
foopy71
0
foopy72
0
foopy73
0
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 70 73`; do echo foopy$i ; ssh root@foopy$i 'ps auxwww | grep buildbot | grep -v grep | wc -l ' ; done
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[15@61 67`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock[C[C[C[C[C[C[C[C[C[4Pls[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2P[3P[1@7[1@0[1@ [1@7[1@3
foopy70
panda-0470/watcher.lock
panda-0471/watcher.lock
panda-0472/watcher.lock
panda-0473/watcher.lock
panda-0474/watcher.lock
panda-0475/watcher.lock
panda-0477/watcher.lock
panda-0478/watcher.lock
panda-0479/watcher.lock
panda-0480/watcher.lock
panda-0481/watcher.lock
panda-0482/watcher.lock
panda-0629/watcher.lock
foopy71
panda-0081/watcher.lock
panda-0483/watcher.lock
panda-0484/watcher.lock
panda-0485/watcher.lock
panda-0486/watcher.lock
panda-0487/watcher.lock
panda-0488/watcher.lock
panda-0491/watcher.lock
panda-0492/watcher.lock
panda-0493/watcher.lock
panda-0494/watcher.lock
panda-0495/watcher.lock
panda-0630/watcher.lock
foopy72
panda-0496/watcher.lock
panda-0497/watcher.lock
panda-0498/watcher.lock
panda-0499/watcher.lock
panda-0500/watcher.lock
panda-0502/watcher.lock
panda-0503/watcher.lock
panda-0504/watcher.lock
panda-0505/watcher.lock
panda-0506/watcher.lock
panda-0507/watcher.lock
panda-0508/watcher.lock
foopy73
panda-0509/watcher.lock
panda-0510/watcher.lock
panda-0511/watcher.lock
panda-0512/watcher.lock
panda-0513/watcher.lock
panda-0514/watcher.lock
panda-0515/watcher.lock
panda-0516/watcher.lock
panda-0517/watcher.lock
panda-0518/watcher.lock
panda-0519/watcher.lock
panda-0520/watcher.lock
panda-0521/watcher.lock
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ for i in `seq 70 73`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX ls XX/watcher.lock ' ; done[11Pps auxwww | grep buildbot | grep -v grep | wc -l[C[C[C[C[C[C[C[C[C
[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[15@61 67`; do echo foopy$i ; ssh root@foopy$i 'cd /builds; ls | grep panda | xargs -IXX rm -rf XX/watcher.lock[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[2P[3P[1@7[1@0[1@ [1@7[1@3
foopy70
foopy71
foopy72
foopy73
]0;selena@taiping: ~/repos/buildselena@taiping:~/repos/build$ exit

Script done on Thu 07 May 2015 11:02:34 AM PDT
Problem was that we were using @reboot to clean up stale lockfiles, and upon testing, the foopys are not honoring our request to remove the lockfiles upon reboot.

Currently discussing a remedy.
Component: Buildduty → General Automation
QA Contact: bugspam.Callek → catlee
(In reply to Selena Deckelmann :selenamarie :selena from comment #3)
> Problem was that we were using @reboot to clean up stale lockfiles, and upon
> testing, the foopys are not honoring our request to remove the lockfiles
> upon reboot.
> 
> Currently discussing a remedy.

We are no longer using foopies as of today, so wontfix
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.