during my smoketest, I worked on the pandas when they had names like: panda-032.build.scl1.mozilla.com we reimaged the machines (with the same image) and changed the name to: panda-0032.build.scl1.mozilla.com I don't know what else what changed in the last month, but this is the one thing I do know. I also know that the name is 33 characters instead of 32...that could be a boundary that we don't understand with the panda boards.
I've modified inventory and the A and PTR record to say pan-32.build.scl1.mozilla.com. Give it 15-20m to update everywhere.
I am done debugging, my hunch was incorrect, but I couldn't have made the progress I did on stabilizing the pandas without this...thank you very much. We can put this back to panda-0032 when you can work it into your schedule.
Hostname changed back in inventory and dns.