Created attachment 8376159 [details] python_crash_log.txt We've had python crash on mm-osx-108-1 Attached is the python crash log. Jenkins also lost connection to the node. The client was still up, and I was able to restore connection to master without needing to restart it.
Created attachment 8376163 [details] jenkins_log.txt Also the Jenkins log is attached. Fair to say that the testrun failed because of the Python crash. But we also failed to send notifications via IRC and email. From googling the exception code it seems several people had python crash like this across several OSX versions. No definitive answers.
Looks like this is a crash somewhere under PyGC_Collect. Not sure why we are running 2.7.2 on that box. We might have to check all of them and upgrade to the latest Python 2.7 release. We should do some testing on staging first. And it might be that new environments (at least for WIndows) would have to be created. Andrei, would you mind to check which version of Python we are running on the different OS X platforms? Thanks.
Another crash on the same box. Please can you respond to Henrik's question, Andrei?
Created attachment 8382809 [details] python_versions.txt We're using python from 2.7.1 up to 2.7.5 Each OSX version is using the same python version across all 4 machines. Attached is the full list, tl;dr (both staging and production machines): 10.6 2.7.3 10.7 2.7.1 10.8 2.7.2 10.9 2.7.5 Latest stable python (2) version is now 2.7.6 So yeah, we might want to upgrade. --- We are running an even older version on our 10.7 boxes. There's also the possibility that the machine itself has some problems since both times it crashed it was on the same box. Anyway, an upgrade would still be good. We should probably upgrade staging first to see if there are any problems. I can take care of this.
Thanks Andrei, can you please also check the Ubuntu and Windows VMs.
Created attachment 8383532 [details] python_versions.txt Updated list. On Ubuntu we have: 12.04: 2.7.3 13.10: 2.7.5+ (I don't know what this + means) On Windows we bundle python with mozmill-env afaik and it's currently 2.7.3
Please raise bugs for updating these platforms to the latest stable versions.
We haven't seen another crash. We also updated Python on all OSX machines. Please reopen if another crash happens.
I would even call this fixed now. The crash was with a kinda old version of Python, so an upgrade from 2.7.2 to 2.7.5 is a huge step.