972802 - Python crash

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Andrei Eftimie]

Attachment: python_crash_log.txt

We've had python crash on mm-osx-108-1
Attached is the python crash log.

Jenkins also lost connection to the node. The client was still up, and I was able to restore connection to master without needing to restart it.

Comment 1

[Andrei Eftimie]

Attachment: jenkins_log.txt

Also the Jenkins log is attached.
Fair to say that the testrun failed because of the Python crash.

But we also failed to send notifications via IRC and email.

From googling the exception code it seems several people had python crash like this across several OSX versions. No definitive answers.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Looks like this is a crash somewhere under PyGC_Collect. Not sure why we are running 2.7.2 on that box. We might have to check all of them and upgrade to the latest Python 2.7 release. We should do some testing on staging first. And it might be that new environments (at least for WIndows) would have to be created.

Andrei, would you mind to check which version of Python we are running on the different OS X platforms? Thanks.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Another crash on the same box. Please can you respond to Henrik's question, Andrei?

Comment 4

[Andrei Eftimie]

Attachment: python_versions.txt

We're using python from 2.7.1 up to 2.7.5
Each OSX version is using the same python version across all 4 machines.

Attached is the full list, tl;dr (both staging and production machines):
10.6 2.7.3
10.7 2.7.1
10.8 2.7.2
10.9 2.7.5

Latest stable python (2) version is now 2.7.6
So yeah, we might want to upgrade.

---

We are running an even older version on our 10.7 boxes.
There's also the possibility  that the machine itself has some problems since both times it crashed it was on the same box.

Anyway, an upgrade would still be good.
We should probably upgrade staging first to see if there are any problems.
I can take care of this.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Thanks Andrei, can you please also check the Ubuntu and Windows VMs.

Comment 6

[Andrei Eftimie]

Attachment: python_versions.txt

Updated list.

On Ubuntu we have:
12.04: 2.7.3 
13.10: 2.7.5+ (I don't know what this + means)

On Windows we bundle python with mozmill-env afaik and it's currently 2.7.3

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Please raise bugs for updating these platforms to the latest stable versions.

Comment 8

[Andrei Eftimie]

We haven't seen another crash.
We also updated Python on all OSX machines.

Please reopen if another crash happens.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I would even call this fixed now. The crash was with a kinda old version of Python, so an upgrade from 2.7.2 to 2.7.5 is a huge step.