837983 - Need simplejson or Python 2.7 on fedora test slaves

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

As part of getting mochitests running with mozbase components, we need access to either simplejson or Python 2.7 on fedora test slaves.

A test try run that tries to import simplejson or json (from Python 2.7) passes on all platforms except fedora 32/64, on which it fails:

Traceback (most recent call last):
  File "mochitest/runtests.py", line 32, in <module>
    import simplejson as json
ImportError: No module named simplejson

https://tbpl.mozilla.org/?tree=Try&rev=17b17c573706

It looks like this was already done as part of bug 584947 for Python 2.5, but I believe the slaves use Python 2.6 now, and apparently simplejson wasn't installed for that version of Python on all platforms.

Note that switching to Python 2.7 on fedora would solve this issue equally well.

If we make mozharness a requirement for this work, then this issue is less important, since we can install simplejson from puppetagain in that environment.

Comment 1

[Gregory Szorc [:gps]]

Switching to Python 2.7 instead of merely installing simplejson is highly preferred since we are standardizing on Python 2.7 everywhere.

Comment 2

[Jeff Hammel]

> If we make mozharness a requirement for this work, then this issue is less important, since we can install simplejson from puppetagain in that environment.

Specifically, using pip (etc) via mozharness's virtualenv front end from http://puppetagain.pub.build.mozilla.org/data/python/packages/ using --find-links.

Comment 3

[Jeff Hammel]

(In reply to Gregory Szorc [:gps] from comment #1)
> Switching to Python 2.7 instead of merely installing simplejson is highly
> preferred since we are standardizing on Python 2.7 everywhere.

+1. We want this anyway and the more effort we can devote to forward momentum vs side-stepping the better

Comment 4

[Andrew Halberstadt [:ahal]]

(In reply to Jonathan Griffin (:jgriffin) from comment #0)
> It looks like this was already done as part of bug 584947 for Python 2.5,
> but I believe the slaves use Python 2.6 now, and apparently simplejson
> wasn't installed for that version of Python on all platforms.

The python json module was included in Python 2.6, so they must be running 2.5.  Not that it matters either way, we still want 2.7

Comment 5

[Rail Aliiev [:rail]]

FTR, Fedora slaves use Python 2.5 installed to cltbld's $HOME directory (!) because ~/bin is the first directory in PATH. Python 2.6 is avalable as well (/usr/bin/python).

Comment 6

[Andrew Halberstadt [:ahal]]

Is there a way to explicitly set the python path via buildbot config?

Comment 7

[Rail Aliiev [:rail]]

I'd better remove ~/bin from PATH. Buildbot itself is run via /tools/buildbot-0.8.4-pre-moz2/bin/python (2.6.2).

[cltbld@talos-r3-fed64-002 ~]$ ls -la ~/bin
total 14504
drwxr-xr-x  2 cltbld cltbld     4096 2012-05-14 09:47 .
drwxr-xr-x 39 cltbld cltbld     4096 2013-02-05 07:23 ..
-rwxrwxr-x  1 cltbld cltbld       98 2010-01-14 17:06 idle
-rwxrwxr-x  1 cltbld cltbld      634 2010-01-19 23:37 manhole
-rwxrwxr-x  1 cltbld cltbld      648 2010-01-19 23:37 mktap
-rwxr-xr-x  1 cltbld cltbld 10423144 2012-05-14 09:47 node.exe
-rwxrwxr-x  1 cltbld cltbld       83 2010-01-14 17:06 pydoc
lrwxrwxrwx  1 cltbld cltbld        9 2010-01-19 23:35 python -> python2.5
-rwxr-xr-x  1 cltbld cltbld  4351665 2010-01-14 17:07 python2.5
-rwxr-xr-x  1 cltbld cltbld     1426 2010-01-14 17:08 python2.5-config
lrwxrwxrwx  1 cltbld cltbld       16 2010-01-19 23:35 python-config -> python2.5-config
-rwxrwxr-x  1 cltbld cltbld    18053 2010-01-14 17:06 smtpd.py
-rwxrwxr-x  1 cltbld cltbld      582 2010-01-19 23:37 tap2deb
-rwxrwxr-x  1 cltbld cltbld      544 2010-01-19 23:37 tap2rpm
-rwxrwxr-x  1 cltbld cltbld      652 2010-01-19 23:37 tapconvert
-rwxrwxr-x  1 cltbld cltbld      564 2010-01-19 23:37 tkmktap
-rwxrwxr-x  1 cltbld cltbld      727 2010-01-19 23:37 trial
-rwxrwxr-x  1 cltbld cltbld      794 2010-01-19 23:37 twistd

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Rail Aliiev [:rail] from comment #7)
> I'd better remove ~/bin from PATH. Buildbot itself is run via
> /tools/buildbot-0.8.4-pre-moz2/bin/python (2.6.2).
> 
> [cltbld@talos-r3-fed64-002 ~]$ ls -la ~/bin
...
> -rwxr-xr-x  1 cltbld cltbld 10423144 2012-05-14 09:47 node.exe

Is this available elsewhere ? Perhaps puppet should be putting it into /usr/local/bin/ or somewhere in /tools which is on the PATH.

Comment 9

[Rail Aliiev [:rail]]

Yes, node.exe isn't used anymore, see bug 831771. It'd be safe to remove it.

Comment 10

[Rail Aliiev [:rail]]

https://bugzilla.mozilla.org/attachment.cgi?id=706429&action=edit to be precise

Comment 11

[cmtalbert]

We need to solve this sooner rather than later. I don't care how we solve this, only that it happens before it blocks us from doing mozbase refactoring this quarter (we need this by the end of next week). I see that you have solutions:
* Change the slaves so that they use 2.7 for everything (and install 2.7 if it isn't already)
* Land desktop unit tests in mozharness
* Use AWS Ubuntu for all fedora slaves

Do we know if this is going to be a problem on windows as well?

Comment 12

[:aceman]

We seem to recently get this on Thunderbird-trunk https://tbpl.mozilla.org/?tree=Thunderbird-Trunk. Can it be moved forward?

Comment 13

[Mark Banner (:standard8)]

(In reply to :aceman from comment #12)
> We seem to recently get this on Thunderbird-trunk

Ah, so either we need to get the Thunderbird builders on Ubuntu, or we need to get fedora upgrade to 2.7.

I suspect this bug is now "fixed" for Firefox.

Comment 14

[Mark Banner (:standard8)]

I've branched out the Thunderbird requirement into bug 887716, as it is unclear if this bug is still valid for FF (I think its obsolete now).

Comment 15

[Chris AtLee [:catlee]]

We have simplejson on these slaves now (from bug 887716). These machines aren't used for much any more, and we're actively trying to move tests off of them.