Closed Bug 908090 Opened 11 years ago Closed 7 years ago

ImportError: No module named json, on SeaMonkey Linux/Windows (all) test runs, due to (still) using Python 2.5

Categories

(SeaMonkey :: Testing Infrastructure, defect)

defect
Not set
blocker

Tracking

(seamonkey2.20 unaffected, seamonkey2.21 unaffected, seamonkey2.22? affected, seamonkey2.23 affected)

RESOLVED FIXED
Tracking Status
seamonkey2.20 --- unaffected
seamonkey2.21 --- unaffected
seamonkey2.22 ? affected
seamonkey2.23 --- affected

People

(Reporter: mcsmurf, Assigned: ewong)

References

Details

Attachments

(1 file, 1 obsolete file)

So currently all the mochitest on Windows and Linux fail due to this error (as example Linux):
"python mochitest/runtests.py --appname=seamonkey/seamonkey-bin --utility-path=bin --extra-profile-file=bin/plugins --certificate-path=certs --autorun --close-when-done --console-level=INFO --symbols-path=symbols --chrome
[...]
Traceback (most recent call last):
  File "mochitest/runtests.py", line 23, in <module>
    from automation import Automation
  File "/builds/slave/test/build/mochitest/automation.py", line 5, in <module>
    import json
ImportError: No module named json
program finished with exit code 1
elapsedTime=0.079714
TinderboxPrint: mochitest-chrome<br/><em class="testfail">T-FAIL</em>
Unknown Error: command finished with exit code: 1"

AFAIK this happens due to mochitests using an old python version. Linux and Windows builders should already have python 2.6 or 2.7 installed, which include the json module. So just some path vars need to be fixed.
Hardware: x86_64 → All
I've looked a bit around in buildbot-config and buildbotcustom...is there no easy way to define the PATH for the mochitest test runs? I see that http://hg.mozilla.org/build/buildbotcustom/file/b9ea9639c354/env.py could be modified to set the path for the unittests, but not sure if this is the best or correct solution.
Callek: What do you think of my solution in Comment 1 to modify env.py?
Flags: needinfo?(bugspam.Callek)
I checked "Win7 debug":
all tests (B M(1 2 3 4 5 oth) R(C R) X) are affected.

Though 'X' reports a different (python) error:
{{
  File "xpcshell/runxpcshelltests.py", line 142
    except Exception as e:
                      ^
SyntaxError: invalid syntax
program finished with exit code 1
}}

It looks like all these boxes (still) use Python 2.5:
{{
  PATH=D:\mozilla-build\msys\local\bin;d:\mozilla-build\wget;d:\mozilla-build\7zip;d:\mozilla-build\blat261\full;d:\mozilla-build\python25;d:\mozilla-build\svn-win32-1.6.3\bin;d:\mozilla-build\upx203w;d:\mozilla-build\emacs-22.3\bin;d:\mozilla-build\info-zip;d:\mozilla-build\nsis-2.22;d:\mozilla-build\nsis-2.33u;d:\mozilla-build\hg;d:\mozilla-build\python25\Scripts;d:\mozilla-build\kdiff3;d:\mozilla-build\vim\vim72;d:\mozilla-build\yasm;.;D:\mozilla-build\msys\local\bin;D:\mozilla-build\msys\mingw\bin;D:\mozilla-build\msys\bin;d:\sdks\v7.0\bin;d:\msvs8\Common7\IDE;d:\msvs8\VC\BIN;d:\msvs8\Common7\Tools;d:\msvs8\Common7\Tools\bin;d:\msvs8\VC\PlatformSDK\bin;d:\msvs8\SDK\v2.0\bin;c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;d:\msvs8\VC\VCPackages;c:\WINDOWS\system32;c:\WINDOWS;c:\WINDOWS\System32\Wbem;c:\Program Files\Intel\DMIX;d:\mozilla-build\python25;d:\mercurial;c:\Program Files\Microsoft SQL Server\90\Tools\binn\;d:\sdks\tegra042\tools;d:\sdks\tegra042\platformlibs\bin\winxp\x86\release;d:\sdks\tegra042\3rdparty\bin\winxp\x86\release;d:\mozilla-build\python25\scripts;d:\mozilla-build\hg\bin;d:\mozilla-build\moztools\bin

  _=d:/mozilla-build/python25/python
}}


(In reply to Frank Wein [:mcsmurf] (afk for a few days) from comment #2)
> Callek: What do you think of my solution in Comment 1 to modify env.py?

Do SeaMonkey need to go that way (again)?
How did MoCo/Firefox do the upgrade?
Can't SeaMonkey boxes use a newer python version (+/-) "by default"?
Severity: major → critical
Component: Build Config → Testing Infrastructure
Summary: ImportError: No module named json on SeaMonkey Linux/Windows mochitest test runs → ImportError: No module named json, on SeaMonkey Linux/Windows (all) test runs, due to (still) using Python 2.5
Adding keyword so that it appears on tbpl
(In reply to Frank Wein [:mcsmurf] (afk for a few days) from comment #2)
> Callek: What do you think of my solution in Comment 1 to modify env.py?

I'd rather mirror what is done on MoCo side if possible, I don't have tme to check there right now though, so I'm willing to give it a shot.
Flags: needinfo?(bugspam.Callek)
I think the MoCo way these days is to run this script http://mxr.mozilla.org/build/source/mozharness/scripts/desktop_unittest.py for the tests:
"/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --cfg unittests/linux_unittest.py --mochitest-suite plain1 --download-symbols ondemand" (from a Linux mochitest 1 box).
and on a Windows mochitest 1 box:
"========= Started 'c:/mozilla-build/python27/python -u ...' (results: 0, elapsed: 24 mins, 47 secs) (at 2013-09-13 09:38:40.891918) =========
'c:/mozilla-build/python27/python' '-u' 'scripts/scripts/desktop_unittest.py' '--cfg' 'unittests/win_unittest.py' '--mochitest-suite' 'plain1' '--download-symbols' 'ondemand'"

Not sure if the MoCo boxen use absolute or relative python path for those commands. On the Windows box python 2.7 is included in the path:
PATH=C:\Program Files\NVIDIA Corporation\PhysX\Common;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\mozilla-build\python27;C:\mozilla-build\python27\Scripts;C:\mozilla-build\msys\bin;C:\mozilla-build\vim\vim72;C:\mozilla-build\wget;C:\mozilla-build\info-zip;C:\CoreUtils\bin;C:\mozilla-build\buildbotve\scripts;c:\Program Files\Microsoft Windows Performance Toolkit\;C:\mozilla-build\hg
(comparison SeaMonkey Windows PATH
PATH=D:\mozilla-build\msys\local\bin;d:\mozilla-build\wget;d:\mozilla-build\7zip;d:\mozilla-build\blat261\full;d:\mozilla-build\python25;d:\mozilla-build\svn-win32-1.6.3\bin;d:\mozilla-build\upx203w;d:\mozilla-build\emacs-22.3\bin;d:\mozilla-build\info-zip;d:\mozilla-build\nsis-2.22;d:\mozilla-build\nsis-2.33u;d:\mozilla-build\hg;d:\mozilla-build\python25\Scripts;d:\mozilla-build\kdiff3;d:\mozilla-build\vim\vim72;d:\mozilla-build\yasm;.;D:\mozilla-build\msys\local\bin;D:\mozilla-build\msys\mingw\bin;D:\mozilla-build\msys\bin;d:\sdks\v7.0\bin;d:\msvs8\Common7\IDE;d:\msvs8\VC\BIN;d:\msvs8\Common7\Tools;d:\msvs8\Common7\Tools\bin;d:\msvs8\VC\PlatformSDK\bin;d:\msvs8\SDK\v2.0\bin;c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;d:\msvs8\VC\VCPackages;c:\WINDOWS\system32;c:\WINDOWS;c:\WINDOWS\System32\Wbem;c:\Program Files\Intel\DMIX;d:\mozilla-build\python25;d:\mercurial;c:\Program Files\Microsoft SQL Server\90\Tools\binn\;d:\sdks\tegra042\tools;d:\sdks\tegra042\platformlibs\bin\winxp\x86\release;d:\sdks\tegra042\3rdparty\bin\winxp\x86\release;d:\mozilla-build\python25\scripts;d:\mozilla-build\hg\bin;d:\mozilla-build\moztools\bin)

On Linux boxen python is not in the path:
PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

Or does it find python because the context is set to /tools/buildbot/bin/?

Anyway, at least for Windows boxen I suggest to replace "d:\mozilla-build\python25" with "d:\mozilla-build\python27" in the default path, there's no reason to still have python 2.5 there.
(In reply to Frank Wein [:mcsmurf] from comment #6)

To update the user env that is used was exactly what I was assuming (MoCo did) ;-)
(And to use desktop_unittest.py would be good too, if not already the case.)
Not sure if MoCo boxen still use env.py for unittests as the file at http://mxr.mozilla.org/build/source/buildbotcustom/env.py still mentions python2.5 for win32-ref-platform and win32-unittest does not set/override any PATH.
So, now that Python has been updated on all buildboxen, it's time to look at this again. Current state of this problem on Linux and Windows is basically still the same:
Mochitest 1 log http://ftp.mozilla.org/pub/mozilla.org/seamonkey/tinderbox-builds/comm-central-trunk-win32-debug/1389083935/comm-central-trunk-win32-debug-unittest-mochitests-1-bm01-universal-build499.txt.gz
"  PATH=D:\mozilla-build\msys\local\bin;d:\mozilla-build\wget;d:\mozilla-build\7zip;d:\mozilla-build\blat261\full;d:\mozilla-build\python25;d:\mozilla-build\svn-win32-1.6.3\bin;d:\mozilla-build\upx203w;d:\mozilla-build\emacs-22.3\bin;d:\mozilla-build\info-zip;d:\mozilla-build\nsis-2.22;d:\mozilla-build\nsis-2.33u;d:\mozilla-build\hg;d:\mozilla-build\python25\Scripts;d:\mozilla-build\kdiff3;d:\mozilla-build\vim\vim72;d:\mozilla-build\yasm;.;D:\mozilla-build\msys\local\bin;D:\mozilla-build\msys\mingw\bin;D:\mozilla-build\msys\bin;d:\sdks\v7.0\bin;d:\msvs8\Common7\IDE;d:\msvs8\VC\BIN;d:\msvs8\Common7\Tools;d:\msvs8\Common7\Tools\bin;d:\msvs8\VC\PlatformSDK\bin;d:\msvs8\SDK\v2.0\bin;c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;d:\msvs8\VC\VCPackages;c:\WINDOWS\system32;c:\WINDOWS;c:\WINDOWS\System32\Wbem;c:\Program Files\Intel\DMIX;d:\mozilla-build\python25;d:\mercurial;c:\Program Files\Microsoft SQL Server\90\Tools\binn\;d:\sdks\tegra042\tools;d:\sdks\tegra042\platformlibs\bin\winxp\x86\release;d:\sdks\tegra042\3rdparty\bin\winxp\x86\release;d:\mozilla-build\python25\scripts;d:\mozilla-build\hg\bin;d:\mozilla-build\moztools\bin
[...]
ImportError: No module named json"

Linux mochitest 1 log:
"  PATH=/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/home/seabld/bin
[...]
ImportError: No module named json"

As already said, I'm not sure where the "python25" part on the Windows testbox is coming from, on Linux the problem might be a wrong symbol link in "/tools/python/bin".
Conclusions on the Linux boxen from IRC:
/tools/python points to /tools/python-2.5.1 which is why it is using the old Python version. Changing the symbol link to /tools/python27 should be fine in my opinion, but I don't know everything about these systems. CentOS system depends on Python 2.4 in some way (especially depending on the installed modules), but /usr/bin/python would not be changed. Python 2.7 should be backwards compatible regarding Python 2.x.
Ok.. here's what I've 'done'.

I traced through the buildbotcustom code and after a bit of tracing, I came across 
http://hg.mozilla.org/build/buildbotcustom/file/b9ea9639c354/process/factory.py#l6775

which then I arrived here:

http://hg.mozilla.org/build/buildbotcustom/file/b9ea9639c354/misc.py#l454.

Looking all over the place, I couldn't find in our buildbot-configs where we actually set
unittest-env for each platform.

At first, I fudged it with a modification to http://hg.mozilla.org/build/buildbotcustom/file/b9ea9639c354/env.py#l81 by:
    80     "MOZ_HIDE_RESULTS_TABLE": '1'
    81 }

to
    80     "MOZ_HIDE_RESULTS_TABLE": '1',
    81     "PATH": '/tools/python-2.7.3/bin' 
    82 }

then reconfig'd the master and ran the /Linux comm-central-trunk debug test mochitests-1/5,
and that put us squarely in bug 853720.  So the JSON error is gone.

So the solution that I've come up with is to add unittest-env to whatever platform
that needs the mochitests done. i.e. macosx64-debug and win32-debug.

I added:
            'unittest-env': {
                'MOZ_OBJDIR': OBJDIR,
                'DISPLAY': ':2',
                'LD_LIBRARY_PATH': '%s/mozilla/dist/bin' % OBJDIR,
                'XPCOM_DEBUG_BREAK': 'stack-and-abort',
                'MOZ_CRASHREPORTER_NO_REPORT': '1',
                'CCACHE_DIR': '/builds/ccache',
                'CCACHE_COMPRESS': '1',
                'CCACHE_UMASK': '002',
                'PATH': '/tools/python-2.7.3/bin:/tools/python-2.7.2/bin:/tools/python-2.6.5/bin:${PATH}',
            },

to linux-debug's section in config.py. (It's basically a c&p of 'env')

Now why we didn't have unittest-env set.. I don't know.  There's probably a better solution to this.

Callek?
Flags: needinfo?(bugspam.Callek)
Depends on: 919126
Attached patch proposed patch (v1) (obsolete) — Splinter Review
Assignee: nobody → ewong
Status: NEW → ASSIGNED
Attachment #8358708 - Flags: review?(bugspam.Callek)
Please note that the current tests are working because I had manually applied the patch to the config.py and reconfig'd the master.  However, I reverted my changes 
because the proper way of doing it is to actually push the attached patch
to the repo.

So while it's working right now (because I haven't reconfig'd the master), it's
going to go back to the same error because I'll be doing the 2.24b1 patch
first.  Once I reconfig the master for the 2.24b1 patch, this patch (or another 
patch depending on what Callek says) will need to be pushed and the master
reconfig'd in order to have the unittests go back to complaining about
mozcrash.

This is mainly a heads up for everyone, just in case someone wonders why 
the unittests are back to complaining about JSON.
Bah.

Was going over the code and the buildbot configs as to why without the
patch, t's WFM now.  Then it hit me.

Basically, all our boxes are now using Python 2.7+ (and not the Python
2.5 when this bug was filed).  Ergo, this is no longer a problem as
JSON is included with 2.7

And so, I'm closing this as WFM.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Flags: needinfo?(bugspam.Callek)
Resolution: --- → WORKSFORME
Attachment #8358708 - Attachment is obsolete: true
Attachment #8358708 - Flags: review?(bugspam.Callek)
While fighting the buildbotcustom code to make Mock work, I came across
the Windows c-b tests that were failing.  While we *do* have Python2.7 installed,
the unittests NEED the patch.  

I'm reopening this and resurrecting the patch from the graveyard.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Status: REOPENED → ASSIGNED
Attachment #8358708 - Attachment is obsolete: false
Attachment #8358708 - Flags: review?(bugspam.Callek)
Comment on attachment 8358708 [details] [diff] [review]
proposed patch (v1)

Review of attachment 8358708 [details] [diff] [review]:
-----------------------------------------------------------------

not the approach I want.

We should do something like: http://hg.mozilla.org/build/buildbotcustom/rev/70e92b0482f4

Note the current version of that is http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#5383
Attachment #8358708 - Flags: review?(bugspam.Callek) → review-
Blocks: 1072713
Attachment #8358708 - Attachment is obsolete: true
Attachment #8577734 - Flags: review?(bugspam.Callek)
Removing keyword, since it's intended for test failures that are seen on Treeherder (and otherwise you'll end up with bugs being marked as worksforme if no occurrences are seen on Treeherder).
Comment on attachment 8577734 [details] [diff] [review]
proposed patch (v2)

Fixed in bug 1072713
Attachment #8577734 - Flags: review?(bugspam.Callek)
Status: ASSIGNED → RESOLVED
Closed: 10 years ago7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.