Closed
Bug 908090
Opened 11 years ago
Closed 7 years ago
ImportError: No module named json, on SeaMonkey Linux/Windows (all) test runs, due to (still) using Python 2.5
Categories
(SeaMonkey :: Testing Infrastructure, defect)
SeaMonkey
Testing Infrastructure
Tracking
(seamonkey2.20 unaffected, seamonkey2.21 unaffected, seamonkey2.22? affected, seamonkey2.23 affected)
RESOLVED
FIXED
Tracking | Status | |
---|---|---|
seamonkey2.20 | --- | unaffected |
seamonkey2.21 | --- | unaffected |
seamonkey2.22 | ? | affected |
seamonkey2.23 | --- | affected |
People
(Reporter: mcsmurf, Assigned: ewong)
References
Details
Attachments
(1 file, 1 obsolete file)
1.90 KB,
patch
|
Details | Diff | Splinter Review |
So currently all the mochitest on Windows and Linux fail due to this error (as example Linux): "python mochitest/runtests.py --appname=seamonkey/seamonkey-bin --utility-path=bin --extra-profile-file=bin/plugins --certificate-path=certs --autorun --close-when-done --console-level=INFO --symbols-path=symbols --chrome [...] Traceback (most recent call last): File "mochitest/runtests.py", line 23, in <module> from automation import Automation File "/builds/slave/test/build/mochitest/automation.py", line 5, in <module> import json ImportError: No module named json program finished with exit code 1 elapsedTime=0.079714 TinderboxPrint: mochitest-chrome<br/><em class="testfail">T-FAIL</em> Unknown Error: command finished with exit code: 1" AFAIK this happens due to mochitests using an old python version. Linux and Windows builders should already have python 2.6 or 2.7 installed, which include the json module. So just some path vars need to be fixed.
Reporter | ||
Updated•11 years ago
|
Hardware: x86_64 → All
Reporter | ||
Comment 1•11 years ago
|
||
I've looked a bit around in buildbot-config and buildbotcustom...is there no easy way to define the PATH for the mochitest test runs? I see that http://hg.mozilla.org/build/buildbotcustom/file/b9ea9639c354/env.py could be modified to set the path for the unittests, but not sure if this is the best or correct solution.
Reporter | ||
Comment 2•11 years ago
|
||
Callek: What do you think of my solution in Comment 1 to modify env.py?
Flags: needinfo?(bugspam.Callek)
Comment 3•11 years ago
|
||
I checked "Win7 debug": all tests (B M(1 2 3 4 5 oth) R(C R) X) are affected. Though 'X' reports a different (python) error: {{ File "xpcshell/runxpcshelltests.py", line 142 except Exception as e: ^ SyntaxError: invalid syntax program finished with exit code 1 }} It looks like all these boxes (still) use Python 2.5: {{ PATH=D:\mozilla-build\msys\local\bin;d:\mozilla-build\wget;d:\mozilla-build\7zip;d:\mozilla-build\blat261\full;d:\mozilla-build\python25;d:\mozilla-build\svn-win32-1.6.3\bin;d:\mozilla-build\upx203w;d:\mozilla-build\emacs-22.3\bin;d:\mozilla-build\info-zip;d:\mozilla-build\nsis-2.22;d:\mozilla-build\nsis-2.33u;d:\mozilla-build\hg;d:\mozilla-build\python25\Scripts;d:\mozilla-build\kdiff3;d:\mozilla-build\vim\vim72;d:\mozilla-build\yasm;.;D:\mozilla-build\msys\local\bin;D:\mozilla-build\msys\mingw\bin;D:\mozilla-build\msys\bin;d:\sdks\v7.0\bin;d:\msvs8\Common7\IDE;d:\msvs8\VC\BIN;d:\msvs8\Common7\Tools;d:\msvs8\Common7\Tools\bin;d:\msvs8\VC\PlatformSDK\bin;d:\msvs8\SDK\v2.0\bin;c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;d:\msvs8\VC\VCPackages;c:\WINDOWS\system32;c:\WINDOWS;c:\WINDOWS\System32\Wbem;c:\Program Files\Intel\DMIX;d:\mozilla-build\python25;d:\mercurial;c:\Program Files\Microsoft SQL Server\90\Tools\binn\;d:\sdks\tegra042\tools;d:\sdks\tegra042\platformlibs\bin\winxp\x86\release;d:\sdks\tegra042\3rdparty\bin\winxp\x86\release;d:\mozilla-build\python25\scripts;d:\mozilla-build\hg\bin;d:\mozilla-build\moztools\bin _=d:/mozilla-build/python25/python }} (In reply to Frank Wein [:mcsmurf] (afk for a few days) from comment #2) > Callek: What do you think of my solution in Comment 1 to modify env.py? Do SeaMonkey need to go that way (again)? How did MoCo/Firefox do the upgrade? Can't SeaMonkey boxes use a newer python version (+/-) "by default"?
Severity: major → critical
Component: Build Config → Testing Infrastructure
Keywords: intermittent-failure
Summary: ImportError: No module named json on SeaMonkey Linux/Windows mochitest test runs → ImportError: No module named json, on SeaMonkey Linux/Windows (all) test runs, due to (still) using Python 2.5
Reporter | ||
Comment 4•11 years ago
|
||
Adding keyword so that it appears on tbpl
Keywords: intermittent-failure
Comment 5•11 years ago
|
||
(In reply to Frank Wein [:mcsmurf] (afk for a few days) from comment #2) > Callek: What do you think of my solution in Comment 1 to modify env.py? I'd rather mirror what is done on MoCo side if possible, I don't have tme to check there right now though, so I'm willing to give it a shot.
Flags: needinfo?(bugspam.Callek)
Reporter | ||
Comment 6•11 years ago
|
||
I think the MoCo way these days is to run this script http://mxr.mozilla.org/build/source/mozharness/scripts/desktop_unittest.py for the tests: "/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --cfg unittests/linux_unittest.py --mochitest-suite plain1 --download-symbols ondemand" (from a Linux mochitest 1 box). and on a Windows mochitest 1 box: "========= Started 'c:/mozilla-build/python27/python -u ...' (results: 0, elapsed: 24 mins, 47 secs) (at 2013-09-13 09:38:40.891918) ========= 'c:/mozilla-build/python27/python' '-u' 'scripts/scripts/desktop_unittest.py' '--cfg' 'unittests/win_unittest.py' '--mochitest-suite' 'plain1' '--download-symbols' 'ondemand'" Not sure if the MoCo boxen use absolute or relative python path for those commands. On the Windows box python 2.7 is included in the path: PATH=C:\Program Files\NVIDIA Corporation\PhysX\Common;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\mozilla-build\python27;C:\mozilla-build\python27\Scripts;C:\mozilla-build\msys\bin;C:\mozilla-build\vim\vim72;C:\mozilla-build\wget;C:\mozilla-build\info-zip;C:\CoreUtils\bin;C:\mozilla-build\buildbotve\scripts;c:\Program Files\Microsoft Windows Performance Toolkit\;C:\mozilla-build\hg (comparison SeaMonkey Windows PATH PATH=D:\mozilla-build\msys\local\bin;d:\mozilla-build\wget;d:\mozilla-build\7zip;d:\mozilla-build\blat261\full;d:\mozilla-build\python25;d:\mozilla-build\svn-win32-1.6.3\bin;d:\mozilla-build\upx203w;d:\mozilla-build\emacs-22.3\bin;d:\mozilla-build\info-zip;d:\mozilla-build\nsis-2.22;d:\mozilla-build\nsis-2.33u;d:\mozilla-build\hg;d:\mozilla-build\python25\Scripts;d:\mozilla-build\kdiff3;d:\mozilla-build\vim\vim72;d:\mozilla-build\yasm;.;D:\mozilla-build\msys\local\bin;D:\mozilla-build\msys\mingw\bin;D:\mozilla-build\msys\bin;d:\sdks\v7.0\bin;d:\msvs8\Common7\IDE;d:\msvs8\VC\BIN;d:\msvs8\Common7\Tools;d:\msvs8\Common7\Tools\bin;d:\msvs8\VC\PlatformSDK\bin;d:\msvs8\SDK\v2.0\bin;c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;d:\msvs8\VC\VCPackages;c:\WINDOWS\system32;c:\WINDOWS;c:\WINDOWS\System32\Wbem;c:\Program Files\Intel\DMIX;d:\mozilla-build\python25;d:\mercurial;c:\Program Files\Microsoft SQL Server\90\Tools\binn\;d:\sdks\tegra042\tools;d:\sdks\tegra042\platformlibs\bin\winxp\x86\release;d:\sdks\tegra042\3rdparty\bin\winxp\x86\release;d:\mozilla-build\python25\scripts;d:\mozilla-build\hg\bin;d:\mozilla-build\moztools\bin) On Linux boxen python is not in the path: PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games Or does it find python because the context is set to /tools/buildbot/bin/? Anyway, at least for Windows boxen I suggest to replace "d:\mozilla-build\python25" with "d:\mozilla-build\python27" in the default path, there's no reason to still have python 2.5 there.
Comment 7•11 years ago
|
||
(In reply to Frank Wein [:mcsmurf] from comment #6) To update the user env that is used was exactly what I was assuming (MoCo did) ;-) (And to use desktop_unittest.py would be good too, if not already the case.)
Severity: critical → blocker
status-seamonkey2.20:
--- → unaffected
status-seamonkey2.21:
--- → unaffected
status-seamonkey2.22:
--- → affected
status-seamonkey2.23:
--- → affected
tracking-seamonkey2.22:
--- → ?
Reporter | ||
Comment 8•11 years ago
|
||
Not sure if MoCo boxen still use env.py for unittests as the file at http://mxr.mozilla.org/build/source/buildbotcustom/env.py still mentions python2.5 for win32-ref-platform and win32-unittest does not set/override any PATH.
Reporter | ||
Comment 9•10 years ago
|
||
So, now that Python has been updated on all buildboxen, it's time to look at this again. Current state of this problem on Linux and Windows is basically still the same: Mochitest 1 log http://ftp.mozilla.org/pub/mozilla.org/seamonkey/tinderbox-builds/comm-central-trunk-win32-debug/1389083935/comm-central-trunk-win32-debug-unittest-mochitests-1-bm01-universal-build499.txt.gz " PATH=D:\mozilla-build\msys\local\bin;d:\mozilla-build\wget;d:\mozilla-build\7zip;d:\mozilla-build\blat261\full;d:\mozilla-build\python25;d:\mozilla-build\svn-win32-1.6.3\bin;d:\mozilla-build\upx203w;d:\mozilla-build\emacs-22.3\bin;d:\mozilla-build\info-zip;d:\mozilla-build\nsis-2.22;d:\mozilla-build\nsis-2.33u;d:\mozilla-build\hg;d:\mozilla-build\python25\Scripts;d:\mozilla-build\kdiff3;d:\mozilla-build\vim\vim72;d:\mozilla-build\yasm;.;D:\mozilla-build\msys\local\bin;D:\mozilla-build\msys\mingw\bin;D:\mozilla-build\msys\bin;d:\sdks\v7.0\bin;d:\msvs8\Common7\IDE;d:\msvs8\VC\BIN;d:\msvs8\Common7\Tools;d:\msvs8\Common7\Tools\bin;d:\msvs8\VC\PlatformSDK\bin;d:\msvs8\SDK\v2.0\bin;c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;d:\msvs8\VC\VCPackages;c:\WINDOWS\system32;c:\WINDOWS;c:\WINDOWS\System32\Wbem;c:\Program Files\Intel\DMIX;d:\mozilla-build\python25;d:\mercurial;c:\Program Files\Microsoft SQL Server\90\Tools\binn\;d:\sdks\tegra042\tools;d:\sdks\tegra042\platformlibs\bin\winxp\x86\release;d:\sdks\tegra042\3rdparty\bin\winxp\x86\release;d:\mozilla-build\python25\scripts;d:\mozilla-build\hg\bin;d:\mozilla-build\moztools\bin [...] ImportError: No module named json" Linux mochitest 1 log: " PATH=/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/home/seabld/bin [...] ImportError: No module named json" As already said, I'm not sure where the "python25" part on the Windows testbox is coming from, on Linux the problem might be a wrong symbol link in "/tools/python/bin".
Reporter | ||
Comment 10•10 years ago
|
||
Conclusions on the Linux boxen from IRC: /tools/python points to /tools/python-2.5.1 which is why it is using the old Python version. Changing the symbol link to /tools/python27 should be fine in my opinion, but I don't know everything about these systems. CentOS system depends on Python 2.4 in some way (especially depending on the installed modules), but /usr/bin/python would not be changed. Python 2.7 should be backwards compatible regarding Python 2.x.
Assignee | ||
Comment 11•10 years ago
|
||
Ok.. here's what I've 'done'. I traced through the buildbotcustom code and after a bit of tracing, I came across http://hg.mozilla.org/build/buildbotcustom/file/b9ea9639c354/process/factory.py#l6775 which then I arrived here: http://hg.mozilla.org/build/buildbotcustom/file/b9ea9639c354/misc.py#l454. Looking all over the place, I couldn't find in our buildbot-configs where we actually set unittest-env for each platform. At first, I fudged it with a modification to http://hg.mozilla.org/build/buildbotcustom/file/b9ea9639c354/env.py#l81 by: 80 "MOZ_HIDE_RESULTS_TABLE": '1' 81 } to 80 "MOZ_HIDE_RESULTS_TABLE": '1', 81 "PATH": '/tools/python-2.7.3/bin' 82 } then reconfig'd the master and ran the /Linux comm-central-trunk debug test mochitests-1/5, and that put us squarely in bug 853720. So the JSON error is gone. So the solution that I've come up with is to add unittest-env to whatever platform that needs the mochitests done. i.e. macosx64-debug and win32-debug. I added: 'unittest-env': { 'MOZ_OBJDIR': OBJDIR, 'DISPLAY': ':2', 'LD_LIBRARY_PATH': '%s/mozilla/dist/bin' % OBJDIR, 'XPCOM_DEBUG_BREAK': 'stack-and-abort', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_COMPRESS': '1', 'CCACHE_UMASK': '002', 'PATH': '/tools/python-2.7.3/bin:/tools/python-2.7.2/bin:/tools/python-2.6.5/bin:${PATH}', }, to linux-debug's section in config.py. (It's basically a c&p of 'env') Now why we didn't have unittest-env set.. I don't know. There's probably a better solution to this. Callek?
Flags: needinfo?(bugspam.Callek)
Assignee | ||
Comment 12•10 years ago
|
||
Assignee | ||
Comment 13•10 years ago
|
||
Please note that the current tests are working because I had manually applied the patch to the config.py and reconfig'd the master. However, I reverted my changes because the proper way of doing it is to actually push the attached patch to the repo. So while it's working right now (because I haven't reconfig'd the master), it's going to go back to the same error because I'll be doing the 2.24b1 patch first. Once I reconfig the master for the 2.24b1 patch, this patch (or another patch depending on what Callek says) will need to be pushed and the master reconfig'd in order to have the unittests go back to complaining about mozcrash. This is mainly a heads up for everyone, just in case someone wonders why the unittests are back to complaining about JSON.
Assignee | ||
Comment 14•10 years ago
|
||
Bah. Was going over the code and the buildbot configs as to why without the patch, t's WFM now. Then it hit me. Basically, all our boxes are now using Python 2.7+ (and not the Python 2.5 when this bug was filed). Ergo, this is no longer a problem as JSON is included with 2.7 And so, I'm closing this as WFM.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Flags: needinfo?(bugspam.Callek)
Resolution: --- → WORKSFORME
Assignee | ||
Updated•10 years ago
|
Attachment #8358708 -
Attachment is obsolete: true
Attachment #8358708 -
Flags: review?(bugspam.Callek)
Assignee | ||
Comment 15•10 years ago
|
||
While fighting the buildbotcustom code to make Mock work, I came across the Windows c-b tests that were failing. While we *do* have Python2.7 installed, the unittests NEED the patch. I'm reopening this and resurrecting the patch from the graveyard.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Assignee | ||
Updated•10 years ago
|
Status: REOPENED → ASSIGNED
Assignee | ||
Updated•10 years ago
|
Attachment #8358708 -
Attachment is obsolete: false
Attachment #8358708 -
Flags: review?(bugspam.Callek)
Comment 16•10 years ago
|
||
Comment on attachment 8358708 [details] [diff] [review] proposed patch (v1) Review of attachment 8358708 [details] [diff] [review]: ----------------------------------------------------------------- not the approach I want. We should do something like: http://hg.mozilla.org/build/buildbotcustom/rev/70e92b0482f4 Note the current version of that is http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#5383
Attachment #8358708 -
Flags: review?(bugspam.Callek) → review-
Assignee | ||
Comment 17•9 years ago
|
||
Attachment #8358708 -
Attachment is obsolete: true
Assignee | ||
Updated•9 years ago
|
Attachment #8577734 -
Flags: review?(bugspam.Callek)
Comment 19•9 years ago
|
||
Removing keyword, since it's intended for test failures that are seen on Treeherder (and otherwise you'll end up with bugs being marked as worksforme if no occurrences are seen on Treeherder).
Keywords: intermittent-failure
Assignee | ||
Comment 20•7 years ago
|
||
Comment on attachment 8577734 [details] [diff] [review] proposed patch (v2) Fixed in bug 1072713
Attachment #8577734 -
Flags: review?(bugspam.Callek)
Assignee | ||
Updated•7 years ago
|
Status: ASSIGNED → RESOLVED
Closed: 10 years ago → 7 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•