Can't run mach reftest on Windows 10 (ImportError: No module named marionette_test)

NEW
Unassigned

Status

()

Core
mach
a year ago
a year ago

People

(Reporter: birtles, Unassigned)

Tracking

Trunk
x86_64
Windows 10
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [Solution: execute "./mach clobber python"])

(Reporter)

Description

a year ago
When running |mach reftest| or |mach crashtest| on Windows 10 with today's trunk I get:

> $ mach reftest
> REFTEST INFO | Checking for orphan ssltunnel processes...
> REFTEST INFO | Checking for orphan xpcshell processes...
> REFTEST INFO | Running with e10s: True
> REFTEST INFO | Application command: c:\moz\src2\obj-debug\dist\bin\firefox.exe -marionette -profile c:\users\brian\appdata\local\temp\tmpcouqff.mozrunner
> Error running mach:
> 
>     ['reftest']
> 
> The error occurred in code that was called by the mach command. This is either
> a bug in the called code itself or in the way that mach is calling it.
> 
> You should consider filing a bug for this issue.
> 
> If filing a bug, please include the full output of mach, including this error
> message.
> 
> The details of the failure are as follows:
> 
> ImportError: No module named marionette_test
> 
>   File "c:\moz\src2\layout/tools/reftest/mach_commands.py", line 319, in run_reftest
>     return self._run_reftest(**kwargs)
>   File "c:\moz\src2\layout/tools/reftest/mach_commands.py", line 349, in _run_reftest
>     return reftest.run_desktop_test(**kwargs)
>   File "c:\moz\src2\layout/tools/reftest/mach_commands.py", line 205, in run_desktop_test
>     rv = runreftest.run_test_harness(parser, args)
>   File "c:\moz\src2\layout/tools/reftest\runreftest.py", line 734, in run_test_harness
>     return reftest.runTests(options.tests, options)
>   File "c:\moz\src2\layout/tools/reftest\runreftest.py", line 426, in runTests
>     return self.runSerialTests(manifests, options, cmdargs)
>   File "c:\moz\src2\layout/tools/reftest\runreftest.py", line 681, in runSerialTests
>     debuggerInfo=debuggerInfo)
>   File "c:\moz\src2\layout/tools/reftest\runreftest.py", line 621, in runApp
>     marionette = Marionette(**marionette_args)
>   File "c:\moz\src2\layout/tools/reftest\runreftest.py", line 43, in reraise
>     raise(e)
> Assertion failure: false (Tried and failed to create an XPCOM log), at c:/moz/src2/xpcom/base/nsTraceRefcnt.cpp:685
> #01: InitTraceLog (c:\moz\src2\xpcom\base\nstracerefcnt.cpp:715)
> #02: NS_LogAddRef (c:\moz\src2\xpcom\base\nstracerefcnt.cpp:1036)
> #03: nsStringBuffer::Alloc (c:\moz\src2\xpcom\string\nssubstring.cpp:223)
> #04: nsAString_internal::MutatePrep (c:\moz\src2\xpcom\string\nstsubstring.cpp:149)
> #05: nsAString_internal::SetCapacity (c:\moz\src2\xpcom\string\nstsubstring.cpp:687)
> #06: nsAString_internal::SetLength (c:\moz\src2\xpcom\string\nstsubstring.cpp:725)
> #07: AppendUTF8toUTF16 (c:\moz\src2\xpcom\string\nsreadableutils.cpp:354)
> #08: AppendUTF8toUTF16 (c:\moz\src2\xpcom\string\nsreadableutils.cpp:333)
> #09: CopyUTF8toUTF16 (c:\moz\src2\xpcom\string\nsreadableutils.cpp:138)
> #10: NS_CStringToUTF16 (c:\moz\src2\xpcom\build\nsxpcomstrings.cpp:334)
> #11: NS_CStringToUTF16 (c:\moz\src2\xpcom\glue\standalone\nsxpcomglue.cpp:735)
> #12: NS_ConvertUTF8toUTF16_external::NS_ConvertUTF8toUTF16_external (c:\moz\src2\obj-debug\dist\include\nsstringapi.h:1211)
> #13: InitXPCOMGlue (c:\moz\src2\browser\app\nsbrowserapp.cpp:393)
> #14: NS_internal_main (c:\moz\src2\browser\app\nsbrowserapp.cpp:450)
> #15: wmain (c:\moz\src2\toolkit\xre\nswindowswmain.cpp:115)
> #16: __scrt_common_main_seh (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253)
> #17: BaseThreadInitThunk[C:\WINDOWS\System32\KERNEL32.DLL +0x162c4]
> #18: RtlSubscribeWnfStateChangeNotification[C:\WINDOWS\SYSTEM32\ntdll.dll +0x60719]
> #19: RtlSubscribeWnfStateChangeNotification[C:\WINDOWS\SYSTEM32\ntdll.dll +0x606e4]

After this firefox crashes.

Xidorn reports that he is seeing the same.

However, running the same on a checkout based on https://hg.mozilla.org/mozilla-central/rev/a69583d2dbc6 works as expected.

Looking through the log I see changes to support artifact builds and bug 1319705 but nothing else sticks out.
(Reporter)

Comment 1

a year ago
Henrik, could bug 1319705 be related to this?
Flags: needinfo?(hskupin)
(Reporter)

Updated

a year ago
Summary: Can't run mach reftest on Windows 10 → Can't run mach reftest on Windows 10 (ImportError: No module named marionette_test)
I eventually know what the reason is. Anyone meets this issue should run './mach clobber python'.

The issue here is probably that, some files are moved in bug 1319705, but corresponding .pyc files are not removed, and Python tries to use those files first, which leads to this issue. The problematic file here seems to be testing/marionette/client/marionette/__init__.pyc.

We probably want some approach to automatically do 'clobber python' like autoclobber. This kind of issues are pretty confusing and could take lots of time to debug when people come across.
Given this issue is actually that .pyc files are not removed correctly, I'm moving this bug to Core/Build Config in hope that some people there have idea to fix the fundamental issue.
Component: Marionette → Build Config
Product: Testing → Core
Also cancel ni? whimboo since it isn't really an issue of the testing harness.
Flags: needinfo?(hskupin)
(Reporter)

Comment 5

a year ago
Also, you will need to enable the hg purge extension to run './mach clobber python'.

Agree it would be good to automate this. I lost quite a bit of time to this.
Keywords: regression, regressionwindow-wanted
Whiteboard: Run "./mach clobber python" to fix this issue
(In reply to Brian Birtles (:birtles) from comment #5)
> Also, you will need to enable the hg purge extension to run './mach clobber
> python'.

It seems mercurial-setup doesn't setup that properly. Filed bug 1321470 for that.
(In reply to Brian Birtles (:birtles) from comment #1)
> Henrik, could bug 1319705 be related to this?

Just to come back to this question... Nothing in my patch actually touched anything Marionette Test related. It only moved an optional package into the Marionette subfolder. So not sure why you are actually seeing this exactly now with the above patch being landed.
I am seeing a similar issue with mochitest:

$ ./mach mochitest devtools/client/responsive.html

gives

Application command: /Users/jryans/projects/mozilla/gecko-dev-3/obj-firefox-release-artifact/dist/Nightly.app/Contents/MacOS/firefox -marionette -foreground -profile /var/folders/j6/jhbppx_x1050lwg8bmjxkt240000gp/T/tmpvuonp8.mozrunner
runtests.py | Application pid: 17202
TEST-INFO | started process Main app process
Traceback (most recent call last):
  File "/Users/jryans/projects/mozilla/gecko-dev-3/obj-firefox-release-artifact/_tests/testing/mochitest/runtests.py", line 2392, in doTests
    marionette_args=marionette_args,
  File "/Users/jryans/projects/mozilla/gecko-dev-3/obj-firefox-release-artifact/_tests/testing/mochitest/runtests.py", line 2004, in runApp
    self.marionette = Marionette(**marionette_args)
  File "/Users/jryans/projects/mozilla/gecko-dev-3/obj-firefox-release-artifact/_tests/testing/mochitest/runtests.py", line 57, in reraise
    raise(e)
ImportError: No module named marionette_test
Whiteboard: Run "./mach clobber python" to fix this issue → [Solution: execute "./mach clobber python"]
I did try `./mach clobber python`, but I guess that doesn't work as well for a git repo.

:aswan suggested `find . -name \*.pyc -exec rm {} \;` which worked for me.
(In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #9)
> I did try `./mach clobber python`, but I guess that doesn't work as well for
> a git repo.
> 
> :aswan suggested `find . -name \*.pyc -exec rm {} \;` which worked for me.

It should work: https://dxr.mozilla.org/mozilla-central/rev/cd4cdcc9ad6c45dad8b8d8c0d40e459db2bca8a1/python/mozbuild/mozbuild/mach_commands.py#697-704

If it doesn't, you should file a bug to mach. (Actually this bug should be in mach component as well?)
Component: Build Config → mach
The `find` version also visits objdirs, while the `git clean ...` version does not.  Not sure if that was relevant for my case (I thought I had done a regular clobber too...).
Whiteboard: [Solution: execute "./mach clobber python"] → [Solution: execute "v"]
Comment hidden (obsolete)
You need to log in before you can comment on or make changes to this bug.