1321468 - Can't run mach reftest on Windows 10 (ImportError: No module named marionette_test)

Status: NEW

(Firefox Build System :: Mach Core, enhancement)

Description

[Brian Birtles (:birtles, Feb 26~ parental leave)]

When running |mach reftest| or |mach crashtest| on Windows 10 with today's trunk I get:

> $ mach reftest
> REFTEST INFO | Checking for orphan ssltunnel processes...
> REFTEST INFO | Checking for orphan xpcshell processes...
> REFTEST INFO | Running with e10s: True
> REFTEST INFO | Application command: c:\moz\src2\obj-debug\dist\bin\firefox.exe -marionette -profile c:\users\brian\appdata\local\temp\tmpcouqff.mozrunner
> Error running mach:
> 
>     ['reftest']
> 
> The error occurred in code that was called by the mach command. This is either
> a bug in the called code itself or in the way that mach is calling it.
> 
> You should consider filing a bug for this issue.
> 
> If filing a bug, please include the full output of mach, including this error
> message.
> 
> The details of the failure are as follows:
> 
> ImportError: No module named marionette_test
> 
>   File "c:\moz\src2\layout/tools/reftest/mach_commands.py", line 319, in run_reftest
>     return self._run_reftest(**kwargs)
>   File "c:\moz\src2\layout/tools/reftest/mach_commands.py", line 349, in _run_reftest
>     return reftest.run_desktop_test(**kwargs)
>   File "c:\moz\src2\layout/tools/reftest/mach_commands.py", line 205, in run_desktop_test
>     rv = runreftest.run_test_harness(parser, args)
>   File "c:\moz\src2\layout/tools/reftest\runreftest.py", line 734, in run_test_harness
>     return reftest.runTests(options.tests, options)
>   File "c:\moz\src2\layout/tools/reftest\runreftest.py", line 426, in runTests
>     return self.runSerialTests(manifests, options, cmdargs)
>   File "c:\moz\src2\layout/tools/reftest\runreftest.py", line 681, in runSerialTests
>     debuggerInfo=debuggerInfo)
>   File "c:\moz\src2\layout/tools/reftest\runreftest.py", line 621, in runApp
>     marionette = Marionette(**marionette_args)
>   File "c:\moz\src2\layout/tools/reftest\runreftest.py", line 43, in reraise
>     raise(e)
> Assertion failure: false (Tried and failed to create an XPCOM log), at c:/moz/src2/xpcom/base/nsTraceRefcnt.cpp:685
> #01: InitTraceLog (c:\moz\src2\xpcom\base\nstracerefcnt.cpp:715)
> #02: NS_LogAddRef (c:\moz\src2\xpcom\base\nstracerefcnt.cpp:1036)
> #03: nsStringBuffer::Alloc (c:\moz\src2\xpcom\string\nssubstring.cpp:223)
> #04: nsAString_internal::MutatePrep (c:\moz\src2\xpcom\string\nstsubstring.cpp:149)
> #05: nsAString_internal::SetCapacity (c:\moz\src2\xpcom\string\nstsubstring.cpp:687)
> #06: nsAString_internal::SetLength (c:\moz\src2\xpcom\string\nstsubstring.cpp:725)
> #07: AppendUTF8toUTF16 (c:\moz\src2\xpcom\string\nsreadableutils.cpp:354)
> #08: AppendUTF8toUTF16 (c:\moz\src2\xpcom\string\nsreadableutils.cpp:333)
> #09: CopyUTF8toUTF16 (c:\moz\src2\xpcom\string\nsreadableutils.cpp:138)
> #10: NS_CStringToUTF16 (c:\moz\src2\xpcom\build\nsxpcomstrings.cpp:334)
> #11: NS_CStringToUTF16 (c:\moz\src2\xpcom\glue\standalone\nsxpcomglue.cpp:735)
> #12: NS_ConvertUTF8toUTF16_external::NS_ConvertUTF8toUTF16_external (c:\moz\src2\obj-debug\dist\include\nsstringapi.h:1211)
> #13: InitXPCOMGlue (c:\moz\src2\browser\app\nsbrowserapp.cpp:393)
> #14: NS_internal_main (c:\moz\src2\browser\app\nsbrowserapp.cpp:450)
> #15: wmain (c:\moz\src2\toolkit\xre\nswindowswmain.cpp:115)
> #16: __scrt_common_main_seh (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253)
> #17: BaseThreadInitThunk[C:\WINDOWS\System32\KERNEL32.DLL +0x162c4]
> #18: RtlSubscribeWnfStateChangeNotification[C:\WINDOWS\SYSTEM32\ntdll.dll +0x60719]
> #19: RtlSubscribeWnfStateChangeNotification[C:\WINDOWS\SYSTEM32\ntdll.dll +0x606e4]

After this firefox crashes.

Xidorn reports that he is seeing the same.

However, running the same on a checkout based on https://hg.mozilla.org/mozilla-central/rev/a69583d2dbc6 works as expected.

Looking through the log I see changes to support artifact builds and bug 1319705 but nothing else sticks out.

Comment 1

[Brian Birtles (:birtles, Feb 26~ parental leave)]

Henrik, could bug 1319705 be related to this?

Comment 2

[Xidorn Quan [:xidorn] UTC+11]

I eventually know what the reason is. Anyone meets this issue should run './mach clobber python'.

The issue here is probably that, some files are moved in bug 1319705, but corresponding .pyc files are not removed, and Python tries to use those files first, which leads to this issue. The problematic file here seems to be testing/marionette/client/marionette/__init__.pyc.

We probably want some approach to automatically do 'clobber python' like autoclobber. This kind of issues are pretty confusing and could take lots of time to debug when people come across.

Comment 3

[Xidorn Quan [:xidorn] UTC+11]

Given this issue is actually that .pyc files are not removed correctly, I'm moving this bug to Core/Build Config in hope that some people there have idea to fix the fundamental issue.

Comment 4

[Xidorn Quan [:xidorn] UTC+11]

Also cancel ni? whimboo since it isn't really an issue of the testing harness.

Comment 5

[Brian Birtles (:birtles, Feb 26~ parental leave)]

Also, you will need to enable the hg purge extension to run './mach clobber python'.

Agree it would be good to automate this. I lost quite a bit of time to this.

Comment 6

[Xidorn Quan [:xidorn] UTC+11]

(In reply to Brian Birtles (:birtles) from comment #5)
> Also, you will need to enable the hg purge extension to run './mach clobber
> python'.

It seems mercurial-setup doesn't setup that properly. Filed bug 1321470 for that.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Brian Birtles (:birtles) from comment #1)
> Henrik, could bug 1319705 be related to this?

Just to come back to this question... Nothing in my patch actually touched anything Marionette Test related. It only moved an optional package into the Marionette subfolder. So not sure why you are actually seeing this exactly now with the above patch being landed.

Comment 8

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

I am seeing a similar issue with mochitest:

$ ./mach mochitest devtools/client/responsive.html

gives

Application command: /Users/jryans/projects/mozilla/gecko-dev-3/obj-firefox-release-artifact/dist/Nightly.app/Contents/MacOS/firefox -marionette -foreground -profile /var/folders/j6/jhbppx_x1050lwg8bmjxkt240000gp/T/tmpvuonp8.mozrunner
runtests.py | Application pid: 17202
TEST-INFO | started process Main app process
Traceback (most recent call last):
  File "/Users/jryans/projects/mozilla/gecko-dev-3/obj-firefox-release-artifact/_tests/testing/mochitest/runtests.py", line 2392, in doTests
    marionette_args=marionette_args,
  File "/Users/jryans/projects/mozilla/gecko-dev-3/obj-firefox-release-artifact/_tests/testing/mochitest/runtests.py", line 2004, in runApp
    self.marionette = Marionette(**marionette_args)
  File "/Users/jryans/projects/mozilla/gecko-dev-3/obj-firefox-release-artifact/_tests/testing/mochitest/runtests.py", line 57, in reraise
    raise(e)
ImportError: No module named marionette_test

Comment 9

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

I did try `./mach clobber python`, but I guess that doesn't work as well for a git repo.

:aswan suggested `find . -name \*.pyc -exec rm {} \;` which worked for me.

Comment 10

[Xidorn Quan [:xidorn] UTC+11]

(In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #9)
> I did try `./mach clobber python`, but I guess that doesn't work as well for
> a git repo.
> 
> :aswan suggested `find . -name \*.pyc -exec rm {} \;` which worked for me.

It should work: https://dxr.mozilla.org/mozilla-central/rev/cd4cdcc9ad6c45dad8b8d8c0d40e459db2bca8a1/python/mozbuild/mozbuild/mach_commands.py#697-704

If it doesn't, you should file a bug to mach. (Actually this bug should be in mach component as well?)

Comment 11

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

The `find` version also visits objdirs, while the `git clean ...` version does not.  Not sure if that was relevant for my case (I thought I had done a regular clobber too...).

Comment 12

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

whoops, sorry