1173691 - bug 1155338 removed the app-override switch from ./mach mochitest

Status: RESOLVED INVALID

(Testing :: Mochitest, defect)

Description

[:Gijs (he/him)]

As in summary. Now I can no longer run tests on my VM using just an up-to-date tree and a tinderbox build. This is annoying because it means I need to do install MSVS2013 and do a build on said underpowered VM, which will take me all day.

This seems to have been removed in bug 1155338 but I can't tell if that was on purpose or by accident, and if the former, *why* it was removed and/or if there is now an alternative

Comment 1

[Andrew Halberstadt [:ahal]]

Do you mean --appname? If so, it's actually still there, just hidden from help for UX reasons:
https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/mochitest_options.py?from=mochitest_options.py#73

I have a patch landing soon (bug 1171602) that will make it so --appname only gets suppressed if an objdir is detected. But until then you can still use it like normal. Sorry for the confusion.

Comment 2

[:Gijs (he/him)]

(In reply to Andrew Halberstadt [:ahal] from comment #1)
> Do you mean --appname? If so, it's actually still there, just hidden from
> help for UX reasons:
> https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/
> mochitest_options.py?from=mochitest_options.py#73

No, I really mean --app-override. I added it back in bug 1004410 and blogged about it here http://www.gijsk.com/blog/2014/05/run-mochitests-against-an-arbitrary-build-or-against-the-distribution-directory/ .

It sounds like it got renamed?


> I have a patch landing soon (bug 1171602) that will make it so --appname
> only gets suppressed

What does this mean? That it doesn't show up under ./mach help mochitest?

> if an objdir is detected. But until then you can still
> use it like normal. Sorry for the confusion.

That sounds like it would break if I use --disable-compile-environment because I would have an objdir - just no binaries.

Comment 3

[Andrew Halberstadt [:ahal]]

(In reply to :Gijs Kruitbosch from comment #2)
> No, I really mean --app-override. I added it back in bug 1004410 and blogged
> about it here
> http://www.gijsk.com/blog/2014/05/run-mochitests-against-an-arbitrary-build-
> or-against-the-distribution-directory/ .
> 
> It sounds like it got renamed?

Oh I see, --app-override was defined in the mach_commands.py.

The mochitest harness has had --appname since the beginning, and as far as I could tell --app-override just did the exact same thing. I decided to keep the former as it has been around longer. I also added the 'dist' capability to --appname for parity:
https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/mochitest_options.py#536

> > I have a patch landing soon (bug 1171602) that will make it so --appname
> > only gets suppressed
> 
> What does this mean? That it doesn't show up under ./mach help mochitest?
>
> That sounds like it would break if I use --disable-compile-environment
> because I would have an objdir - just no binaries.

Yes, suppress just means it is hidden from help, but you can still pass suppressed arguments in via the command line. The utilityPath is auto-detected based on --appname, so your use case above should work.

tl;dr: --appname should work as a drop-in replacement for --app-override in all cases. But if you have trouble using it, please let me know.

Comment 4

[Andrew Halberstadt [:ahal]]

Also, if you feel strongly that --appname should never be suppressed that's fine. It is more useful than most of the other arguments that got hidden.

Comment 5

[:Gijs (he/him)]

(In reply to Andrew Halberstadt [:ahal] from comment #3)
> tl;dr: --appname should work as a drop-in replacement for --app-override in
> all cases. But if you have trouble using it, please let me know.


Gijs Kruitbosch@WIN-9TKK8JRE825 /c/dev/fx-team
$ ./mach mochitest --appname "c:\\Users\\Gijs Kruitbosch\\Desktop\\fx-trunk-as-aurora-failure\\firefox\\firefox.exe" browser/components/customi zableui/test/browser_968565_insert_before_hidden_items.js
c:\mozilla-build\mozmake\mozmake.EXE -j4 -s install-tests
From _tests: Kept 38559 existing; Added/updated 5; Removed 0 files and 0 directories.

######
### Now running mochitest-browser.
######

Checking for orphan ssltunnel processes...
Checking for orphan xpcshell processes...
SUITE-START | Running 1 tests
dir: browser/components/customizableui/test
runtests.py | Failed to copy c:\dev\fx-team\obj-i686-pc-mingw32\dist\plugins to profile
[u'c:\\dev\\fx-team\\obj-i686-pc-mingw32\\dist\\bin\\certutil.exe', '-N', '-d', 'c:\\users\\gijskr~1\\appdata\\local\\temp\\tmpwnzqno.mozrunner', '-f', 'c:\\users\\gijskr~1\\appdata\\local\\temp\\tmpwnzqno.mozrunner\\.crtdbpw']
Error running mach:

    ['mochitest', '--appname', 'c:\\Users\\Gijs Kruitbosch\\Desktop\\fx-trunk-as-aurora-failure\\firefox\\firefox.exe', 'browser/components/customizableui/test/browser_968565_insert_before_hidden_items.js']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

WindowsError: [Error 2] The system cannot find the file specified.

  File "c:\dev\fx-team\testing/mochitest/mach_commands.py", line 563, in run_mochitest_general
    **harness_args)
  File "c:\dev\fx-team\testing/mochitest/mach_commands.py", line 352, in run_desktop_test
    result = mochitest.run_test_harness(options)
  File "c:\dev\fx-team\obj-i686-pc-mingw32\_tests\testing\mochitest\runtests.py", line 2612, in run_test_harness
    result = runner.runTests(options)
  File "c:\dev\fx-team\obj-i686-pc-mingw32\_tests\testing\mochitest\runtests.py", line 2129, in runTests
    result = self.runMochitests(options, tests_in_dir, onLaunch)
  File "c:\dev\fx-team\obj-i686-pc-mingw32\_tests\testing\mochitest\runtests.py", line 2049, in runMochitests
    result = self.doTests(options, onLaunch, testsToRun)
  File "c:\dev\fx-team\obj-i686-pc-mingw32\_tests\testing\mochitest\runtests.py", line 2184, in doTests
    self.manifest = self.buildProfile(options)
  File "c:\dev\fx-team\obj-i686-pc-mingw32\_tests\testing\mochitest\runtests.py", line 1441, in buildProfile
    certificateStatus = self.fillCertificateDB(options)
  File "c:\dev\fx-team\obj-i686-pc-mingw32\_tests\testing\mochitest\runtests.py", line 1310, in fillCertificateDB
    [certutil, "-N", "-d", certdbPath, "-f", pwfilePath], env=toolsEnv)
  File "c:\dev\fx-team\obj-i686-pc-mingw32\_tests\testing\mochitest\runtests.py", line 282, in call
    process.run()
  File "c:\dev\fx-team\testing/mozbase/mozprocess\mozprocess\processhandler.py", line 727, in run
    self.proc = self.Process([self.cmd] + self.args, **args)
  File "c:\dev\fx-team\testing/mozbase/mozprocess\mozprocess\processhandler.py", line 114, in __init__
    universal_newlines, startupinfo, creationflags)
  File "c:\mozilla-build\python\lib\subprocess.py", line 711, in __init__
    errread, errwrite)
  File "c:\dev\fx-team\testing/mozbase/mozprocess\mozprocess\processhandler.py", line 268, in _execute_child
    cwd, startupinfo)
  File "c:\dev\fx-team\testing/mozbase/mozprocess\mozprocess\winprocess.py", line 184, in ErrCheckCreateProcess
    ErrCheckBool(result, func, args)
  File "c:\dev\fx-team\testing/mozbase/mozprocess\mozprocess\winprocess.py", line 51, in ErrCheckBool
    raise WinError()
Exception AttributeError: "'Process' object has no attribute '_job'" in <bound method Process.__del__ of <mozprocess.processhandler.Process object at 0x10D7AD50>> ignored




The same thing happens if I pass a unix-like mozilla-build-bash "/c/Users/..." path instead.

Passing the same executable path to "file" tells me that they are executables.

I don't actually know which file windows says doesn't exist, because this stack and the error message don't really tell me...

Comment 6

[:Gijs (he/him)]

(In reply to :Gijs Kruitbosch from comment #5)
> Passing the same executable path to "file" tells me that they are
> executables.
> 
> I don't actually know which file windows says doesn't exist, because this
> stack and the error message don't really tell me...

D'oh, I need to read better. Looks like the code that copies the plugins folder should be OK with having nothing to copy.

... except that when I create a folder there, the command still fails...

Comment 7

[:Gijs (he/him)]

OK, so it looks like running mochitests without having actually built Firefox just doesn't work (certutil.exe is missing, etc.) even using --appname, and that probably needs its own bugreport.

Comment 8

[Andrew Halberstadt [:ahal]]

Looks like we should be able to divine options.utilityPath based on options.app if it's set. Can you print(options.utilityPath) around line 1308 in runtests.py?

I'm not sure this was regressed by bug 1155338 because --app-override just set options.app the same way --appname does:
https://hg.mozilla.org/mozilla-central/diff/2a4e56526e67/testing/mochitest/mach_commands.py#l1.387

But mochitest is such a ball of spaghetti it's hard to tell.

Comment 9

[:Gijs (he/him)]

(In reply to Andrew Halberstadt [:ahal] from comment #8)
> Looks like we should be able to divine options.utilityPath based on
> options.app if it's set. Can you print(options.utilityPath) around line 1308
> in runtests.py?

I tried also passing the utilityPath but that didn't help; certutil isn't actually in our "normal" dist packages, AFAICT. Would you prefer to keep further discussion of this here? I was thinking I should file a new bug... either works.

Comment 10

[Andrew Halberstadt [:ahal]]

Please file a new bug and cc me. Thanks!