Closed Bug 1216575 Opened 6 years ago Closed 6 years ago

Currently ./mach mochitest is failing with 'OSError: [Errno 2] No such file or directory'

Categories

(Testing :: Mochitest, defect)

Unspecified
macOS
defect
Not set
normal

Tracking

(firefox44 fixed)

RESOLVED FIXED
mozilla44
Tracking Status
firefox44 --- fixed

People

(Reporter: jwatt, Unassigned)

References

Details

(Whiteboard: [see comment 3 for workaround])

Attachments

(1 file)

$ ./mach mochitest dom/html/test/forms
From _tests: Kept 41512 existing; Added/updated 0; Removed 0 files and 0 directories.

######
### Now running mochitest-chrome.
######

Checking for orphan ssltunnel processes...
Checking for orphan xpcshell processes...
SUITE-START | Running 2 tests
dir: dom/html/test/forms
[u'/Users/jw/mozilla/trees/obj/i2-debug/dist/NightlyDebug.app/Contents/Resources/certutil', '-N', '-d', '/var/folders/_3/769fqy7n59s0qc63997hs4qw0000gn/T/tmpgd4HTO.mozrunner', '-f', '/var/folders/_3/769fqy7n59s0qc63997hs4qw0000gn/T/tmpgd4HTO.mozrunner/.crtdbpw']
Error running mach:

    ['mochitest', 'dom/html/test/forms']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

OSError: [Errno 2] No such file or directory

  File "/Users/jw/mozilla/trees/i2/testing/mochitest/mach_commands.py", line 588, in run_mochitest_general
    **harness_args)
  File "/Users/jw/mozilla/trees/i2/testing/mochitest/mach_commands.py", line 339, in run_desktop_test
    result = mochitest.run_test_harness(options)
  File "/Users/jw/mozilla/trees/obj/i2-debug/_tests/testing/mochitest/runtests.py", line 2598, in run_test_harness
    result = runner.runTests(options)
  File "/Users/jw/mozilla/trees/obj/i2-debug/_tests/testing/mochitest/runtests.py", line 2111, in runTests
    result = self.runMochitests(options, tests_in_dir, onLaunch)
  File "/Users/jw/mozilla/trees/obj/i2-debug/_tests/testing/mochitest/runtests.py", line 2037, in runMochitests
    result = self.doTests(options, onLaunch, testsToRun)
  File "/Users/jw/mozilla/trees/obj/i2-debug/_tests/testing/mochitest/runtests.py", line 2189, in doTests
    self.manifest = self.buildProfile(options)
  File "/Users/jw/mozilla/trees/obj/i2-debug/_tests/testing/mochitest/runtests.py", line 1431, in buildProfile
    certificateStatus = self.fillCertificateDB(options)
  File "/Users/jw/mozilla/trees/obj/i2-debug/_tests/testing/mochitest/runtests.py", line 1300, in fillCertificateDB
    [certutil, "-N", "-d", certdbPath, "-f", pwfilePath], env=toolsEnv)
  File "/Users/jw/mozilla/trees/obj/i2-debug/_tests/testing/mochitest/runtests.py", line 280, in call
    process.run()
  File "/Users/jw/mozilla/trees/i2/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 727, in run
    self.proc = self.Process([self.cmd] + self.args, **args)
  File "/Users/jw/mozilla/trees/i2/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 114, in __init__
    universal_newlines, startupinfo, creationflags)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
Both dist/bin/certutil and dist/NightlyDebug.app/Contents/MacOS/certutil exist in my OBJDIR.
For some reason we're trying to use dist/NightlyDebug.app/Contents/Resources/certutil but that doesn't exist.
Until recently options.utilityPath was set to point to obj/dist/bin, but now it's pointing to dist/NightlyDebug.app/Contents/Resources.

To work around this issue run |./mach mochitest --utility-path=$OBJDIR/dist/bin <test-path>|
I'm seeing this too on the latest fx-team on osx
I get this same issue on OSX, but it works fine on Linux.
OS: Unspecified → Mac OS X
Might this have happened when upgrading to El Capitan? I'm hitting this too and I very recently upgraded. Not 100% sure that's when it started to happen.
(In reply to Jonas Sicking (:sicking) from comment #6)
> Might this have happened when upgrading to El Capitan? I'm hitting this too
> and I very recently upgraded. Not 100% sure that's when it started to happen.

I don't think so. I haven't upgraded, and also I can roll back my tree to a day ago and it stops happening. I was starting to bisect but it was taking forever to rebuild the tree.
Whiteboard: [see comment 3 for workaround]
This is "regressed" by bug 1216171. In that bug I was fixing an obviously wrong import error and assuming that the underlying code was correct, but I guess that was a bad assumption.

Here is the corresponding line:
https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/base.py#283

That if statement was originally added in bug 1060562. I don't own or know much about osx, so I'm not sure why it was added, or what it's trying to do. Spohl, does that if statement also need to be checking for a particular version of osx?

I do know that in bug 1216171, :erahm needed that if statement to get triggered in order for things to work, so removing it altogether isn't an option.
Flags: needinfo?(spohl.mozilla.bugs)
(In reply to Andrew Halberstadt [:ahal] from comment #8)
> I do know that in bug 1216171, :erahm needed that if statement to get
> triggered in order for things to work, so removing it altogether isn't an
> option.

Spoke with :ahal on irc. It looks like the if statement always evaluated to false prior to bug 1216171. This means that bindir defaulted to dist/bin, which seemingly worked for most of our tests. :erahm, do your compiled cppunittests not get placed into dist/bin? Could you tell us what files couldn't be found during your tests?

If you have steps to reproduce locally I'd like to take a look. Thanks!
Flags: needinfo?(spohl.mozilla.bugs) → needinfo?(erahm)
Also, if only cppunittests need dist/NightlyDebug.app/Contents/Resources, and everything else requires dist/bin, we should just remove the if statement in the meantime, as way more people are broken by it as is.
Bug 1216575 - [mozbuild] Remove if statement that sets bindir to resource dir on osx, r=spohl
Attachment #8676838 - Flags: review?(spohl.mozilla.bugs)
https://reviewboard.mozilla.org/r/22819/#review20287

Agreed, let's remove this and fix cppunittests separately.
Comment on attachment 8676838 [details]
MozReview Request: Bug 1216575 - [mozbuild] Remove if statement that sets bindir to resource dir on osx, r=spohl

https://reviewboard.mozilla.org/r/22821/#review20291
Attachment #8676838 - Flags: review?(spohl.mozilla.bugs) → review+
(In reply to Stephen A Pohl [:spohl] from comment #9)
> (In reply to Andrew Halberstadt [:ahal] from comment #8)
> > I do know that in bug 1216171, :erahm needed that if statement to get
> > triggered in order for things to work, so removing it altogether isn't an
> > option.
> 
> Spoke with :ahal on irc. It looks like the if statement always evaluated to
> false prior to bug 1216171. This means that bindir defaulted to dist/bin,
> which seemingly worked for most of our tests. :erahm, do your compiled
> cppunittests not get placed into dist/bin? Could you tell us what files
> couldn't be found during your tests?
> 
> If you have steps to reproduce locally I'd like to take a look. Thanks!

STR:
On a mac (running 10.10.5):
1) |mach build|
2) |mach cppunittest $OBJ_DIR/dist/bin/TestHashtables|

There's an exception about the xre_path not being a directory:
> Exception: ('xre_path does not exist: %s', u'/Users/ericrahm/dev/mozilla-central/obj-x86_64-apple-darwin14.5.0/dist/Resources')

ahal pointed out this is b/c it's hardcoded in cppunittest [1] to append 'Resources' to the path.

By pointing to the proper app bundle we made cppunitests happy (Resources exists) but I guess made everyone else sad. Removing that line seems to work for me, but I can't guarantee it won't break other tests.

[1] https://dxr.mozilla.org/mozilla-central/rev/605de27d4e7f530159842649c02075c578d7a4a5/testing/runcppunittests.py#235
Flags: needinfo?(erahm)
Blocks: 1217144
https://hg.mozilla.org/mozilla-central/rev/874761c630c2
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla44
You need to log in before you can comment on or make changes to this bug.