Closed Bug 857332 Opened 10 years ago Closed 10 years ago

No such file or directory: u'./objdir-droid/.mozbuild/warnings.json' running `mach build`

Categories

(Firefox Build System :: Mach Core, enhancement)

x86
macOS
enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla23

People

(Reporter: nalexander, Assigned: mbrubeck)

References

(Blocks 1 open bug)

Details

Attachments

(1 file, 2 obsolete files)

And indeed, there is no .mozbuild directory in objdir-droid.  This was from running `./mach build` in $topsrcdir, against a recently merged services-central.

...
49:20.60 761 compiler warnings present.
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

IOError: [Errno 2] No such file or directory: u'./objdir-droid/.mozbuild/warnings.json'

  File "/Users/ncalexan/Mozilla/mcgit/python/mozbuild/mozbuild/mach_commands.py", line 132, in build
    warnings_database.save_to_file(warnings_path)
  File "/Users/ncalexan/Mozilla/mcgit/python/mozbuild/mozbuild/compilation/warnings.py", line 268, in save_to_file
    with open(filename, 'wb') as fh:
Blind guess: the directory doesn't exist? Since 'wb' means you should be making the file...

Then again, I get a different error:

│python
Python 2.7.3 (default, Sep 26 2012, 21:53:58) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> with open('/home/jhammel/foo', 'wb') as f:
...     f.write('bar')
... 
>>> with open('/home/jhammel/foo/bar/fleem', 'wb') as f:
...     f.write('bar')
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 20] Not a directory: '/home/jhammel/foo/bar/fleem'
(In reply to Jeff Hammel [:jhammel] from comment #1)
> Blind guess: the directory doesn't exist? Since 'wb' means you should be
> making the file...

Yes, that is correct.  See first line of description.
I wonder if it is a good idea to have a convenience method in mozfile that will create a directory if it does not exist.  My vote: likely.
My guess is this is a current working directory issue. I'm guessing cwd changes to objdir when it resolves the relative path for warnings.json.

How is MOZ_OBJDIR defined in your mozconfig?

We really need a |mach dump-config| or some other command to facilitate debugging. Or, perhaps mach should dump this information automatically on error to make error reports easier to sort through...
(In reply to Gregory Szorc [:gps] from comment #4)
> My guess is this is a current working directory issue. I'm guessing cwd
> changes to objdir when it resolves the relative path for warnings.json.
> 
> How is MOZ_OBJDIR defined in your mozconfig?

mk_add_options MOZ_OBJDIR=./objdir-droid

So changing CWD sounds likely.  Point is, this used to work.
I haven't reproduced this bug to verify the fix, but if it is caused by MOZ_OBJDIR being a relative path (as some other recent problems were, like bug 855709), then this should fix it.
Assignee: nobody → mbrubeck
Status: NEW → ASSIGNED
Attachment #732589 - Flags: review?(gps)
Comment on attachment 732589 [details] [diff] [review]
return topobjdir as an absolute path

Review of attachment 732589 [details] [diff] [review]:
-----------------------------------------------------------------

I'm pretty sure this will break at least one person's build or workflow. However, I'm not sure how. There are just so many ways people can build the tree. Some things to consider:

* Effect of cwd on objdir resolution. It's quite possible somebody does: mkdir tmpdir; cd tmpdir; /path/to/mozilla-central/mach build. Before, since cwd was the objdir, I'm pretty sure topobjdir would resolve to tmpdir/<relative path from mozconfig>. Never mind this workflow is silly because objdir would change depending on cwd. But, you never know what people do...

* Effect of symlinks on objdir resolution. Some may manage objdirs by having separate directories symlinked into the same abspath. Depending on how we resolve topsrcdir and/or if we resolve symlinks, this workflow may break.

That being said, I think <topsrcdir>/<relobjdir> makes the most sense. So, it has my tentative r+. Final r+ will be granted after a simple test is added.
Attachment #732589 - Flags: review?(gps)
We can also revert the change from bug 855709.
Attachment #732589 - Attachment is obsolete: true
Attachment #732598 - Flags: review?(gps)
Comment on attachment 732598 [details] [diff] [review]
return topobjdir as an absolute path (v2)

Sorry, didn't see your last comment when I posted this.
Attachment #732598 - Flags: review?(gps)
Attached patch patch v3Splinter Review
Now with tests.

(In reply to Gregory Szorc [:gps] from comment #7)
> I'm pretty sure this will break at least one person's build or workflow.

Aw geez, you're right.  But I'm not sure there's any sane way we can support both those who expect "MOZ_OBJDIR=relative/path" to be srcdir-relative and those who expect it to be CWD-relative...
Attachment #732598 - Attachment is obsolete: true
Attachment #732609 - Flags: review?(gps)
I don't see a way to support both methods (unless you leave it up to a config option).

I /think/ what we're doing here is defining an undefined aspect of the build system. I'm totally OK with this.

As for reverting other changes, there are *many* parts of the build code where we normalize to abspath. I look forward to changing these to asserts.
Blocks: 856204
Comment on attachment 732609 [details] [diff] [review]
patch v3

Review of attachment 732609 [details] [diff] [review]:
-----------------------------------------------------------------

Let's hope this works...

::: python/mozbuild/mozbuild/base.py
@@ +58,5 @@
>      def topobjdir(self):
>          if self._topobjdir is None:
>              topobj = self.mozconfig['topobjdir'] or 'obj-@CONFIG_GUESS@'
> +            if not os.path.isabs(topobj):
> +                topobj = os.path.abspath(os.path.join(self.topsrcdir, topobj))

How about:

  topobjd = os.path.normpath(os.path.abspath(os.path.join(self.topsrcdir, topobj)))

I hate ".." showing up in paths. Semantically it /should/ be the same, right?
Attachment #732609 - Flags: review?(gps) → review+
Green on Try: https://tbpl.mozilla.org/?tree=Try&rev=3c4127c60002

(In reply to Gregory Szorc [:gps] from comment #12)
> How about:
> 
>   topobjd = os.path.normpath(os.path.abspath(os.path.join(self.topsrcdir,
> topobj)))

abspath normalizes its arguments already, so normpath here would be a no-op.  I'd prefer to leave it out.

(Or if we are certain that topsrcdir is already absolute, we could leave out the abspath call instead.)
(In reply to Matt Brubeck (:mbrubeck) from comment #13)
> abspath normalizes its arguments already...

(I mean, it normalizes its *return value* already.)
So it does! I didn't realize that! I've been bitten by non-normalized paths so often I typically throw in normpaths as a last line of defense.
https://hg.mozilla.org/mozilla-central/rev/4a6fd35a9a53
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla23
Sorry to say I just ran into this with a freshly merged trunk:

[xykon-2:~/devel/hg/mozilla-central] bkelly% hg log python/mozbuild/ | grep 'Bug 857332'
summary:     Bug 857332 - Normalize topobjdir to an absolute path in mozbuild.base [r=gps]

The original error message I received was:

IOError: [Errno 2] No such file or directory: u'/Users/bkelly/devel/hg/mozilla-central/obj-x86_64-apple-darwin11.4.2/.mozbuild/warnings.json'

  File "/Users/bkelly/devel/hg/mozilla-central/python/mozbuild/mozbuild/mach_commands.py", line 115, in build
    warnings_database.save_to_file(warnings_path)
  File "/Users/bkelly/devel/hg/mozilla-central/python/mozbuild/mozbuild/compilation/warnings.py", line 268, in save_to_file
    with open(filename, 'wb') as fh:
(In reply to Ben Kelly from comment #18)
> IOError: [Errno 2] No such file or directory:
> u'/Users/bkelly/devel/hg/mozilla-central/obj-x86_64-apple-darwin11.4.2/.
> mozbuild/warnings.json'

Does the "/Users/bkelly/devel/hg/mozilla-central/obj-x86_64-apple-darwin11.4.2" directory exist?  Does it have a ".mozbuild" subdirectory?  Do you get this error when running "mach build" with no other arguments?

Assuming the paths above are at the expected locations, the .mozbuild directory should be created by this call to _ensure_objdir_exists in _run_make:
http://hg.mozilla.org/mozilla-central/file/b0d842380959/python/mozbuild/mozbuild/base.py#l218

And so it should work as long as _run_make is called before save_to_file here.  I can't see any obvious way for that to fail:
http://hg.mozilla.org/mozilla-central/file/b0d842380959/python/mozbuild/mozbuild/mach_commands.py#l115
Depends on: 859349
Aha!  I just reproduced this error myself.  I filed a new bug 859349 for the specific case that I ran into, since I'm not sure if there are other causes for the same error.
It looks like you already tracked this down, but to answer your questions:

> Does the "/Users/bkelly/devel/hg/mozilla-central/obj-x86_64-apple-darwin11.4.2"
> directory exist?  Does it have a ".mozbuild" subdirectory?  Do you get this error
> when running "mach build" with no other arguments?

After receiving this error I found that obj-x86_64-apple-darwin11.4.2 did exist,
but the .mozbuild directory was missing.

Does that make sense given the issue you found?
(In reply to Ben Kelly from comment #22)
> After receiving this error I found that obj-x86_64-apple-darwin11.4.2 did
> exist,
> but the .mozbuild directory was missing.
> 
> Does that make sense given the issue you found?

Yes, that matches what I saw in bug 859349, and even if it had a different root cause it should be fixed by that patch.
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.