Closed Bug 901811 Opened 8 years ago Closed 8 years ago

ObjdirMismatchException using @CONFIG_GUESS@ in objdir

Categories

(Firefox Build System :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla26

People

(Reporter: markh, Assigned: gps)

References

(Blocks 1 open bug)

Details

Attachments

(2 files, 2 obsolete files)

Running mach, I suddenly started getting:
...
The details of the failure are as follows:

ObjdirMismatchException: Objdir mismatch: o:\src\mm\mozilla-hg\mc-socialapi-landing\obj-i686-pc-mingw32 != o:/src/mm/mozilla-hg/mc-socialapi-landing/obj-@CONFIG_GUESS@
...
  File "o:\src\mm\mozilla-hg\mc-socialapi-landing\python/mozbuild\mozbuild\base.py", line 163, in from_environment
    raise ObjdirMismatchException(topobjdir, config['topobjdir'])

--
.mozconfig includes the line:

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-@CONFIG_GUESS@

--
output of "mach environment" (from bug 894973)

platform:
        Windows-7-6.1.7601-SP1
python version:
        2.7.4 (default, Apr  6 2013, 19:54:46) [MSC v.1500 32 bit (Intel)]
python prefix:
        c:\mozilla-build\python
mach cwd:
        o:\src\mm\mozilla-hg\mc-socialapi-landing
os cwd:
        o:\src\mm\mozilla-hg\mc-socialapi-landing
mach directory:
        o:\src\mm\mozilla-hg\mc-socialapi-landing
state directory:
        o:/home/.mozbuild
object directory:
        o:\src\mm\mozilla-hg\mc-socialapi-landing/obj-i686-pc-mingw32
mozconfig path:
        o:\src\mm\mozilla-hg\mc-socialapi-landing\.mozconfig
mozconfig objdir:
        o:\src\mm\mozilla-hg\mc-socialapi-landing/obj-@CONFIG_GUESS@
mozconfig configure args:
        --enable-application=browser
        --disable-optimize
        --enable-debug
        --enable-tests
mozconfig extra make args:
        AUTOCLOBBER=1
mozconfig make flags:
        -
        j
        1
        2
config topsrcdir:
        o:/src/mm/mozilla-hg/mc-socialapi-landing
config topobjdir:
        o:\src\mm\mozilla-hg\mc-socialapi-landing\obj-i686-pc-mingw32
I guess we need to throw in a normpath somewhere...

Until then, you should be able to work around this by using a backslash in your mozconfig or by not defining an objdir in your mozconfig.
Component: mach → Build Config
(In reply to Gregory Szorc [:gps] from comment #1)
> I guess we need to throw in a normpath somewhere...
> 
> Until then, you should be able to work around this by using a backslash in
> your mozconfig or by not defining an objdir in your mozconfig.

I don't think the problem is the lack of a backslash or a normpath, but instead that we are simply using config['topobjdir'] while it still has the literal '@CONFIG_GUESS@' in it.  Simply replacing that with the literal 'i686-pc-mingw32' (and without adding or changing any slashes) works around the problem.
I can reproduce this on OS X as well.
OS: Windows 7 → All
Hardware: x86_64 → All
This is really weird, actually, because I'm using an environment variable for my .mozconfig, and am running the command (./mach mochitest-browser foo/bar/baz) from my srcdir, not my objdir.
Adding a print(topobjdir) in base.py before it throws the exception, I'm seeing it invoked once with 'None' and once with the correct normalized path to my objdir. I don't know why it does that.
... but fixing the paths to make them match gets me:

gkruitbosch-16516:ux gkruitbosch$ ./mach mochitest-browser browser/components/customizableui/test/
Error running mach:

    ['mochitest-browser', 'browser/components/customizableui/test/']

The error occurred in the implementation of the invoked mach command.

This should never occur and is likely a bug in the implementation of that
command. Consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ImportError: No module named automation

  File "/Users/gkruitbosch/dev/ux/testing/mochitest/mach_commands.py", line 280, in run_mochitest_browser
    return self.run_mochitest(test_file, 'browser', **kwargs)
  File "/Users/gkruitbosch/dev/ux/testing/mochitest/mach_commands.py", line 299, in run_mochitest
    **kwargs)
  File "/Users/gkruitbosch/dev/ux/testing/mochitest/mach_commands.py", line 85, in run_mochitest_test
    from automation import Automation

(should this be a different bug? I don't know.)

:-(
(In reply to :Gijs Kruitbosch from comment #6)
> ... but fixing the paths to make them match gets me:
> ImportError: No module named automation

This was a pebkac, because I mistyped the objdir in my mozconfig, which was therefore empty. The error message could have been better ("You haven't built at this path yet"), but fine.

Now that the path is correct, and I can run ./mach mochitest-browser path/to/test, I can no longer run ./mach build path/to/stuff/that/changed, because it errors out with:

gkruitbosch-16516:ux gkruitbosch$ ./mach build browser/components/customizableui/
 0:00.19 /usr/bin/make -C rowser/components/customizableui -j8 -s
 0:00.20 make: *** rowser/components/customizableui: No such file or directory.  Stop.

(the 'b' seems to randomly go missing from the path?)

Again, not sure if this is related or not.
(In reply to :Gijs Kruitbosch (PTO Aug 8-Aug 18) from comment #7)
> (In reply to :Gijs Kruitbosch from comment #6)
> > ... but fixing the paths to make them match gets me:
> > ImportError: No module named automation
> 
> This was a pebkac, because I mistyped the objdir in my mozconfig, which was
> therefore empty. The error message could have been better ("You haven't
> built at this path yet"), but fine.
> 
> Now that the path is correct, and I can run ./mach mochitest-browser
> path/to/test, I can no longer run ./mach build path/to/stuff/that/changed,
> because it errors out with:
> 
> gkruitbosch-16516:ux gkruitbosch$ ./mach build
> browser/components/customizableui/
>  0:00.19 /usr/bin/make -C rowser/components/customizableui -j8 -s
>  0:00.20 make: *** rowser/components/customizableui: No such file or
> directory.  Stop.
> 
> (the 'b' seems to randomly go missing from the path?)
> 
> Again, not sure if this is related or not.

And this seems unrelated but to do with whether or not the path specified has a trailing slash. Filed bug 902389.
I'm going to look into this today.
Assignee: nobody → gps
This sounded like a trivial fix but the patch bloated up a bit.
Essentially the problem was MozbuildObject.from_environment() and
MozbuildObject's .topobjdir getter had slightly different code paths for
resolving topobjdir from mozconfigs. This patch hopefully fixes that.

Before, @CONFIG_GUESS@ just wasn't getting substituted in
MozbuildObject.from_environment(). This was causing the topobjdir sanity
checking comparison to fail and causing errors for users with
@CONFIG_GUESS@ in their mozconfig. We now substitute @CONFIG_GUESS@ in
from_environment() by using the same code path as the .topobjdir getter.

The included test case failed before the patch and now passes.

https://tbpl.mozilla.org/?tree=Try&rev=9f312c918fd9
Attachment #788277 - Flags: review?(mshal)
Duplicate of this bug: 902484
I had a bad patch in my try push, so https://tbpl.mozilla.org/?tree=Try&rev=a608c8f25785
Comment on attachment 788277 [details] [diff] [review]
Unify code paths for load mozconfigs

Is this the right patch? You mentioned that the test case passes, but it's failing in the try jobs (as well as when I run it locally).
Attachment #788277 - Flags: review?(mshal) → review-
markh: I was trying to reproduce this on Windows with this mozconfig:

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj2-@CONFIG_GUESS@

However, I don't hit the ObjdirMismatchException (before applying gps' patch). I just ran a './mach build'

Can you walk me through the steps to reproduce it?
Flags: needinfo?(mhammond)
FWIW, ted pointed me here since I'm hitting the same exception, and I'm not using @CONFIG_GUESS@. I'm simply using:

mk_add_options MOZ_OBJDIR=./objdir-droid

(This line is given as an example in https://wiki.mozilla.org/Android?title=Mobile/Fennec/Android#Setup_Fennec_mozconfig.)

Printing out the absolute paths, I see that:

          topobjdir is /data/mozilla/central/objdir-droid
config['topobjdir'] is /data/mozilla/central/objdir-droid/objdir-droid
BTW, I'm building Fennec. I see this only when trying to run robocop tests (make mochitest-robocop); I do not see this when doing a normal mach build.
Brian: Please paste the contents of your objdir/mozinfo.json file.
Flags: needinfo?(bnicholson)
(In reply to Gregory Szorc [:gps] from comment #17)
> Brian: Please paste the contents of your objdir/mozinfo.json file.

{"bin_suffix": "", "tests_enabled": true, "bits": 32, "appname": "fennec", "mozconfig": "/home/brian/Dropbox/mozilla/mozconfig-droid", "topsrcdir": "/data/mozilla/central", "toolkit": "android", "crashreporter": true, "debug": false, "asan": false, "os": "android", "processor": "arm"}
Flags: needinfo?(bnicholson)
(In reply to Michael Shal [:mshal] from comment #14)
> markh: I was trying to reproduce this on Windows with this mozconfig:
> 
> mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj2-@CONFIG_GUESS@
> 
> However, I don't hit the ObjdirMismatchException (before applying gps'
> patch). I just ran a './mach build'

'build' seems to work, but I get it trying to run, eg, './mach mochitest-browser browser/base/content/test/social/'
Brian: I suspect this is a current working directory issue. Can you paste exact steps to reproduce. Start your shell at topsrcdir and include cd's, etc.

FWIW, I could probably ask you to file a followup, but I want this bug to be the last time we have to track down the lingering issues with mozconfig loading.
(In reply to Gregory Szorc [:gps] from comment #20)
> Brian: I suspect this is a current working directory issue. Can you paste
> exact steps to reproduce. Start your shell at topsrcdir and include cd's,
> etc.

Sure. Using the attached mozconfig:

cd /data/mozilla/central
./mach build
./mach package
./mach install
export MOZ_HOST_BIN=/data/mozilla/central/objdir-linux/dist/bin/
cd objdir-droid
make mochitest-robocop

Output:
Traceback (most recent call last):
  File "_tests/testing/mochitest/runtestsremote.py", line 19, in <module>
    from runtests import Mochitest
  File "/data/mozilla/central/objdir-droid/_tests/testing/mochitest/runtests.py", line 25, in <module>
    from mochitest_options import MochitestOptions
  File "/data/mozilla/central/objdir-droid/_tests/testing/mochitest/mochitest_options.py", line 17, in <module>
    build_obj = MozbuildObject.from_environment()
  File "/data/mozilla/central/python/mozbuild/mozbuild/base.py", line 165, in from_environment
    raise ObjdirMismatchException(topobjdir, config['topobjdir'])
mozbuild.base.ObjdirMismatchException: Objdir mismatch: /data/mozilla/central/objdir-droid != ./objdir-droid
make: *** [mochitest-robocop] Error 1
Brian: I wrote an explicit test for relative-defined objdirs with cwd inside the objdir and it passes with my patch. Please confirm my patch fixes things for you.
Flags: needinfo?(bnicholson)
Let's see if this has better results on Try...

https://tbpl.mozilla.org/?tree=Try&rev=83293931fd36
Attachment #788277 - Attachment is obsolete: true
(In reply to Gregory Szorc [:gps] from comment #22)
> Brian: I wrote an explicit test for relative-defined objdirs with cwd inside
> the objdir and it passes with my patch. Please confirm my patch fixes things
> for you.

I was able to reproduce the ObjdirMismatchException by loosely following bnicholson's instructions. I have:

.mozconfig:
mk_add_options MOZ_OBJDIR=./objdir-droid

Then did:

./mach build
./mach package (not sure if this was necessary)
cd objdir-droid
python ../build/pymake/make.py mochitest

I also confirmed that your latest patch fixes the error (at least, some mochitests start running). Hopefully try is happy this time :)
I got the info I needed to reproduce the error.
Flags: needinfo?(mhammond)
Yep, I can also confirm that the patch fixes the issue for me.
Flags: needinfo?(bnicholson)
Random thought: if I'm unable to figure out how to get the tests to pass on automation, would reviewers consider skipping the tests for the initial landing? The way I see it, this bug is interfering with the ability of some people to optimally perform work and that takes precedence over test coverage in the short term.
Yes, let's get this landed. Testing these code paths is a PITA.
Let's get this pushed so developers don't have a hard time.
Attachment #790815 - Flags: review?(ted)
Attachment #790264 - Attachment is obsolete: true
Comment on attachment 790815 [details] [diff] [review]
Unify code paths for load mozconfigs

Review of attachment 790815 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/base.py
@@ +185,5 @@
> +            topobjdir = os.path.abspath(os.path.join(topsrcdir, topobjdir))
> +
> +        if '@CONFIG_GUESS@' in topobjdir:
> +            topobjdir = topobjdir.replace('@CONFIG_GUESS@',
> +                MozbuildObject.resolve_config_guess(mozconfig, topsrcdir))

Is doing this test after the abspath test going to screw things up? @CONFIG_GUESS@ at the beginning of the objdir path wouldn't be treated as an abspath right?
Attachment #790815 - Flags: review?(ted) → review+
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #32)
> ::: python/mozbuild/mozbuild/base.py
> @@ +185,5 @@
> > +            topobjdir = os.path.abspath(os.path.join(topsrcdir, topobjdir))
> > +
> > +        if '@CONFIG_GUESS@' in topobjdir:
> > +            topobjdir = topobjdir.replace('@CONFIG_GUESS@',
> > +                MozbuildObject.resolve_config_guess(mozconfig, topsrcdir))
> 
> Is doing this test after the abspath test going to screw things up?
> @CONFIG_GUESS@ at the beginning of the objdir path wouldn't be treated as an
> abspath right?

Good catch. I moved the abspath after the .replace() and verified all tests still pass.
https://hg.mozilla.org/integration/fx-team/rev/734f0dc3958e

Inbound was closed. And since this issue impacts mochitest running, I figured fx-team was as good of an alternative as there is.
Status: NEW → ASSIGNED
BTW: So for now comm-central users (Thunderbird, SeaMonkey) are left a bit in the rain regarding running tests? I know that some people are working on really integrating comm-central with mozilla-central, but I guess this might still take a while. As an example for what does not work atm:
"TEST_PATH=suite/common/tests/browser/ pymake mochitest-browser-chrome
[...]
mozbuild.base.ObjdirMismatchException: Objdir mismatch: c:\mozilla\tree-hg\objdirs\seamonkey-objdir\mozilla != c:\mozilla\tree-hg\comm-central\objdirs\seamonkey-objdir"

when executing that command in the objdir or in objdir/mozilla (does not matter). Hm, I guess I should file a follow-up bug on that... :)
Uh-huh, I just noticed the config_topobjdir path is completely wrong (I built with the patch in this bug here). I'll build a Firefox build for now and report back, I use a hardlink for the objdirs on my HDD. Maybe that matters, too.
https://hg.mozilla.org/mozilla-central/rev/734f0dc3958e
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla26
Depends on: 907434
Blocks: 912114
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.