905824 - ObjdirMismatchException when symlinks are involved

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P1)

Description

[Mats Palmgren (inactive)]

# mach mochitest-plain 
Error running mach:

    ['mochitest-plain']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ObjdirMismatchException: Objdir mismatch: /md1/OBJDIR/home/mats/moz/inbound != /OBJDIR/home/mats/moz/inbound

  File "/home/mats/moz/inbound/testing/mochitest/mach_commands.py", line 267, in run_mochitest_plain
    return self.run_mochitest(test_file, 'plain', **kwargs)
  File "/home/mats/moz/inbound/testing/mochitest/mach_commands.py", line 298, in run_mochitest
    **kwargs)
  File "/home/mats/moz/inbound/testing/mochitest/mach_commands.py", line 109, in run_mochitest_test
    runner = mochitest.Mochitest(automation)
  File "/OBJDIR/home/mats/moz/inbound/_tests/testing/mochitest/runtests.py", line 453, in __init__
    super(Mochitest, self).__init__()
  File "/OBJDIR/home/mats/moz/inbound/_tests/testing/mochitest/runtests.py", line 187, in __init__
    mozinfo.find_and_update_from_json(*dirs)
  File "/home/mats/moz/inbound/testing/mozbase/mozinfo/mozinfo/mozinfo.py", line 148, in find_and_update_from_json
    build = MozbuildObject.from_environment()
  File "/home/mats/moz/inbound/python/mozbuild/mozbuild/base.py", line 163, in from_environment
    raise ObjdirMismatchException(topobjdir, config['topobjdir'])
# ls -l /OBJDIR
lrwxrwxrwx 1 root root 11 Feb 26  2012 /OBJDIR -> /md1/OBJDIR

Comment 1

[Mats Palmgren (inactive)]

This setup used to work recently.

Comment 2

[Gregory Szorc [:gps]]

It's a regression from the mochitest harness changing how it loads mozconfigs. Please try the patch in bug 901811. If that fixes it, please dupe. Else, please add a comment with the output of |mach environment| and we'll get to the bottom of this.

Comment 3

[Mats Palmgren (inactive)]

Attachment: environment

The patch in bug 901811 did not help - same error.
This is the output of "./mach environment" (with that patch applied).

BTW, building and running firefox works fine.
"mach reftest" also works.

Comment 4

[Gregory Szorc [:gps]]

Please also upload your mozconfig and information about your directory structure, symlinks and all. Please also include steps to reproduce, including working directories. The goal is to give us enough information so that we can write a unit test that reproduces your environment (and hopefully the error).

Comment 5

[Mats Palmgren (inactive)]

The attachment has all the paths, the only symlink I have is the one in comment 0:
/OBJDIR -> /md1/OBJDIR

/md1/OBJDIR and all the other paths are real directories, not symlinks.

The contents of the local (source dir) .mozconfig is:
. $HOME/.mozconfig
ac_add_options --enable-installer

and $HOME/.mozconfig has:
mk_add_options MOZ_OBJDIR=/OBJDIR@TOPSRCDIR@

and some standard stuff like --enable-debug etc.
I'm not aware of anything else that is special in my environment.
Let me know if you can reproduce it with that info.

Comment 6

[Mats Palmgren (inactive)]

Bug still occurs... where you able to reproduce it?

Comment 7

[Gregory Szorc [:gps]]

I haven't looked at this one yet. Will try to look at it today. If you want a quick workaround, comment out the raising of ObjdirMismatchException from python/mozbuild/mozbuild/base.py. It should be relatively safe for most development scenarios.

Comment 9

[Gregory Szorc [:gps]]

Attachment: Handle symlinks when validating build environment

I'm optimistic this will fix things.

Turns out the implementation of samepath() was wrong because it was
looking for an API on os.path that AFAICT never existed! I also think
I've identified the root cause of automation test failures: they're
picking up mozinfo.json from the virtualenv. So I added a backdoor to
disable that. Will post another patch that attempts to enable disabled
tests with this workaround.

https://tbpl.mozilla.org/?tree=Try&rev=85aa11e17f9d

Comment 10

[Gregory Szorc [:gps]]

https://hg.mozilla.org/projects/build-system/rev/b702bb90a87e

Comment 11

[Gregory Szorc [:gps]]

https://hg.mozilla.org/mozilla-central/rev/b702bb90a87e