818656 - mozconfig loading busted on Windows due to path encoding

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(no longer active)]

ehsan@BEHEMOTH ~/moz/mozgit
$ ./mach build
Error running mach:

    ['build']

The error occured in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

CalledProcessError: Command '[u'C:/mozilla-build/msys/bin/sh.exe', u'-c', u'c:/Users/ehsan/moz/mozgit/python/mozbuild/mozbuild/mozconfig_loa
der c:\\Users\\ehsan\\moz\\mozgit c:/Users/ehsan/moz/mozgit/.mozconfig']' returned non-zero exit status 1

  File "c:\Users\ehsan\moz\mozgit\python/mozbuild/mozbuild/mach_commands.py", line 31, in build
    warnings_path = self._get_state_filename('warnings.json')
  File "c:\Users\ehsan\moz\mozgit\python/mozbuild\mozbuild\base.py", line 107, in _get_state_filename
    path = self.statedir
  File "c:\Users\ehsan\moz\mozgit\python/mozbuild\mozbuild\base.py", line 79, in statedir
    return os.path.join(self.topobjdir, '.mozbuild')
  File "c:\Users\ehsan\moz\mozgit\python/mozbuild\mozbuild\base.py", line 50, in topobjdir
    if self.mozconfig['topobjdir'] is None:
  File "c:\Users\ehsan\moz\mozgit\python/mozbuild\mozbuild\base.py", line 65, in mozconfig
    self._mozconfig = loader.read_mozconfig()
  File "c:\Users\ehsan\moz\mozgit\python/mozbuild\mozbuild\mozconfig.py", line 177, in read_mozconfig
    cwd=self.topsrcdir, env=env)
  File "c:\mozilla-build\python\lib\subprocess.py", line 544, in check_output
    raise CalledProcessError(retcode, cmd, output=output)

ehsan@BEHEMOTH ~/moz/mozgit
$ python --version
Python 2.7.2


This is the latest trunk.

Comment 1

[Gregory Szorc [:gps]]

Your mozconfig is exiting with a non-0 exit code. Apply the patch in bug 818545 and you should see a better error message.

See also http://gregoryszorc.com/blog/2012/12/05/changes-to-how-mach-loads-mozconfigs/

Comment 2

[(no longer active)]

ehsan@BEHEMOTH ~/moz/mozgit
$ ./mach build
Error loading mozconfig: c:/Users/ehsan/moz/mozgit/.mozconfig

Evaluation of your mozconfig exited with an error. This could be triggered
by a command inside your mozconfig failing. Please change your mozconfig
to not error and/or to catch errors in executed commands.

mozconfig output:

c:/Users/ehsan/moz/mozgit/.mozconfig: line 2: c:Usersehsanmozmozgit/browser/config/mozconfig: No such file or directory

ehsan@BEHEMOTH ~/moz/mozgit
$ cat .mozconfig
#export NSDISTMODE=copy
. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-debug
ac_add_options --enable-debug
ac_add_options --enable-trace-malloc
ac_add_options --disable-optimize

(Note the corrupted path in the error message.

Comment 3

[Gregory Szorc [:gps]]

I hate Windows style paths and execution within sh.exe so much.

I think I have enough to go by to reproduce and fix.

Comment 4

[(no longer active)]

Thanks! :-)  Let me know if you needed more info from me.

Comment 5

[Gregory Szorc [:gps]]

I bet if I ran the unit tests on Windows they would fail. I bet things regressed from bug 818543.

Comment 6

[(no longer active)]

(In reply to comment #5)
> I bet if I ran the unit tests on Windows they would fail. I bet things
> regressed from bug 818543.

Are you saying that we don't run them on every check-in?  ;-)

Comment 7

[Gregory Szorc [:gps]]

(In reply to Ehsan Akhgari [:ehsan] from comment #6)
> (In reply to comment #5)
> > I bet if I ran the unit tests on Windows they would fail. I bet things
> > regressed from bug 818543.
> 
> Are you saying that we don't run them on every check-in?  ;-)

The mach and mozbuild unit tests aren't enabled because both require Python 2.7. Bug 818646.

Comment 8

[Ed Morley [:emorley]]

(In reply to Gregory Szorc [:gps] from comment #5)
> I bet if I ran the unit tests on Windows they would fail. I bet things
> regressed from bug 818543.

How do I run the unit test suite?

I couldn't find anything on https://developer.mozilla.org/en-US/docs/Developer_Guide/mach

Comment 9

[Gregory Szorc [:gps]]

The easy way to run the test suite is:

1) Add the following line to build/virtualenv/packages.txt

    mach.pth:python/mach

2) Run configure and abort once the virtualenv has been repopulated
3) source objdir/_virtualenv/bin/activate (Scripts/activate on Windows)
4) pip install nose
5) cd python/mozbuild && nosetests -s -v
6) cd python/mach && nosetests -s -v

I think there are some known failures on Windows right now *sigh*. For this bug, you care about:

  nosetests -s -v python/mozbuild/mozbuild/test/test_mozconfig.py

You should also be able to use Python's built-in unit runner by doing something like:

  python -m unittest mozbuild.test.test_mozconfig

(From inside the virtualenv, of course.) Python's test runner is a bit more fragile than nose and I haven't used it in ages, so I'm not sure if it works.

I should add a mach command to run the unit tests!

Comment 10

[Gregory Szorc [:gps]]

Attachment: Normalize path separators, v1

Sure enough, the unit tests failed on Windows with path components not agreeing!

I /think/ this will fix things. Review to Mike. I'd like feedback from somebody who encountered this before I commit it. If you give f+, please cancel f? on the other - I don't want to waste anyone's time!

Comment 11

[(no longer active)]

I'll test it tomorrow when I'm back at the office (that's where my Windows box is), if nobody beats me to it.

Comment 12

[Gregory Szorc [:gps]]

https://hg.mozilla.org/mozilla-central/rev/5f626f86b374

I accidentally. It didn't regress any unit tests and I'm happy with the test coverage, so I'm pretty confident with this making it in before f+. Reopen if this doesn't fix it for you.

Comment 13

[(no longer active)]

Comment on attachment 689058 [details] [diff] [review]
Normalize path separators, v1

This fixed it for me.  Thanks!