773423 - WebRTC build breaks when MOZ_OBJDIR is defined outside of TOPSRCDIR on Windows

Status: RESOLVED WORKSFORME

(Core :: WebRTC, defect)

Description

[Justin Lebar (not reading bugmail)]

My build setup on Windows 7:

  srcdir is z:/src
  objdir is c:/objdir/ff-git-windows/debug

This configuration dies during the WebRTC build with:

> make.py[6]: Entering directory 'c:\objdir\ff-git-windows\debug\media\webrtc\trun
k'
> No rule to make target '../../../../../../../z:src/media/webrtc/trunk/src/voice_
engine/voice_engine.gyp' needed by ['../../../../../../../z:src/media/webrtc/tru
nk/src/voice_engine/voice_engine.gyp']
> No rule to make target '../../../../../../../z:src/media/webrtc/trunk/src/common
_video/common_video.gyp' needed by ['../../../../../../../z:src/media/webrtc/tru
nk/src/common_video/common_video.gyp']

and so on.  The apparent problem is that it's using "z:src/", which is not a real directory.

Here's my full mozconfig:

> . $topsrcdir/browser/config/mozconfig
> ac_add_options --disable-angle
> ac_add_options --disable-webgl
> 
> mk_add_options MOZ_OBJDIR=c:/objdir/ff-git-windows/debug
> ac_add_options --enable-debug
> ac_add_options --disable-optimize
> ac_add_options --enable-trace-malloc

I'm building with the following command:

> /c/objdir/ff-git-windows/debug$ /z/src/build/pymake/make.py -j8 -f /z/src/client.mk

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

Related to bug 772201 (symlinked objdirs on linux/mac)

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

Attachment: use absolute paths for windows, different for make vs pymake

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 642175 [details] [diff] [review]
use absolute paths for windows, different for make vs pymake

Or I'll take a review from any other build peer.... :-)

This undoes the "ignore windows" aspect of bug 772201, and with great pain generates correct absolute paths (in the header - topsrcdir/srcdir/VPATH) in all cases on all platforms.

Comment 4

[Mike Hommey [:glandium]]

Comment on attachment 642175 [details] [diff] [review]
use absolute paths for windows, different for make vs pymake

Ted knows these msys vs windows issues better than i do.

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 642175 [details] [diff] [review]
use absolute paths for windows, different for make vs pymake

Review of attachment 642175 [details] [diff] [review]:
-----------------------------------------------------------------

My only real worry here is that this is going to break people who don't use client.mk and also build with pymake. (There are a fair amount of people that just configure && make.) If we make this decision in configure and we aren't running from client.mk, then we'll have no indication that we should be using Windows paths.

I guess the onus is on the user to provide the right kind of paths in that scenario? 

Really, there must be a way to do this properly, since the existing Makefile.in-to-Makefile writing logic handles this properly.

::: configure.in
@@ +8898,5 @@
>  
>  if test -n "$MOZ_WEBRTC"; then
>     AC_MSG_RESULT("generating WebRTC Makefiles...")
> +   # msys (on windows) does EVIL things to parameters to python it thinks are msys paths.
> +   # Since we already have the path in the format we want (c:/foo for pymake, 

nit: trailing whitespace at the end of this line

@@ +8902,5 @@
> +   # Since we already have the path in the format we want (c:/foo for pymake, 
> +   # /c/foo for make), we just want to stop msys from mucking with the string.
> +   # The underscores will be removed in mozmake.py.  We need the correct form of
> +   # absolute path for building topsrcdir= in Makefiles.
> +   MOZ_TOPSRCDIR="_${MOZ_TOPSRCDIR:0:2}_${MOZ_TOPSRCDIR:2}"

This is horrible, but I don't have a better solution for you.

Comment 6

[Mike Hommey [:glandium]]

(In reply to Ted Mielczarek [:ted] from comment #5)
> Comment on attachment 642175 [details] [diff] [review]
> use absolute paths for windows, different for make vs pymake
> 
> Review of attachment 642175 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> My only real worry here is that this is going to break people who don't use
> client.mk and also build with pymake. (There are a fair amount of people
> that just configure && make.) If we make this decision in configure and we
> aren't running from client.mk, then we'll have no indication that we should
> be using Windows paths.
> 
> I guess the onus is on the user to provide the right kind of paths in that
> scenario? 
> 
> Really, there must be a way to do this properly, since the existing
> Makefile.in-to-Makefile writing logic handles this properly.

wouldn't bug 778236 essentially fix this?

Comment 7

[(not currently active) Ted Mielczarek]

Most likely, yes.

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

Checked in:
https://hg.mozilla.org/integration/mozilla-inbound/rev/152d19d479b2

Comment 10

[Doug Turner (:dougt)]

Backed out:
  https://hg.mozilla.org/integration/mozilla-inbound/rev/3e6ed8c7aaa6

This busted linux clobber builds:

generating WebRTC Makefiles...
/builds/mozilla-inbound/configure: 27333: /builds/mozilla-inbound/configure: Bad substitution
*** Fix above errors and then restart with               "make -f client.mk build"
make[2]: *** [configure] Error 1
make[2]: Leaving directory `/builds/mozilla-inbound'
make[1]: *** [/builds/mozilla-inbound/objdir-ff-dbg/Makefile] Error 2
make[1]: Leaving directory `/builds/mozilla-inbound'
make: *** [build] Error 2


Line 27333 is:

   MOZ_TOPSRCDIR="_${MOZ_TOPSRCDIR:0:2}_${MOZ_TOPSRCDIR:2}"


I think this is only suppose to be substituted on windows.  I am sure it is an easy fix, but we need to keep the tree building.

Comment 11

[Randell Jesup [:jesup] (needinfo me)]

Attachment: use absolute paths for windows, different for make vs pymake

Comment 12

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 656334 [details] [diff] [review]
use absolute paths for windows, different for make vs pymake

The issue turned out to be Ubuntu using 'dash' for /bin/sh, and the change used a bash-ism  for getting a substr: ${var:start:len}

The recommended solution at https://wiki.ubuntu.com/DashAsBinSh (using awk) works fine on the commandline for me, but fails in configure (returns "0").  

So instead I made the insertion of the _'s conditional on the platform, which is annoying and non-optimal, but should work.

Comment 13

[(not currently active) Ted Mielczarek]

Comment on attachment 656334 [details] [diff] [review]
use absolute paths for windows, different for make vs pymake

Review of attachment 656334 [details] [diff] [review]:
-----------------------------------------------------------------

::: configure.in
@@ +8700,5 @@
> +   if test "$MOZ_WIDGET_TOOLKIT" == "windows"; then
> +     # Windows only because the other OSes don't have the problem; MUST
> +     # match the receiving side in mozmake.py!
> +     MOZ_TOPSRCDIR="_${MOZ_TOPSRCDIR:0:2}_${MOZ_TOPSRCDIR:2}"
> +   fi

I wonder if rather than jumping through these hoops if we shouldn't just write out a simple .gyp file here that defines MOZ_TOPSRCDIR, and forcibly include it from the gyp commandline. We'd have to modify mozmake.py to pull the variable out of the right place, but it would avoid this nonsense.

Comment 14

[Mike Hommey [:glandium]]

(In reply to Ted Mielczarek [:ted] from comment #13)
> I wonder if rather than jumping through these hoops if we shouldn't just
> write out a simple .gyp file here that defines MOZ_TOPSRCDIR, and forcibly
> include it from the gyp commandline. We'd have to modify mozmake.py to pull
> the variable out of the right place, but it would avoid this nonsense.

bug 778236 is where we're going.

Comment 15

[Frank Wein [:mcsmurf]]

BTW: This patch no longer applies on mozilla-central. I tried to fix this manually, but then another issue occurred (something about use_system_libvpx not being defined).

Comment 17

[Ed Morley [:emorley]]

jessup said bug 798814 is a dupe of this, but my srcdir and objdir are on the same drive; how can I work around this in the meantime (other than having to sift through the patch)?

Comment 18

[Randell Jesup [:jesup] (needinfo me)]

Actually the summary is incorrect, it happens if the objdir has an absolute path.

Comment 19

[Ed Morley [:emorley]]

(In reply to Randell Jesup [:jesup] from comment #18)
> Actually the summary is incorrect, it happens if the objdir has an absolute
> path.

I was previously using an absolute path (see bug 798814), so tried switching to:
mk_add_options MOZ_OBJDIR=../../obj-inbound

...which still doesn't work, failing with the same error as in bug 798814 :-(

Comment 20

[Frank Wein [:mcsmurf]]

I cannot confirm Comment 19 (relative MOZ_OBJDIR being broken, too). If you have src- and objdir on two different partitions (like I do), one can use the NTFS symbolic link feature as a workaround for now:
Enter "mklink /d C:\mozilla\tree-hg\objdirs F:\mozilla" on the Windows command line (as Administrator) to create an NTFS symbolic link on the same partition as the srcdir, this works on Windows Vista and above (I have my srcdir in C:\mozilla\tree-hg\comm-central on an SSD, my objdirs are in F:\mozilla on a normal harddrive). As MOZ_OBJDIR I've set "../objdirs" in my .mozconfig.

Comment 21

[Frank Wein [:mcsmurf]]

Actually I've set "../objdirs/seamonkey-objdir" as objdir, but that should not really matter.

Comment 22

[ABE Hiroki (:hATrayflood)]

see also, Bug 775939

Comment 23

[Ed Morley [:emorley]]

Please can we do something about this. Not being able to build locally without disabling webrtc is a PITA :-)

Comment 24

[Ed Morley [:emorley]]

(We've also just had someone on #developers scratching their heads over a broken local build & it turned out to be this)

Comment 25

[Ed Morley [:emorley]]

Adjusting summary given comment 19.

Comment 26

[ABE Hiroki (:hATrayflood)]

(In reply to Ed Morley [:edmorley UTC+0] from comment #23)
> Please can we do something about this. Not being able to build locally
> without disabling webrtc is a PITA :-)

Attachment #696621 [details] [diff] of Bug 775939 maybe resolves this bus.

Comment 27

[ABE Hiroki (:hATrayflood)]

s/this bus./this bug./

Comment 28

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: use absolute paths for windows, different for make vs pymake (v2)

The former patch has been bitrotted and doesn't apply anymore. I have updated it so it can be applied to recent code. Randell, can you please have a look at if everything is still ok and that no other build system changes invalidated parts of the code? When we are ok I will ask Ted for review.

I haven't tested it now but will do in a bit.

Comment 29

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: use absolute paths for windows, different for make vs pymake (v2.1)

Hah, as I spoke about there was a change lately. Now updated to tip.

Comment 30

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 711348 [details] [diff] [review]
use absolute paths for windows, different for make vs pymake (v2.1)

This doesn't work. We somewhat mangle the topsrcdir, most likely when stripping of the _ set in configure.in.

No rule to make target '../../../:/ozilla/code/firefox/nightly/media/webrtc/trun
k/src/common_audio/common_audio.gyp' needed by ['../../../:/ozilla/code/firefox/
nightly/media/webrtc/trunk/src/common_audio/common_audio.gyp']

Comment 31

[Honza Bambas (:mayhemer)]

Result of using the patch for obj dir in the source:

No rule to make target '../../../:/ozilla/src/mozilla-central/media/webrtc/trunk/webrtc/common_video/common_video.gyp

I cannot build w/o webrtc as well as with webrtc.  I'm really done!!!

Comment 32

[(not currently active) Ted Mielczarek]

Let's just make this write out Windows-style paths and stop supporting gmake. We have a patch to require pymake in bug 828317 anyway.

Comment 33

[Ed Morley [:emorley]]

Have now been unable to build on Windows for >6 months :-(

Comment 34

[Justin Lebar (not reading bugmail)]

Randell, if we don't fix this, it sends a pretty strong message that the next time WebRTC breaks something, we should back it out immediately, because otherwise the issue won't get fixed for months.  I'm not sure you want to establish that precedent...

Comment 35

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I tried to fix it with Randells patch but wasn't able to do so given that I don't know that much about the build system yet.

Comment 36

[(not currently active) Ted Mielczarek]

I think bug 775939 might actually fix this. We dropped support for GNU make on Windows (bug 828317), so writing out absolute paths should be fine now.

Comment 37

[[e-mail]]

@Ted: Will the fix from https://bugzilla.mozilla.org/show_bug.cgi?id=775939 land in Beta?

Comment 38

[(not currently active) Ted Mielczarek]

No, because bug 828317 only got fixed for Firefox 24. You can certainly apply that patch locally, though.

Comment 39

[Maire Reavy [:mreavy]]

Is this bug still relevant?

Comment 40

[Randell Jesup [:jesup] (needinfo me)]

This seems to work now.