Closed Bug 773423 Opened 12 years ago Closed 9 years ago

WebRTC build breaks when MOZ_OBJDIR is defined outside of TOPSRCDIR on Windows

Categories

(Core :: WebRTC, defect)

Product:
Core
Component:
WebRTC
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED WORKSFORME
Blocking Flags:
backlog parking-lot

People

(Reporter: justin.lebar+bug, Unassigned)

References

Details

(Whiteboard: [WebRTC], [blocking-webrtc-])

Attachments

(1 file, 3 obsolete files)

use absolute paths for windows, different for make vs pymake (v2.1)
8.61 KB, patch
Details | Diff | Splinter Review 
My build setup on Windows 7:

  srcdir is z:/src
  objdir is c:/objdir/ff-git-windows/debug

This configuration dies during the WebRTC build with:

> make.py[6]: Entering directory 'c:\objdir\ff-git-windows\debug\media\webrtc\trun
k'
> No rule to make target '../../../../../../../z:src/media/webrtc/trunk/src/voice_
engine/voice_engine.gyp' needed by ['../../../../../../../z:src/media/webrtc/tru
nk/src/voice_engine/voice_engine.gyp']
> No rule to make target '../../../../../../../z:src/media/webrtc/trunk/src/common
_video/common_video.gyp' needed by ['../../../../../../../z:src/media/webrtc/tru
nk/src/common_video/common_video.gyp']

and so on.  The apparent problem is that it's using "z:src/", which is not a real directory.

Here's my full mozconfig:

> . $topsrcdir/browser/config/mozconfig
> ac_add_options --disable-angle
> ac_add_options --disable-webgl
> 
> mk_add_options MOZ_OBJDIR=c:/objdir/ff-git-windows/debug
> ac_add_options --enable-debug
> ac_add_options --disable-optimize
> ac_add_options --enable-trace-malloc

I'm building with the following command:

> /c/objdir/ff-git-windows/debug$ /z/src/build/pymake/make.py -j8 -f /z/src/client.mk
Summary: WebRTC build breaks when srcdir and objdir are on different drives → WebRTC build breaks when srcdir and objdir are on different drives on Windows
Related to bug 772201 (symlinked objdirs on linux/mac)
Comment on attachment 642175 [details] [diff] [review]
use absolute paths for windows, different for make vs pymake

Or I'll take a review from any other build peer.... :-)

This undoes the "ignore windows" aspect of bug 772201, and with great pain generates correct absolute paths (in the header - topsrcdir/srcdir/VPATH) in all cases on all platforms.
Attachment #642175 - Flags: review?(mh+mozilla)
Comment on attachment 642175 [details] [diff] [review]
use absolute paths for windows, different for make vs pymake

Ted knows these msys vs windows issues better than i do.
Attachment #642175 - Flags: review?(mh+mozilla) → review?(ted.mielczarek)
Comment on attachment 642175 [details] [diff] [review]
use absolute paths for windows, different for make vs pymake

Review of attachment 642175 [details] [diff] [review]:
-----------------------------------------------------------------

My only real worry here is that this is going to break people who don't use client.mk and also build with pymake. (There are a fair amount of people that just configure && make.) If we make this decision in configure and we aren't running from client.mk, then we'll have no indication that we should be using Windows paths.

I guess the onus is on the user to provide the right kind of paths in that scenario? 

Really, there must be a way to do this properly, since the existing Makefile.in-to-Makefile writing logic handles this properly.

::: configure.in
@@ +8898,5 @@
>  
>  if test -n "$MOZ_WEBRTC"; then
>     AC_MSG_RESULT("generating WebRTC Makefiles...")
> +   # msys (on windows) does EVIL things to parameters to python it thinks are msys paths.
> +   # Since we already have the path in the format we want (c:/foo for pymake, 

nit: trailing whitespace at the end of this line

@@ +8902,5 @@
> +   # Since we already have the path in the format we want (c:/foo for pymake, 
> +   # /c/foo for make), we just want to stop msys from mucking with the string.
> +   # The underscores will be removed in mozmake.py.  We need the correct form of
> +   # absolute path for building topsrcdir= in Makefiles.
> +   MOZ_TOPSRCDIR="_${MOZ_TOPSRCDIR:0:2}_${MOZ_TOPSRCDIR:2}"

This is horrible, but I don't have a better solution for you.
Attachment #642175 - Flags: review?(ted.mielczarek) → review+
(In reply to Ted Mielczarek [:ted] from comment #5)
> Comment on attachment 642175 [details] [diff] [review]
> use absolute paths for windows, different for make vs pymake
> 
> Review of attachment 642175 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> My only real worry here is that this is going to break people who don't use
> client.mk and also build with pymake. (There are a fair amount of people
> that just configure && make.) If we make this decision in configure and we
> aren't running from client.mk, then we'll have no indication that we should
> be using Windows paths.
> 
> I guess the onus is on the user to provide the right kind of paths in that
> scenario? 
> 
> Really, there must be a way to do this properly, since the existing
> Makefile.in-to-Makefile writing logic handles this properly.

wouldn't bug 778236 essentially fix this?
Most likely, yes.
QA Contact: jsmith
Whiteboard: [WebRTC], [blocking-webrtc-]
Backed out:
  https://hg.mozilla.org/integration/mozilla-inbound/rev/3e6ed8c7aaa6

This busted linux clobber builds:

generating WebRTC Makefiles...
/builds/mozilla-inbound/configure: 27333: /builds/mozilla-inbound/configure: Bad substitution
*** Fix above errors and then restart with               "make -f client.mk build"
make[2]: *** [configure] Error 1
make[2]: Leaving directory `/builds/mozilla-inbound'
make[1]: *** [/builds/mozilla-inbound/objdir-ff-dbg/Makefile] Error 2
make[1]: Leaving directory `/builds/mozilla-inbound'
make: *** [build] Error 2


Line 27333 is:

   MOZ_TOPSRCDIR="_${MOZ_TOPSRCDIR:0:2}_${MOZ_TOPSRCDIR:2}"


I think this is only suppose to be substituted on windows.  I am sure it is an easy fix, but we need to keep the tree building.
Attachment #642175 - Attachment is obsolete: true
Comment on attachment 656334 [details] [diff] [review]
use absolute paths for windows, different for make vs pymake

The issue turned out to be Ubuntu using 'dash' for /bin/sh, and the change used a bash-ism  for getting a substr: ${var:start:len}

The recommended solution at https://wiki.ubuntu.com/DashAsBinSh (using awk) works fine on the commandline for me, but fails in configure (returns "0").  

So instead I made the insertion of the _'s conditional on the platform, which is annoying and non-optimal, but should work.
Attachment #656334 - Flags: review?(ted.mielczarek)
Comment on attachment 656334 [details] [diff] [review]
use absolute paths for windows, different for make vs pymake

Review of attachment 656334 [details] [diff] [review]:
-----------------------------------------------------------------

::: configure.in
@@ +8700,5 @@
> +   if test "$MOZ_WIDGET_TOOLKIT" == "windows"; then
> +     # Windows only because the other OSes don't have the problem; MUST
> +     # match the receiving side in mozmake.py!
> +     MOZ_TOPSRCDIR="_${MOZ_TOPSRCDIR:0:2}_${MOZ_TOPSRCDIR:2}"
> +   fi

I wonder if rather than jumping through these hoops if we shouldn't just write out a simple .gyp file here that defines MOZ_TOPSRCDIR, and forcibly include it from the gyp commandline. We'd have to modify mozmake.py to pull the variable out of the right place, but it would avoid this nonsense.
(In reply to Ted Mielczarek [:ted] from comment #13)
> I wonder if rather than jumping through these hoops if we shouldn't just
> write out a simple .gyp file here that defines MOZ_TOPSRCDIR, and forcibly
> include it from the gyp commandline. We'd have to modify mozmake.py to pull
> the variable out of the right place, but it would avoid this nonsense.

bug 778236 is where we're going.
BTW: This patch no longer applies on mozilla-central. I tried to fix this manually, but then another issue occurred (something about use_system_libvpx not being defined).
jessup said bug 798814 is a dupe of this, but my srcdir and objdir are on the same drive; how can I work around this in the meantime (other than having to sift through the patch)?
Actually the summary is incorrect, it happens if the objdir has an absolute path.
Summary: WebRTC build breaks when srcdir and objdir are on different drives on Windows → WebRTC build breaks when objdir has an absolute path on Windows
(In reply to Randell Jesup [:jesup] from comment #18)
> Actually the summary is incorrect, it happens if the objdir has an absolute
> path.

I was previously using an absolute path (see bug 798814), so tried switching to:
mk_add_options MOZ_OBJDIR=../../obj-inbound

...which still doesn't work, failing with the same error as in bug 798814 :-(
Severity: normal → critical
I cannot confirm Comment 19 (relative MOZ_OBJDIR being broken, too). If you have src- and objdir on two different partitions (like I do), one can use the NTFS symbolic link feature as a workaround for now:
Enter "mklink /d C:\mozilla\tree-hg\objdirs F:\mozilla" on the Windows command line (as Administrator) to create an NTFS symbolic link on the same partition as the srcdir, this works on Windows Vista and above (I have my srcdir in C:\mozilla\tree-hg\comm-central on an SSD, my objdirs are in F:\mozilla on a normal harddrive). As MOZ_OBJDIR I've set "../objdirs" in my .mozconfig.
Actually I've set "../objdirs/seamonkey-objdir" as objdir, but that should not really matter.
Please can we do something about this. Not being able to build locally without disabling webrtc is a PITA :-)
(We've also just had someone on #developers scratching their heads over a broken local build & it turned out to be this)
Adjusting summary given comment 19.
Summary: WebRTC build breaks when objdir has an absolute path on Windows → WebRTC build breaks when objdir has an absolute path on Windows (or even a relative path too)
(In reply to Ed Morley [:edmorley UTC+0] from comment #23)
> Please can we do something about this. Not being able to build locally
> without disabling webrtc is a PITA :-)

Attachment #696621 [details] [diff] of Bug 775939 maybe resolves this bus.
s/this bus./this bug./
Summary: WebRTC build breaks when objdir has an absolute path on Windows (or even a relative path too) → WebRTC build breaks when MOZ_OBJDIR is defined on Windows
Summary: WebRTC build breaks when MOZ_OBJDIR is defined on Windows → WebRTC build breaks when MOZ_OBJDIR is defined outside of TOPSRCDIR on Windows
The former patch has been bitrotted and doesn't apply anymore. I have updated it so it can be applied to recent code. Randell, can you please have a look at if everything is still ok and that no other build system changes invalidated parts of the code? When we are ok I will ask Ted for review.

I haven't tested it now but will do in a bit.
Assignee: nobody → hskupin
Attachment #656334 - Attachment is obsolete: true
Status: NEW → ASSIGNED
Attachment #656334 - Flags: review?(ted)
Attachment #711343 - Flags: feedback?(rjesup)
Hah, as I spoke about there was a change lately. Now updated to tip.
Attachment #711343 - Attachment is obsolete: true
Attachment #711343 - Flags: feedback?(rjesup)
Attachment #711348 - Flags: feedback?(rjesup)
Comment on attachment 711348 [details] [diff] [review]
use absolute paths for windows, different for make vs pymake (v2.1)

This doesn't work. We somewhat mangle the topsrcdir, most likely when stripping of the _ set in configure.in.

No rule to make target '../../../:/ozilla/code/firefox/nightly/media/webrtc/trun
k/src/common_audio/common_audio.gyp' needed by ['../../../:/ozilla/code/firefox/
nightly/media/webrtc/trunk/src/common_audio/common_audio.gyp']
Attachment #711348 - Flags: feedback?(rjesup)
Result of using the patch for obj dir in the source:

No rule to make target '../../../:/ozilla/src/mozilla-central/media/webrtc/trunk/webrtc/common_video/common_video.gyp

I cannot build w/o webrtc as well as with webrtc.  I'm really done!!!
Let's just make this write out Windows-style paths and stop supporting gmake. We have a patch to require pymake in bug 828317 anyway.
Have now been unable to build on Windows for >6 months :-(
Randell, if we don't fix this, it sends a pretty strong message that the next time WebRTC breaks something, we should back it out immediately, because otherwise the issue won't get fixed for months.  I'm not sure you want to establish that precedent...
I tried to fix it with Randells patch but wasn't able to do so given that I don't know that much about the build system yet.
Assignee: hskupin → nobody
Status: ASSIGNED → NEW
I think bug 775939 might actually fix this. We dropped support for GNU make on Windows (bug 828317), so writing out absolute paths should be fine now.
@Ted: Will the fix from https://bugzilla.mozilla.org/show_bug.cgi?id=775939 land in Beta?
No, because bug 828317 only got fixed for Firefox 24. You can certainly apply that patch locally, though.
Is this bug still relevant?
backlog: --- → parking-lot
Flags: needinfo?(rjesup)
QA Contact: jsmith
This seems to work now.
Status: NEW → RESOLVED
Closed: 9 years ago
Flags: needinfo?(rjesup)
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: