Closed Bug 1402915 Opened 7 years ago Closed 6 years ago

Clang build fails because needed env vars are not set if the compiler is not msvc

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Platform:
Unspecified
Windows
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox60 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla60
Tracking Flags:
Tracking Status
firefox60 --- fixed

People

(Reporter: marco, Assigned: away)

References

Details

Attachments

(1 file)

Possible patch?
1.40 KB, patch
froydnj
: review+
kats
: feedback+
Details | Diff | Splinter Review 
I'm using Clang 5.0.0 (with a couple of custom patches that I'm sure can't affect the build), MozillaBuild 3.0 and Visual Studio 2015 Update 3.

The build is failing with "midl : command line error MIDL1001 : cannot open input file objidl.idl".

I was able to build earlier today with a older mozilla-central changeset that didn't require SDK version 10.0.10586.0.
I both updated MozillaBuild (from 2.2.0 to 3.0) and installed the required SDK, and I couldn't build anymore. 
So, it's hard to say where the problem is coming from (since everything was updated at the same time).

I've tried to uninstall and reinstall VS2015 with no luck.

If I disable accessibility, I no longer see the midl error, but I see an error while compiling ("no member named 'memalign' in the global namespace").
Component: General → Build Config
Product: Firefox → Core
It looks like it doesn't find winver.h (if I disable accessibility).

I guess the build system is not setting the env variables that the vcvars script used to set.
The SDK is detected correctly:
 0:04.72 checking for Windows SDK... 0x0a00 in 'C:\Program Files (x86)\Windows Kits\10\'
 0:04.72 checking for Universal CRT SDK... 10.0.10586.0 in 'C:\Program Files (x86)\Windows Kits\10\'
If I manually set vc_path, I get the following error while compiling:
 0:08.70 c:/FD/mozilla-central/memory/build/malloc_decls.h(52,1):  error: no member named 'memalign' in the global namespace; did you mean simply 'memalign'?
 0:08.70 MALLOC_DECL(memalign, void *, size_t, size_t)
 0:08.70 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 0:08.70 c:/FD/mozilla-central/memory/build/fallback.cpp(15,12):  note: expanded from macro 'MALLOC_DECL'
 0:08.70     return ::name(ARGS_HELPER(ARGS, ##__VA_ARGS__)); \
 0:08.70            ^~
 0:08.70 c:/FD/mozilla-central/memory/build/malloc_decls.h(52,13):  note: 'memalign' declared here
 0:08.70 MALLOC_DECL(memalign, void *, size_t, size_t)
 0:08.70             ^
 0:08.70 1 error generated.
 0:08.72 mozmake.EXE[5]: *** [c:/FD/mozilla-central/config/rules.mk:1065: Unified_cpp_memory_build0.obj] Error 1
 0:08.72 mozmake.EXE[4]: *** [c:/FD/mozilla-central/config/recurse.mk:73: memory/build/target] Error 2
As expected, downgrading to MozillaBuild 2.2.0 solves the problems with the env vars.
Summary: Clang build with VS2015 Update 3 fails because of MIDL-related error → Clang build with VS2015 Update 3 fails because needed env vars are not set if the compiler is not msvc
(In reply to Marco Castelluccio [:marco] from comment #4)
> If I manually set vc_path, I get the following error while compiling:
>  0:08.70 c:/FD/mozilla-central/memory/build/malloc_decls.h(52,1):  error: no
> member named 'memalign' in the global namespace; did you mean simply
> 'memalign'?
>  0:08.70 MALLOC_DECL(memalign, void *, size_t, size_t)
>  0:08.70 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>  0:08.70 c:/FD/mozilla-central/memory/build/fallback.cpp(15,12):  note:
> expanded from macro 'MALLOC_DECL'
>  0:08.70     return ::name(ARGS_HELPER(ARGS, ##__VA_ARGS__)); \
>  0:08.70            ^~
>  0:08.70 c:/FD/mozilla-central/memory/build/malloc_decls.h(52,13):  note:
> 'memalign' declared here
>  0:08.70 MALLOC_DECL(memalign, void *, size_t, size_t)
>  0:08.70             ^
>  0:08.70 1 error generated.
>  0:08.72 mozmake.EXE[5]: *** [c:/FD/mozilla-central/config/rules.mk:1065:
> Unified_cpp_memory_build0.obj] Error 1
>  0:08.72 mozmake.EXE[4]: *** [c:/FD/mozilla-central/config/recurse.mk:73:
> memory/build/target] Error 2

This is bug 1402647, so it's unrelated.
Since I have installed only VC17 I'm gonna test it with mozbuild 2.2.0 to see if using the VC15 pipeline works.
I think the problem is that setting the compiler to clang skips all the moz.configure changes that set up the build environment. Various MSVC directories aren't being added to various environment variables and clang-cl can't find necessary files.

Bug 1401627 may help with some of this. But it's not the full solution since moz.configure needs to accept clang-cl as an msvc substitute.
Depends on: 1401627
(In reply to Gregory Szorc [:gps] from comment #8)
> I think the problem is that setting the compiler to clang skips all the
> moz.configure changes that set up the build environment. Various MSVC
> directories aren't being added to various environment variables and clang-cl
> can't find necessary files.

Yes, see comment 3. If I manually set the vc_path the necessary env vars are defined.
This apparently affects latest MSVC2017 and MozillaBuild 3.1.1 too.
The title of this bug keeps confusing me, so let me make sure I understand: the problem here is not "the clang build fails with VS2015u3" but rather "the clang build fails with MozillaBuild3 and/or VS2017" -- is that right?

As I understand it, a machine with only VS2015 (the original, not VS2017's extra toolset) and MozillaBuild 2.x, ought to work. At least that's been my experience.

If that's the case, then this bug should be about making the new-style MSVC detection (where we use vswhere and configure logic rather than vsNNNN batch files) do the right thing when the compiler is clang-cl. I agree this is a very much needed fix. I've been going to great lengths to hack around this all the time.
(In reply to David Major [:dmajor] from comment #11)
> The title of this bug keeps confusing me, so let me make sure I understand:
> the problem here is not "the clang build fails with VS2015u3" but rather
> "the clang build fails with MozillaBuild3 and/or VS2017" -- is that right?

I have only tested an environment with VS2015u3 and MozillaBuild 3 myself, but according to gcp
it fails with VS2017 and MozillaBuild 3 too.
I don't know if an environment with VS2017 and MozillaBuild 2 works.

> As I understand it, a machine with only VS2015 (the original, not VS2017's
> extra toolset) and MozillaBuild 2.x, ought to work. At least that's been my
> experience.

Correct. VS2015u3 and MozillaBuild 2 works for me.

> If that's the case, then this bug should be about making the new-style MSVC
> detection (where we use vswhere and configure logic rather than vsNNNN batch
> files) do the right thing when the compiler is clang-cl. I agree this is a
> very much needed fix. I've been going to great lengths to hack around this
> all the time.

Agreed!
Summary: Clang build with VS2015 Update 3 fails because needed env vars are not set if the compiler is not msvc → Clang build fails because needed env vars are not set if the compiler is not msvc
Blocks: 1428349
gps: This bug is becoming increasingly painful as more people are moving to MozillaBuild 3 and VS2017 and as we have more clang-based projects going on. I wish I could fix it myself but it's beyond my build knowledge. Could you please recommend an owner for this?
Flags: needinfo?(gps)
froydnj: I've seen you making some noise about MSVC things in #build recently. And I've heard from backchannels that people in your realm of the org are starting to have issues with this. Would you be willing to take a look?
Flags: needinfo?(gps) → needinfo?(nfroyd)
(In reply to Gregory Szorc [:gps] from comment #14)
> froydnj: I've seen you making some noise about MSVC things in #build
> recently. And I've heard from backchannels that people in your realm of the
> org are starting to have issues with this. Would you be willing to take a
> look?

I will never make noise about MSVC things again.

I will take a look later this week.  ni? myself to remind me.  Does anybody have a set of steps to take to get to a setup that causes problems?  And/or a list of particular setups that cause problems?
Flags: needinfo?(nfroyd)
Oh pick me! pick me!

I have Mozilla Build 3.0, MSVC 2017. Clean checkout of gecko-dev from github at e73379a010 (which is tip as of my writing this). Mozconfig and build output from ./mach build are at https://gist.github.com/anonymous/d5c2e48692180d273a77161b1c40f412

I have clang installed to /c/Program Files/LLVM and the bin folder is on my PATH. The clang version is 6.0.0 (trunk), I just downloaded the prebuilt thing from their website.

Let me know if you need more info!
(Looks like you accidentally removed your ni in comment 15? Putting it back...)
Flags: needinfo?(nfroyd)
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #16)
> Oh pick me! pick me!
> 
> I have Mozilla Build 3.0, MSVC 2017. Clean checkout of gecko-dev from github
> at e73379a010 (which is tip as of my writing this). Mozconfig and build
> output from ./mach build are at
> https://gist.github.com/anonymous/d5c2e48692180d273a77161b1c40f412
> 
> I have clang installed to /c/Program Files/LLVM and the bin folder is on my
> PATH. The clang version is 6.0.0 (trunk), I just downloaded the prebuilt
> thing from their website.
> 
> Let me know if you need more info!

Can you gist your config.status along with that?
Flags: needinfo?(bugmail)
Here you go: https://gist.github.com/anonymous/90983d7d4d4c076132252951dee76be2

(This was done with a newer gecko-dev version, and in a different working dir, so things might not line up exactly with the previous gist).
Flags: needinfo?(bugmail)
Just FWIW, I think gps summarized the problem accurately:

(In reply to Gregory Szorc [:gps] from comment #8)
> I think the problem is that setting the compiler to clang skips all the
> moz.configure changes that set up the build environment. Various MSVC
> directories aren't being added to various environment variables and clang-cl
> can't find necessary files.

MozillaBuild 3.0 doesn't do any detection of Visual C++ paths, so it doesn't set PATH/LIB/INCLUDE at all. We added code to configure to do this detection:
* Locating VC installations: https://dxr.mozilla.org/mozilla-central/rev/6d8f470b2579e7570f14e3db557264dc075dd654/build/moz.configure/toolchain.configure#556
* Setting LIB/INCLUDE/etc: https://dxr.mozilla.org/mozilla-central/source/build/moz.configure/windows.configure

However, some of those bits have explicit MSVC checks:
https://dxr.mozilla.org/mozilla-central/rev/6d8f470b2579e7570f14e3db557264dc075dd654/build/moz.configure/windows.configure#250
https://dxr.mozilla.org/mozilla-central/rev/6d8f470b2579e7570f14e3db557264dc075dd654/build/moz.configure/windows.configure#287

It may be as simple as fixing those to un-break this.
Attached patch Possible patch? — — Splinter Review 
Thanks for the pointers Ted!

Things appear to work if I just hardcode a search for cl.exe when we're doing a clang-cl build. I'm not entirely sure whether this is an ugly hack or a mere acceptance of the fact that we still rely on having an MSVC installation.

I'd like for someone to tell me I'm not crazy before I move this forward...
(In reply to David Major [:dmajor] from comment #21)
> Things appear to work if I just hardcode a search for cl.exe when we're
> doing a clang-cl build. I'm not entirely sure whether this is an ugly hack
> or a mere acceptance of the fact that we still rely on having an MSVC
> installation.

I think we want to rely on having an MSVC installation for users.  In automation it's a little different, but automation is also a solved problem today, so...

> I'd like for someone to tell me I'm not crazy before I move this forward...

Kats, does dmajor's fix work for you?
Flags: needinfo?(bugmail)
It certainly gets a lot further. However it still fails. Build output (not the whole thing, but an incremental re-run) is at https://gist.github.com/anonymous/7011f74dc2a9f867b7d07a4925b36167 - it shows an error in one of the Windows Kits headers, and I included that bit as well.
Flags: needinfo?(bugmail)
I see that you were using SDK 16299 which has known incompatibilities with clang-cl (bug 1413675). Is it any better with 15063?
Flags: needinfo?(bugmail)
Uninstalling SDK 16299 (so that it used SDK 15063) fixed the problem, and the build completed successfully! \o/
Flags: needinfo?(bugmail)
Comment on attachment 8950723 [details] [diff] [review]
Possible patch?

Review of attachment 8950723 [details] [diff] [review]:
-----------------------------------------------------------------

Assuming this passes automation as well as local builds, let's go ahead and land this.  We should probably have configure detect incompatible SDKs for clang cl as well, but that can be a followup bug.
Attachment #8950723 - Flags: review+
I love bugs assigned to me that I don't have to fix!
Assignee: nobody → dmajor
Status: NEW → ASSIGNED
Flags: needinfo?(nfroyd)
Pushed by dmajor@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/da5d7ba9a855
In clang-cl builds, use MSVC paths for INCLUDE/LIB/etc. r=froydnj
https://hg.mozilla.org/mozilla-central/rev/da5d7ba9a855
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox60: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: