Closed Bug 1367995 Opened 3 years ago Closed 1 year ago

Building geckodriver on local Windows builds fails with: failed to run custom build command for `miniz-sys v0.1.9`

Categories

(Testing :: Marionette, defect, P3)

All
Windows
defect

Tracking

(firefox-esr52 unaffected, firefox53 unaffected, firefox54 unaffected, firefox55 fixed)

RESOLVED WORKSFORME
Tracking Status
firefox-esr52 --- unaffected
firefox53 --- unaffected
firefox54 --- unaffected
firefox55 --- fixed

People

(Reporter: Paenglab, Assigned: ato)

References

Details

Attachments

(2 files)

0:56.06 error: failed to run custom build command for `miniz-sys v0.1.9`
 0:56.09 process didn't exit successfully: `z:\Mozilla\TB\testing\geckodriver\.\release\build\miniz-sys-0d42c5494ba5bc47\build-script-build` (exit code: 101)
 0:56.10 --- stdout
 0:56.10 TARGET = Some("i686-pc-windows-msvc")
 0:56.10 OPT_LEVEL = Some("3")
 0:56.10 TARGET = Some("i686-pc-windows-msvc")
 0:56.10 HOST = Some("x86_64-pc-windows-msvc")
 0:56.10 TARGET = Some("i686-pc-windows-msvc")
 0:56.10 TARGET = Some("i686-pc-windows-msvc")
 0:56.10 HOST = Some("x86_64-pc-windows-msvc")
 0:56.11 CC_i686-pc-windows-msvc = None
 0:56.11 CC_i686_pc_windows_msvc = None
 0:56.11 TARGET_CC = None
 0:56.11 CC = None
 0:56.12 TARGET = Some("i686-pc-windows-msvc")
 0:56.12 HOST = Some("x86_64-pc-windows-msvc")
 0:56.12 CFLAGS_i686-pc-windows-msvc = None
 0:56.12 CFLAGS_i686_pc_windows_msvc = None
 0:56.12 TARGET_CFLAGS = None
 0:56.12 CFLAGS = None
 0:56.13 PROFILE = Some("release")
 0:56.13 running: "cl.exe" "/nologo" "/MD" "/O2" "/Foz:\\Mozilla\\TB\\testing\\geckodriver\\.\\i686-pc-windows-msvc\\release\\build\\miniz-sys-18005000ddedadf4\\out\\miniz.o" "/c" "miniz.c"
 0:56.13
 0:56.13
 0:56.13 failed to execute command: Das System kann die angegebene Datei nicht finden. (os error 2)
 0:56.13 Is `cl.exe` not installed? (see https://github.com/alexcrichton/gcc-rs#compile-time-requirements for help)

I tried to set the path explained in https://github.com/alexcrichton/gcc-rs#compile-time-requirements, but didn't success.

What do I need to do? I'm not a compiler specialist.
Me too. Unable to compile suite. Is probably caused by Bug 1340637. From the patch it looks like 
it doesn't take the comm-central dir into accound and tries to find the source in the topmost sourcedir:

/Foz:\\Mozilla\\TB\\testing\\

Should probably be /Foz:\\Mozilla\\TB\\mozilla\\testing\\ in case of comm-central builds. Best locally would probably to have an option to disable it. I see not much if any use for it locally. That still would still ned a fix for the offcial TB and SM builds if it is needed there.
Andreas and Ted, it seems that changes in bug 1340637 busted local builds, at least for Thunderbird for some people.

I tried local Firefox and Thunderbird 32bit debug builds and they seem to work.

Richard, can you please attach a longer log of the first 56 seconds to see where the error happens. My build is happily compiling ;-)
Flags: needinfo?(ted)
Flags: needinfo?(ato)
I don't have the log anymore. But before where no errors. If needed I have a older log (the initial build which failed) with output before the error.
Attached file 1367995.txt - Log
I spoke too early, it fails for me, too, after eight(!!) minutes.

If it's a "top source dir" issue, then FF local build won't be affected. I'm just wondering why server builds on treeherder still work for C-C. I've just retriggered a Mac opt
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=abc0bcd3caec68343fce1ff4fcb700edd6c72076&selectedJob=102237328
and that worked.
>> /Foz:\\Mozilla\\TB\\testing\\geckodriver

This is the object dir so I take it back that it might be a toplevel source problem. If the script tries to run the x86 cl.exe for cross compile it might miss some files from the x64 version. The server uses the tooltool package and might not be affected. Anyway just speculation.
A quick search indicates cl.exe is not exposed in the Windows command prompt by default unless you enable the ‘SDK command line’ or install the C++ requirements in Visual Studio.  Possibly you need to start a Visual Studio command prompt that sets all the correct path variables for it to pick up cl.exe correctly.

We need cl.exe to compile some C libraries as part of a Rust dependency used by testing/geckodriver.

Note that with https://bugzilla.mozilla.org/show_bug.cgi?id=1368035 we are making the compilation of testing/geckodriver opt-in rather than mandatory on local developer machines.  Compiling it in CI will still be required as it is used to run the WPT wdspec tests.
Flags: needinfo?(ato)
I local Firefox build fails as well. So basically you've broken all local compiles on Windows for FF, TB and SM.

So can you please come up with a fix or workaround real quick.
Severity: normal → blocker
Flags: needinfo?(ato)
Summary: error: failed to run custom build command for `miniz-sys v0.1.9` → local Windows builds for Firefox and Thunderbird broken: error: failed to run custom build command for `miniz-sys v0.1.9`
Hardware: Unspecified → All
I am temporarily disabling compilation of testing/geckodriver in https://bugzilla.mozilla.org/show_bug.cgi?id=1368084 until https://bugzilla.mozilla.org/show_bug.cgi?id=1368035, to make it opt-in for local developers, is fixed.
Assignee: nobody → ato
Status: NEW → ASSIGNED
Depends on: 1368084
Flags: needinfo?(ted)
Flags: needinfo?(ato)
Blocks: 1340637
Keywords: regression
Duplicate of this bug: 1367935
(In reply to Andreas Tolfsen ‹:ato› from comment #6)
> A quick search indicates cl.exe is not exposed in the Windows command prompt
> by default unless you enable the ‘SDK command line’ or install the C++
> requirements in Visual Studio.  Possibly you need to start a Visual Studio
> command prompt that sets all the correct path variables for it to pick up
> cl.exe correctly.

The problem is not this. As I mentioned in bug 1367935 comment 4, I'm able to build geckodriver via cargo directly from the source code in the same environment, which means that it is mach which hides the necessary environment variables.

It reminds me of bug 1350001 which removed almost all environment variables for Windows when calling into cargo, which then broke stylo build (see bug 1355464). They could be related.
(In reply to Jorg K (GMT+2) from comment #11)
> Created attachment 8871793 [details] [diff] [review]
> Patch by FRG to get the compile going again ;-) - final solution may differ.

That's the same workaround I came up with in bug 1367935 comment 2 :)
(In reply to Xidorn Quan [:xidorn] UTC+10 from comment #10)
> (In reply to Andreas Tolfsen ‹:ato› from comment #6)
> > A quick search indicates cl.exe is not exposed in the Windows command prompt
> > by default unless you enable the ‘SDK command line’ or install the C++
> > requirements in Visual Studio.  Possibly you need to start a Visual Studio
> > command prompt that sets all the correct path variables for it to pick up
> > cl.exe correctly.
> 
> The problem is not this. As I mentioned in bug 1367935 comment 4, I'm able
> to build geckodriver via cargo directly from the source code in the same
> environment, which means that it is mach which hides the necessary
> environment variables.
> 
> It reminds me of bug 1350001 which removed almost all environment variables
> for Windows when calling into cargo, which then broke stylo build (see bug
> 1355464). They could be related.

This is very interesting information.  I will follow up on this.

In the meantime I’m resolving this bug since the build problems on m-c are fixed by https://bugzilla.mozilla.org/show_bug.cgi?id=1368084.
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
> This is very interesting information.  I will follow up on this.

If you need to call cl.exe directly also take into account the the x86 cross compiler needs the x64 libs and that VS2017 also has a different directory structure. 

https://bugzilla.mozilla.org/show_bug.cgi?id=1318143#c34
xidorn (or someone else who can reproduce this): can you `cargo run` this test app from the same shell where you're building Firefox? It's just using the same version of the gcc crate that we have vendored, and printing the MSVC paths it returns.

On my Windows machine, which has MSVC 2015 and 2017 installed, in a MozillaBuild start-shell.bat shell it prints:
CC_x86_64-pc-windows-msvc = None
CC_x86_64_pc_windows_msvc = None
HOST_CC = None
CC = None
CFLAGS_x86_64-pc-windows-msvc = None
CFLAGS_x86_64_pc_windows_msvc = None
HOST_CFLAGS = None
CFLAGS = None
64-bit MSVC path: "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\amd64\\cl.exe"
CC_i686-pc-windows-msvc = None
CC_i686_pc_windows_msvc = None
TARGET_CC = None
CC = None
CFLAGS_i686-pc-windows-msvc = None
CFLAGS_i686_pc_windows_msvc = None
TARGET_CFLAGS = None
CFLAGS = None
32-bit MSVC path: "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\amd64_x86\\cl.exe"
Flags: needinfo?(xidorn+moz)
$ cargo run
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading gcc v0.3.42
   Compiling gcc v0.3.42
   Compiling msvc-test v0.1.0 (file:///J:/temp/rust-msvc-test-master/rust-msvc-test-master)
    Finished dev [unoptimized + debuginfo] target(s) in 12.78 secs
     Running `target\debug\msvc-test.exe`
CC_x86_64-pc-windows-msvc = None
CC_x86_64_pc_windows_msvc = None
HOST_CC = None
CC = None
CFLAGS_x86_64-pc-windows-msvc = None
CFLAGS_x86_64_pc_windows_msvc = None
HOST_CFLAGS = None
CFLAGS = None
64-bit MSVC path: "d:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\amd64\\cl.exe"
CC_i686-pc-windows-msvc = None
CC_i686_pc_windows_msvc = None
TARGET_CC = None
CC = None
CFLAGS_i686-pc-windows-msvc = None
CFLAGS_i686_pc_windows_msvc = None
TARGET_CFLAGS = None
CFLAGS = None
32-bit MSVC path: "d:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\amd64\\cl.exe"

(I installed MSVC 2015 on D: because my C: is a SSD)
This seems wrong:
32-bit MSVC path: "d:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\amd64\\cl.exe"

You don't have any MSVC environment variables set in that shell, do you?
What are "MSVC environment variables"? I'm just running start-shell-msvc2015-x64.bat.
If I run start-shell-msvc2015.bat, I got:

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.4 secs
     Running `target\debug\msvc-test.exe`
CC_x86_64-pc-windows-msvc = None
CC_x86_64_pc_windows_msvc = None
HOST_CC = None
CC = None
CFLAGS_x86_64-pc-windows-msvc = None
CFLAGS_x86_64_pc_windows_msvc = None
HOST_CFLAGS = None
CFLAGS = None
64-bit MSVC path: "d:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\amd64_x86\\cl.exe"
CC_i686-pc-windows-msvc = None
CC_i686_pc_windows_msvc = None
TARGET_CC = None
CC = None
CFLAGS_i686-pc-windows-msvc = None
CFLAGS_i686_pc_windows_msvc = None
TARGET_CFLAGS = None
CFLAGS = None
32-bit MSVC path: "d:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\amd64_x86\\cl.exe"
By the way, I backed out bug 1368084 and I got the following error:
 1:58.69 failed to execute command: 隰厄ソス・ョ螢ケ・・ケァ蠕娯螺郢晁シ斐<郢ァ・、郢晢スォ邵コ迹夲スヲ荵昶命邵コ荵晢ス顔クコ・セ邵コ蟶呻ス鍋クイ・ス(os error 2)
Looks like a wrong encoding is used.
Target Milestone: --- → mozilla55
start-shell-msvc2015-x64.bat gives me:

$ cargo run
   Compiling gcc v0.3.42
   Compiling msvc-test v0.1.0 (file:///C:/mozilla-source/rust-msvc-test)
    Finished dev [unoptimized + debuginfo] target(s) in 3.21 secs
     Running `target\debug\msvc-test.exe`
CC_x86_64-pc-windows-msvc = None
CC_x86_64_pc_windows_msvc = None
HOST_CC = None
CC = None
CFLAGS_x86_64-pc-windows-msvc = None
CFLAGS_x86_64_pc_windows_msvc = None
HOST_CFLAGS = None
CFLAGS = None
64-bit MSVC path: "c:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\amd64\\cl.exe"
CC_i686-pc-windows-msvc = None
CC_i686_pc_windows_msvc = None
TARGET_CC = None
CC = None
CFLAGS_i686-pc-windows-msvc = None
CFLAGS_i686_pc_windows_msvc = None
TARGET_CFLAGS = None
CFLAGS = None
32-bit MSVC path: "c:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\amd64\\cl.exe"


Seems to be similar to emk's.
Flags: needinfo?(xidorn+moz)
(In reply to Andreas Tolfsen ‹:ato› from comment #13)
> In the meantime I’m resolving this bug since the build problems on m-c are
> fixed by https://bugzilla.mozilla.org/show_bug.cgi?id=1368084.

Is there a bug that tracks the underlying issue so that we can re-enable geckodriver building on Windows?
I don't think it's a good idea to continue works on a closed bug.
Apparently it does not take care of failures in local builds.
(In reply to Masatoshi Kimura [:emk] from comment #23)
> (In reply to Andreas Tolfsen ‹:ato› from comment #13)
> > In the meantime I’m resolving this bug since the build problems on m-c are
> > fixed by https://bugzilla.mozilla.org/show_bug.cgi?id=1368084.
> 
> Is there a bug that tracks the underlying issue so that we can re-enable
> geckodriver building on Windows?

We can use this bug to track that, certainly.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Severity: blocker → normal
Component: Build Config → Marionette
Keywords: regression
Product: Core → Testing
Summary: local Windows builds for Firefox and Thunderbird broken: error: failed to run custom build command for `miniz-sys v0.1.9` → Building geckodriver on local Windows builds fails with: failed to run custom build command for `miniz-sys v0.1.9`
Version: unspecified → Trunk
Target Milestone: mozilla55 → ---
Oh, I got the correct result if I run start-shell.bat:
$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.4 secs
     Running `target\debug\msvc-test.exe`
CC_x86_64-pc-windows-msvc = None
CC_x86_64_pc_windows_msvc = None
HOST_CC = None
CC = None
CFLAGS_x86_64-pc-windows-msvc = None
CFLAGS_x86_64_pc_windows_msvc = None
HOST_CFLAGS = None
CFLAGS = None
64-bit MSVC path: "D:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\amd64\\cl.exe"
CC_i686-pc-windows-msvc = None
CC_i686_pc_windows_msvc = None
TARGET_CC = None
CC = None
CFLAGS_i686-pc-windows-msvc = None
CFLAGS_i686_pc_windows_msvc = None
TARGET_CFLAGS = None
CFLAGS = None
32-bit MSVC path: "D:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\amd64_x86\\cl.exe"

But the current latest release of MozillaBuild does not support building from start-shell.bat yet. I have to use start-shell-msvc2015(-x64).bat.
(In reply to Masatoshi Kimura [:emk] from comment #27)
> But the current latest release of MozillaBuild does not support building
> from start-shell.bat yet. I have to use start-shell-msvc2015(-x64).bat.

Hm, I was wrong. I could build Firefox with geckodriver enabled.
start-shell-msvc2015(-x64).bat will set $VCINSTALLDIR, so the tool detection will not run:
https://github.com/alexcrichton/gcc-rs/blob/0d87a9b739a9b0086015d5356de729f1dc1fb010/src/windows_registry.rs#L70-L74
Priority: -- → P3
I think this problem has been resolved.  At least we have had no
complaints that enabling geckodriver in the default local build has
caused any problems for Windows developers.
Status: REOPENED → RESOLVED
Closed: 3 years ago1 year ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.