1646363 - mach build throws KeyError: ''$srcdir/mozilla-central/obj-x86_64-pc-mingw32/dom/bindings'

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P3)

Description

[kenrick95]

Attachment: error.log

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0

Steps to reproduce:

On Windows 10 (64-bit).

Run mach bootstrap, choose 2 (Firefox Desktop, non-artifact), and then run mach build.

Actual results:

It throws error. More detailed logs attached

Traceback (most recent call last):
  File "$srcdir/obj-x86_64-pc-mingw32/config.status", line 837, in <module>
    config_status(**args)
  File "$srcdir\python\mozbuild\mozbuild\config_status.py", line 145, in config_status
    definitions = list(definitions)
  File "$srcdir\python\mozbuild\mozbuild\frontend\emitter.py", line 193, in emit
    objs = list(self._emit_libs_derived(contexts))
  File "$srcdir\python\mozbuild\mozbuild\frontend\emitter.py", line 227, in _emit_libs_derived
    collection = cls(contexts[root])
KeyError: '$srcdir/obj-x86_64-pc-mingw32/dom/bindings'

Expected results:

It should not throw error

Note that if I add ac_add_options --enable-artifact-builds at mozconfig, I am able to build (using Artifact Build) successfully

Comment 1

[Mike Hommey [:glandium]]

Smells like a problem with the case of the drive letter.

Comment 2

[kenrick95]

Okay looks like I have new information that might be related to this bug:

My repository is located at D:\Cloud\Mercurial\mozilla-central and the mozilla-build folder is left at its default path at C:\mozilla-build\

So, as suggested by the Windows setup docs, I created a shortcut to C:\mozilla-build\start-shell.bat on my desktop.

Now if I use that shortcut, not modifying anything, it will start a MSYS shell at $HOME, to go to the mozilla-central repo, I have to cd /d/Cloud/Mercurial/mozilla-central/, and then to start a build, I typed mach build

At this point, the build (whether it is artifact build, or non-artifact build) can be run successfully.

So since I want to skip that "cd" command every time, I edited the shortcut to specify "Start in: D:\Cloud\Mercurial\mozilla-central", and also edited C:\mozilla-build\msys\etc\profile where I commented out the line cd "$HOME".

This way, when I use that shortcut, the MSYS shell started in /d/Cloud/Mercurial/mozilla-central/ already.

At this point, if I run mach build, only the artifact build is successful, the non-artifact build fails with error as described in the ticket initial description.

So yeah, if I revert my changes at that C:\mozilla-build\msys\etc\profile and have to bear running cd to the correct location, the non-artifact build actually proceeds normally!

I hope this provide some useful information on finding the root cause. Let me know if you need more information. Thanks :)

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

That's really helpful, thanks! I have a reproduce working locally, and it looks like an issue with drive letter capitalization. I'm currently determining what causes different parts of the build to use different drive letter cases.

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

I've got a workaround for you: put your settings back the way you want (cd $HOME removed from profile), then ./mach clobber. This will purge the local build configuration, allowing you to build from scratch. At this point, it should work properly.

Explanation

Windows is ... weird. Depending on whether you cd into your other drive or "start your shell" there, the drive letter is a different capitalization. You can test this theory by:

1. Put mozilla-build back to how it is at install-time
2. Run start-shell.bat
3. cd /d/Cloud/Mercurial/mozilla-central
4. python3 -c "import os; print(os.getcwd())"

   • "d:\Cloud\Mercurial\mozilla-central"

5. Edit your shortcut so it starts in D:\<snip>\mozilla-central, remove the cd $HOME from profile.
6. Run start-shell.bat
7. python3 -c "import os; print(os.getcwd())"

   • "D:\Cloud\Mercurial\mozilla-central"

Why the build failed

I'm guessing that the first time you did ./mach build, it was after manually performing that cd. When the "configure" stage runs, a config.status file is written into the objdir (obj-x86_64-pc-mingw32). This file has some paths, such as c:/Cloud/Mercurial/mozilla-central/obj-x86_64-pc-mingw32/dom/bindings. Later, after you changed your settings and ran ./mach build, python thinks your drive letter is capitalized. When it does some string comparisons against the existing config.status values, they unexpectedly don't match up, and the script balks.

Should the build be resilient to different drive letter casings?

I'm not sure about this, this seems like a pretty large undertaking to do. We have a workaround (clobber if your build environment changes ... even if it feels like such a small change shouldn't have affected the build environment). Ricky, what do you think? One lightweight option we can do is to encode the drive letter case at configure time, then assert against on every subsequent build and throw a helpful error if it doesn't match up nicely.

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Ricky has mentioned that we should ensure the values in config.status are normcase()'d, and the same for the build values that are being compared against them.

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1646363: "mach build" should be resilient to drive letter capitalization r?rstewart

Depending on how MozillaBuild is invoked, python may think that the drive letter is a different case.
Since some paths are saved between builds in "config.status", this inconsistency can cause issues.

Though this patch doesn't solve the underlying inconsistency, it resolves the crash that was occurring.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5653dc323bc1
"mach build" should be resilient to drive letter capitalization r=rstewart

Comment 8

[Cosmin Sabou [:CosminS]]

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&revision=5653dc323bc1c67815f35c1865abea409418e554&failure_classification_id=2&selectedTaskRun=PFdGnG3cQFaZLixy_uzR7Q.0

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=308077957&repo=autoland
https://treeherder.mozilla.org/logviewer.html#?job_id=308077930&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/38f0f9287b897fea1845d4613e9a836771da258e

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

Updated the patch, have a try job running here

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/aa02739c58d6
"mach build" should be resilient to drive letter capitalization r=rstewart

Comment 11

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/aa02739c58d6