Open Bug 1762385 Opened 3 years ago Updated 3 years ago

Validate that newlines aren't in PATH

Categories

(Firefox Build System :: Mach Core, enhancement, P5)

Product:
Firefox Build System
Component:
Mach Core
Type:
enhancement

Tracking

(Not tracked)

Status:
REOPENED

People

(Reporter: aryx, Unassigned)

References

(Regression)

Details

$ ./mach build installers-de
 0:01.81 Clobber not needed.
  Parallelism determined by memory: using 8 jobs for 8 cores based on 15.9 GiB RAM and estimated job size of 1.0 GiB
 0:02.32 C:/Users/Eichelhaeher/.mozbuild/mozmake/mozmake.exe -j8 -s installers-de
 0:02.34 config/autoconf.mk:290: *** missing separator.  Stop.
 0:02.37 0 compiler warnings present.
Hey! Builds initiated with `mach build $A_SPECIFIC_TARGET` may not always work, even if the code being built is correct. Consider doing a bare `mach build` instead.

with

ac_add_options --enable-application=browser
ac_add_options --with-branding=browser/branding/official
ac_add_options --with-l10n-base=../../l10n-central

as mozconfig set with export MOZCONFIG=mozconfig-name

Works as expected with MozillaBuild 3.4.

Operating system Windows 8.1.

Hmm, I'm not able to reproduce this, but I haven't manually set up my l10n-base.

:aryx can you try the following?

  1. Try using the auto-managed l10n-base by pruning your mozconfig a bit:
ac_add_options --enable-application=browser
ac_add_options --with-branding=browser/branding/official
  1. If that^ doesn't work, then there's an issue on your local machine, and I'd recommend clobbering and checking out most up-to-date central
  2. Otherwise, please share your steps for having a manually-managed l10n-base and I'll follow the same steps and try to repro.
Flags: needinfo?(aryx.bugmail)

Thank you for the quick reply.

  1. Try using the auto-managed l10n-base by pruning your mozconfig a bit:
ac_add_options --enable-application=browser
ac_add_options --with-branding=browser/branding/official

This doesn't change the console output.

  1. If that^ doesn't work, then there's an issue on your local machine, and I'd recommend clobbering and checking out most up-to-date central

Similar for this (also had this done before reporting the bug). Active revision is https://hg.mozilla.org/mozilla-central/rev/260e22f03e984e0ced16b6c5ff63201cdef0a1f6

  1. Otherwise, please share your steps for having a manually-managed l10n-base and I'll follow the same steps and try to repro.

mozilla-central source folder: C:\Mozilla\Coding\Code\mozilla-central
l10n-central folder: C:\Mozilla\Coding\Code\l10n-central
^ contains a folder de which is a junction point to C:\Mozilla\L10n\Repositories\mozilla-central-de

Flags: needinfo?(aryx.bugmail)

Thanks! I'm away tomorrow, but I'll dig into this next week :)

This doesn't change the console output.

Hmm, that's a bit of a surprise.
As a "random guess", when you installed MozillaBuild, did you fully uninstall the old version first? If not, it can cause some implicit issues.

Either way, maybe I'll have more luck repro-ing this on my Windows host next week rather than my in-VM test today.

Flags: needinfo?(mhentges)

Yes, the old MozillaBuild folder got renamed to MozillaBuild-3.4 and is kept as backup. Removing the .mozbuild also didn't alter behavior.

An attempt with C:\Mozilla\Coding\l10n-central\de as l10n folder without the use of a junction point didn't fix it.

Ok, I've got a replica of your exact setup here locally:

  • MozillaBuild 4.0
  • mozilla-central: 260e22f03e98
  • l10n-central/de: 3434414ac67d
  • Junction point: C:\dev\l10n-central\de -> C:\dev\firefox-l10n-de
  • mozconfig:
ac_add_options --enable-application=browser
ac_add_options --with-branding=browser/branding/official
ac_add_options --with-l10n-base=../../l10n-central

However, ./mach clobber && ./mach build && ./mach package && ./mach build installers-de works for me locally.

Let's tackle this from a different direction to see why it's failing for you, at which point we might learn what makes your environment different from mine.

Can you do the following for me?

  1. Make sure that your l10n-central/de has the same revision as mine
  2. Do ./mach build && ./mach package, then use the verbose flag: ./mach build -j 1 -v installers-de. (the -j 1 is to make the logs more consistent).

Also, what's the result of sha256sum ~/.mozbuild/mozmake/mozmake.exe? Mine is 264014c3f66f3bf36054aea806524c527c765e023f5ab6b68ec0c7446dd3e6f1.

Flags: needinfo?(mhentges) → needinfo?(aryx.bugmail)

tl;dr: The value of the PATH variable in obj-x86_64-pc-mingw32\config\autoconf.mk gets split across 3 lines without any escapes for the linebreaks. If the backend change check gets disabled and the linebreaks removed, ./mach build works again (even the general build command is affected but I don't use is on Windows and only repack download binaries).

The unfixed autoconf.mk looks like this:

PATH = C:/Mozilla/Coding/Code/mozilla-central/obj-x86_64-pc-mingw32\_virtualenvs\build\Scripts;C:\Users\MyUsername\.mozbuild\srcdirs\mozilla-central-31702f4897ac\_virtualenvs\mach\Scripts;C:\Program
Files
(x86)\Hilfen\Gow\bin\;C:\Mozilla\Coding\Buildtools\MozillaBuild\bin;...

There are other directories in PATH which contain whitespaces, and copy and pasting the PATH values from system control into an editor doesn't highlight unexpected (non-printable) characters. It's only the first directory for which the value is spread across multiple lines.

Fwiw, mozmake.exe has the same checksum as mentioned in the previous comment.

Flags: needinfo?(aryx.bugmail)

Interesting, thanks for the write-up!

and copy and pasting the PATH values from system control into an editor doesn't highlight unexpected (non-printable) characters.

Hmm, that's helpful. What does echo $PATH print if run from MozillaBuild directly?
If it prints a line break, then we've got a MozillaBuild environment issue. If not, then the generation of autoconf.mk needs to be investigated.

Flags: needinfo?(aryx.bugmail)

Gets printed as one line.

Flags: needinfo?(aryx.bugmail)

Are there any escape characters in that line? Please share it here in Bugzilla.

I did some digging into how config/autoconf.mk is created, and it looks like this:

  1. The source is in autoconf.mk.in, and the specific line that's expanded problematically is here
  2. Here is where ALLSUBSTS is expanded
  3. ALLSUBSTS is originally defined here

So, TL;DR, the information that I'd like you to share is:

  1. The full output of echo $PATH
  2. The contents of $objdir/config.statusd/substs/PATH
  3. (If the above doesn't appear to have a surprise newline) If possible, the source of the unexpected newline if you trace ALLSUBSTS as defined above. If you'd prefer to pair program that debugging, I'm available for a Zoom meet :)
Flags: needinfo?(aryx.bugmail)

The issue starts earlier: a print(os.environ.get("PATH")) at the start of main() in configure.py already shows the issue. On the other hand, checking the value in ConfigureSandbox doesn't report the unexpected behavior.

Launching python3 or obj-x86_64-pc-mingw32/_virtualenvs/build/Scripts/python.exe and checking os.environ yields a sane list.

Flags: needinfo?(aryx.bugmail)

I wonder if it's _maybe_activate_mozillabuild_environment() causing trouble.
Here's some other places to hook in some print()s:

So, TL;DR, the information that I'd like you to share is:

Note that this discussion could go easier if you share the raw output I'm requesting, for a few reasons:

  1. If you accidentally miss something, two eyes are better than one
  2. We have a record trail
  3. Future eyes can also investigate if they're having a similar issue.
Flags: needinfo?(aryx.bugmail)

(In reply to Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout) from comment #10)
Comment 10 was incorrect grepping - the issue is always present in the shell started with start-shell.bat, even without running any code from mozilla-central.

$ python3
Python 3.9.10 (tags/v3.9.10:f2f3f53, Jan 17 2022, 15:14:21) [MSC v.1929 64 bit (
AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ.get("PATH")
'C:\\Program\nFiles\n(x86)\\Hilfen\\Gow\\bin\\;C:\\Mozilla\\Coding\\Buildtools\\
MozillaBuild\\bin;C:\\Mozilla\\Coding\\Buildtools\\MozillaBuild\\kdiff3;C:\\Mozi
lla\\Coding\\Buildtools\\MozillaBuild\\python3;C:\\Mozilla\\Coding\\Buildtools\\
MozillaBuild\\python3\\Scripts;C:\\Mozilla\\Coding\\Buildtools\\MozillaBuild\\ms
ys2\\usr\\local\\bin;C:\\Mozilla\\Coding\\Buildtools\\MozillaBuild\\msys2\\usr\\
bin;C:\\Mozilla\\Coding\\Buildtools\\MozillaBuild\\msys2\\usr\\bin;C:\\Mozilla\\
Coding\\Buildtools\\MozillaBuild\\msys2\\opt\\bin;C:\\Program Files\\Common File
s\\Oracle\\Java\\javapath;C:\\Program Files (x86)\\Common Files\\Oracle\\Java\\j
avapath;C:\\CustomSoftware\\Coding\\Python\\Python310\\Scripts;C:\\CustomSoftwar
e\\Coding\\Python\\Python310;C:\\Python310\\Scripts;C:\\Python310;C:\\CustomSoft
ware\\Coding\\Python\\Python39\\Scripts;C:\\CustomSoftware\\Coding\\Python\\Pyth
on39;C:\\CustomSoftware\\Coding\\Komodo Edit;C:\\CustomSoftware\\Coding\\Python\
\Python38\\Scripts;C:\\CustomSoftware\\Coding\\Python\\Python38;C:\\CustomSoftwa
re\\Coding\\Python\\Python37\\Scripts;C:\\CustomSoftware\\Coding\\Python\\Python
37;C:\\CustomSoftware\\Coding\\Python\\Python36\\Scripts;C:\\CustomSoftware\\Cod
ing\\Python\\Python36;C:\\Program Files (x86)\\Intel\\iCLS Client;C:\\Program Fi
les\\Intel\\iCLS Client;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\
\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0;C:\\Program Files\\Intel\\I
ntel(R) Management Engine Components\\DAL;C:\\Program Files\\Intel\\Intel(R) Man
agement Engine Components\\IPT;C:\\Program Files (x86)\\Intel\\Intel(R) Manageme
nt Engine Components\\DAL;C:\\Program Files (x86)\\Intel\\Intel(R) Management En
gine Components\\IPT;C:\\Program Files\\Microsoft SQL Server\\110\\Tools\\Binn;C
:\\Program Files\\Microsoft SQL Server\\120\\Tools\\Binn;C:\\CustomSoftware\\Med
ia Editing\\FFmpeg\\bin;C:\\CustomSoftware\\Helpers\\File Search\\Sift;C:\\Progr
am Files (x86)\\Hilfen\\Gow\\bin;C:\\Users\\Eichelhaeher\\.dnx\\bin;C:\\Program
Files\\Microsoft DNX\\Dnvm;C:\\Users\\Eichelhaeher\\.dnx\\bin;C:\\Program Files\
\Microsoft DNX\\Dnvm;C:\\Program Files\\Microsoft SQL Server\\130\\Tools\\Binn;C
:\\Program Files (x86)\\Windows Kits\\8.1\\Windows Performance Toolkit;C:\\Custo
mSoftware\\Coding\\Go\\bin;C:\\Program Files (x86)\\Windows Kits\\10\\Windows Pe
rformance Toolkit;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\
\Program Files (x86)\\WiFi\\bin;C:\\Program Files\\Common Files\\Intel\\Wireless
Common;C:\\ProgramData\\chocolatey\\bin;C:\\Program Files (x86)\\Sicherheit\\Ver
schluesselung\\GnuPG\\bin;C:\\CustomSoftware\\Coding\\JavaScript\\nodejs;C:\\Use
rs\\Eichelhaeher\\AppData\\Roaming\\ActiveState\\bin;C:\\CustomSoftware\\Coding\
\Komodo Edit;C:\\Program Files (x86)\\Common Files\\Oracle\\Java\\javapath;C:\\C
ustomSoftware\\Coding\\Python\\Python38\\Scripts;C:\\CustomSoftware\\Coding\\Pyt
hon\\Python38;C:\\CustomSoftware\\Coding\\Python\\Python37\\Scripts;C:\\CustomSo
ftware\\Coding\\Python\\Python37;C:\\CustomSoftware\\Coding\\Python\\Python36\\S
cripts;C:\\CustomSoftware\\Coding\\Python\\Python36;C:\\Program Files (x86)\\Int
el\\iCLS Client;C:\\Program Files\\Intel\\iCLS Client;C:\\Windows\\system32;C:\\
Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1
.0;C:\\Program Files\\Intel\\Intel(R) Management Engine Components\\DAL;C:\\Prog
ram Files\\Intel\\Intel(R) Management Engine Components\\IPT;C:\\Program Files (
x86)\\Intel\\Intel(R) Management Engine Components\\DAL;C:\\Program Files (x86)\
\Intel\\Intel(R) Management Engine Components\\IPT;C:\\Program Files\\Microsoft
SQL Server\\110\\Tools\\Binn;C:\\Program Files\\Microsoft SQL Server\\120\\Tools
\\Binn;C:\\CustomSoftware\\Media Editing\\FFmpeg\\bin;C:\\CustomSoftware\\Helper
s\\File Search\\Sift;C:\\Program Files (x86)\\Hilfen\\Gow\\bin;C:\\Users\\Eichel
haeher\\.dnx\\bin;C:\\Program Files\\Microsoft DNX\\Dnvm;C:\\Users\\Eichelhaeher
\\.dnx\\bin;C:\\Program Files\\Microsoft DNX\\Dnvm;C:\\Program Files\\Microsoft
SQL Server\\130\\Tools\\Binn;C:\\Program Files (x86)\\Windows Kits\\8.1\\Windows
 Performance Toolkit;C:\\Program Files\\nodejs;C:\\CustomSoftware\\Coding\\Go\\b
in;C:\\CustomSoftware\\Helpers\\Notebook Fan Control;C:\\Program Files (x86)\\In
crediBuild;C:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolki
t;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Program Files (
x86)\\Sicherheit\\Verschluesselung\\GnuPG\\bin;C:\\Program Files (x86)\\WiFi\\bi
n;C:\\Program Files\\Common Files\\Intel\\WirelessCommon;C:\\Users\\Eichelhaeher
\\AppData\\Roaming\\npm;C:\\Mozilla\\Coding\\Buildtools\\MozillaBuild\\msys2\\us
r\\bin\\site_perl;C:\\Mozilla\\Coding\\Buildtools\\MozillaBuild\\msys2\\usr\\bin
\\vendor_perl;C:\\Mozilla\\Coding\\Buildtools\\MozillaBuild\\msys2\\usr\\bin\\co
re_perl;C:\\Program Files\\nodejs\\;C:\\Users\\Eichelhaeher\\AppData\\Roaming\\n
pm\\;C:\\Users\\Eichelhaeher\\.cargo\\bin;C:\\CustomSoftware\\Coding\\Version Co
ntrol\\Git\\bin;C:\\CustomSoftware\\Coding\\Go\\bin;C:\\Users\\Eichelhaeher\\go\
\bin'

But if ran in a normal Windows console:

C:\Users\Eichelhaeher>c:\Mozilla\Coding\Buildtools\MozillaBuild\python3\python.e
xe
Python 3.9.10 (tags/v3.9.10:f2f3f53, Jan 17 2022, 15:14:21) [MSC v.1929 64 bit (
AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ.get("PATH")
'C:\\Program Files\\Common Files\\Oracle\\Java\\javapath;C:\\Program Files (x86)
\\Common Files\\Oracle\\Java\\javapath;C:\\CustomSoftware\\Coding\\Python\\Pytho
n310\\Scripts\\;C:\\CustomSoftware\\Coding\\Python\\Python310\\;C:\\Python310\\S
cripts\\;C:\\Python310\\;C:\\CustomSoftware\\Coding\\Python\\Python39\\Scripts\\
;C:\\CustomSoftware\\Coding\\Python\\Python39\\;C:\\CustomSoftware\\Coding\\Komo
do Edit\\;C:\\CustomSoftware\\Coding\\Python\\Python38\\Scripts\\;C:\\CustomSoft
ware\\Coding\\Python\\Python38\\;C:\\CustomSoftware\\Coding\\Python\\Python37\\S
cripts\\;C:\\CustomSoftware\\Coding\\Python\\Python37\\;C:\\CustomSoftware\\Codi
ng\\Python\\Python36\\Scripts\\;C:\\CustomSoftware\\Coding\\Python\\Python36\\;C
:\\Program Files (x86)\\Intel\\iCLS Client\\;C:\\Program Files\\Intel\\iCLS Clie
nt\\;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\
System32\\WindowsPowerShell\\v1.0\\;C:\\Program Files\\Intel\\Intel(R) Managemen
t Engine Components\\DAL;C:\\Program Files\\Intel\\Intel(R) Management Engine Co
mponents\\IPT;C:\\Program Files (x86)\\Intel\\Intel(R) Management Engine Compone
nts\\DAL;C:\\Program Files (x86)\\Intel\\Intel(R) Management Engine Components\\
IPT;C:\\Program Files\\Microsoft SQL Server\\110\\Tools\\Binn\\;C:\\Program File
s\\Microsoft SQL Server\\120\\Tools\\Binn\\;C:\\CustomSoftware\\Media Editing\\F
Fmpeg\\bin;C:\\CustomSoftware\\Helpers\\File Search\\Sift;C:\\Program Files (x86
)\\Hilfen\\Gow\\bin;C:\\Users\\Eichelhaeher\\.dnx\\bin;C:\\Program Files\\Micros
oft DNX\\Dnvm\\;C:\\Users\\Eichelhaeher\\.dnx\\bin;C:\\Program Files\\Microsoft
DNX\\Dnvm\\;C:\\Program Files\\Microsoft SQL Server\\130\\Tools\\Binn\\;C:\\Prog
ram Files (x86)\\Windows Kits\\8.1\\Windows Performance Toolkit\\;C:\\CustomSoft
ware\\Coding\\Go\\bin;C:\\Program Files (x86)\\Windows Kits\\10\\Windows Perform
ance Toolkit\\;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Pr
ogram Files (x86)\\WiFi\\bin\\;C:\\Program Files\\Common Files\\Intel\\WirelessC
ommon\\;C:\\ProgramData\\chocolatey\\bin;C:\\Program Files (x86)\\Sicherheit\\Ve
rschluesselung\\GPG4win\\..\\GnuPG\\bin;C:\\CustomSoftware\\Coding\\JavaScript\\
nodejs\\;C:\\Users\\Eichelhaeher\\AppData\\Roaming\\ActiveState\\bin;C:\\CustomS
oftware\\Coding\\Komodo Edit\\;C:\\Program Files (x86)\\Common Files\\Oracle\\Ja
va\\javapath;C:\\CustomSoftware\\Coding\\Python\\Python38\\Scripts\\;C:\\CustomS
oftware\\Coding\\Python\\Python38\\;C:\\CustomSoftware\\Coding\\Python\\Python37
\\Scripts\\;C:\\CustomSoftware\\Coding\\Python\\Python37\\;C:\\CustomSoftware\\C
oding\\Python\\Python36\\Scripts\\;C:\\CustomSoftware\\Coding\\Python\\Python36\
\;C:\\Program Files (x86)\\Intel\\iCLS Client\\;C:\\Program Files\\Intel\\iCLS C
lient\\;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Window
s\\System32\\WindowsPowerShell\\v1.0\\;C:\\Program Files\\Intel\\Intel(R) Manage
ment Engine Components\\DAL;C:\\Program Files\\Intel\\Intel(R) Management Engine
 Components\\IPT;C:\\Program Files (x86)\\Intel\\Intel(R) Management Engine Comp
onents\\DAL;C:\\Program Files (x86)\\Intel\\Intel(R) Management Engine Component
s\\IPT;C:\\Program Files\\Microsoft SQL Server\\110\\Tools\\Binn\\;C:\\Program F
iles\\Microsoft SQL Server\\120\\Tools\\Binn\\;C:\\CustomSoftware\\Media Editing
\\FFmpeg\\bin;C:\\CustomSoftware\\Helpers\\File Search\\Sift;C:\\Program Files (
x86)\\Hilfen\\Gow\\bin;C:\\Users\\Eichelhaeher\\.dnx\\bin;C:\\Program Files\\Mic
rosoft DNX\\Dnvm\\;C:\\Users\\Eichelhaeher\\.dnx\\bin;C:\\Program Files\\Microso
ft DNX\\Dnvm\\;C:\\Program Files\\Microsoft SQL Server\\130\\Tools\\Binn\\;C:\\P
rogram Files (x86)\\Windows Kits\\8.1\\Windows Performance Toolkit\\;C:\\Program
 Files\\nodejs\\;C:\\CustomSoftware\\Coding\\Go\\bin;C:\\CustomSoftware\\Helpers
\\Notebook Fan Control\\;C:\\Program Files (x86)\\IncrediBuild;C:\\Program Files
 (x86)\\Windows Kits\\10\\Windows Performance Toolkit\\;C:\\Program Files (x86)\
\NVIDIA Corporation\\PhysX\\Common;C:\\Program Files (x86)\\Sicherheit\\Verschlu
esselung\\GPG4win\\..\\GnuPG\\bin;C:\\Program Files (x86)\\WiFi\\bin\\;C:\\Progr
am Files\\Common Files\\Intel\\WirelessCommon\\;C:\\Users\\Eichelhaeher\\AppData
\\Roaming\\npm'
Flags: needinfo?(aryx.bugmail)

The Command Prompt/Powershell environments might be automatically replacing \n in the PATH, I'm not sure.
I'd recommend looking at the definition of C:\Program\nFiles\n(x86)\Hilfen\Gow\bin in your system properties - try replacing the whitespace with a hard space to see if that fixes it.

If not, some other debugging you can do is:

  • Try manually invoking MSYS2 with msys2_shell.cmd -no-start -defterm -full-path, see what path\to\python3 ... print(PATH) outputs
  • Remove files from MozillaBuild's msys2/etc/profile.d to see if they're affecting this (which I doubt)
Flags: needinfo?(aryx.bugmail)

Uninstalling gow fixed it - replacing whitespaces hadn't changed behavior (not sure if rebooting reset the whitespaces but a removal of gow from PATH + reboot saw it added back). Unknown why it became an issue with MozillaBuild upgrade.

Flags: needinfo?(aryx.bugmail)

MozillaBuild 3.4 and before didn't maintain the PATH (causing cargo and other tools to be unavailable).
Now that MozillaBuild 4.0+ doesn't purge the PATH, weirdness from your environment can creep in.

Status: NEW → RESOLVED
Type: defect → enhancement
Closed: 3 years ago
Component: MozillaBuild → Mach Core
Priority: -- → P5
Product: mozilla.org → Firefox Build System
Resolution: --- → FIXED
Summary: './mach build installers-de' fails with MozillaBuild 4.0: config/autoconf.mk:290: *** missing separator. Stop. → Validate that newlines aren't in PATH
Version: other → unspecified

Hmm, I initially marked as invalid, but then unselected that, yet Bugzilla didn't allow reverting the intention to close. Weird.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Has Regression Range: --- → yes
You need to log in before you can comment on or make changes to this bug.