1722098 - fzf isn't automatically installed with MozillaBuild

Status: RESOLVED FIXED

(Developer Infrastructure :: Try, defect, P3)

Description

[Robin Steuber (they/them) [:bytesized]]

STR:

1. Download the fzf binary and copy it to a location within the $PATH.
2. Just to be thorough, add the directory to the Windows Path Environment Variable
3. Run ./mach try fuzzy

Expect Results
It should not complain that fzf is not available.

Actual Results
It complains that fzf is not available.

> ./mach try fuzzy
Already up to date.
No prebuilt binary for MINGW32_NT-6.2 i686 ...
go executable not found. Installation failed.
Failed to install fzf.

Please install fzf manually following the appropriate instructions for your
platform:

    https://github.com/junegunn/fzf#installation

Only the binary is required, if you do not wish to install the shell and
editor integrations, download the appropriate binary and put it on your $PATH:

    https://github.com/junegunn/fzf/releases


> which fzf
/c/Users/bytesized/bin/fzf.exe

> fzf --version
0.27.2 (e086f0b)

I am running Windows 10.

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Hmm, I believe that we locate fzf by looking in ~/.mozbuild/fzf. Perhaps we should overriding that behaviour and asking to search the path, but that is probably a separate discussion (compatibility with different versions, etc).

Have you run ./mach bootstrap recently? What's in your ~/.mozbuild/fzf? What about ~/.mozbuild/fzf/bin/ (for me, just tmux-fzf)?

Comment 2

[Robin Steuber (they/them) [:bytesized]]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #1)

Hmm, I believe that we locate fzf by looking in ~/.mozbuild/fzf. Perhaps we should overriding that behaviour and asking to search the path, but that is probably a separate discussion (compatibility with different versions, etc).

Then you might want to consider changing this message:

Only the binary is required, if you do not wish to install the shell and
editor integrations, download the appropriate binary and put it on your $PATH:

Have you run ./mach bootstrap recently?

It's been a while. I don't remember precisely when the last time was. I just ran it again though, and it does not seem to have changed anything. ./mach try fuzzy still fails the same way.

What's in your ~/.mozbuild/fzf?

> ls -l ~/.mozbuild/fzf
total 142K
-rw-r--r-- 1 bytesized Administrators  22K Jul 23 14:39 ADVANCED.md
-rw-r--r-- 1 bytesized Administrators 1.3K Jul 23 14:39 BUILD.md
-rw-r--r-- 1 bytesized Administrators  38K Jul 23 14:39 CHANGELOG.md
-rw-r--r-- 1 bytesized Administrators  489 Jul 23 14:39 Dockerfile
-rw-r--r-- 1 bytesized Administrators 1.1K Jul 23 14:39 LICENSE
-rw-r--r-- 1 bytesized Administrators 4.3K Jul 23 14:39 Makefile
-rw-r--r-- 1 bytesized Administrators  18K Jul 23 14:39 README-VIM.md
-rw-r--r-- 1 bytesized Administrators  22K Jul 23 14:39 README.md
drwxr-xr-x 2 bytesized Administrators    0 Jul 23 14:39 bin
drwxr-xr-x 2 bytesized Administrators    0 Jul 23 14:39 doc
-rw-r--r-- 1 bytesized Administrators  532 Jul 23 14:39 go.mod
-rw-r--r-- 1 bytesized Administrators 3.0K Jul 23 14:39 go.sum
-rwxr-xr-x 1 bytesized Administrators 9.8K Jul 23 14:39 install
-rw-r--r-- 1 bytesized Administrators 1.9K Jul 23 14:39 install.ps1
-rw-r--r-- 1 bytesized Administrators  242 Jul 23 14:39 main.go
drwxr-xr-x 3 bytesized Administrators    0 Jul 23 14:39 man
drwxr-xr-x 2 bytesized Administrators    0 Jul 23 14:39 plugin
drwxr-xr-x 2 bytesized Administrators 4.0K Jul 23 14:39 shell
drwxr-xr-x 6 bytesized Administrators 8.0K Jul 23 14:39 src
drwxr-xr-x 2 bytesized Administrators    0 Jul 23 14:39 test
-rwxr-xr-x 1 bytesized Administrators 2.5K Jul 23 14:39 uninstall

What about ~/.mozbuild/fzf/bin/ (for me, just tmux-fzf)?

> ls -l ~/.mozbuild/fzf/bin
total 7.0K
-rwxr-xr-x 1 bytesized Administrators 6.3K Jul 23 14:39 fzf-tmux

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

FWIW, I've fired up my Windows machine here and have a couple findings:

• Normally, fzf is found at C:\mozilla-build\bin\fzf.exe. Do you have one there? Note that you should be on MozillaBuild 3.3, and it could be worth upgrading if you're not there.
• Keep in mind that your MozillaBuild $PATH may be different from your Windows %PATH%, though based on your which fzf printing a MozillaBuild-y path, it looks like you're already keeping that in mind 👍
  • Side-note: my assumption that the $PATH wasn't respected was false: if MozillaBuild can which fzf, then Mach should be able to, too. The messaging is correct. My mistake earlier about the messaging :)
• I've got to run here, but if you want to dig in a little bit, then see what's going on here - is distutils.spawn.find_executable() failing to find fzf, or is something else behaving incorrectly?

I'll be away until Thursday next week, but hopefully this is work-aroundable by at least setting up a MozillaBuild with an fzf.exe binary.

Comment 4

[Robin Steuber (they/them) [:bytesized]]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #3)

FWIW, I've fired up my Windows machine here and have a couple findings:

• Normally, fzf is found at C:\mozilla-build\bin\fzf.exe. Do you have one there? Note that you should be on MozillaBuild 3.3, and it could be worth upgrading if you're not there.

That file did not exist for me. I updated MozillaBuild to the newest version. That file still does not exist.

• I've got to run here, but if you want to dig in a little bit, then see what's going on here - is distutils.spawn.find_executable() failing to find fzf, or is something else behaving incorrectly?

It appears to me that distutils.spawn.find_executable() is failing to find the executable in my path.

> python3 -c "import distutils.spawn; print(distutils.spawn.find_executable('fzf'))"
None

I'm not familiar with that function, so I couldn't say why. I've had problems before with ~ expansion, but it shouldn't be that this time because I replaced them all with $HOME when I had that problem. I can see in my $PATH that $HOME was expanded properly. So AFAICT my $PATH shouldn't be causing any issues.

I'll be away until Thursday next week, but hopefully this is work-aroundable by at least setting up a MozillaBuild with an fzf.exe binary.

No hurry. I usually use ./mach try chooser, but I always end up running way more tests than what I need that way. It appears that ./mach try fuzzy can take a path, so I was hoping to try out only running the tests from the directories that are relevant to my work. But I can continue using ./mach try chooser in the meantime.

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Oh, my mistake, MozillaBuild doesn't ship with fzf.
I can reproduce this after installing a fresh MozillaBuild.

This fails because:

• We use the fzf install script
• The architecture of MozillaBuild machines is grasped with uname -sm: MINGW32_NT-6.2 i686 (since MozillaBuild is 32bit).
• fzf only has 64bit releases, so our 32bit MozillaBuild doesn't match any options
• We fail to install fzf, so we dump some messages and stop.

I am not sure how this used to work in the past, and I'm looking forward to seeing how it worked on my other machine (I'll be able to investigate that this weekend). I don't recall ever installing fzf manually.

---

Regarding the $PATH issue, would you mind digging in a little bit with pdb? My best guess right now is that the $PATH is being modified, maybe.
Anyways, if you wouldn't mind trying the following:

1. which fzf
2. echo $PATH
3. ./mach --debug-command try fuzzy
4. b tools\tryselect\selectors\fuzzy.py:230 (note: I'm on revision bdf05015fbff, which may affect the breakpoint location)
5. c
6. s
7. Now, play around with find_executable(). Debugger commands can be found here. The minimum information that may be helpful here is:
   • Is path sourced from os.environ, or os.confstr("CS_PATH")?
   • What is paths set to?
   • Why isn't fzf found? :)

---

TL;DR: there's two issues:

• Auto-installation of fzf is failing, and I'll look into this more next week.
• distutils.spawn.find_executable() can't find fzf on your path, and some local debugging would go a long way in finding out why.

Comment 6

[Robin Steuber (they/them) [:bytesized]]

Gah! The problem was the exact problem that I was saying it wasn't in Comment 4. It appears that I missed one of the ~ to $HOME replacements when I fixed that, and that happened to be the exact location where I put fzf.

Once I fixed that, ./mach try fuzzy started working. It does appear that distutils.spawn.find_executable() does not do ~ expansion.

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Nice, good eye! Glad it's working for you now :)
I'll tweak this ticket to be about the automatic installation of MozillaBuild now.
Thanks Kirk!

Comment 9

[Andrew Halberstadt [:ahal]]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #5)

I am not sure how this used to work in the past, and I'm looking forward to seeing how it worked on my other machine (I'll be able to investigate that this weekend). I don't recall ever installing fzf manually.

This used to work because fzf used to provide 32 bit binaries.

Comment 10

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

I hit this for Updatebot, - specifically the i686 issue Mitchell described above. It happened in 0.23.0: "Built with Go 1.15.2: We no longer provide 32-bit binaries"

Running install.ps1 avoids fzf complaining about the architecture, it just downloads the 64-bit binary. However, when I do that I did hit an issue upstream, which I created a PR to fix. If that gets accepted or some other work-around exists, we should be able to invoke install.ps1 instead of install and resolve this.

Comment 11

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

I'm going to resolve this by (a) pinning a specific version of fzf and (b) hardcoding a download link rather than using their script. Will work on it in February.

Comment 12

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

Attachment: Bug 1722098: Fix the fzf version and download it directly r?ahal

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/140a45a4d8dd
Fix the fzf version and download it directly r=ahal

Comment 15

[Cristina Cozmuta (:CrissCozmuta)]

https://hg.mozilla.org/mozilla-central/rev/140a45a4d8dd

Comment 16

[Julian Descottes [:jdescottes]]

After this bug landed I can no longer push to try. I got a prompt to update fzf and now have the following error:

Error running mach:

    ['try', '--preset', 'devtools']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file try| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

OSError: [Errno 8] Exec format error: '/Users/jdescottes/.mozbuild/fzf/fzf'

  File "/Users/jdescottes/Development/hg/fx-team-artifact/tools/tryselect/mach_commands.py", line 236, in try_default
    return command_context._mach_context.commands.dispatch(
  File "/Users/jdescottes/Development/hg/fx-team-artifact/python/mach/mach/registrar.py", line 183, in dispatch
    return self._run_command_handler(handler, context, **kwargs)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/python/mach/mach/registrar.py", line 116, in _run_command_handler
    result = fn(instance, **kwargs)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/tools/tryselect/mach_commands.py", line 343, in try_fuzzy
    return run(command_context, **kwargs)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/tools/tryselect/mach_commands.py", line 201, in run
    return mod.run(**kwargs)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/tools/tryselect/selectors/fuzzy.py", line 451, in run
    fzf = fzf_bootstrap(update)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/tools/tryselect/selectors/fuzzy.py", line 361, in fzf_bootstrap
    if fzf_bin and should_force_fzf_update(fzf_bin):  # Case (1)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/tools/tryselect/selectors/fuzzy.py", line 333, in should_force_fzf_update
    fzf_version = get_fzf_version(fzf_bin)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/tools/tryselect/selectors/fuzzy.py", line 321, in get_fzf_version
    fzf_version = subprocess.check_output(cmd)
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/jdescottes/Development/hg/fx-team-artifact/third_party/python/sentry_sdk/sentry_sdk/integrations/stdlib.py", line 190, in sentry_patched_popen_init
    rv = old_popen_init(self, *a, **kw)  # type: ignore
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)

Any idea?

Comment 17

[Julian Descottes [:jdescottes]]

This seems to break for others in my team, also on macos

Comment 18

[Nicolas Chevobbe [:nchevobbe]]

I can't get ./mach try fuzzy to work anymore:

./mach try fuzzy
Artifact builds enabled, pass --no-artifact to disable
Error running mach:

    ['try', 'fuzzy']

[…]

The details of the failure are as follows:

OSError: [Errno 8] Exec format error: '/Users/nchevobbe/.mozbuild/fzf/fzf'

  File "/Users/nchevobbe/Projects/mozilla-central/tools/tryselect/mach_commands.py", line 343, in try_fuzzy
    return run(command_context, **kwargs)
  File "/Users/nchevobbe/Projects/mozilla-central/tools/tryselect/mach_commands.py", line 201, in run
    return mod.run(**kwargs)
  File "/Users/nchevobbe/Projects/mozilla-central/tools/tryselect/selectors/fuzzy.py", line 451, in run
    fzf = fzf_bootstrap(update)
  File "/Users/nchevobbe/Projects/mozilla-central/tools/tryselect/selectors/fuzzy.py", line 361, in fzf_bootstrap
    if fzf_bin and should_force_fzf_update(fzf_bin):  # Case (1)
  File "/Users/nchevobbe/Projects/mozilla-central/tools/tryselect/selectors/fuzzy.py", line 333, in should_force_fzf_update
    fzf_version = get_fzf_version(fzf_bin)
  File "/Users/nchevobbe/Projects/mozilla-central/tools/tryselect/selectors/fuzzy.py", line 321, in get_fzf_version
    fzf_version = subprocess.check_output(cmd)
  File "/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/nchevobbe/Projects/mozilla-central/third_party/python/sentry_sdk/sentry_sdk/integrations/stdlib.py", line 190, in sentry_patched_popen_init
    rv = old_popen_init(self, *a, **kw)  # type: ignore
  File "/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)

Sentry event ID: 1a621796e237432d939b8970aa2eee58

Comment 19

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

If you're on mac, you got the linux binaries. I made this mistake because the darwin binaries aren't included in the checksum file (I have no idea why.)

You can unbbreak yourself by replacing ~/.mozbuild/fzf/fzf with the fzf from https://github.com/junegunn/fzf/releases/download/0.29.0/fzf-0.29.0-darwin_amd64.zip or this one if you're on an ARM M1.

It is confusing to me why people got a prompt though. You should have only gotten a prompt if we couldn't find fzf or fzf was not just outdated, but majorly outdated. The second shouldn't happen (it's super old and you couldn't use fuzzy prior if you were that outdated.) The former could happen if you were using fzf from a system dir instead of ~/.mozbuild/fzf/ but the detection logic for that didn't change so I don't know why it wouldn't find your old fzf you were using.... (Unless you passed --update which would do the update with no prompt.)

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks for the hints but nevertheless I cannot get it working when trying to push a commit to try on the esr91 branch. I always get the failure that it's not a git repository. What can I do to get this working? Or does that require some kind of backport? I'm on MacOS Monterey (aarch64).

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As I was informed by Julian I have to completely remove the ~/.mozbuild/fzf folder before trying to push to try on esr91. Only that way mach try fuzzy will automatically clone the fzf repository which is still needed on that branch.

Comment 22

[Julian Descottes [:jdescottes]]

Some background on the issues on macos.

Before Bug 1722098 landed, .mozbuild/fzf contained a git repository, and the binary was built in .mozbuild/fzf/bin. After Bug 1722098, we only expect the binary, and we expect it directly under .mozbuild/fzf.

When you try to use fuzzy for the first time after moving to a changeset which has the changes from this bug (and from Bug 1756813), it will try to look for the binary in .mozbuild/fzf, it won't find it (since it's most likely under /bin) and will prompt you to download a new fzf. If you accept, .mozbuild/fzf will now only contain a fzf binary.

And now if you go back to an older revision (old mozilla central, or beta, esr etc...) and try to use fuzzy, it will fail because it can't find the binary under .mozbuild/fzf/bin. It will fail with:

fatal: not a git repository (or any of the parent directories): .git
Update fzf failed.

My suggestion to workaround this was to fully delete the fzf folder in such a case so that the local git repo would be recreated. But it's annoying because as you jump around between old changesets and new changesets you have to do this repeatedly.

A better solution seems to be to make a copy (or alias?) of the binary .mozbuild/fzf/fzf in .mozbuild/fzf/bin/fzf. This way it should work with all versions.