1671438 - cbindgen not up-to-date after |mach bootstrap|

Status: RESOLVED WORKSFORME

(Firefox Build System :: Mach Core, defect, P3)

Description

[ISHIKAWA, Chiaki]

cbindgen not up-to-date after |mach bootstrap|

I got a strange outdated cbindgen version issue IMMEDIATELY after
|mach bootstrap|

Could be related to:
Bug 1498267 Opened 2 years ago Closed 2 years ago
cbindgen provided with |mach bootstrap| is too old

Bug 1494287 Opened 2 years ago Closed 2 years ago
mach bootstrap on macOS doesn't update cbindgen

---

What happened.

I ran
|mach bootstrap| after fixing hg credential.
( Bug 1670031 hgmo SSL certificate renewal work (Oct 2020) )

Then run |mach configure| and got outdated cbindgen !?

From the log of |mach configure|:

checking for cbindgen...
DEBUG: <truncated - see config.log for full output>
DEBUG: Executing: `/home/ishikawa/.mozbuild/cbindgen/cbindgen --version`
DEBUG: /home/ishikawa/.mozbuild/cbindgen/cbindgen has version 0.14.3
DEBUG: trying cbindgen: /home/ishikawa/.cargo/bin/cbindgen
DEBUG: Executing: `/home/ishikawa/.cargo/bin/cbindgen --version`
DEBUG: /home/ishikawa/.cargo/bin/cbindgen has version 0.14.3
DEBUG: trying cbindgen: /home/ishikawa/.cargo/bin/cbindgen
DEBUG: Executing: `/home/ishikawa/.cargo/bin/cbindgen --version`
DEBUG: /home/ishikawa/.cargo/bin/cbindgen has version 0.14.3
DEBUG: trying cbindgen: /home/ishikawa/.cargo/bin/cbindgen
DEBUG: Executing: `/home/ishikawa/.cargo/bin/cbindgen --version`
DEBUG: /home/ishikawa/.cargo/bin/cbindgen has version 0.14.3
DEBUG: trying cbindgen: /home/ishikawa/.cargo/bin/cbindgen
DEBUG: Executing: `/home/ishikawa/.cargo/bin/cbindgen --version`
DEBUG: /home/ishikawa/.cargo/bin/cbindgen has version 0.14.3
DEBUG: trying cbindgen: /home/ishikawa/.mozbuild/cbindgen/cbindgen
DEBUG: Executing: `/home/ishikawa/.mozbuild/cbindgen/cbindgen --version`
DEBUG: /home/ishikawa/.mozbuild/cbindgen/cbindgen has version 0.14.3
DEBUG: trying cbindgen: /home/ishikawa/.mozbuild/cbindgen/cbindgen
DEBUG: Executing: `/home/ishikawa/.mozbuild/cbindgen/cbindgen --version`
DEBUG: /home/ishikawa/.mozbuild/cbindgen/cbindgen has version 0.14.3
ERROR: cbindgen version 0.14.3 is too old. At least version 0.15.0 is required.

Please update using 'cargo install cbindgen --force' or running
'./mach bootstrap', after removing the existing executable located at
/home/ishikawa/.mozbuild/cbindgen/cbindgen.

*** Fix above errors and then restart with\
               "./mach build"
gmake: *** [client.mk:115: configure] Error 1

So I removed ~/.mozbuild/cbindgen/cbindgen and ran

cargo install cbindgen --force

Then |mach configure| ran to completion.

But something is fishy here.
So I investigated and checked the log from the initial |mach
bootstrap|, and found the cbindgen update.
But please note that it uses local cache(?!)

I re-ran |mach bootstrap| and now I see the downloading of update archive.

Output of egrep "(cbindgen|bootstrap)" /tmp/t.log (local log)
Previous |mach bootstrap|

++ /NREF-COMM-CENTRAL/mozilla/mach --log-no-times bootstrap
 0:04.54 Setting up artifact cbindgen.tar.xz
 0:04.54 Using artifact from local cache: /home/ishikawa/.mozbuild/toolchains/8fed8f17110695a3-cbindgen.tar.xz
 0:04.58 rm tree: /home/ishikawa/.mozbuild/cbindgen
 0:04.58 untarring "/home/ishikawa/.mozbuild/cbindgen.tar.xz"
 0:05.21 Purged artifact /home/ishikawa/.mozbuild/toolchains/d0d626601d459ac9-cbindgen.tar.xz

Latest |mach bootstrap|
Output of egrep "(cbindgen|bootstrap)" /tmp/t.log (local log)

++ /NREF-COMM-CENTRAL/mozilla/mach --log-no-times bootstrap
 0:07.70 Purged artifact /home/ishikawa/.mozbuild/toolchains/70af6eeaa9c16f33-cbindgen.tar.xz
 0:04.23 Setting up artifact cbindgen.tar.xz
 0:04.23 Downloading artifact to local cache: /home/ishikawa/.mozbuild/toolchains/4dc577c341b2214b-cbindgen.tar.xz
 0:04.40 rm tree: /home/ishikawa/.mozbuild/cbindgen
 0:04.40 untarring "/home/ishikawa/.mozbuild/cbindgen.tar.xz"
...

What gives?

It could be that there MAY BE the updating of certain files on the hg
server side going on at the moment, and the problem may have been
ephemeral.

But then the order of update is screwed up IMHO.:

The (to-be-downloaded) archives ought to be updated FIRST, and THEN
the version requirement declaration in configure ought to be UPDATED.

BTW, can people agree on exactly where the binaries required or
installed by the commands suggested by |mach| after a version mismatch
failure are stored?

cargo and |mach bootstrap| store them in different places. Ugh...

Eg. Where are cbindgen binaries?

|cargo|: ~/.cargo/bin
|make bootstrap| : ~/.mozbuild/cbindgen/

which cbindgen
/home/ishikawa/.cargo/bin/cbindgen <--- cargo stores it here 
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$ cbindgen --version
cbindgen 0.15.0
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$ ~/.mozbuild/cbindgen/cbindgen --version
cbindgen 0.15.0
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$ 

TIA

Comment 1

[ISHIKAWA, Chiaki]

Obviously |mach| failed to check the version of cbindgen under ~/.mozbuild/cbindgen/cbindgen against the one on the server and failed to download the newer version.

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Hey, thanks for the bug report here! There's definitely some shenanigans going on here 🤔

BTW, can people agree on exactly where the binaries required or
installed by the commands suggested by |mach| after a version mismatch
failure are stored?

bootstrap will always just put cbindgen at ~/.mozbuild/cbindgen/cbindgen, and will never put it in ~/.cargo.
However, when mach configure locates cbindgen, it is very flexible in the locations it checks (~/.mozbuild/cbindgen/, ~/.cargo/bin, and more). This explains why your manual removal of ~/.mozbuild/cbindgen/cbindgen and manual installation via cargo allowed mach configure to work.

It could be that there MAY BE the updating of certain files on the hg
server side going on at the moment, and the problem may have been
ephemeral.

I don't think that's the case. When you bootstrap, we choose which version of each artifact to install according to both this definition and the hashes of some related files you have locally. So, even if there's changes on the hg server-side, the version of cbindgen that is downloaded on your machine is decided by the files on your machine.

Between your two different mach bootstrap runs, did you checkout a different revision or edit toolchains.yml?

Comment 3

[ISHIKAWA, Chiaki]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #2)

Hey, thanks for the bug report here! There's definitely some shenanigans going on here 🤔

BTW, can people agree on exactly where the binaries required or
installed by the commands suggested by |mach| after a version mismatch
failure are stored?

bootstrap will always just put cbindgen at ~/.mozbuild/cbindgen/cbindgen, and will never put it in ~/.cargo.
However, when mach configure locates cbindgen, it is very flexible in the locations it checks (~/.mozbuild/cbindgen/, ~/.cargo/bin, and more). This explains why your manual removal of ~/.mozbuild/cbindgen/cbindgen and manual installation via cargo allowed mach configure to work.

I see. Funny, though, at one point in the past, I had to put ~/.mozbuild/cbindgen into my PATH (under linux) so that |mach configure| would pickup cbindgen there for execution. I don't know if the addition of ~/.mozbuild/cbindgen to PATH is done internally by |mach| now.
My script to invoke |mach| is full of such addition of ~/.mozbuild subdirectories to PATH...

It could be that there MAY BE the updating of certain files on the hg
server side going on at the moment, and the problem may have been
ephemeral.

I don't think that's the case. When you bootstrap, we choose which version of each artifact to install according to both this definition and the hashes of some related files you have locally. So, even if there's changes on the hg server-side, the version of cbindgen that is downloaded on your machine is decided by the files on your machine.

Between your two different mach bootstrap runs, did you checkout a different revision or edit toolchains.yml?

No I did not.
I am a bit hazy now, but I update the M-C and C-C tree locally first.
Then since I wanted to push jobs eventually to try-comm-central, I wanted to update the hg credential.
So I ran |make vcs-setup|.
But the initial |mach vcs-setup| failed due to missing correct credential despite what I read in Bug 1670031.
(|mach| aborted during its operation. I had to manually insert the correct credential into ~/.hgrc and retried |mach vcs-setup| just in case something was left in an intermediate state due to abort.)

Then what happened was then I ran |mach configure| for build when it detected outdated
cbindgen.
So I ran |make bootstrap| but this did NOT update the binaries, at least cbindgen, as it should have.
So the next |make configure| still failed.
That is when I got confused and suspicious and proceeded carefully and later I filed this bugzilla.

Strange, isn't it?

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

My script to invoke |mach| is full of such addition of ~/.mozbuild subdirectories to PATH...

I think that you should be able to clear out those ~/.mozbuild PATH additions now :)
If that causes problems, then it could be worth reporting here so we can do some digging.

But the initial |mach vcs-setup| failed due to missing correct credential despite what I read in Bug 1670031.

That's strange, I wonder if you were using the new fingerprint before the change went live? Unsure about this one, but if it sounds like you were able to resolve the problem locally, which is good 👍

So I ran |make bootstrap| but this did NOT update the binaries, at least cbindgen, as it should have.

That is bizarre, and a bug, I agree.
Unfortunately, this is tough to diagnose, now that it's working for you again.
Next time this happens, can you back up your .cargo and .mozbuild directories before manually fixing the issue? I'd like to be able to dig into how/why mach bootstrap wasn't able to update the cbindgen binaries 🤔

I'm going to close this ticket for the moment, but if/when you run into this problem again, please reopen it :)