1744910 - mach bootstrap bustage on Fedora/CentOS/RockyLinux: Permission denied: '/var/cache/dnf/appstream-62ae9a0bbea44fbe/repodata/gen'

Status: VERIFIED FIXED

(Firefox Build System :: Bootstrap Configuration, defect)

Description

[Alex Vincent [:WeirdAl]]

Attachment: Dockerfile

I'm providing a Dockerfile for podman use so developers can reproduce this in RockyLinux 8.5, without contaminating their own systems.

Steps to reproduce:

1. Make sure you have an up-to-date mozilla-central clone.
2. Download the Dockerfile I'm attaching as a sibling to mozilla-central, and cd to that parent directory
3. In the directory above mozilla-central: podman build -t test:firefox -f Dockerfile . This will copy mozilla-central into the RockyLinux 8.5 derivative image, which will take several minutes. (The alternative is pulling the repository, which is probably even more expensive on bandwidth alone.)
4. podman run --rm -t -i test:firefox.
5. In the container, echo "nonroot" | sudo -S tail /dev/null && ./mach --no-interactive bootstrap --application-choice browser

Expected results: mach bootstrap runs correctly.
Actual results:

Error running mach:

    ['--no-interactive', 'bootstrap', '--application-choice', 'browser']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Failed to determine currently-installed package groups: "Last metadata expiration check: 0:16:21 ago on Wed Dec  8 06:39:04 2021.
[Errno 13] Permission denied: '/var/cache/dnf/appstream-62ae9a0bbea44fbe/repodata/gen'
"

  File "/home/nonroot/firefox-build/mozilla-central/python/mozboot/mozboot/mach_commands.py", line 42, in bootstrap
    bootstrapper.bootstrap(command_context.settings)
  File "/home/nonroot/firefox-build/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 330, in bootstrap
    self.instance.install_system_packages()
  File "/home/nonroot/firefox-build/mozilla-central/python/mozboot/mozboot/centosfedora.py", line 73, in install_system_packages
    self.dnf_groupinstall(*self.group_packages)
  File "/home/nonroot/firefox-build/mozilla-central/python/mozboot/mozboot/base.py", line 480, in dnf_groupinstall
    f'Failed to determine currently-installed package groups: "{installed.stdout}"'

hg bisect running commands like this (with the revisions appropriately adjusted on each pass) narrow it down to bug 1744277:

hg bisect -b a4e59edd7e36
hg bisect -g 7ae01be458cc
./mach --no-interactive bootstrap --application-choice browser
exit

On the one hand, RockyLinux is clearly a tier 2 platform (one I added support for in bug 1717650). On the other hand, it's supposed to be equivalent to CentOS 8 (which is about to go EOL).

Comment 1

[Jonathan Watt [:jwatt]]

Do you get the same error if you run dnf --cacheonly group list --installed manually? If so, what about if you drop the --cacheonly flag?

Comment 2

[Jonathan Watt [:jwatt]]

I just had this happen to me too. After investigating it seems my suspicion about the above 'Permission denied' error looking like an issue with dnf's --cacheonly flag is correct. Remove it, and the command succeeds. I've filed an upstream bug against dnf.

I added the --cacheonly flag in the code I landed to avoid unnecessarily spending time waiting for the online checks that dnf sometimes does (when the cache is stale?). If --cacheonly can cause intermittent failures though, I guess we remove it for now.

Comment 3

[Jonathan Watt [:jwatt]]

Attachment: Bug 1744910 - For now, don't use --cacheonly with dnf. r=mhentges

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1744277

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c06799010c25
For now, don't use --cacheonly with dnf. r=mhentges

Comment 6

[Jonathan Watt [:jwatt]]

Sorry for the disruption, Alex.

Comment 7

[Alex Vincent [:WeirdAl]]

It happens. I only noticed it because I was trying to update after the FF91.4.0 release. (I'm trying to get our infrastructure improvements too.)

Comment 8

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/c06799010c25

Comment 9

[Alex Vincent [:WeirdAl]]

Verified using the steps I provided above.