1245969 - Automated testing for `mach bootstrap`

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect)

Description

[Gregory Szorc [:gps]]

We currently don't have automated test coverage that `mach bootstrap` works. This bug is about creating that automation.

Comment 1

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=78cf9b4874fb

Comment 2

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=913555a0f3e0

Comment 3

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7f017ed84688

Comment 4

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8eb928aa1475

Comment 5

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=89d6e5f04c15

Comment 6

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2a30171bf3ff

Comment 7

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4382062e87ea

Comment 8

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=18e473915496

Comment 9

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d0b08702a03d

Comment 10

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=60bd86485928

Comment 11

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=99933750115e

Comment 12

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=38c1a7fd7494

Comment 13

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cdbc2e48209c

Comment 14

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3f128456aad7

Comment 15

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e2268568765b

Comment 16

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f6b30b956194

Comment 17

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=fe38ca3fe109

Comment 18

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ce8ac426fb1f

Comment 19

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=dd1f37c715c0

Comment 20

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d45fbaf3f71b

Comment 21

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6f18422dd777

Comment 22

[Gregory Szorc [:gps]]

Attachment: Bug 1245969 - TaskCluster tasks for performing system bootstrap;

(THIS PATCH IS FEEDBACK QUALITY AND ISN'T READY FOR FULL REVIEW)

We've had `mach bootstrap` and the one-line wget + `python bootstrap`
bootstrap tool for years. Unfortunately, it's never been formally
tested. TaskCluster enables us to finally do that.

This commit introduces support for testing the system bootstrap tool
on various Linux distros.

The bootstrap tasks are implemented as generic non-build/test tasks.
They are part of the "bootstrap" tag, so you can include "-j bootstrap"
in your try syntax to schedule them.

TODO

* Arch does not work (interactive prompt is hanging execution)
* Gentoo does not work
* Figure out how to schedule every ~24 hours
* Support Fennec bootstrap
* Tidy up TreeHerder names/groups?
* Work with sheriffs on TreeHerder visibility policy

Review commit: https://reviewboard.mozilla.org/r/35345/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/35345/

Comment 23

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8720527 [details]
Bug 1245969 - TaskCluster tasks for performing system bootstrap;

https://reviewboard.mozilla.org/r/35345/#review32141

It would be better to define docker images for each of these tasks, with the system setup complete, and then just run the VCS clone (please use tc-vcs to take advantage of caching) and `mach configure` in the task itself.  In terms of in-tree complexity, it's about the same, but has the advantage of not pulling down loads of software for every run of the task.  Plus it nicely separates system configuration from the task itself.

Note, too, that TaskCluster jobs run as root (there's a bug to make that configurable..), so you'll probably want to create a user in your docker image and then switch to that user before cloning and running mach.

You can schedule these on a periodic basis with the taskcluster hooks service.  We're also looking at something like .cron.yml in the root of the gecko tree to allow more ad-hoc scheduling.

Comment 24

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6f6c39f698d1

Comment 25

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8cdd933ed222

Comment 26

[Gregory Szorc [:gps]]

Attachment: Bug 1245969 - Use working yasm rpm URLs

The previously listed server wasn't working :(

Review commit: https://reviewboard.mozilla.org/r/94158/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/94158/

Comment 27

[Gregory Szorc [:gps]]

Attachment: Bug 1245969 - Support fetching bootstrap files from an arbitrary revision

This is needed so we can test the bootstrapper on Try.

Review commit: https://reviewboard.mozilla.org/r/94160/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/94160/

Comment 28

[Gregory Szorc [:gps]]

Comment on attachment 8720527 [details]
Bug 1245969 - TaskCluster tasks for performing system bootstrap;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/35345/diff/1-2/

Comment 29

[Ralph Giles (:rillian)]

Comment on attachment 8720527 [details]
Bug 1245969 - TaskCluster tasks for performing system bootstrap;

https://reviewboard.mozilla.org/r/35345/#review94352

::: taskcluster/ci/bootstrap/kind.yml:74
(Diff revision 2)
> +            docker-image: "debian:jessie"
> +        run:
> +            system-setup: "apt-get update && apt-get -y install python2.7 wget"
> +            python: python2.7
> +
> +    bootstrap/fedora22:

Fedora 22 entered end-of-life in July. I'd suggest Feodra 23 and 24. https://lists.fedoraproject.org/archives/list/[e-mail]/thread/GCMM2KQQRY5PKFY42S3K772BDRENXZ7U/

Fedora 25 comes out next week, which will obsolete 23 as well, but there's no tag for it on dockerhub yet.

::: taskcluster/ci/bootstrap/kind.yml:110
(Diff revision 2)
> +                && echo ">=x11-libs/cairo-1.14.2 X" >> /etc/portage/package.use/firefox
> +                && emerge -UD world
> +            python: python2.7
> +
> +    bootstrap/ubuntu1404:
> +        description: "Ubuntu 16.04 Bootstrap Support"

Target should be `ubuntu1604` to match the image?

::: taskcluster/taskgraph/transforms/job/bootstrap.py:87
(Diff revision 2)
> +
> +    script_args = [
> +        run['system-setup'].strip(),
> +        'wget %s' % bsurl,
> +        bs,
> +        # Many distros run an ancient Mercurial that doesn't do cloen bundles.

s/cloen/clone/

Comment 30

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8720527 [details]
Bug 1245969 - TaskCluster tasks for performing system bootstrap;

https://reviewboard.mozilla.org/r/35345/#review94684

With this in place, can we get rid of the "artifact build" builds?  I think those are doing a similar thing, right (verifying that the pieces necessary for an artifact build are in place)?

I like this!

::: taskcluster/ci/bootstrap/kind.yml:24
(Diff revision 2)
> +    run:
> +        using: bootstrap

Minor: I kind of prefer to have this co-located with the other contents of the `run` key. Otherwise the reader is left without much context for the other `run` keys.

::: taskcluster/taskgraph/transforms/job/bootstrap.py:39
(Diff revision 2)
> [e-mail]
> +def clone_applications(config, jobs):

Transform should be in `taskcluster/taskgraph/transforms/bootstrap.py`.

Comment 31

[Gregory Szorc [:gps]]

dustin: it's been a year since we visited this problem. How would you recommend we solve it today?

Recap: We don't want these tasks to run on every push. But because results are definitely not deterministic, we want them to run periodically. I was thinking we could use SCHEDULES to ensure we run them when python/bootstrap changes. And maybe supplement that with a cron scheduling? I just don't know the state of periodic scheduling on mozilla-central these days. I think it was in flux last I talked to bstack?

Comment 32

[Dustin J. Mitchell [:dustin] (he/him)]

Periodic scheduling is pretty solid -- it's used for nightlies and a number of other things.  So yeah, I think that approach makes a lot of sense.

Comment 33

[Gregory Szorc [:gps]]

bstack: the desire for having CI coverage of `mach bootstrap` has come up again recently. These tasks are special in that they ideally start from a vanilla-as-possible OS install. I think it would be ideal if we could define them as Linux tasks that ran in a VM that was imaged from a fresh distro install. Do generic-worker or taskcluster-worker yet support running Linux tasks as root in a VM - not Docker - environment? And do you think we should attempt to shoehorn these tasks into docker-worker or wait for something better?

Comment 34

[Brian Stack [:bstack]]

Afaik, taskcluster-worker has experimental qemu support but that's as far as we are w.r.t. using a vm instead of docker. :jonasfj, what is the status of qemu support?

Could another possibility to have a 99% fresh ami but with generic worker and livelog binaries dropped in? Maybe even scriptworker would work in that case. I'm guessing that if docker is weird enough compared to a normal distribution install then maybe the ami are too?

I think I'm missing up above where the desire that these run in a VM rather than docker comes from? Are the docker images just plain weird compared to the actual disributions?

Comment 35

[Gregory Szorc [:gps]]

Distro Docker images are stripped down installs (they are optimized for size). And we occasionally run into issues where certain packages don't install correctly inside Docker or that Docker does weird things to the environment. I'd feel better if we were able to tell a task to execute in a VM "image" (which could be versioned in source control), as I think it would be more representative of end-user systems.

Now, how we'd go about producing and consuming said image, I'm not sure. But if we could tell a task to start with a VM or filesystem snapshot and execute a process therein in a "real" OS environment (not Docker), then we'd be in a better position.

Comment 36

[Phabricator Automation]

Attachment: Bug 1245969 - TaskCluster tasks for `mach bootstrap`; r?glandium

We've had `mach bootstrap` and the one-line wget + `python bootstrap`
bootstrap tool for years. Unfortunately, it's never been formally
tested. TaskCluster enables us to finally do that.

This commit introduces support for testing the system bootstrap tool
on various Linux distros.

The bootstrap tasks are implemented as their own kind because they
are somewhat special.

Each OS is defined once along with its base Docker images, a command
to pre-bootstrap the Docker environment (typically to install Python
and wget), and the name of the Python binary to use.

A transform "splits" each job into 4 separate jobs, each one testing a
different bootstrap application: firefox and fennec with normal and
artifact variations.

The emitted task is a giant pile of shell commands that essentially:

1) Perform (hopefully minimal) pre-bootstrap setup
2) Download bootstrap.py from the active revision
3) Invoke the bootstrapper
4) Clone the repo
5) Run `mach build`

Several tasks are broken:

* Arch is completely broken due to attempting to run a command as root
* CentOS7 is completely broken attempting a group install of a missing
  group.
* Debian, Ubuntu, and Fedora aren't installing Clang (likely due to the
  bootstrapper not installing it in non-interactive mode).
* Probably a handful of other failures.

The tasks are all tier 2. And they aren't enabled by default or picked
up by any dependencies. The only way to trigger them is via
`mach try fuzzy --full`.

Considering I've been sitting on this patch for ~2 years and that others
have wanted to test bootstrap in CI, I think getting a somewhat working
version of the CI tasks landed is a good first step and unblocks a lot
of incremental improvements to testing bootstrap. While there are
many shortcomings of this code, I think we should land now and iterate
later.

Comment 37

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months, so the assignee is being reset.

Comment 39

[Mike Hommey [:glandium]]

Attachment: Bug 1245969 - Add tasks that perform a bootstrap on various bare Linux environments.

Comment 40

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/03596bbe6b15
Add tasks that perform a bootstrap on various bare Linux environments. r=taskgraph-reviewers,firefox-build-system-reviewers,jlorenzo,ahochheiden

Comment 41

[Iulian Moraru]

Backed out for causing bootstrap bustages.

Push with failures

Failure log

Backout link

[taskcluster 2022-11-25 02:47:59.507Z] Task ID: Wpd92JnGSse7_JlWFfNREg
[taskcluster 2022-11-25 02:47:59.507Z] Worker ID: 8584270821800645696
[taskcluster 2022-11-25 02:47:59.507Z] Worker Group: us-central1
[taskcluster 2022-11-25 02:47:59.507Z] Worker Node Type: projects/970387039909/machineTypes/n2-custom-16-73728
[taskcluster 2022-11-25 02:47:59.507Z] Worker Pool: gecko-3/b-linux-gcp
[taskcluster 2022-11-25 02:47:59.507Z] Worker Version: 38.0.4
[taskcluster 2022-11-25 02:47:59.507Z] Public IP: 34.121.40.147
[taskcluster 2022-11-25 02:47:59.507Z] Hostname: gecko-3-b-linux-gcp-lpesyk6ztaq-q5h9qhvjga
7d81aaeee7fa - Started downloading
7d81aaeee7fa - Downloaded in 0 seconds
0d1706afb11e - Started downloading
0d1706afb11e - Downloaded in 1.902 seconds
Digest: sha256:353e92f6b8bcd32ae55780feda22ae68cce47c2f41f0c41c2ee596a2e969c1f4
Status: Downloaded newer image for archlinux:latest
[taskcluster 2022-11-25 02:48:06.001Z] === Task Starting ===
+ pacman -Sy --noconfirm mercurial python-pip
error: failed to initialize alpm library:
(root: /, dbpath: /var/lib/pacman/)
could not find or read directory
[taskcluster 2022-11-25 02:48:07.318Z] Unsuccessful task run with exit code: 255 completed in 7.812 seconds

Comment 42

[Mike Hommey [:glandium]]

So, archlinux failed with:

+ pacman -Sy --noconfirm mercurial python-pip
error: failed to initialize alpm library:
(root: /, dbpath: /var/lib/pacman/)
could not find or read directory

debian12 failed with:

W: GPG error: http://deb.debian.org/debian bookworm InRelease: At least one invalid signature was encountered.
E: The repository 'http://deb.debian.org/debian bookworm InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: GPG error: http://deb.debian.org/debian-security bookworm-security InRelease: At least one invalid signature was encountered.
E: The repository 'http://deb.debian.org/debian-security bookworm-security InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: GPG error: http://deb.debian.org/debian bookworm-updates InRelease: At least one invalid signature was encountered.
E: The repository 'http://deb.debian.org/debian bookworm-updates InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code

ubuntu* failed with:

W: http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://security.ubuntu.com/ubuntu jammy-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://security.ubuntu.com/ubuntu jammy-security InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy-updates InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy-backports InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy-backports InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code

opensuse-tumbleweed failed with:

+ zypper install -y mercurial python3-pip
terminate called after throwing an instance of 'std::system_error'
  what():  Operation not permitted

gentoo failed with:

Using keys from /usr/share/openpgp-keys/gentoo-release.asc
!!! Manifest verification impossible due to keyring problem:
OpenPGP key import failed:
gpg: keybox '/tmp/gemato.1ms72rou/pubring.kbx' created
gpg: key DB6B8C1F96D8BF6D: 2 signatures not checked due to missing keys
gpg: /tmp/gemato.1ms72rou/trustdb.gpg: trustdb created
gpg: key DB6B8C1F96D8BF6D: public key "Gentoo ebuild repository signing key (Automated Signing Key) <[e-mail]>" imported
gpg: can't connect to the agent: End of file
gpg: key 9E6438C817072058: 3 signatures not checked due to missing keys
gpg: key 9E6438C817072058: public key "Gentoo Linux Release Engineering (Gentoo Linux Release Signing Key) <[e-mail]>" imported
gpg: can't connect to the agent: IPC connect call failed
gpg: key BB572E0E2D182910: 1 signature not checked due to a missing key
gpg: key BB572E0E2D182910: public key "Gentoo Linux Release Engineering (Automated Weekly Release Key) <[e-mail]>" imported
gpg: can't connect to the agent: End of file
gpg: key A13D0EF1914E7A72: 1 signature not checked due to a missing key
gpg: key A13D0EF1914E7A72: public key "Gentoo repository mirrors (automated git signing key) <[e-mail]>" imported
gpg: can't connect to the agent: IPC connect call failed
gpg: Total number processed: 4
gpg:               imported: 4
gpg: no need for a trustdb check with 'direct' trust model


 * IMPORTANT: 8 news items need reading for repository 'gentoo'.
 * Use eselect news read to view new items.


Action: sync for repo: gentoo, returned code = 1

fedora* failed with:

+ dnf install -y mercurial python3-pip
Fedora 36 - x86_64                              0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'fedora':
  - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64 [getaddrinfo() thread failed to start]
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64 [getaddrinfo() thread failed to start]

These tasks work just fine on try. In fact, if you take any of them on firefox-ci-tc, edit it to replace gecko-3/b-linux-gcp with gecko-1/b-linux-gcp, without changing anything else, they all work. Whatever is happening that causes these failures (and weirdly enough, no failure on the debian10, debian11, opensuse-leap, rockylinux8 or voidlinux tasks) comes from whatever difference there is at the docker/host level between gecko-3/b-linux-gcp and gecko-1/b-linux-gcp.

Comment 43

[Julien Cristau [:jcristau]]

Michelle or David, do you know if there are differences between the l1 and l3 images that could explain comment 42?

Comment 44

[Michelle Goossens [:masterwayz]]

The images are built using https://github.com/taskcluster/monopacker
To explain how it works in a comment is a bit much, but I checked, and other than some secrets, there should not be any differences between the two.

However, I think when we run tasks on a level 3 repo, the tasks are provisioned in a different GCP project that may have restrictions on outbound network connectivity, this is just a wild guess (and I can't check this manually as I don't have the access yet so I am going to defer to dhouse for when he returns next week).

Comment 45

[Michelle Goossens [:masterwayz]]

(See above comment)

Comment 46

[Andrew Halberstadt [:ahal]]

In theory, the images should be identical other than the CoT private key.

Is it possible that some tool or library got implicitly updated (due to lack of pinning) when we did https://bugzilla.mozilla.org/show_bug.cgi?id=1768865 ? If it works on L1 but not L3, maybe we just need to generate a new L3 image as well to trigger the same update there.

Perhaps a policy of always updating them both together would be good for reproducibility going forward.

Comment 47

[Michelle Goossens [:masterwayz]]

I'll generate some new images.

Comment 48

[Mike Hommey [:glandium]]

(In reply to Michelle Goossens [:masterwayz] from comment #47)

I'll generate some new images.

Did this happen?

Comment 49

[Michelle Goossens [:masterwayz]]

The image I had made I cannot use so I'll build a new one without those changes.
It's building now.

Comment 50

[Michelle Goossens [:masterwayz]]

Leaving NI in place so I don't have to spent two hours looking through my email

Comment 51

[Michelle Goossens [:masterwayz]]

Attachment: Bug 1245969 - Use new level 3 GCP builder image with updates r=#releng-reviewers

Comment 52

[Michelle Goossens [:masterwayz]]

https://hg.mozilla.org/ci/ci-configuration/rev/7a1d3c870d87

Comment 53

[Michelle Goossens [:masterwayz]]

Attachment: Backed out 1 changesets (Bug 1245969)

Backed out changeset 7a1d3c870d87 (bug 1245969)

Comment 54

[Michelle Goossens [:masterwayz]]

Backed out one changeset for: image is busted.

Nov 30 13:52:35Z gecko-3-decision-gcp-bemiofzkt9uykqadbjtdtq docker-worker Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /home/ubuntu/docker-worker/node_modules/fs-ext/build/Release/fs-ext.node)
Nov 30 13:52:35Z gecko-3-decision-gcp-bemiofzkt9uykqadbjtdtq docker-worker     at Object.Module._extensions..node (node:internal/modules/cjs/loader:1239:18)
Nov 30 13:52:35Z gecko-3-decision-gcp-bemiofzkt9uykqadbjtdtq docker-worker     at Module.load (node:internal/modules/cjs/loader:1033:32)
Nov 30 13:52:35Z gecko-3-decision-gcp-bemiofzkt9uykqadbjtdtq docker-worker     at Function.Module._load (node:internal/modules/cjs/loader:868:12)
Nov 30 13:52:35Z gecko-3-decision-gcp-bemiofzkt9uykqadbjtdtq docker-worker     at Module.require (node:internal/modules/cjs/loader:1057:19)
Nov 30 13:52:35Z gecko-3-decision-gcp-bemiofzkt9uykqadbjtdtq docker-worker     at require (node:internal/modules/cjs/helpers:103:18)
Nov 30 13:52:35Z gecko-3-decision-gcp-bemiofzkt9uykqadbjtdtq docker-worker     at Object.<anonymous> (/home/ubuntu/docker-worker/node_modules/fs-ext/fs-ext.js:22:15)
Nov 30 13:52:35Z gecko-3-decision-gcp-bemiofzkt9uykqadbjtdtq docker-worker     at Module._compile (node:internal/modules/cjs/loader:1155:14)
Nov 30 13:52:35Z gecko-3-decision-gcp-bemiofzkt9uykqadbjtdtq docker-worker     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1209:10)
Nov 30 13:52:35Z gecko-3-decision-gcp-bemiofzkt9uykqadbjtdtq docker-worker     at Module.load (node:internal/modules/cjs/loader:1033:32)
Nov 30 13:52:35Z gecko-3-decision-gcp-bemiofzkt9uykqadbjtdtq docker-worker     at Function.Module._load (node:internal/modules/cjs/loader:868:12)
Nov 30 13:52:35Z gecko-3-decision-gcp-bemiofzkt9uykqadbjtdtq docker-worker 2022/11/30 13:51:24 exit status 1

https://hg.mozilla.org/ci/ci-configuration/rev/8cfd8bde6eb7

Comment 55

[Michelle Goossens [:masterwayz]]

Attachment: Bug 1245969 - Create level 1 pool for image testing r=#releng-reviewers

Comment 56

[Michelle Goossens [:masterwayz]]

Still working on the image! Will let you know once it is fixed.

Comment 57

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/b0beb00b2141
Create level 1 pool for image testing r=releng-reviewers,gbrown

Comment 58

[Michelle Goossens [:masterwayz]]

Setting this so the bug doesn't close with my regular bugherder run.

Comment 59

[Michelle Goossens [:masterwayz]]

https://hg.mozilla.org/ci/ci-configuration/rev/b0beb00b2141

Comment 60

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/ae27e7b9ac33
Use new level 3 GCP builder image with updates r=releng-reviewers,gbrown

Comment 61

[Michelle Goossens [:masterwayz]]

Attachment: Backed out 1 changesets (Bug 1245969)

Backed out changeset ae27e7b9ac33 (bug 1245969)

Comment 62

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/590fae29acd0
Backed out 1 changesets

Comment 63

[Mike Hommey [:glandium]]

Do you have an update?

Comment 64

[Michelle Goossens [:masterwayz]]

Yesterday I landed a shiny new level 3 image attached to a different bug, forgot to say it here, sorry.
We have a new image in place right now should you want to try it again.

Comment 65

[Mike Hommey [:glandium]]

I retriggered one of the jobs that failed on the first landing, and it worked. Yoohoo \o/

Comment 66

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/675d7821269c
Add tasks that perform a bootstrap on various bare Linux environments. r=taskgraph-reviewers,firefox-build-system-reviewers,jlorenzo,ahochheiden

Comment 67

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/675d7821269c

Comment 68

[Michelle Goossens [:masterwayz]]

Glad that it's fixed now!