Open Bug 1245969 Opened 6 years ago Updated 3 years ago

Automated testing for `mach bootstrap`

Categories

(Firefox Build System :: Bootstrap Configuration, defect)

defect
Not set
normal

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: gps, Assigned: gps)

References

Details

Attachments

(4 files)

We currently don't have automated test coverage that `mach bootstrap` works. This bug is about creating that automation.
Depends on: 1246025
Depends on: 1246033
Depends on: 1246040
(THIS PATCH IS FEEDBACK QUALITY AND ISN'T READY FOR FULL REVIEW)

We've had `mach bootstrap` and the one-line wget + `python bootstrap`
bootstrap tool for years. Unfortunately, it's never been formally
tested. TaskCluster enables us to finally do that.

This commit introduces support for testing the system bootstrap tool
on various Linux distros.

The bootstrap tasks are implemented as generic non-build/test tasks.
They are part of the "bootstrap" tag, so you can include "-j bootstrap"
in your try syntax to schedule them.

TODO

* Arch does not work (interactive prompt is hanging execution)
* Gentoo does not work
* Figure out how to schedule every ~24 hours
* Support Fennec bootstrap
* Tidy up TreeHerder names/groups?
* Work with sheriffs on TreeHerder visibility policy

Review commit: https://reviewboard.mozilla.org/r/35345/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/35345/
Attachment #8720527 - Flags: review?(dustin)
Comment on attachment 8720527 [details]
Bug 1245969 - TaskCluster tasks for performing system bootstrap;

https://reviewboard.mozilla.org/r/35345/#review32141

It would be better to define docker images for each of these tasks, with the system setup complete, and then just run the VCS clone (please use tc-vcs to take advantage of caching) and `mach configure` in the task itself.  In terms of in-tree complexity, it's about the same, but has the advantage of not pulling down loads of software for every run of the task.  Plus it nicely separates system configuration from the task itself.

Note, too, that TaskCluster jobs run as root (there's a bug to make that configurable..), so you'll probably want to create a user in your docker image and then switch to that user before cloning and running mach.

You can schedule these on a periodic basis with the taskcluster hooks service.  We're also looking at something like .cron.yml in the root of the gecko tree to allow more ad-hoc scheduling.
Attachment #8720527 - Flags: review?(dustin)
Depends on: 1251352
Comment on attachment 8720527 [details]
Bug 1245969 - TaskCluster tasks for performing system bootstrap;

https://reviewboard.mozilla.org/r/35345/#review94352

::: taskcluster/ci/bootstrap/kind.yml:74
(Diff revision 2)
> +            docker-image: "debian:jessie"
> +        run:
> +            system-setup: "apt-get update && apt-get -y install python2.7 wget"
> +            python: python2.7
> +
> +    bootstrap/fedora22:

Fedora 22 entered end-of-life in July. I'd suggest Feodra 23 and 24. https://lists.fedoraproject.org/archives/list/announce@lists.fedoraproject.org/thread/GCMM2KQQRY5PKFY42S3K772BDRENXZ7U/

Fedora 25 comes out next week, which will obsolete 23 as well, but there's no tag for it on dockerhub yet.

::: taskcluster/ci/bootstrap/kind.yml:110
(Diff revision 2)
> +                && echo ">=x11-libs/cairo-1.14.2 X" >> /etc/portage/package.use/firefox
> +                && emerge -UD world
> +            python: python2.7
> +
> +    bootstrap/ubuntu1404:
> +        description: "Ubuntu 16.04 Bootstrap Support"

Target should be `ubuntu1604` to match the image?

::: taskcluster/taskgraph/transforms/job/bootstrap.py:87
(Diff revision 2)
> +
> +    script_args = [
> +        run['system-setup'].strip(),
> +        'wget %s' % bsurl,
> +        bs,
> +        # Many distros run an ancient Mercurial that doesn't do cloen bundles.

s/cloen/clone/
Comment on attachment 8720527 [details]
Bug 1245969 - TaskCluster tasks for performing system bootstrap;

https://reviewboard.mozilla.org/r/35345/#review94684

With this in place, can we get rid of the "artifact build" builds?  I think those are doing a similar thing, right (verifying that the pieces necessary for an artifact build are in place)?

I like this!

::: taskcluster/ci/bootstrap/kind.yml:24
(Diff revision 2)
> +    run:
> +        using: bootstrap

Minor: I kind of prefer to have this co-located with the other contents of the `run` key. Otherwise the reader is left without much context for the other `run` keys.

::: taskcluster/taskgraph/transforms/job/bootstrap.py:39
(Diff revision 2)
> +@transforms.add
> +def clone_applications(config, jobs):

Transform should be in `taskcluster/taskgraph/transforms/bootstrap.py`.
Attachment #8720527 - Flags: review?(dustin) → review-
dustin: it's been a year since we visited this problem. How would you recommend we solve it today?

Recap: We don't want these tasks to run on every push. But because results are definitely not deterministic, we want them to run periodically. I was thinking we could use SCHEDULES to ensure we run them when python/bootstrap changes. And maybe supplement that with a cron scheduling? I just don't know the state of periodic scheduling on mozilla-central these days. I think it was in flux last I talked to bstack?
Flags: needinfo?(dustin)
Periodic scheduling is pretty solid -- it's used for nightlies and a number of other things.  So yeah, I think that approach makes a lot of sense.
Flags: needinfo?(dustin)
Product: Core → Firefox Build System
Component: General → Bootstrap Configuration
bstack: the desire for having CI coverage of `mach bootstrap` has come up again recently. These tasks are special in that they ideally start from a vanilla-as-possible OS install. I think it would be ideal if we could define them as Linux tasks that ran in a VM that was imaged from a fresh distro install. Do generic-worker or taskcluster-worker yet support running Linux tasks as root in a VM - not Docker - environment? And do you think we should attempt to shoehorn these tasks into docker-worker or wait for something better?
Flags: needinfo?(bstack)
Depends on: 1469441
Afaik, taskcluster-worker has experimental qemu support but that's as far as we are w.r.t. using a vm instead of docker. :jonasfj, what is the status of qemu support?

Could another possibility to have a 99% fresh ami but with generic worker and livelog binaries dropped in? Maybe even scriptworker would work in that case. I'm guessing that if docker is weird enough compared to a normal distribution install then maybe the ami are too?

I think I'm missing up above where the desire that these run in a VM rather than docker comes from? Are the docker images just plain weird compared to the actual disributions?
Flags: needinfo?(bstack) → needinfo?(jopsen)
Distro Docker images are stripped down installs (they are optimized for size). And we occasionally run into issues where certain packages don't install correctly inside Docker or that Docker does weird things to the environment. I'd feel better if we were able to tell a task to execute in a VM "image" (which could be versioned in source control), as I think it would be more representative of end-user systems.

Now, how we'd go about producing and consuming said image, I'm not sure. But if we could tell a task to start with a VM or filesystem snapshot and execute a process therein in a "real" OS environment (not Docker), then we'd be in a better position.
Depends on: 1383929
We've had `mach bootstrap` and the one-line wget + `python bootstrap`
bootstrap tool for years. Unfortunately, it's never been formally
tested. TaskCluster enables us to finally do that.

This commit introduces support for testing the system bootstrap tool
on various Linux distros.

The bootstrap tasks are implemented as their own kind because they
are somewhat special.

Each OS is defined once along with its base Docker images, a command
to pre-bootstrap the Docker environment (typically to install Python
and wget), and the name of the Python binary to use.

A transform "splits" each job into 4 separate jobs, each one testing a
different bootstrap application: firefox and fennec with normal and
artifact variations.

The emitted task is a giant pile of shell commands that essentially:

1) Perform (hopefully minimal) pre-bootstrap setup
2) Download bootstrap.py from the active revision
3) Invoke the bootstrapper
4) Clone the repo
5) Run `mach build`

Several tasks are broken:

* Arch is completely broken due to attempting to run a command as root
* CentOS7 is completely broken attempting a group install of a missing
  group.
* Debian, Ubuntu, and Fedora aren't installing Clang (likely due to the
  bootstrapper not installing it in non-interactive mode).
* Probably a handful of other failures.

The tasks are all tier 2. And they aren't enabled by default or picked
up by any dependencies. The only way to trigger them is via
`mach try fuzzy --full`.

Considering I've been sitting on this patch for ~2 years and that others
have wanted to test bootstrap in CI, I think getting a somewhat working
version of the CI tasks landed is a good first step and unblocks a lot
of incremental improvements to testing bootstrap. While there are
many shortcomings of this code, I think we should land now and iterate
later.
Flags: needinfo?(jopsen)
You need to log in before you can comment on or make changes to this bug.