1563864 - Ubuntu 18.04 investigation

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Edwin Takahashi (:egao | infrequent contributor)]

Currently, the CI system runs Ubuntu 16.04 that has been 'hacked' to inherit from a last known good configuration, as detailed in bug 1511527 and bug 1503756.

Ubuntu 16.04 was released in April 2016 so it is now at least 3 years old.

The CI system should be upgraded to either Ubuntu 18.04 (available now) or Ubuntu 20.04 (planned April 2020). Ideally, the migration path to Ubuntu 18.04 could serve as a trial run for the Ubuntu 20.04 upgrade path as a proper methodology around this process is built.

Comment 1

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: ubuntu 18.04 - wip

I was able to get the ubuntu18.04 docker image to successfully build, see:
https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=36c472e4eb1012b35821676bd8e3b1a7aca93740

The diff that has worked so far is attached as `ubuntu 18.04 - wip`.

With the ubuntu 18.04 images, a sample of tests were run.

*reftests*
- some unexpected-fail, some unexpected-pass
- overall, easy to deal with

*jsreftests*
- all pass

*crashtest*
- all pass

*mochitest-media*
- multiple failures; may be related to DRM restriction on 18.04
- most failrues are `NotFoundError`
- some tests are `NotAllowedError`

*web-platform-wdspec*
- most failures relate to screen resolution

*web-platform-reftest*
- need some fuzzing of results

Comment 2

[Edwin Takahashi (:egao | infrequent contributor)]

After fuzzing the expected outcomes in https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=a8e202ad0 and https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=b82dff8b63163c790f060e240a6366ffe9a897bb, I was able to achieve a green run for web-platform-test-reftests and reftests, with crashtest already green.

Comment 3

[Edwin Takahashi (:egao | infrequent contributor)]

In bug 1565332, attempts were made to use Debian 10 as the new distribution of choice to run CI tasks.

As of https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=119fc9a78c7c62c7bace963a4bb88673ee9cab51 shows that this endeavor holds some promise.

Comment 4

[Edwin Takahashi (:egao | infrequent contributor)]

Closing this bug as I have made significant investment into Debian 10 and that is the future of linux CI at mozilla.

Comment 5

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1563864 - add ubuntu18.04 dockerfile and enable codepath to specify image for use in linux testing

Comment 6

[Edwin Takahashi (:egao | infrequent contributor)]

Reopening bug due to some changes prompted by further investigation.

deterministic build

One of the big reasons for initially going with Debian 10 was the deterministic build capability. Debian emphasizes this on their wiki. This was to avoid a repeat of Bug 1503756 where some changes in the package busted all of the Linux related CI.

Thanks to investigative work by :aerickson it has been noted that Docker tag can be useful to implement a deterministic build:
https://medium.com/@tariq.m.islam/container-deployments-a-lesson-in-deterministic-ops-a4a467b14a03

---

window manager and audio driver issues

Both Ubuntu 18.04 and Debian 10 have migrated to the GNOME window manager, which presents a few problems for the tests.

Current test are written with the implicit assumption that Unity is the desktop window manager, but it has now been discontinued. This leads to some issues with window size detection, placement of some buttons and/or context menus as well as rendering of some text and fonts.

Furthermore, both Debian 10 and Ubuntu 18.04 therefor require a workaround posted against https://bugzilla.mozilla.org/show_bug.cgi?id=1596526 due to the switch to GNOME shell relating to pulseaudio and pacftl. So both systems are on equal footing here.

For some tests/suites, Ubuntu 18.04 has less failures, such as reftest, web-platform-tests-reftests and mochitest-devtools. Debian 10 tends to have more issues with reftests that will require some additional greening work.

---

market share

Last reason to reopen this bug is the market penetration of the distros.

For Firefox Desktop on Linux, it is difficult to definitively say the proportion of market share but all telemetry, product manager opinions and anecdotal evidence points to Ubuntu taking the lion's share of the market, up to 80%.

This is a strong argument in favor of continuing to use the Ubuntu distribution as baseline in CI, so we are testing against the most probable operating system for the target market.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a6ea48838875
add ubuntu18.04 dockerfile and enable codepath to specify image for use in linux testing r=jmaher

Comment 8

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/a6ea48838875

Comment 9

[Mike Hommey [:glandium]]

it has been noted that Docker tag can be useful to implement a deterministic build

FWIW, that's only true for the base image. As soon as you apt-get install something, you're back to non-determinism. We get determinism from Debian by using snapshot.debian.org at fixed dates as an apt source (well, the debian10-test image doesn't, but it should, like all the other debian-based docker images we have in tree), but Ubuntu doesn't have that.

Comment 10

[Edwin Takahashi (:egao | infrequent contributor)]

:glandium - won't the date-tagged ones do what you're saying?

from ubuntu:bionic-20191029 as mentioned in https://hub.docker.com/_/ubuntu/

Comment 11

[Mike Hommey [:glandium]]

No, all that does is take a fixed image, but then if you apt-get install, it pulls from the normal Ubuntu repositories. So apt-get install in 2 months is not going to install the same things as today, and we're back to square one.