Closed Bug 1563864 Opened 1 year ago Closed 10 months ago

Ubuntu 18.04 investigation

Categories

(Firefox Build System :: Task Configuration, task)

Desktop
Linux
task
Not set
normal

Tracking

(firefox72 fixed)

RESOLVED FIXED
mozilla72
Tracking Status
firefox72 --- fixed

People

(Reporter: egao, Assigned: egao)

References

Details

Attachments

(2 files)

Currently, the CI system runs Ubuntu 16.04 that has been 'hacked' to inherit from a last known good configuration, as detailed in bug 1511527 and bug 1503756.

Ubuntu 16.04 was released in April 2016 so it is now at least 3 years old.

The CI system should be upgraded to either Ubuntu 18.04 (available now) or Ubuntu 20.04 (planned April 2020). Ideally, the migration path to Ubuntu 18.04 could serve as a trial run for the Ubuntu 20.04 upgrade path as a proper methodology around this process is built.

Assignee: nobody → egao
OS: Unspecified → Linux
Hardware: Unspecified → Desktop
Attached file ubuntu 18.04 - wip
I was able to get the ubuntu18.04 docker image to successfully build, see:
https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=36c472e4eb1012b35821676bd8e3b1a7aca93740

The diff that has worked so far is attached as `ubuntu 18.04 - wip`.

With the ubuntu 18.04 images, a sample of tests were run.

*reftests*
- some unexpected-fail, some unexpected-pass
- overall, easy to deal with

*jsreftests*
- all pass

*crashtest*
- all pass

*mochitest-media*
- multiple failures; may be related to DRM restriction on 18.04
- most failrues are `NotFoundError`
- some tests are `NotAllowedError`

*web-platform-wdspec*
- most failures relate to screen resolution

*web-platform-reftest*
- need some fuzzing of results

After fuzzing the expected outcomes in https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=a8e202ad0 and https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=b82dff8b63163c790f060e240a6366ffe9a897bb, I was able to achieve a green run for web-platform-test-reftests and reftests, with crashtest already green.

Depends on: linux1804

In bug 1565332, attempts were made to use Debian 10 as the new distribution of choice to run CI tasks.

As of https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=119fc9a78c7c62c7bace963a4bb88673ee9cab51 shows that this endeavor holds some promise.

Closing this bug as I have made significant investment into Debian 10 and that is the future of linux CI at mozilla.

Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → WONTFIX
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---

Reopening bug due to some changes prompted by further investigation.

deterministic build

One of the big reasons for initially going with Debian 10 was the deterministic build capability. Debian emphasizes this on their wiki. This was to avoid a repeat of Bug 1503756 where some changes in the package busted all of the Linux related CI.

Thanks to investigative work by :aerickson it has been noted that Docker tag can be useful to implement a deterministic build:
https://medium.com/@tariq.m.islam/container-deployments-a-lesson-in-deterministic-ops-a4a467b14a03


window manager and audio driver issues

Both Ubuntu 18.04 and Debian 10 have migrated to the GNOME window manager, which presents a few problems for the tests.

Current test are written with the implicit assumption that Unity is the desktop window manager, but it has now been discontinued. This leads to some issues with window size detection, placement of some buttons and/or context menus as well as rendering of some text and fonts.

Furthermore, both Debian 10 and Ubuntu 18.04 therefor require a workaround posted against https://bugzilla.mozilla.org/show_bug.cgi?id=1596526 due to the switch to GNOME shell relating to pulseaudio and pacftl. So both systems are on equal footing here.

For some tests/suites, Ubuntu 18.04 has less failures, such as reftest, web-platform-tests-reftests and mochitest-devtools. Debian 10 tends to have more issues with reftests that will require some additional greening work.


market share

Last reason to reopen this bug is the market penetration of the distros.

For Firefox Desktop on Linux, it is difficult to definitively say the proportion of market share but all telemetry, product manager opinions and anecdotal evidence points to Ubuntu taking the lion's share of the market, up to 80%.

This is a strong argument in favor of continuing to use the Ubuntu distribution as baseline in CI, so we are testing against the most probable operating system for the target market.

Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a6ea48838875
add ubuntu18.04 dockerfile and enable codepath to specify image for use in linux testing r=jmaher
Status: REOPENED → RESOLVED
Closed: 1 year ago10 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla72

it has been noted that Docker tag can be useful to implement a deterministic build

FWIW, that's only true for the base image. As soon as you apt-get install something, you're back to non-determinism. We get determinism from Debian by using snapshot.debian.org at fixed dates as an apt source (well, the debian10-test image doesn't, but it should, like all the other debian-based docker images we have in tree), but Ubuntu doesn't have that.

:glandium - won't the date-tagged ones do what you're saying?

from ubuntu:bionic-20191029 as mentioned in https://hub.docker.com/_/ubuntu/

No, all that does is take a fixed image, but then if you apt-get install, it pulls from the normal Ubuntu repositories. So apt-get install in 2 months is not going to install the same things as today, and we're back to square one.

You need to log in before you can comment on or make changes to this bug.