These are the package versions difference from the upgrade that happens when rebuilding the docker images: 32,33c32,33 < ii apt 1.2.27 amd64 commandline package manager < ii apt-utils 1.2.27 amd64 package management related utility programs --- > ii apt 1.2.29 amd64 commandline package manager > ii apt-utils 1.2.29 amd64 package management related utility programs 73c73 < ii chromium-codecs-ffmpeg-extra 69.0.3497.81-0ubuntu0.16.04.1 amd64 Extra ffmpeg codecs for the Chromium Browser --- > ii chromium-codecs-ffmpeg-extra 70.0.3538.77-0ubuntu0.16.04.1 amd64 Extra ffmpeg codecs for the Chromium Browser 103c103 < ii curl 7.47.0-1ubuntu2.9 amd64 command line tool for transferring data with URL syntax --- > ii curl 7.47.0-1ubuntu2.11 amd64 command line tool for transferring data with URL syntax 122c122 < ii distro-info-data 0.28ubuntu0.8 all information about the distributions' releases (data files) --- > ii distro-info-data 0.28ubuntu0.9 all information about the distributions' releases (data files) 130c130 < ii dpkg-dev 1.18.4ubuntu1.4 all Debian package development tools --- > ii dpkg-dev 1.18.4ubuntu1.5 all Debian package development tools 150c150 < ii firefox 62.0.3+build1-0ubuntu0.16.04.2 amd64 Safe and easy web browser from Mozilla --- > ii firefox 63.0+build2-0ubuntu0.16.04.2 amd64 Safe and easy web browser from Mozilla 240,241c240,241 < ii ghostscript 9.25~dfsg+1-0ubuntu0.16.04.1 amd64 interpreter for the PostScript language and for PDF < ii ghostscript-x 9.25~dfsg+1-0ubuntu0.16.04.1 amd64 interpreter for the PostScript language and for PDF - X11 support --- > ii ghostscript 9.25~dfsg+1-0ubuntu0.16.04.2 amd64 interpreter for the PostScript language and for PDF > ii ghostscript-x 9.25~dfsg+1-0ubuntu0.16.04.2 amd64 interpreter for the PostScript language and for PDF - X11 support 268c268 < ii gir1.2-packagekitglib-1.0 0.8.17-4ubuntu6~gcc5.4ubuntu1.3 amd64 GObject introspection data for the PackageKit GLib library --- > ii gir1.2-packagekitglib-1.0 0.8.17-4ubuntu6~gcc5.4ubuntu1.4 amd64 GObject introspection data for the PackageKit GLib library 462c462 < ii libapt-inst2.0:amd64 1.2.27 amd64 deb package format runtime library --- > ii libapt-inst2.0:amd64 1.2.29 amd64 deb package format runtime library 464c464 < ii libapt-pkg5.0:amd64 1.2.27 amd64 package management runtime library --- > ii libapt-pkg5.0:amd64 1.2.29 amd64 package management runtime library 618,620c618,620 < ii libcurl3:amd64 7.47.0-1ubuntu2.9 amd64 easy-to-use client-side URL transfer library (OpenSSL flavour) < ii libcurl3-gnutls:amd64 7.47.0-1ubuntu2.9 amd64 easy-to-use client-side URL transfer library (GnuTLS flavour) < ii libcurl4-openssl-dev:amd64 7.47.0-1ubuntu2.9 amd64 development files and documentation for libcurl (OpenSSL flavour) --- > ii libcurl3:amd64 7.47.0-1ubuntu2.11 amd64 easy-to-use client-side URL transfer library (OpenSSL flavour) > ii libcurl3-gnutls:amd64 7.47.0-1ubuntu2.11 amd64 easy-to-use client-side URL transfer library (GnuTLS flavour) > ii libcurl4-openssl-dev:amd64 7.47.0-1ubuntu2.11 amd64 development files and documentation for libcurl (OpenSSL flavour) 653c653 < ii libdpkg-perl 1.18.4ubuntu1.4 all Dpkg perl modules --- > ii libdpkg-perl 1.18.4ubuntu1.5 all Dpkg perl modules 844,845c844,845 < ii libgs9:amd64 9.25~dfsg+1-0ubuntu0.16.04.1 amd64 interpreter for the PostScript language and for PDF - Library < ii libgs9-common 9.25~dfsg+1-0ubuntu0.16.04.1 all interpreter for the PostScript language and for PDF - common files --- > ii libgs9:amd64 9.25~dfsg+1-0ubuntu0.16.04.2 amd64 interpreter for the PostScript language and for PDF - Library > ii libgs9-common 9.25~dfsg+1-0ubuntu0.16.04.2 all interpreter for the PostScript language and for PDF - common files 1008c1008 < ii libmad0:amd64 0.15.1b-8ubuntu1 amd64 MPEG audio decoder library --- > ii libmad0:amd64 0.15.1b-9ubuntu16.04.1 amd64 MPEG audio decoder library 1056c1056 < ii libmysqlclient20:amd64 5.7.23-0ubuntu0.16.04.1 amd64 MySQL database client library --- > ii libmysqlclient20:amd64 5.7.24-0ubuntu0.16.04.1 amd64 MySQL database client library 1141c1141 < ii libpackagekit-glib2-16:amd64 0.8.17-4ubuntu6~gcc5.4ubuntu1.3 amd64 Library for accessing PackageKit using GLib --- > ii libpackagekit-glib2-16:amd64 0.8.17-4ubuntu6~gcc5.4ubuntu1.4 amd64 Library for accessing PackageKit using GLib 1147c1147 < ii libpam-systemd:amd64 229-4ubuntu21.4 amd64 system and service manager - PAM module --- > ii libpam-systemd:amd64 229-4ubuntu21.5 amd64 system and service manager - PAM module 1382,1383c1382,1383 < ii libsystemd0:amd64 229-4ubuntu21.4 amd64 systemd utility library < ii libsystemd0:i386 229-4ubuntu21.4 i386 systemd utility library --- > ii libsystemd0:amd64 229-4ubuntu21.5 amd64 systemd utility library > ii libsystemd0:i386 229-4ubuntu21.5 i386 systemd utility library 1423,1424c1423,1424 < ii libudev1:amd64 229-4ubuntu21.4 amd64 libudev shared library < ii libudev1:i386 229-4ubuntu21.4 i386 libudev shared library --- > ii libudev1:amd64 229-4ubuntu21.5 amd64 libudev shared library > ii libudev1:i386 229-4ubuntu21.5 i386 libudev shared library 1715c1715 < ii mysql-common 5.7.23-0ubuntu0.16.04.1 all MySQL database common files, e.g. /etc/mysql/my.cnf --- > ii mysql-common 5.7.24-0ubuntu0.16.04.1 all MySQL database common files, e.g. /etc/mysql/my.cnf 1737,1741c1737,1741 < ii openjdk-8-jdk:i386 8u181-b13-0ubuntu0.16.04.1 i386 OpenJDK Development Kit (JDK) < ii openjdk-8-jdk-headless:i386 8u181-b13-0ubuntu0.16.04.1 i386 OpenJDK Development Kit (JDK) (headless) < ii openjdk-8-jre:i386 8u181-b13-0ubuntu0.16.04.1 i386 OpenJDK Java runtime, using Hotspot JIT < ii openjdk-8-jre-headless:amd64 8u181-b13-0ubuntu0.16.04.1 amd64 OpenJDK Java runtime, using Hotspot JIT (headless) < ii openjdk-8-jre-headless:i386 8u181-b13-0ubuntu0.16.04.1 i386 OpenJDK Java runtime, using Hotspot JIT (headless) --- > ii openjdk-8-jdk:i386 8u181-b13-1ubuntu0.16.04.1 i386 OpenJDK Development Kit (JDK) > ii openjdk-8-jdk-headless:i386 8u181-b13-1ubuntu0.16.04.1 i386 OpenJDK Development Kit (JDK) (headless) > ii openjdk-8-jre:i386 8u181-b13-1ubuntu0.16.04.1 i386 OpenJDK Java runtime, using Hotspot JIT > ii openjdk-8-jre-headless:amd64 8u181-b13-1ubuntu0.16.04.1 amd64 OpenJDK Java runtime, using Hotspot JIT (headless) > ii openjdk-8-jre-headless:i386 8u181-b13-1ubuntu0.16.04.1 i386 OpenJDK Java runtime, using Hotspot JIT (headless) 1934c1934 < ii secureboot-db 1.1 amd64 Secure Boot updates for DB and DBX --- > ii secureboot-db 1.4~ubuntu0.16.04.1 amd64 Secure Boot updates for DB and DBX 1971c1971 < ii systemd 229-4ubuntu21.4 amd64 system and service manager --- > ii systemd 229-4ubuntu21.5 amd64 system and service manager 1996c1996 < ii tzdata 2018e-0ubuntu0.16.04 all time zone and daylight-saving time data --- > ii tzdata 2018g-0ubuntu0.16.04 all time zone and daylight-saving time data 2018c2018 < ii udev 229-4ubuntu21.4 amd64 /dev/ and hotplug management daemon --- > ii udev 229-4ubuntu21.5 amd64 /dev/ and hotplug management daemon One of those presumably breaks some expectation from the test.

Summary: Perma Linux mda webrtc failures with "NotFoundError: The object can not be found here." after hg server certificate updated and docker images rebuilt → Perma Linux mda webrtc failures with "NotFoundError: The object can not be found here." after Ubuntu 16.04 docker image upgrade

Stefan Hindli [:stefan_hindli]

Reporter

Updated

•

6 years ago

Flags: needinfo?(gps)

Stefan Hindli [:stefan_hindli]

Reporter

Updated

•

6 years ago

Depends on: 1502531

Stefan Hindli [:stefan_hindli]

Reporter

Comment 10

•

6 years ago

Comment 24

•

6 years ago

Apparently every time we switch the major distro version of the main test image, it causes a ton of test fallout. Apparently it can be several weeks of people time to sort through. I believe jmaher can share horror stories. It is probably easier to coerce the existing images into being deterministic. I've explored running a snapshotted Apt mirror previously so we can remove the 3rd party dependency completely. However, this requires running new infrastructure indefinitely. There was also talk of potentially using a caching HTTP proxy to capture the files requested by Apt and reusing that cache/snapshot on subsequent image builds. I think solutions are moderate effort (1+ week) and are out of scope for this bug. But we shouldn't let them fall off our radar because this kind of stop-the-world-for-several-weeks issue stemming from non-deterministic and non-hermetic Docker builds causes significant disruption and happens a few times per year.

Alex Chronopoulos [:achronop]

Comment 25

•

6 years ago

Gah, I don't believe it ... The problem was that in that system the "null device" is called "dummy device". We hardcode the device name and it is the one expected to be found and be used by getUserMedia request. In [1] is a green try with that change. I need to find a smarter solution for that, though. [1] https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=211709478&revision=5dc96597847b2b4f2e4039f6d456fec067692e3b

Gregory Szorc [:gps]

Assignee

Comment 26

•

6 years ago

Excellent news on finding the problem! Since this issue is blocking the landing of a handful of other patches, I think a hacky solution to unblock others is justified. I would gladly r+ a patch that makes the device name change along with a change forcing the desktop1604-test image to be rebuilt so others are unblocked. Such a change would be to change any file under docker/desktop1604-test. It could be as simple as adding a comment to the Dockerfile file. Or if you'll have a proper patch without much effort, we can wait a bit. But I'd prefer to not wait more than a few hours.

Alex Chronopoulos [:achronop]

Comment 27

•

6 years ago

In that case you can just use the patch from the try and go on. Fell free to use r=me on that. However, it's necessary to tight the change to the new image otherwise tests on the old image will break. A proper solution would be to experiment with PulseAudio commands in order to set the name of the device and see how this affects the name of monitor device. I found something relevant in [1]. However, I need to experiment a little with that and test it on try which will be more than a few hours. [1] https://unix.stackexchange.com/questions/128371/pulseaudio-naming-a-device

Gregory Szorc [:gps]

Assignee

Comment 28

•

6 years ago

Attached file Bug 1503756 - Change name of media device; r=achronop — Details

Regenerating the ubuntu1604-test Docker image pulls in a new package version that changes the name of a media device. This media device name is currently hard coded in the mochitest test harness. This commit makes a change to force regeneration of the Docker image and updates the hardcoded device name to reflect the new version.

Gregory Szorc [:gps]

Assignee

Comment 29

•

6 years ago

Patch is queued. Autoland is closed. Presumably others can queue patches that would regenerate desktop1604-test now, since Lando should process landings in FIFO order. Although it is safer to wait to avoid excessive backout risk.

Mike Hommey [:glandium]

Comment 30

•

6 years ago

Note that freezing the testing images can be seen as both good *and* bad. Good because it would avoid problems like this one popping up randomly, but bad because when those things happen it *does* mean Ubuntu 16.04 *has* changed in ways that break tests, which may or may not also mean that there are problems in the wild with Ubuntu 16.04. OTOH, we don't test other versions of Ubuntu, so...

Pulsebot

•

6 years ago

There was a 2nd instance of "Null Output" being used for the media.cubeb.output_device pref. https://treeherder.mozilla.org/#/jobs?repo=try&revision=ae8401009d4bd34f3f74ba27cc4bd93a591edbb2 is a Try push changing that as well and triggering the new failures that caused the backout. If that comes back green, I'll get that reviewed and landed.

Flags: needinfo?(gps)

Gregory Szorc [:gps]

Assignee

Comment 35

•

6 years ago

My Try push is yielding failures on Android x86 :( https://treeherder.mozilla.org/#/jobs?repo=try&author=gszorc%40mozilla.com&selectedJob=212009567

Gregory Szorc [:gps]

Assignee

Comment 36

•

6 years ago

bc: I'm a bit out of my league dealing with ADB issues. Is there any chance you can help us diagnose the Android 7.0 x86 opt failures in the Try push in comment 34? It obviously has something to do with regenerating the desktop1604-test Docker image. But unlike the other diagnosed failure in this bug, it looks like we can't connect to ADB at all with the new Docker image. It's possible we may have a 2nd failure as a result of rebuilding the Docker image :/

Flags: needinfo?(bob)

Bob Clary [:bc] (inactive)

Comment 37

•

•

6 years ago

Sorry, I don't understand how that would help me run an image interactively. I downloaded the image to an ubuntu vm and attempted to run it but all I can get are no such file or directory when attempting to run commands. I'm at a loss on how to debug this.

Flags: needinfo?(bob)

Mike Hommey [:glandium]

Comment 48

•

6 years ago

With an interactive task, you can get shell access. (once it's running, you can go to artifacts and open the shell.html file)

Bob Clary [:bc] (inactive)

Comment 49

•

6 years ago

fwiw, I've been pending for 3 hours. Hopefully I'll catch it when it starts to run.

Geoff Brown [:gbrown]

Comment 50

•

6 years ago

I've had a similar experience to bc: Can't get an interactive task, possibly because of a backlog. I tried downloading image.tar.zst / zstd -d image.tar.zst / docker import image.tar / docker run -ti <image> /bin/bash (and variants) but the docker run always fails: 'docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/bin/bash\": stat /bin/bash: no such file or directory": unknown.'

Gregory Szorc [:gps]

Assignee

Comment 51

•

6 years ago

Try using `mach taskcluster-load-image <taskid>` to import the Docker image.

Geoff Brown [:gbrown]

Comment 52

•

6 years ago

Thanks. 'mach taskcluster-load-image --task-id UqXjBDhuRw2Q50IFq3OmfQ' worked -- I can docker run now.

Geoff Brown [:gbrown]

Comment 53

•

6 years ago

With the new image, Xsession is crashing, even in Linux and Android 4.3 tasks; I suspect the x86 emulator is more sensitive to its graphical environment, while Linux and Android 4.3 tests can muddle through. Xsession is started at https://dxr.mozilla.org/mozilla-central/rev/b0b856065d5b7ad2996f707e6e797d0d72afd803/taskcluster/scripts/tester/test-linux.sh#141 If you look at any of the Linux or Android tests from gps' original push, you see: https://taskcluster-artifacts.net/Z7cTKkyMS6-4EwZqsJ4T-A/0/public/logs/live_backing.log [task 2018-11-15T17:58:55.477Z] /builds/worker/bin/test-linux.sh: line 191: 36 Trace/breakpoint trap /etc/X11/Xsession 2>&1 Running the docker image locally, I see: root@83a42b9d7153:~# /etc/X11/Xsession Trace/breakpoint trap (core dumped) Tasks have a xsession-errors.log artifact such as https://taskcluster-artifacts.net/Z7cTKkyMS6-4EwZqsJ4T-A/0/public/test/public/xsession-errors.log Xsession: X session started for worker at Thu Nov 15 17:39:48 UTC 2018 localuser:worker being added to access control list openConnection: connect: No such file or directory cannot connect to brltty at :0 /builds/worker/Desktop was removed, reassigning DESKTOP to homedir /builds/worker/Downloads was removed, reassigning DOWNLOAD to homedir /builds/worker/Templates was removed, reassigning TEMPLATES to homedir /builds/worker/Public was removed, reassigning PUBLICSHARE to homedir gnome-session-is-accelerated: llvmpipe detected. gnome-session-binary[36]: ERROR: Failed to connect to system bus: Could not connect: No such file or directory aborting... Also, screenshots show an all black desktop with the new image, and 'ps -ef' is missing many of the gvfs/evolution/etc processes associated with the window manager. None of this points to an obvious solution, but hopefully I've narrowed down the scope of the problem?

Flags: needinfo?(gbrown)

Mike Hommey [:glandium]

Comment 54

•

6 years ago

I guess part of the problem is that we're running tests in an environment that doesn't run any of the daemons that normally run on an Ubuntu system (here, probably the dbus system daemon)

Geoff Brown [:gbrown]

Comment 55

•

6 years ago

Comparing process lists from system-info.log artifacts from a normal Linux test: https://taskcluster-artifacts.net/Se57jPcNSduPOHHVwTHCDw/0/public/test_info//system-info.log to one from gps' try push: https://taskcluster-artifacts.net/Z7cTKkyMS6-4EwZqsJ4T-A/0/public/test_info//system-info.log They both have dbus-daemon and dbus-launch processes, but gps' is missing PID 100 ssh-agent ['/usr/bin/ssh-agent', '/usr/bin/dbus-launch', '--exit-with-session', '/usr/bin/im-launch', 'x-session-manager']

Mike Hommey [:glandium]

Comment 56

•

6 years ago

ssh-agent is there, just with different arguments. There's another ton missing, though: PID 159 gnome-screensaver ['/usr/bin/gnome-screensaver', '--no-daemon'] created at 2018-11-16 22:43:18.690000 PID 175 dconf-service ['/usr/lib/dconf/dconf-service'] created at 2018-11-16 22:43:18.940000 PID 241 nautilus ['nautilus', '-n'] created at 2018-11-16 22:43:21.690000 PID 246 indicator-session-service ['/usr/lib/x86_64-linux-gnu/indicator-session/indicator-session-service'] created at 2018-11-16 22:43:21.700000 PID 248 indicator-power-service ['/usr/lib/x86_64-linux-gnu/indicator-power/indicator-power-service'] created at 2018-11-16 22:43:21.710000 PID 257 indicator-datetime-service ['/usr/lib/x86_64-linux-gnu/indicator-datetime/indicator-datetime-service'] created at 2018-11-16 22:43:21.760000 PID 268 indicator-application-service ['/usr/lib/x86_64-linux-gnu/indicator-application/indicator-application-service'] created at 2018-11-16 22:43:21.770000 PID 272 indicator-keyboard-service ['/usr/lib/x86_64-linux-gnu/indicator-keyboard/indicator-keyboard-service'] created at 2018-11-16 22:43:21.790000 PID 273 indicator-bluetooth-service ['/usr/lib/x86_64-linux-gnu/indicator-bluetooth/indicator-bluetooth-service'] created at 2018-11-16 22:43:21.790000 PID 292 gvfs-udisks2-volume-monitor ['/usr/lib/gvfs/gvfs-udisks2-volume-monitor'] created at 2018-11-16 22:43:21.870000 PID 301 gvfs-mtp-volume-monitor ['/usr/lib/gvfs/gvfs-mtp-volume-monitor'] created at 2018-11-16 22:43:21.900000 PID 307 evolution-source-registry ['/usr/lib/evolution/evolution-source-registry'] created at 2018-11-16 22:43:21.910000 PID 309 initctl [] created at 2018-11-16 22:43:21.910000 PID 318 gvfs-afc-volume-monitor ['/usr/lib/gvfs/gvfs-afc-volume-monitor'] created at 2018-11-16 22:43:21.930000 PID 325 gvfs-gphoto2-volume-monitor ['/usr/lib/gvfs/gvfs-gphoto2-volume-monitor'] created at 2018-11-16 22:43:21.960000 PID 330 gvfs-goa-volume-monitor ['/usr/lib/gvfs/gvfs-goa-volume-monitor'] created at 2018-11-16 22:43:21.970000 PID 338 evolution-calendar-factory ['/usr/lib/evolution/evolution-calendar-factory'] created at 2018-11-16 22:43:22.140000 PID 354 gvfsd-trash ['/usr/lib/gvfs/gvfsd-trash', '--spawner', ':1.1', '/org/gtk/gvfs/exec_spaw/0'] created at 2018-11-16 22:43:22.440000 PID 360 evolution-calendar-factory-subprocess ['/usr/lib/evolution/evolution-calendar-factory-subprocess', '--factory', 'contacts', '--bus-name', 'org.gnome.evolution.dataserver.Subprocess.Backend.Calendarx338x2', '--own-path', '/org/gnome/evolution/dataserver/Subprocess/Backend/Calendar/338/2'] created at 2018-11-16 22:43:22.470000 PID 373 evolution-addressbook-factory ['/usr/lib/evolution/evolution-addressbook-factory'] created at 2018-11-16 22:43:22.650000 PID 375 evolution-calendar-factory-subprocess ['/usr/lib/evolution/evolution-calendar-factory-subprocess', '--factory', 'local', '--bus-name', 'org.gnome.evolution.dataserver.Subprocess.Backend.Calendarx338x3', '--own-path', '/org/gnome/evolution/dataserver/Subprocess/Backend/Calendar/338/3'] created at 2018-11-16 22:43:22.650000 PID 384 evolution-addressbook-factory-subprocess ['/usr/lib/evolution/evolution-addressbook-factory-subprocess', '--factory', 'local', '--bus-name', 'org.gnome.evolution.dataserver.Subprocess.Backend.AddressBookx373x2', '--own-path', '/org/gnome/evolution/dataserver/Subprocess/Backend/AddressBook/373/2'] created at 2018-11-16 22:43:22.800000 PID 399 gvfsd-metadata ['/usr/lib/gvfs/gvfsd-metadata'] created at 2018-11-16 22:43:22.860000 PID 439 zeitgeist-datahub ['zeitgeist-datahub'] created at 2018-11-16 22:43:41.810000 PID 446 sh ['/bin/sh', '-c', '/usr/lib/x86_64-linux-gnu/zeitgeist/zeitgeist-maybe-vacuum; /usr/bin/zeitgeist-daemon'] created at 2018-11-16 22:43:41.820000 PID 450 zeitgeist-daemon ['/usr/bin/zeitgeist-daemon'] created at 2018-11-16 22:43:41.830000 PID 456 zeitgeist-datah [] created at 2018-11-16 22:43:41.850000 PID 467 update-notifier ['update-notifier'] created at 2018-11-16 22:44:21.810000

Cosmin Sabou [:CosminS]

•

6 years ago

Can one of you three recommend the next steps here? I see this bug and while it appears to cause android failures, :gbrown pointed out there are many other issues with the image in general.

Flags: needinfo?(ted)

Flags: needinfo?(mh+mozilla)

Flags: needinfo?(gps)

Andrew Halberstadt [:ahal]

•

6 years ago

•

6 years ago

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #67) > Also we can try using `xenial-*` tags from > https://hub.docker.com/r/library/ubuntu/tags/ instead of using "FROM: > ubuntu:16.04". That's an interesting idea, but would probably require some package pinning to avoid some of the packages from the original docker image being upgraded.

Gregory Szorc [:gps]

Assignee

•

6 years ago

We decided to create a one-off image using the last known good base image of desktop1604-test as a workaround. I've filed bug 1511527 to track unforking the image. I'm actively working on this. So assigning to me. When I started work an hour or two ago, I thought I'd be done by now. But I'm hitting issues with Docker where changes in an upper image layer matching those in the base layer seem to be hitting bugs in Docker (specifically its storage drivers). Docker is a real piece of work...

Assignee: nobody → gps

Status: NEW → ASSIGNED

Gregory Szorc [:gps]

Assignee

Updated

•

6 years ago

Component: WebRTC → Task Configuration

Product: Core → Firefox Build System

Gregory Szorc [:gps]

Assignee

Comment 76

•

6 years ago

Attached file Bug 1503756 - Move ubuntu1604-test-system-setup.sh to desktop1604-test image; r?dustin (obsolete) — Details

It is only used by this image. No need to have it hang out in recipes. Note: this commit landing in isolation will result in the building of a bad Docker image. The commit after this that copies the ubuntu1604-test image is necessary to prevent task bustage.

Gregory Szorc [:gps]

Assignee

Comment 77

•

6 years ago

Attached file Bug 1503756 - Inherit from working desktop1604-test image to work around non-deterministic rebuilding; r?dustin — Details

DO NOT LAND UNTIL indygreg/ REFERENCE IS REMOVED FROM Dockerfile! The desktop1604-test image doesn't build deterministically, primarily due to not pinning APT package indices. Rebuilding it from scratch today results in at least 2 problems causing test failures. One has to due with a renamed media device. Another is not yet determined but is causing the Android emulator to fail. Various people have spent several hours trying to debug the failures. There are some leads. But no obvious solution is in sight. Various code landings are blocked on this. Enough is enough. This commit creates a copy of the desktop1604-test image and modifies it so it inherits from a base image that contains the last good build of the image. Other minimal modifications were made to enable the image to build with a populated base image. References to the old image have been updated to point to this new image. This is a gross hack. But it gets the job done. It should unblock various code landings. The base image was imported from task K-qJkvIqQQ-EFK8-4zg_nA. The intent is to undo this hack ASAP by fixing the base image. This is tracked by bug 1511527. As part of developing this patch, I ran into at least two different issues building the Docker image using the last known good image as a base. On my desktop machine using btrfs for Docker storage, the image failed to build due to a "failed to export image: failed to create image: failed to get layer XXX: layer does not exist" error. In CI, I ran into separate issues where some `rm` invocations didn't work. Why, I have no clue. I ran a `docker export | docker import` of a running container with the good image to produce a new Docker image with just the filesystem contents. This had the effect of squashing the original Docker image into a single layer. For reasons I can't explain, this made the image build in CI. Welcome to the wonderful world of Docker bugs, everyone.

Phabricator Automation

Updated

6 years ago

Target Milestone: Firefox 65 → mozilla65

Tom Prince [:tomprince]

Comment 81

•

6 years ago

bugherder uplift

https://hg.mozilla.org/releases/mozilla-esr60/rev/68843d2182e8

status-firefox-esr60: --- → fixed

Edwin Takahashi (:egao | infrequent contributor)

Updated

•

5 years ago

Blocks: 1563864

Bug 1503756 - Change name of media device; r=achronop 6 years ago Gregory Szorc [:gps] 47 bytes, text/x-phabricator-request		Details \| Review
Bug 1503756 - Move ubuntu1604-test-system-setup.sh to desktop1604-test image; r?dustin 6 years ago Gregory Szorc [:gps] 47 bytes, text/x-phabricator-request		Details \| Review
Bug 1503756 - Inherit from working desktop1604-test image to work around non-deterministic rebuilding; r?dustin 6 years ago Gregory Szorc [:gps] 47 bytes, text/x-phabricator-request		Details \| Review