Closed Bug 1294264 Opened 8 years ago Closed 6 years ago

Desktop test images are ~4GB, take too long to build and download

Categories

(Taskcluster Graveyard :: Docker Images, defect)

Product:
Taskcluster Graveyard
Component:
Docker Images
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gps, Unassigned)

References

Details

Attachments

(1 file)

Bug 1294264 - Don't install recommended packages in desktop test images;
58 bytes, text/x-review-board-request
Details 
The desktop-test Ubuntu based Docker images are ~4 GB, take ~20 minutes to build, and several minutes to download and apply.

Since the images install the "ubuntu-desktop" package, they pretty much pull in the world in terms of dependencies. Included in the dependencies are a Sudoku Gnome application, Thunderbird, and, - wait for it - Firefox.

I know we should test in environments that resemble end-user environments. But having games and a copy of Firefox in the image feels a bit overkill. Let's see if we can trim out some fat so our automation is leaner and faster.
How would fit other applications suites like LibreOffice in here? Does any of our tests depend on it to be installed? I don't think we have any kind of test which operates with external applications right now. If we could get rid of that we could spent another huge amount of disk space.
I think some of this might be fixed by adjusting the "suggests" functionality?
I think you mean --no-install-recommends.
On a fresh 16.04 Docker image, `apt install ubuntu-desktop` versus apt install --no-install-recommends ubuntu-desktop` is a difference of ~1800 MB, or ~50% size savings.

The list of recommended packages (minus dependencies) not getting installed with --no-install-recommends is:

 cron | cron-daemon rsyslog | system-log-daemon lintian gnome-system-monitor | mate-system-monitor dns-root-data gcc | c-compiler build-essential fakeroot
  libalgorithm-merge-perl cups-filters | foomatic-filters cups cups-client gvfs-backends gnome-user-share libcanberra-pulse evolution-data-server geoclue-ubuntu-geoip
  | geoclue-provider ubuntu-touch-sounds libatm1 libxtables11 isc-dhcp-common indicator-application libcanberra-gtk-module libcanberra-gtk3-module
  libclutter-1.0-common libcogl-common colord cracklib-runtime libfile-fcntllock-perl enchant libtxc-dxtn-s2tc0 | libtxc-dxtn0 hwdata libgtk2.0-bin hud hunspell-en-us
  | hunspell-dictionary | myspell-dictionary usbmuxd krb5-locales modemmanager libfribidi0 python3-aptdaemon.pkcompat | packagekit-system-interface | packagekit
  libpaper-utils qttranslations5-l10n libwacom-bin libwebkit2gtk-4.0-37-gtk2 tcpd xml-core zeitgeist-core | zeitgeist file ppp iptables crda iputils-arping
  network-manager-pptp network-manager-gnome | plasma-widget-networkmanagement | plasma-nm notify-osd-icons netbase rename gir1.2-gst-plugins-base-1.0
  gir1.2-gstreamer-1.0 pm-utils plymouth-theme-ubuntu-text | plymouth-theme pulseaudio-module-x11 rtkit apport python3-bs4 python3-html5lib unattended-upgrades
  a11y-profile-manager-indicator acpi-support activity-log-manager aisleriot app-install-data-partner apport-gtk avahi-autoipd avahi-daemon baobab bluez-cups
  branding-ubuntu brltty cheese cups-bsd cups-filters deja-dup eog evince example-content file-roller firefox fonts-guru fonts-kacst-one fonts-khmeros-core fonts-lao
  fonts-liberation fonts-lklug-sinhala fonts-nanum fonts-noto-cjk fonts-sil-abyssinica fonts-sil-padauk fonts-takao-pgothic fonts-thai-tlwg fonts-tibetan-machine fwupd
  fwupdate fwupdate-signed gedit gnome-accessibility-themes gnome-calculator gnome-calendar gnome-disk-utility gnome-font-viewer gnome-keyring gnome-mahjongg
  gnome-mines gnome-orca gnome-power-manager gnome-screensaver gnome-screenshot gnome-sudoku gnome-system-log gnome-system-monitor gnome-terminal gnupg-agent gucharmap
  gvfs-fuse hplip ibus-gtk ibus-gtk3 ibus-table kerneloops-daemon laptop-detect libnss-mdns libpam-gnome-keyring libproxy1-plugin-gsettings
  libproxy1-plugin-networkmanager libqt4-sql-sqlite libreoffice-calc libreoffice-gnome libreoffice-impress libreoffice-math libreoffice-ogltrans libreoffice-pdfimport
  libreoffice-style-breeze libreoffice-writer libwmf0.2-7-gtk mousetweaks nautilus-sendto nautilus-share network-manager-gnome network-manager-pptp-gnome onboard
  overlay-scrollbar-gtk2 pcmciautils plymouth-theme-ubuntu-logo policykit-desktop-privileges pppconfig pppoeconf printer-driver-brlaser printer-driver-c2esp
  printer-driver-foo2zjs printer-driver-min12xxw printer-driver-ptouch printer-driver-pxljr printer-driver-sag-gdi printer-driver-splix pulseaudio-module-bluetooth
  python3-aptdaemon.pkcompat qt-at-spi remmina rhythmbox seahorse shotwell simple-scan snapd sni-qt speech-dispatcher system-config-printer-gnome thunderbird
  thunderbird-gnome-support totem transmission-gtk ttf-ancient-fonts-symbola ubuntu-docs ubuntu-software unity-accessibility-profiles unity-webapps-common
  usb-creator-gtk vino whoopsie xcursor-themes xdg-utils xul-ext-ubufox zeitgeist-core zeitgeist-datahub dosfstools ntfs-3g eject gdisk unity-scope-zotero
  unity-scope-video-remote unity-lens-applications unity-lens-photos unity-scope-colourlovers unity-lens-video unity-scope-openclipart unity-scope-chromiumbookmarks
  unity-scope-calculator unity-scope-tomboy unity-scope-manpages unity-scope-virtualbox unity-scope-devhelp unity-scope-gdrive unity-scope-texdoc unity-scope-yelp
  unity-lens-files unity-lens-music unity-scope-firefoxbookmarks indicator-appmenu indicator-messages indicator-printers indicator-session session-shortcuts
  cups-pk-helper ubuntu-system-service unity-control-center-faces intel-gpu-tools xfonts-scalable xserver-xorg-video-all | xorg-driver-video gnome-user-guide

Some of those might get pulled in through other dependencies. --no-install-recommends might be a good start...
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2e6613d3f648027c796e4b73c60ef477daaf55f4 contains a try push with --no-install-recommends. Images are about half as large.
we should run some tests on here as well!

I suspect we can reduce the image size more with some fine tuning, it would be nice to have something in the 1.5GB range.
(In reply to Henrik Skupin (:whimboo) from comment #6)
> Sounds great! But why produced the last task a docker image of 2.9GB in size?

Maybe only one of the images was ~50% smaller. I was a bit lazy with my claim.
The question is was that a broken image, or specifically is different. Maybe we should do a binary diff if possible?
here is the list of failures that are perma fail in the try push:
* caps/tests/mochitest/test_bug995943.xul (debug c1)
* layout/base/tests/chrome/test_bug495648.xul (debug c3)
* toolkit/content/tests/chrome/test_arrowpanel.xul (opt/debug c3)
* toolkit/content/tests/widgets/test_videocontrols.html (opt/debug m10)
* /build/tests/reftest/tests/layout/reftests/xul/tree-row-outline-1-ref.xul (opt/debug r8/ru8)
* uriloader/exthandler/tests/unit/test_handlerService.js (debug X10)

I am not sure if these were seen on the base revision.
the xpcshell issue (uriloader/exthandler/tests/unit/test_handlerService.js), seems to not find the appdata directory, nor the ProfD and many other directories.
getting a loaner, I am having trouble reproducing problems.  I pull up a GUI shell since the cmd shell didn't let me run tests.  Here I cannot find a terminal window- the app launcher has no data in it for either term or xterm.  In addition there is no clock or other information on the application bar.  While looking at the app launcher, I see 'Error: HUD service not connected'

I have looked into a few of these failures and nothing specific is standing out.
Attachment #8782942 - Flags: review?(jmaher)
The zstd compression of Docker images that just landed decreases the transferred size of images to 30-50% of original for most images. This shaves several seconds off image upload and download for large images.

There are other improvements to image generation and download linked from bug 1316183 that should give significant speedups as well.
Depends on: 1316183
I think we're happy with this -- they are no longer 4GB :)
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Product: Taskcluster → Taskcluster Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: