Create a new desktop-test-m1 workerType that runs on m1.mediums

RESOLVED FIXED

Status

Taskcluster
Integration
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: dustin, Assigned: dustin)

Tracking

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(2 attachments)

(Assignee)

Description

2 years ago
m3.medium apparently causes test timeouts, even under Buildbot: bug 945981

Updated

2 years ago
Blocks: 1171033
(Assignee)

Comment 1

2 years ago
OK, that successfully ran a task:

  https://tools.taskcluster.net/task-inspector/#FI28ABO2SlSB51Jun2af3A/0

Armen, do you want to switch jobs over to that workerType, or should I?  On this bug, or another?
Flags: needinfo?(armenzg)

Comment 2

2 years ago
This bug should be sufficient.
If you could while I'm out tomorrow that would be great but I'm happy to give it a shot since I have the flow of all patches.
Let me know what works best for you.

This is as simple as changing to the worker type, right?
 "workerType": "desktop-test-m1",
Flags: needinfo?(armenzg)
(Assignee)

Comment 3

2 years ago
Yes, it's that simple.  We can do it for all tasks, or (by changing other files) only for certain suites.  Can you point me to the patches I should layer on top of?
(Assignee)

Comment 4

2 years ago
aiui this ended up mixed into a try push on another bug, so I'll consider this one closed.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Comment 5

2 years ago
The try jobs are failing, and docker is failing to kill unity-2d-shell.  We end up with a process list like

> root       831  0.8  0.4 680424 15664 ?        Ssl  14:14   2:39 /usr/bin/docker -d -g /mnt/var/lib/docker -s aufs
> root      1718  0.0  0.1 124484  5288 ?        Sl   14:15   0:00  \_ docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32768 -container-ip 172.17.0.1 -container-port 60023
> root      1724  0.1  0.1  39320  4216 ?        Ssl  14:15   0:19  \_ /livelog
> root      2053  0.0  0.0      0     0 ?        Ss   14:18   0:00  \_ [sudo]
> ubuntu    2223 89.8  0.0      0     0 ?        R    14:18 270:10      \_ [unity-2d-panel]

and that unity process ignores all signals but still seems to be getting CPU time.  We're debugging now.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 6

2 years ago
Working conjecture is that this is a bug between the PV filesystem drivers and the aufs driver, as aufs seems to be involved in most of the hangs here.

I'm rebuilding the AMI with a newer kernel (3.19.0-43, the latest in the 3.x series) to see if that helps.
(Assignee)

Comment 7

2 years ago
I just deployed the newer kernel to desktop-test-m1.
(Assignee)

Comment 8

2 years ago
Jobs that had failed:
https://tools.taskcluster.net/task-inspector/#I9ExiEGkRLCCp_LukbE0Wg/
https://tools.taskcluster.net/task-inspector/#Ia2bqXMZSj6B4dvTrkk9dQ/
https://tools.taskcluster.net/task-inspector/#NYdTDGRLQ5Cwi7GGPFlVzQ/

Re-runs:
https://tools.taskcluster.net/task-inspector/#A-q92kbLSjuwumEs7uoyrA/
https://tools.taskcluster.net/task-inspector/#H7EQcFnGTj-kBOvdCOnXtQ/
https://tools.taskcluster.net/task-inspector/#Am4cCDpfTOCRZRt7RR0Ykw/
(Assignee)

Comment 9

2 years ago
The instances aren't starting tasks because they don't have any video devices.  Yet:

    pv-builder: Setting up v4l2loopback-dkms (0.8.0-1) ...
    pv-builder: Loading new v4l2loopback-0.8.0 DKMS files...
    pv-builder: First Installation: checking all kernels...
    pv-builder: Building for 3.13.0-74-generic and 3.19.0-43-generic
    pv-builder: Building initial module for 3.13.0-74-generic
    pv-builder: Done.
    pv-builder: 
    pv-builder: v4l2loopback:
    pv-builder: Running module version sanity check.
    pv-builder: - Original module
    pv-builder: - No original module exists within this kernel
    pv-builder: - Installation
    pv-builder: - Installing to /lib/modules/3.13.0-74-generic/updates/dkms/
    pv-builder:
    pv-builder: depmod.....
    pv-builder:
    pv-builder: DKMS: install completed.
    pv-builder: Module build for the currently running kernel was skipped since the
    pv-builder: kernel source for this kernel does not seem to be installed.
    pv-builder: Setting up v4l2loopback-utils (0.8.0-1) ...

so it looks like it *tried* to build it for 3.19.0-43-generic, but never actually did?  Because there are no linux-headers for that version.  However, once those are installed,

DKMS make.log for v4l2loopback-0.8.0 for kernel 3.19.0-43-generic (x86_64)
Fri Jan  8 23:44:49 UTC 2016
Building v4l2-loopback driver...
make -C /lib/modules/3.19.0-43-generic/build M=/var/lib/dkms/v4l2loopback/0.8.0/build modules
make[1]: Entering directory `/usr/src/linux-headers-3.19.0-43-generic'
  CC [M]  /var/lib/dkms/v4l2loopback/0.8.0/build/v4l2loopback.o
/var/lib/dkms/v4l2loopback/0.8.0/build/v4l2loopback.c: In function ‘attr_store_maxopeners’:
/var/lib/dkms/v4l2loopback/0.8.0/build/v4l2loopback.c:486:2: error: implicit declaration of function ‘strict_strtoul’ [-Werror=implicit-function-declaration]
  if (strict_strtoul(buf, 0, &curr))
  ^
cc1: some warnings being treated as errors
make[2]: *** [/var/lib/dkms/v4l2loopback/0.8.0/build/v4l2loopback.o] Error 1
make[1]: *** [_module_/var/lib/dkms/v4l2loopback/0.8.0/build] Error 2
make[1]: Leaving directory `/usr/src/linux-headers-3.19.0-43-generic'
make: *** [v4l2loopback.ko] Error 2

https://bugs.launchpad.net/ubuntu/+source/v4l2loopback/+bug/1455791
(Assignee)

Comment 10

2 years ago
Ah, newer versions of v4l2loopback work with newer kernels, but aren't packaged by Ubuntu.  Yay.  So we'll get v4l2loopback from Git and build it with DKMS.  Fun!

I've also rolled back the max capacity of this workerType to 2, since otherwise the provisioner was spawning lots of instances.
(Assignee)

Comment 11

2 years ago
    pv-builder: Some packages could not be installed. This may mean that you have
    pv-builder: requested an impossible situation or if you are using the unstable
    pv-builder: distribution that some required packages have not yet been created
    pv-builder: or been moved out of Incoming.
    pv-builder: The following information may help to resolve the situation:
    pv-builder:
    pv-builder: The following packages have unmet dependencies:
    pv-builder: dkms : Depends: gcc but it is not installable
    pv-builder: Depends: make but it is not installable or
    pv-builder: E: Unable to correct problems, you have held broken packages.
    pv-builder: build-essential but it is not installable or
    pv-builder: dpkg-dev but it is not going to be installed
    pv-builder: Recommends: fakeroot but it is not installable
    pv-builder: linux-image-extra-3.19.0-43-generic : Depends: crda (>= 1.1.1-1ubuntu2) but it is not installable or
    pv-builder: wireless-crda but it is not installable

so those will need to be solved.  Why is nothing ever even remotely simple???!
(Assignee)

Comment 12

2 years ago
KERNEL_VER=3.19.0-43-generic
sudo apt-get install -y \
    linux-image-${KERNEL_VER} \
    linux-headers-${KERNEL_VER} \
    linux-image-extra-${KERNEL_VER} \
    dkms

works by hand in a 14.04 docker image.  I wonder what the difference is.  Apt's error messages are inscrutible!  "..but it is not going to be installed" -- doesn't it seem like apt could install that, then, like it does all the other dependencies?
(Assignee)

Comment 13

2 years ago
Here's the whole message (not much more):

    pv-builder: + sudo apt-get update -y
    pv-builder: Get:1 http://security.ubuntu.com trusty-security InRelease [64.4 kB]
    pv-builder: Ign http://archive.ubuntu.com trusty InRelease
    pv-builder: Get:2 http://archive.ubuntu.com trusty-updates InRelease [64.4 kB]
    pv-builder: Hit http://archive.ubuntu.com trusty Release.gpg
    pv-builder: Hit http://archive.ubuntu.com trusty Release
    pv-builder: Ign http://archive.ubuntu.com trusty Release
    pv-builder: Get:3 http://security.ubuntu.com trusty-security/main amd64 Packages [403 kB]
    pv-builder: Get:4 http://archive.ubuntu.com trusty-updates/main amd64 Packages [683 kB]
    pv-builder: Get:5 http://security.ubuntu.com trusty-security/restricted amd64 Packages [13.0 kB]
    pv-builder: Get:6 http://security.ubuntu.com trusty-security/universe amd64 Packages [122 kB]
    pv-builder: Get:7 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [15.9 kB]
    pv-builder: Get:8 http://security.ubuntu.com trusty-security/multiverse amd64 Packages [4798 B]
    pv-builder: Get:9 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [333 kB]
    pv-builder: Get:10 http://security.ubuntu.com trusty-security/main Translation-en [220 kB]
    pv-builder: Get:11 http://archive.ubuntu.com trusty-updates/multiverse amd64 Packages [13.0 kB]
    pv-builder: Get:12 http://security.ubuntu.com trusty-security/multiverse Translation-en [2437 B]
    pv-builder: Get:13 https://get.docker.io docker InRelease
    pv-builder: Ign https://get.docker.io docker InRelease
    pv-builder: Get:14 http://archive.ubuntu.com trusty-updates/main Translation-en [342 kB]
    pv-builder: Get:15 http://security.ubuntu.com trusty-security/restricted Translation-en [3206 B]
    pv-builder: Get:16 http://security.ubuntu.com trusty-security/universe Translation-en [71.6 kB]
    pv-builder: Get:17 http://archive.ubuntu.com trusty-updates/multiverse Translation-en [6832 B]
    pv-builder: Get:18 http://archive.ubuntu.com trusty-updates/restricted Translation-en [3699 B]
    pv-builder: Get:19 http://archive.ubuntu.com trusty-updates/universe Translation-en [175 kB]
    pv-builder: Get:20 http://archive.ubuntu.com trusty/main amd64 Packages [1350 kB]
    pv-builder: Get:21 http://archive.ubuntu.com trusty/restricted amd64 Packages [13.0 kB]
    pv-builder: Get:22 http://archive.ubuntu.com trusty/universe amd64 Packages [5859 kB]
    pv-builder: Get:23 https://get.docker.io docker Release
    pv-builder: Get:24 http://archive.ubuntu.com trusty/multiverse amd64 Packages [132 kB]
    pv-builder: Get:25 http://archive.ubuntu.com trusty/main Translation-en [762 kB]
    pv-builder: Get:26 https://get.docker.io docker/main amd64 Packages
    pv-builder: Get:27 http://archive.ubuntu.com trusty/multiverse Translation-en [102 kB]
    pv-builder: Get:28 http://archive.ubuntu.com trusty/restricted Translation-en [3457 B]
    pv-builder: Get:29 http://archive.ubuntu.com trusty/universe Translation-en [4089 kB]
    pv-builder: Ign https://get.docker.io docker/main Translation-en
    pv-builder: Fetched 14.9 MB in 10s (1458 kB/s)
    pv-builder: Reading package lists...
    pv-builder: W: GPG error: http://archive.ubuntu.com trusty Release: Unknown error executing gpgv
    pv-builder:
    pv-builder: ## upgrade the kernel, along with extra (which adds AUFS support)
    pv-builder: KERNEL_VER=3.19.0-43-generic
    pv-builder: + KERNEL_VER=3.19.0-43-generic
    pv-builder: sudo apt-get install -y \
    pv-builder: linux-image-${KERNEL_VER} \
    pv-builder: linux-headers-${KERNEL_VER} \
    pv-builder: linux-image-extra-${KERNEL_VER} \
    pv-builder: dkms
    pv-builder: + sudo apt-get install -y linux-image-3.19.0-43-generic linux-headers-3.19.0-43-generic linux-image-extra-3.19.0-43-generic dkms
    pv-builder: Reading package lists...
    pv-builder: Building dependency tree...
    pv-builder: Reading state information...
    pv-builder: Some packages could not be installed. This may mean that you have
    pv-builder: requested an impossible situation or if you are using the unstable
    pv-builder: distribution that some required packages have not yet been created
    pv-builder: or been moved out of Incoming.
    pv-builder: The following information may help to resolve the situation:
    pv-builder:
    pv-builder: The following packages have unmet dependencies:
    pv-builder: dkms : Depends: gcc but it is not installable
    pv-builder: E: Unable to correct problems, you have held broken packages.
    pv-builder: Depends: make but it is not installable or
    pv-builder: build-essential but it is not installable or
    pv-builder: dpkg-dev but it is not going to be installed
    pv-builder: Recommends: fakeroot but it is not installable
    pv-builder: linux-image-extra-3.19.0-43-generic : Depends: crda (>= 1.1.1-1ubuntu2) but it is not installable or
    pv-builder: wireless-crda but it is not installable

I wonder if the gpg error is related?  That, or possibly I don't have the same repos (universe?) enabled in docker as in the base Ubuntu AMI.
(Assignee)

Comment 14

2 years ago
Catting /etc/apt/sources.list shows, *only* on the pv builder, 

    pv-builder: deb http://archive.ubuntu.com/ubuntu trusty main restricted universe multiverse
    pv-builder: deb http://archive.ubuntu.com/ubuntu trusty-updates main restricted universe multiverse
    pv-builder: deb http://security.ubuntu.com/ubuntu trusty-security main restricted universe multiverse

I'm not sure where that's coming from -- starting up the same AMI from the launch wizard gives the expected sources.list with more repos and linking to the ec2 mirrors.  Rather than try to solve this (packer bug? cloud-tools bug? dunno), I'll just write out the sources.list I want.
(Assignee)

Comment 15

2 years ago
==> Builds finished. The artifacts of successful builds are:
--> hvm-builder: AMIs were created:

us-west-2: ami-2cb6ad4d
--> hvm-builder: 'aws' provider box: deploy/packer/base.box
--> pv-builder: AMIs were created:

us-west-2: ami-feb5ae9f
--> pv-builder: 'aws' provider box: deploy/packer/base.box

In the interim, Greg found that 'apt-get clean' will fix the sources.list, too.  Go figure.
(Assignee)

Comment 16

2 years ago
us-east-1: ami-721a4118
us-west-1: ami-4a8ffa2a
us-west-2: ami-9eaab1ff
--> hvm-builder: 'aws' provider box: app.box
--> pv-builder: AMIs were created:

us-east-1: ami-b91a41d3
us-west-1: ami-d489fcb4
us-west-2: ami-71b7ac10
--> pv-builder: 'aws' provider box: app.box
(Assignee)

Comment 17

2 years ago
That version loaded the wrong kernel in the PV AMI, because the debian install scripts edit /boot/grub/grub.cfg, not the /boot/grub/menu.lst that PV-GRUB loads.  So!  Trying again..

==> Builds finished. The artifacts of successful builds are:
--> hvm-builder: AMIs were created:

us-west-2: ami-7aabb01b
--> hvm-builder: 'aws' provider box: deploy/packer/base.box
--> pv-builder: AMIs were created:

us-west-2: ami-86a9b2e7
--> pv-builder: 'aws' provider box: deploy/packer/base.box

==> Builds finished. The artifacts of successful builds are:
--> hvm-builder: AMIs were created:

us-east-1: ami-41114a2b
us-west-1: ami-308aff50
us-west-2: ami-96afb4f7
--> hvm-builder: 'aws' provider box: app.box
--> pv-builder: AMIs were created:

us-east-1: ami-e8124982
us-west-1: ami-2e8ffa4e
us-west-2: ami-93afb4f2
--> pv-builder: 'aws' provider box: app.box

And a task using the new AMI:

https://tools.taskcluster.net/task-inspector/#L5U1zSnETemjBC3HN3Lu-Q/
(Assignee)

Comment 18

2 years ago
That had a typo ("menu.lsg")

==> Builds finished. The artifacts of successful builds are:
--> hvm-builder: AMIs were created:

us-west-2: ami-eaafb48b
--> hvm-builder: 'aws' provider box: deploy/packer/base.box
--> pv-builder: AMIs were created:

us-west-2: ami-44a9b225
--> pv-builder: 'aws' provider box: deploy/packer/base.box

==> Builds finished. The artifacts of successful builds are:
--> pv-builder: AMIs were created:

us-east-1: ami-fd174c97
us-west-1: ami-7b8aff1b
us-west-2: ami-2ca9b24d
--> pv-builder: 'aws' provider box: app.box
--> hvm-builder: AMIs were created:

us-east-1: ami-51154e3b
us-west-1: ami-c88bfea8
us-west-2: ami-33aeb552
--> hvm-builder: 'aws' provider box: app.box

https://tools.taskcluster.net/task-inspector/#S1bJ6Pi_SIigRsmKJnpW7A/
(Assignee)

Comment 19

2 years ago
Again:

root      1568 23.8  0.7 664040 29904 ?        Ssl  21:32   2:03 /usr/bin/docker -d -g /mnt/var/lib/docker -s aufs
root      1958  0.0  0.3 124492 12208 ?        Sl   21:33   0:00  \_ docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32768 -container-ip 172.17.0.1 -container-port 60023
root      1964  0.1  0.1  40416  7388 ?        Ssl  21:33   0:00  \_ /livelog
root      2383  0.0  0.0      0     0 ?        Ss   21:36   0:00  \_ [sudo]
ubuntu    2541 24.8  0.0      0     0 ?        R    21:37   1:02      \_ [gnome-settings-]
ubuntu    2563 25.0  0.0      0     0 ?        R    21:37   1:02      \_ [unity-2d-shell]

so, if anything, worse now, since gnome settings daemon is appearing.

However, looking at the docker logs from one of those runs:

21:39:07     INFO -  dir: devtools/client/aboutdebugging/test
21:39:09     INFO -  ImportError: No module named pygtk
21:39:09     INFO -  ImportError: No module named pygtk
21:39:09     INFO -  Setting pipeline to PAUSED ...
21:39:09     INFO -  ERROR: Pipeline doesn't want to pause.
21:39:09     INFO -  ERROR: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0: Failed to query attributes of input 0 in device /dev/video0
21:39:09     INFO -  Additional debug info:
21:39:09     INFO -  v4l2_calls.c(142): gst_v4l2_fill_lists (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0:
21:39:09     INFO -  Failed to get 0 in input enumeration for /dev/video0. (25 - Inappropriate ioctl for device)
21:39:09     INFO -  Setting pipeline to NULL ...
21:39:09     INFO -  Freeing pipeline ...
21:39:09     INFO -  Traceback (most recent call last):
21:39:09     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2623, in <module>
21:39:09     INFO -      sys.exit(cli())
21:39:09     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2620, in cli
21:39:09     INFO -      return run_test_harness(options)
21:39:09     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2597, in run_test_harness
21:39:09     INFO -      result = runner.runTests(options)
21:39:09     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2157, in runTests
21:39:09     INFO -      result = self.runMochitests(options, tests_in_dir, onLaunch)
21:39:09     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2106, in runMochitests
21:39:09     INFO -      result = self.doTests(options, onLaunch, testsToRun)
21:39:09     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2203, in doTests
21:39:09     INFO -      devices = findTestMediaDevices(self.log)
21:39:09     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 1448, in findTestMediaDevices
21:39:09     INFO -      'v4l2sink', 'device=%s' % device])
21:39:09     INFO -    File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
21:39:09     INFO -      raise CalledProcessError(retcode, cmd)
21:39:09     INFO -  subprocess.CalledProcessError: Command '['/usr/bin/gst-launch-0.10', 'videotestsrc', 'pattern=green', 'num-buffers=1', '!', 'v4l2sink', 'device=/dev/video0']' returned non-zero exit status 255

So the bad news is, the kernel upgrade didn't help.  The good news is, we might have a lead on why?

On an existing (m3.large) desktop-test instance:

ubuntu@ip-172-31-2-12:~$ docker run --device /dev/video0:/dev/video0 -ti --rm taskcluster/desktop-test:0.5.3
root@taskcluster-worker:~# gst-launch-0.10 'videotestsrc' pattern=green num-buffers=1 \! v4l2sink device=/dev/video0
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 33336501 ns.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

ubuntu@ip-172-31-2-12:~$ dmesg | grep v4l2
[   11.761394] v4l2loopback: module verification failed: signature and/or  required key missing - tainting kernel
[   11.761666] v4l2loopback: number of devices is limited to: 8
[   11.762150] v4l2loopback driver version 0.8.0 loaded

But on a new (m1.medium) desktop-test-m1 instance:

ubuntu@ip-172-31-10-241:~$ docker run --device /dev/video0:/dev/video0 -ti --rm taskcluster/desktop-test:0.5.3
root@taskcluster-worker:~# gst-launch-0.10 'videotestsrc' pattern=green num-buffers=1 \! v4l2sink device=/dev/video0
Setting pipeline to PAUSED ...
ERROR: Pipeline doesn't want to pause.
ERROR: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0: Failed to query attributes of input 0 in device /dev/video0
Additional debug info:
v4l2_calls.c(142): gst_v4l2_fill_lists (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0:
Failed to get 0 in input enumeration for /dev/video0. (25 - Inappropriate ioctl for device)
Setting pipeline to NULL ...
Freeing pipeline ...

ubuntu@ip-172-31-10-241:~$ dmesg | grep v4l2
[   30.997074] v4l2loopback: module verification failed: signature and/or  required key missing - tainting kernel
[   30.997442] v4l2loopback: number of devices is limited to: 8
[   30.999821] v4l2loopback driver version 0.9.1 loaded

probably unrelated, but potentially problematic, dmesg has lots of:

[  986.803165] aufs au_opts_verify:1612:docker[2026]: dirperm1 breaks the protection by the permission bits on the lower branch

Out of a lack of other ideas, I'll try again with the 4.2.0-23 kernel (the latest available for trusty).
(Assignee)

Comment 20

2 years ago
4.2.0-23-generic:

==> Builds finished. The artifacts of successful builds are:
--> hvm-builder: AMIs were created:

us-west-2: ami-ba5349db
--> hvm-builder: 'aws' provider box: deploy/packer/base.box
--> pv-builder: AMIs were created:

us-west-2: ami-225c4643
--> pv-builder: 'aws' provider box: deploy/packer/base.box

==> Builds finished. The artifacts of successful builds are:
--> hvm-builder: AMIs were created:

us-east-1: ami-6c0d5606
us-west-1: ami-70f48110
us-west-2: ami-dd5842bc
--> hvm-builder: 'aws' provider box: app.box
--> pv-builder: AMIs were created:

us-east-1: ami-110d567b
us-west-1: ami-2bf5804b
us-west-2: ami-f7504a96
--> pv-builder: 'aws' provider box: app.box

https://tools.taskcluster.net/task-inspector/#MltRSffdSvyUFEGYg8Wxyg/

Greg has suggested trying the HVM AMIs, too, to see if the errors are the same.
(Assignee)

Comment 21

2 years ago
Same result on PV with 4.2.0-23-generic:

00:48:22     INFO -  Failed to get 0 in input enumeration for /dev/video0. (25 - Inappropriate ioctl for device)

root      1567 19.3  0.7 664040 30080 ?        Ssl  22:57   1:50 /usr/bin/docker -d -g /mnt/var/lib/docker -s aufs
root      1960  0.0  0.3 124492 12020 ?        Sl   22:58   0:00  \_ docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32768 -container-ip 172.17.0.1 -container-port 60023
root      1966  0.1  0.1 105952  6904 ?        Ssl  22:58   0:00  \_ /livelog
root      2375  0.0  0.0      0     0 ?        Ss   23:00   0:00  \_ [sudo]
ubuntu    2556 33.5  0.0      0     0 ?        R    23:01   1:49      \_ [unity-2d-panel]
ubuntu    2557 33.8  0.0      0     0 ?        R    23:01   1:50      \_ [unity-2d-shell]

On HVM with 4.2.0-23-generic:

23:23:25     INFO -  Failed to get 0 in input enumeration for /dev/video0. (25 - Inappropriate ioctl for device)

however, the docker container exits correctly.  So I think upgrading the kernel has made v4l2loopback non-functional, without fixing the apparently-very-PV-related docker failures.
(Assignee)

Comment 22

2 years ago
From meeting notes:

 [dustin] docker, instance types, paravirtualization, and tests
    maybe try on some other instance types?
      help from docker? kernel issues (wedged, un-signalable unity processes)
      #docker on freenode
      try docker update - greg can help dustin
      docker support plans - check in #docker on mozilla to see if someone has that
        https://www.docker.com/pricing
    help from engineering to work on test failures on newer instance types (dougt)
      question of whether to do this work with buildbot on m3's or docker on m3's
      docker issues to land end of week
      remaining non-intermittent issues:
        still have issues with reftests - better with gnome-session, worse with unity
        maybe 1-2 weeks of image issues
      [todo] give selena a list of the tests that we want help with
(Assignee)

Comment 23

2 years ago
I created workerTypes desktop-test-c3 and desktop-test-c3-1cpu with c3.xlarge instances and for the latter isolatedContainers.  Armen's running try pushes on those now.

Building an AMI with Docker-1.9.1:

base:
--> hvm-builder: AMIs were created:
us-west-2: ami-006a7061
--> pv-builder: AMIs were created:
us-west-2: ami-966973f7

app:
--> hvm-builder: AMIs were created:
us-east-1: ami-4cbfe426
us-west-1: ami-cde396ad
us-west-2: ami-d26b71b3
--> pv-builder: AMIs were created:
us-east-1: ami-7fb8e315
us-west-1: ami-3ee2975e
us-west-2: ami-e96a7088

firing those up in AWS manager: https://tools.taskcluster.net/task-inspector/#GBElFJFrSya0d1Xs7DopoQ/
(Assignee)

Comment 24

2 years ago
ubuntu@ip-172-31-0-19:~$ docker --version
Docker version 1.9.1, build a34a1d5
ubuntu@ip-172-31-0-19:~$ uname -r
4.2.0-23-generic

...

20:41:11     INFO -  ImportError: No module named pygtk
20:41:12     INFO -  Setting pipeline to PAUSED ...
20:41:12     INFO -  ERROR: Pipeline doesn't want to pause.
20:41:12     INFO -  ERROR: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0: Failed to query attributes of input 0 in device /dev/video0
20:41:12     INFO -  Additional debug info:
20:41:12     INFO -  v4l2_calls.c(142): gst_v4l2_fill_lists (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0:
20:41:12     INFO -  Failed to get 0 in input enumeration for /dev/video0. (25 - Inappropriate ioctl for device)
20:41:12     INFO -  Setting pipeline to NULL ...
20:41:12     INFO -  Freeing pipeline ...
20:41:12     INFO -  Traceback (most recent call last):
20:41:12     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2623, in <module>
20:41:12     INFO -      sys.exit(cli())
20:41:12     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2620, in cli
20:41:12     INFO -      return run_test_harness(options)
20:41:12     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2597, in run_test_harness
20:41:12     INFO -      result = runner.runTests(options)
20:41:12     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2157, in runTests
20:41:12     INFO -      result = self.runMochitests(options, tests_in_dir, onLaunch)
20:41:12     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2106, in runMochitests
20:41:12     INFO -      result = self.doTests(options, onLaunch, testsToRun)
20:41:12     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2203, in doTests
20:41:12     INFO -      devices = findTestMediaDevices(self.log)
20:41:12     INFO -    File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 1448, in findTestMediaDevices
20:41:12     INFO -      'v4l2sink', 'device=%s' % device])
20:41:12     INFO -    File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
20:41:12     INFO -      raise CalledProcessError(retcode, cmd)
20:41:12     INFO -  subprocess.CalledProcessError: Command '['/usr/bin/gst-launch-0.10', 'videotestsrc', 'pattern=green', 'num-buffers=1', '!', 'v4l2sink', 'device=/dev/video0']' returned non-zero exit status 255
20:41:12    ERROR - Return code: 1
20:41:12    ERROR - No tests run or test summary not found
20:41:12     INFO - TinderboxPrint: mochitest-mochitest-devtools-chrome<br/><em class="testfail">T-FAIL</em>
20:41:12  WARNING - # TBPL FAILURE #
20:41:12  WARNING - setting return code to 2
20:41:12  WARNING - The mochitest suite: mochitest-devtools-chrome ran with return status: FAILURE
20:41:12     INFO - Running post-action listener: _package_coverage_data
20:41:12     INFO - Running post-action listener: _resource_record_post_action
20:41:12     INFO - Running post-run listener: _resource_record_post_run
20:41:13     INFO - Total resource usage - Wall time: 28s; CPU: 100.0%; Read bytes: 52420608; Write bytes: 84410368; Read time: 5100; Write time: 42324
20:41:13     INFO - install - Wall time: 25s; CPU: 100.0%; Read bytes: 155648; Write bytes: 65605632; Read time: 16; Write time: 20116
20:41:13     INFO - run-tests - Wall time: 3s; CPU: 99.0%; Read bytes: 52183040; Write bytes: 9580544; Read time: 5044; Write time: 2296
20:41:13     INFO - Running post-run listener: _upload_blobber_files
20:41:13  WARNING - Blob upload gear skipped. Missing cmdline options.
20:41:13     INFO - Copying logs to upload dir...
20:41:13     INFO - mkdir: /home/worker/workspace/build/upload/logs
20:41:13  WARNING - returning nonzero exit status 2
cleanup
+ cleanup
+ cp /home/worker/.xsession-errors /home/worker/artifacts/public
+ '[' -n 36 ']'
+ '[' false == false ']'
+ kill 36

...

root       942 31.3  2.1 758752 82864 ?        Ssl  20:34   2:27 /usr/bin/docker daemon -g /mnt/var/lib/docker -s aufs
root      1848  0.0  0.6 152732 24168 ?        Sl   20:35   0:00  \_ docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32768 -container-ip 172.17.0.2 -container-port 60023
root      1859  0.2  0.1  56808  7232 ?        Ssl  20:35   0:00  \_ /livelog
root      2282  0.0  0.0      0     0 ?        Ss   20:39   0:00  \_ [sudo]
ubuntu    2431 17.5  0.0      0     0 ?        R    20:39   0:29      \_ [unity-2d-panel]
ubuntu    2432 18.1  0.0      0     0 ?        R    20:39   0:30      \_ [unity-2d-shell]

in other words, 1.9.1 -> no help.

I want to loop back and see if the ioctl issues occur on a plain-vanilla 3.13 PV instance, too -- perhaps that is triggering a bug in v4l2 which is causing the hangs.
(Assignee)

Comment 25

2 years ago
Rebuilding that kernel-3.13, docker-1.6.1 PV image since I'd lost the AMI IDs:

base:
--> hvm-builder: AMIs were created:
us-west-2: ami-8c6278ed
--> pv-builder: AMIs were created:
us-west-2: ami-2f6f754e

app:
--> pv-builder: AMIs were created:
us-east-1: ami-49abf023
us-west-1: ami-5eec993e
us-west-2: ami-36607a57
--> hvm-builder: AMIs were created:
us-east-1: ami-37abf05d
us-west-1: ami-80ed98e0
us-west-2: ami-32637953

https://tools.taskcluster.net/task-inspector/#LLw5sIEeRx2OhM8k2uoVJg/0

which .. succeeded.  Whut?

I've dialed the desktop-test-m1 workerType up to a max of 50 instances.  Armen, can you do a try push to exercise it and post the link here?
Flags: needinfo?(armenzg)
(Assignee)

Comment 26

2 years ago
To be clear, the job succeeded, the processes exited, all good.

ubuntu@ip-172-31-11-201:~$ uname -r
3.13.0-48-generic
ubuntu@ip-172-31-11-201:~$ docker --version
Docker version 1.6.1, build 97cd073
ubuntu@ip-172-31-11-201:~$ curl 169.254.169.254/latest/meta-data/instance-type ; echo
m1.medium

/me doesn't believe his eyes, tries another task re-run..
https://tools.taskcluster.net/task-inspector/#ZVjs4LYkQKCj0ZtVzFqxOQ/
(Assignee)

Comment 27

2 years ago
That one also succeeded.  What am I missing?

Comment 28

2 years ago
Pushed: https://treeherder.mozilla.org/#/jobs?repo=try&revision=287c21e9f37e
Flags: needinfo?(armenzg)

Comment 29

2 years ago
This instead: https://treeherder.mozilla.org/#/jobs?repo=try&revision=df0f0e8baa5f
(Assignee)

Comment 30

2 years ago
The AMIs originally used to test this configuration, back on Jan 7, were

--> pv-builder: AMIs were created:
us-east-1: ami-82c09ee8
us-west-1: ami-7f681d1f
us-west-2: ami-bf6279de

So, in us-west-2, using - for Jan 7 (bad) and + for comment 25 (good), we have

- ami-bf6279de
+ ami-36607a57

and those were built from https://github.com/taskcluster/docker-worker/commit/7ce9448ababc70541d71d39c6b6d4fdd0958a977 (packer has a parent sha1, but the changes in that merge were uncommitted at that time, I think).  The differences between 7ce9448ababc70541d71d39c6b6d4fdd0958a977 and the packer config for the AMI IDs in comment 25 are:

 * different source AMI:
   - ami-47465826      (ubuntu/images/ebs/ubuntu-trusty-14.04-amd64-server-20151218)
   + ami-7f89a64f      (ubuntu/images/ebs/ubuntu-trusty-14.04-amd64-server-20150325)

 * + includes a hard-coded sources.list, since the PV image includes a degenerate one
 * + always runs apt-get update in packages.sh

Comparing instances spun up with those AMIs,

 ubuntu$ uname -r
-3.13.0-74-generic
+3.13.0-48-generic

ubuntu@$ cat /etc/cloud/build.info 
 build_name: server
-serial: 20151218
+serial: 20150325

 ubuntu$ dpkg -l
-ii  accountsservice                     0.6.35-0ubuntu7.2                amd64 
+ii  accountsservice                     0.6.35-0ubuntu7.1                amd64 
-ii  apparmor                            2.8.95~2430-0ubuntu5.3           amd64 
-ii  apport                              2.14.1-0ubuntu3.19               all   
+ii  apparmor                            2.8.95~2430-0ubuntu5.1           amd64 
+ii  apport                              2.14.1-0ubuntu3.8                all   
-ii  apt                                 1.0.1ubuntu2.10                  amd64 
-ii  apt-transport-https                 1.0.1ubuntu2.10                  amd64 
-ii  apt-utils                           1.0.1ubuntu2.10                  amd64 
+ii  apt                                 1.0.1ubuntu2.6                   amd64 
+ii  apt-transport-https                 1.0.1ubuntu2.6                   amd64 
+ii  apt-utils                           1.0.1ubuntu2.6                   amd64 
-ii  base-files                          7.2ubuntu5.3                     amd64 
+ii  base-files                          7.2ubuntu5.2                     amd64 
-ii  bash-completion                     1:2.1-4ubuntu0.1                 all   
+ii  bash-completion                     1:2.1-4                          all   
-ii  bind9-host                          1:9.9.5.dfsg-3ubuntu0.6          amd64 
+ii  bind9-host                          1:9.9.5.dfsg-3ubuntu0.2          amd64 
-ii  bsdutils                            1:2.20.1-5.1ubuntu20.7           amd64 
+ii  bsdutils                            1:2.20.1-5.1ubuntu20.4           amd64 
-ii  cloud-init                          0.7.5-0ubuntu1.15                all   
+ii  cloud-init                          0.7.5-0ubuntu1.5                 all   
-ii  coreutils                           8.21-1ubuntu5.3                  amd64 
+ii  coreutils                           8.21-1ubuntu5.1                  amd64 
-ii  dh-python                           1.20140128-1ubuntu8.2            all   
+ii  dh-python                           1.20140128-1ubuntu8              all   
-ii  dnsutils                            1:9.9.5.dfsg-3ubuntu0.6          amd64 
+ii  dnsutils                            1:9.9.5.dfsg-3ubuntu0.2          amd64 
-ii  dpkg                                1.17.5ubuntu5.5                  amd64 
+ii  dpkg                                1.17.5ubuntu5.3                  amd64 
-ii  e2fslibs:amd64                      1.42.9-3ubuntu1.3                amd64 
-ii  e2fsprogs                           1.42.9-3ubuntu1.3                amd64 
+ii  e2fslibs:amd64                      1.42.9-3ubuntu1.2                amd64 
+ii  e2fsprogs                           1.42.9-3ubuntu1.2                amd64 
-ii  fuse                                2.9.2-4ubuntu4.14.04.1           amd64 
+ii  fuse                                2.9.2-4ubuntu4                   amd64 
-ii  gdisk                               0.8.8-1ubuntu0.1                 amd64 
+ii  gdisk                               0.8.8-1build1                    amd64 
-ii  gnupg                               1.4.16-1ubuntu2.3                amd64 
-ii  gpgv                                1.4.16-1ubuntu2.3                amd64 
+ii  gnupg                               1.4.16-1ubuntu2.1                amd64 
+ii  gpgv                                1.4.16-1ubuntu2.1                amd64 
-ii  grub-common                         2.02~beta2-9ubuntu1.6            amd64 
+ii  grub-common                         2.02~beta2-9ubuntu1              amd64 
-ii  grub-legacy-ec2                     0.7.5-0ubuntu1.15                all   
-ii  grub-pc                             2.02~beta2-9ubuntu1.6            amd64 
-ii  grub-pc-bin                         2.02~beta2-9ubuntu1.6            amd64 
-ii  grub2-common                        2.02~beta2-9ubuntu1.6            amd64 
+ii  grub-legacy-ec2                     0.7.5-0ubuntu1.5                 all   
+ii  grub-pc                             2.02~beta2-9ubuntu1              amd64 
+ii  grub-pc-bin                         2.02~beta2-9ubuntu1              amd64 
+ii  grub2-common                        2.02~beta2-9ubuntu1              amd64 
-ii  initscripts                         2.88dsf-41ubuntu6.2              amd64 
+ii  initscripts                         2.88dsf-41ubuntu6.1              amd64 
-ii  iproute2                            3.12.0-2ubuntu1                  amd64 
+ii  iproute2                            3.12.0-2                         amd64 
-ii  irqbalance                          1.0.6-2ubuntu0.14.04.4           amd64 
-ii  isc-dhcp-client                     4.2.4-7ubuntu12.3                amd64 
-ii  isc-dhcp-common                     4.2.4-7ubuntu12.3                amd64 
+ii  irqbalance                          1.0.6-2ubuntu0.14.04.1           amd64 
+ii  isc-dhcp-client                     4.2.4-7ubuntu12                  amd64 
+ii  isc-dhcp-common                     4.2.4-7ubuntu12                  amd64 
-ii  krb5-locales                        1.12+dfsg-2ubuntu5.2             all   
+ii  krb5-locales                        1.12+dfsg-2ubuntu5.1             all   
-ii  libaccountsservice0:amd64           0.6.35-0ubuntu7.2                amd64 
+ii  libaccountsservice0:amd64           0.6.35-0ubuntu7.1                amd64 
-ii  libapparmor-perl                    2.8.95~2430-0ubuntu5.3           amd64 
-ii  libapparmor1:amd64                  2.8.95~2430-0ubuntu5.3           amd64 
-ii  libapt-inst1.5:amd64                1.0.1ubuntu2.10                  amd64 
+ii  libapparmor-perl                    2.8.95~2430-0ubuntu5.1           amd64 
+ii  libapparmor1:amd64                  2.8.95~2430-0ubuntu5.1           amd64 
+ii  libapt-inst1.5:amd64                1.0.1ubuntu2.6                   amd64 
-ii  libapt-pkg4.12:amd64                1.0.1ubuntu2.10                  amd64 
+ii  libapt-pkg4.12:amd64                1.0.1ubuntu2.6                   amd64 
-ii  libbind9-90                         1:9.9.5.dfsg-3ubuntu0.6          amd64 
-ii  libblkid1:amd64                     2.20.1-5.1ubuntu20.7             amd64 
+ii  libbind9-90                         1:9.9.5.dfsg-3ubuntu0.2          amd64 
+ii  libblkid1:amd64                     2.20.1-5.1ubuntu20.4             amd64 
-ii  libcgmanager0:amd64                 0.24-0ubuntu7.5                  amd64 
+ii  libcgmanager0:amd64                 0.24-0ubuntu7.3                  amd64 
-ii  libcomerr2:amd64                    1.42.9-3ubuntu1.3                amd64 
+ii  libcomerr2:amd64                    1.42.9-3ubuntu1.2                amd64 
-ii  libcurl3-gnutls:amd64               7.35.0-1ubuntu2.5                amd64 
+ii  libcurl3-gnutls:amd64               7.35.0-1ubuntu2.3                amd64 
-ii  libdns100                           1:9.9.5.dfsg-3ubuntu0.6          amd64 
+ii  libdns100                           1:9.9.5.dfsg-3ubuntu0.2          amd64 
-ii  libdrm2:amd64                       2.4.60-2~ubuntu14.04.1           amd64 
+ii  libdrm2:amd64                       2.4.56-1~ubuntu2                 amd64 
-ii  libexpat1:amd64                     2.1.0-4ubuntu1.1                 amd64 
+ii  libexpat1:amd64                     2.1.0-4ubuntu1                   amd64 
-ii  libffi6:amd64                       3.1~rc1+r3.0.13-12ubuntu0.1      amd64 
+ii  libffi6:amd64                       3.1~rc1+r3.0.13-12               amd64 
-ii  libfreetype6:amd64                  2.5.2-1ubuntu2.5                 amd64 
+ii  libfreetype6:amd64                  2.5.2-1ubuntu2.4                 amd64 
-ii  libfuse2:amd64                      2.9.2-4ubuntu4.14.04.1           amd64 
+ii  libfuse2:amd64                      2.9.2-4ubuntu4                   amd64 
-ii  libgcrypt11:amd64                   1.5.3-2ubuntu4.2                 amd64 
+ii  libgcrypt11:amd64                   1.5.3-2ubuntu4.1                 amd64 
-ii  libgnutls-openssl27:amd64           2.12.23-12ubuntu2.3              amd64 
-ii  libgnutls26:amd64                   2.12.23-12ubuntu2.3              amd64 
+ii  libgnutls-openssl27:amd64           2.12.23-12ubuntu2.2              amd64 
+ii  libgnutls26:amd64                   2.12.23-12ubuntu2.2              amd64 
-ii  libgssapi-krb5-2:amd64              1.12+dfsg-2ubuntu5.2             amd64 
+ii  libgssapi-krb5-2:amd64              1.12+dfsg-2ubuntu5.1             amd64 
-ii  libicu52:amd64                      52.1-3ubuntu0.4                  amd64 
+ii  libicu52:amd64                      52.1-3ubuntu0.2                  amd64 
-ii  libisc95                            1:9.9.5.dfsg-3ubuntu0.6          amd64 
-ii  libisccc90                          1:9.9.5.dfsg-3ubuntu0.6          amd64 
-ii  libisccfg90                         1:9.9.5.dfsg-3ubuntu0.6          amd64 
+ii  libisc95                            1:9.9.5.dfsg-3ubuntu0.2          amd64 
+ii  libisccc90                          1:9.9.5.dfsg-3ubuntu0.2          amd64 
+ii  libisccfg90                         1:9.9.5.dfsg-3ubuntu0.2          amd64 
-ii  libk5crypto3:amd64                  1.12+dfsg-2ubuntu5.2             amd64 
+ii  libk5crypto3:amd64                  1.12+dfsg-2ubuntu5.1             amd64 
-ii  libkrb5-3:amd64                     1.12+dfsg-2ubuntu5.2             amd64 
-ii  libkrb5support0:amd64               1.12+dfsg-2ubuntu5.2             amd64 
-ii  libldap-2.4-2:amd64                 2.4.31-1+nmu2ubuntu8.2           amd64 
+ii  libkrb5-3:amd64                     1.12+dfsg-2ubuntu5.1             amd64 
+ii  libkrb5support0:amd64               1.12+dfsg-2ubuntu5.1             amd64 
+ii  libldap-2.4-2:amd64                 2.4.31-1+nmu2ubuntu8             amd64 
-ii  liblwres90                          1:9.9.5.dfsg-3ubuntu0.6          amd64 
+ii  liblwres90                          1:9.9.5.dfsg-3ubuntu0.2          amd64 
-ii  libmount1:amd64                     2.20.1-5.1ubuntu20.7             amd64 
+ii  libmount1:amd64                     2.20.1-5.1ubuntu20.4             amd64 
-ii  libnss3:amd64                       2:3.19.2.1-0ubuntu0.14.04.1      amd64 
-ii  libnss3-nssdb                       2:3.19.2.1-0ubuntu0.14.04.1      all   
-ii  libnuma1:amd64                      2.0.9~rc5-1ubuntu3.14.04.1       amd64 
+ii  libnss3:amd64                       2:3.19.2.1-0ubuntu0.14.04.2      amd64 
+ii  libnss3-nssdb                       2:3.19.2.1-0ubuntu0.14.04.2      all   
+ii  libnuma1:amd64                      2.0.9~rc5-1ubuntu3               amd64 
-ii  libpam-systemd:amd64                204-5ubuntu20.15                 amd64 
+ii  libpam-systemd:amd64                204-5ubuntu20.10                 amd64 
-ii  libparted0debian1:amd64             2.3-19ubuntu1.14.04.1            amd64 
+ii  libparted0debian1:amd64             2.3-19ubuntu1                    amd64 
-ii  libpcre3:amd64                      1:8.31-2ubuntu2.1                amd64 
+ii  libpcre3:amd64                      1:8.31-2ubuntu2                  amd64 
-ii  libpng12-0:amd64                    1.2.50-1ubuntu2.14.04.1          amd64 
+ii  libpng12-0:amd64                    1.2.50-1ubuntu2                  amd64 
-ii  libpolkit-agent-1-0:amd64           0.105-4ubuntu3.14.04.1           amd64 
-ii  libpolkit-backend-1-0:amd64         0.105-4ubuntu3.14.04.1           amd64 
-ii  libpolkit-gobject-1-0:amd64         0.105-4ubuntu3.14.04.1           amd64 
+ii  libpolkit-agent-1-0:amd64           0.105-4ubuntu2                   amd64 
+ii  libpolkit-backend-1-0:amd64         0.105-4ubuntu2                   amd64 
+ii  libpolkit-gobject-1-0:amd64         0.105-4ubuntu2                   amd64 
-ii  libpython2.7:amd64                  2.7.6-8ubuntu0.2                 amd64 
-ii  libpython2.7-minimal:amd64          2.7.6-8ubuntu0.2                 amd64 
-ii  libpython2.7-stdlib:amd64           2.7.6-8ubuntu0.2                 amd64 
+ii  libpython2.7:amd64                  2.7.6-8                          amd64 
+ii  libpython2.7-minimal:amd64          2.7.6-8                          amd64 
+ii  libpython2.7-stdlib:amd64           2.7.6-8                          amd64 
-ii  libpython3.4-minimal:amd64          3.4.3-1ubuntu1~14.04.3           amd64 
-ii  libpython3.4-stdlib:amd64           3.4.3-1ubuntu1~14.04.3           amd64 
+ii  libpython3.4-minimal:amd64          3.4.0-2ubuntu1                   amd64 
+ii  libpython3.4-stdlib:amd64           3.4.0-2ubuntu1                   amd64 
-ii  libsqlite3-0:amd64                  3.8.2-1ubuntu2.1                 amd64 
-ii  libss2:amd64                        1.42.9-3ubuntu1.3                amd64 
-ii  libssl1.0.0:amd64                   1.0.1f-1ubuntu2.16               amd64 
+ii  libsqlite3-0:amd64                  3.8.2-1ubuntu2                   amd64 
+ii  libss2:amd64                        1.42.9-3ubuntu1.2                amd64 
+ii  libssl1.0.0:amd64                   1.0.1f-1ubuntu2.11               amd64 
-ii  libsystemd-daemon0:amd64            204-5ubuntu20.15                 amd64 
-ii  libsystemd-login0:amd64             204-5ubuntu20.15                 amd64 
-ii  libtasn1-6:amd64                    3.4-3ubuntu0.3                   amd64 
+ii  libsystemd-daemon0:amd64            204-5ubuntu20.10                 amd64 
+ii  libsystemd-login0:amd64             204-5ubuntu20.10                 amd64 
+ii  libtasn1-6:amd64                    3.4-3ubuntu0.1                   amd64 
-ii  libudev1:amd64                      204-5ubuntu20.15                 amd64 
+ii  libudev1:amd64                      204-5ubuntu20.10                 amd64 
-ii  libuuid1:amd64                      2.20.1-5.1ubuntu20.7             amd64 
+ii  libuuid1:amd64                      2.20.1-5.1ubuntu20.4             amd64 
-ii  libxext6:amd64                      2:1.3.2-1ubuntu0.0.14.04.1       amd64 
+ii  libxext6:amd64                      2:1.3.2-1                        amd64 
-ii  libxml2:amd64                       2.9.1+dfsg1-3ubuntu4.6           amd64 
+ii  libxml2:amd64                       2.9.1+dfsg1-3ubuntu4.4           amd64 
-ii  linux-headers-3.13.0-74             3.13.0-74.118                    all   
-ii  linux-headers-3.13.0-74-generic     3.13.0-74.118                    amd64 
-ii  linux-headers-generic               3.13.0.74.80                     amd64 
-ii  linux-headers-virtual               3.13.0.74.80                     amd64 
-ii  linux-image-3.13.0-74-generic       3.13.0-74.118                    amd64 
-ii  linux-image-extra-3.13.0-74-generic 3.13.0-74.118                    amd64 
-ii  linux-image-virtual                 3.13.0.74.80                     amd64 
+ii  linux-headers-3.13.0-48             3.13.0-48.80                     all   
+ii  linux-headers-3.13.0-48-generic     3.13.0-48.80                     amd64 
+ii  linux-headers-generic               3.13.0.48.55                     amd64 
+ii  linux-headers-virtual               3.13.0.48.55                     amd64 
+ii  linux-image-3.13.0-48-generic       3.13.0-48.80                     amd64 
+ii  linux-image-extra-3.13.0-48-generic 3.13.0-48.80                     amd64 
+ii  linux-image-virtual                 3.13.0.48.55                     amd64 
-ii  linux-virtual                       3.13.0.74.80                     amd64 
+ii  linux-virtual                       3.13.0.48.55                     amd64 
-ii  login                               1:4.1.5.1-1ubuntu9.1             amd64 
+ii  login                               1:4.1.5.1-1ubuntu9               amd64 
-ii  lshw                                02.16-2ubuntu1.3                 amd64 
+ii  lshw                                02.16-2ubuntu1.2                 amd64 
-ii  mount                               2.20.1-5.1ubuntu20.7             amd64 
+ii  mount                               2.20.1-5.1ubuntu20.4             amd64 
-ii  ntpdate                             1:4.2.6.p5+dfsg-3ubuntu2.14.04.6 amd64 
+ii  ntpdate                             1:4.2.6.p5+dfsg-3ubuntu2.14.04.2 amd64 
-ii  openssh-client                      1:6.6p1-2ubuntu2.3               amd64 
-ii  openssh-server                      1:6.6p1-2ubuntu2.3               amd64 
-ii  openssh-sftp-server                 1:6.6p1-2ubuntu2.3               amd64 
-ii  openssl                             1.0.1f-1ubuntu2.16               amd64 
+ii  openssh-client                      1:6.6p1-2ubuntu2                 amd64 
+ii  openssh-server                      1:6.6p1-2ubuntu2                 amd64 
+ii  openssh-sftp-server                 1:6.6p1-2ubuntu2                 amd64 
+ii  openssl                             1.0.1f-1ubuntu2.11               amd64 
-ii  overlayroot                         0.25ubuntu1.14.04.1              all   
-ii  parted                              2.3-19ubuntu1.14.04.1            amd64 
-ii  passwd                              1:4.1.5.1-1ubuntu9.1             amd64 
-ii  patch                               2.7.1-4ubuntu2.3                 amd64 
+ii  overlayroot                         0.25ubuntu1                      all   
+ii  parted                              2.3-19ubuntu1                    amd64 
+ii  passwd                              1:4.1.5.1-1ubuntu9               amd64 
+ii  patch                               2.7.1-4ubuntu1                   amd64 
-ii  policykit-1                         0.105-4ubuntu3.14.04.1           amd64 
-ii  pollinate                           4.7-0ubuntu1.4                   all   
+ii  policykit-1                         0.105-4ubuntu2                   amd64 
+ii  pollinate                           4.7-0ubuntu1.2                   all   
-ii  ppp                                 2.4.5-5.1ubuntu2.2               amd64 
+ii  ppp                                 2.4.5-5.1ubuntu2.1               amd64 
-ii  python-apt                          0.9.3.5ubuntu2                   amd64 
-ii  python-apt-common                   0.9.3.5ubuntu2                   all   
+ii  python-apt                          0.9.3.5ubuntu1                   amd64 
+ii  python-apt-common                   0.9.3.5ubuntu1                   all   
-ii  python-requests                     2.2.1-1ubuntu0.3                 all   
+ii  python-requests                     2.2.1-1ubuntu0.2                 all   
-ii  python-six                          1.5.2-1ubuntu1                   all   
+ii  python-six                          1.5.2-1                          all   
-ii  python-urllib3                      1.7.1-1ubuntu4                   all   
+ii  python-urllib3                      1.7.1-1ubuntu0.1                 all   
-ii  python2.7                           2.7.6-8ubuntu0.2                 amd64 
-ii  python2.7-minimal                   2.7.6-8ubuntu0.2                 amd64 
+ii  python2.7                           2.7.6-8                          amd64 
+ii  python2.7-minimal                   2.7.6-8                          amd64 
-ii  python3-apport                      2.14.1-0ubuntu3.19               all   
-ii  python3-apt                         0.9.3.5ubuntu2                   amd64 
+ii  python3-apport                      2.14.1-0ubuntu3.8                all   
+ii  python3-apt                         0.9.3.5ubuntu1                   amd64 
-ii  python3-distupgrade                 1:0.220.8                        all   
-ii  python3-gdbm:amd64                  3.4.3-1~14.04.2                  amd64 
+ii  python3-distupgrade                 1:0.220.7                        all   
+ii  python3-gdbm:amd64                  3.4.0-0ubuntu1                   amd64 
-ii  python3-problem-report              2.14.1-0ubuntu3.19               all   
+ii  python3-problem-report              2.14.1-0ubuntu3.8                all   
-ii  python3-software-properties         0.92.37.6                        all   
-ii  python3-update-manager              1:0.196.14                       all   
-ii  python3.4                           3.4.3-1ubuntu1~14.04.3           amd64 
-ii  python3.4-minimal                   3.4.3-1ubuntu1~14.04.3           amd64 
+ii  python3-software-properties         0.92.37.3                        all   
+ii  python3-update-manager              1:0.196.12                       all   
+ii  python3.4                           3.4.0-2ubuntu1                   amd64 
+ii  python3.4-minimal                   3.4.0-2ubuntu1                   amd64 
-ii  software-properties-common          0.92.37.6                        all   
+ii  software-properties-common          0.92.37.3                        all   
-ii  sudo                                1.8.9p5-1ubuntu1.2               amd64 
-ii  systemd-services                    204-5ubuntu20.15                 amd64 
+ii  sudo                                1.8.9p5-1ubuntu1.1               amd64 
+ii  systemd-services                    204-5ubuntu20.10                 amd64 
-ii  sysv-rc                             2.88dsf-41ubuntu6.2              all   
-ii  sysvinit-utils                      2.88dsf-41ubuntu6.2              amd64 
+ii  sysv-rc                             2.88dsf-41ubuntu6.1              all   
+ii  sysvinit-utils                      2.88dsf-41ubuntu6.1              amd64 
-ii  tcpdump                             4.5.1-2ubuntu1.2                 amd64 
+ii  tcpdump                             4.5.1-2ubuntu1.1                 amd64 
-ii  tzdata                              2015g-0ubuntu0.14.04             all   
+ii  tzdata                              2015b-0ubuntu0.14.04             all   
-ii  ubuntu-release-upgrader-core        1:0.220.8                        all   
+ii  ubuntu-release-upgrader-core        1:0.220.7                        all   
-ii  udev                                204-5ubuntu20.15                 amd64 
+ii  udev                                204-5ubuntu20.10                 amd64 
-ii  unattended-upgrades                 0.82.1ubuntu2.4                  all   
+ii  unattended-upgrades                 0.82.1ubuntu2.2                  all   
-ii  update-manager-core                 1:0.196.14                       all   
+ii  update-manager-core                 1:0.196.12                       all   
-ii  util-linux                          2.20.1-5.1ubuntu20.7             amd64 
-ii  uuid-runtime                        2.20.1-5.1ubuntu20.7             amd64 
+ii  util-linux                          2.20.1-5.1ubuntu20.4             amd64 
+ii  uuid-runtime                        2.20.1-5.1ubuntu20.4             amd64 

 ubuntu$ cat /etc/apt/sources.list
 # See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
 # newer versions of the distribution.
-deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty main restricted
-deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty main restricted
+deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty main
+deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty main
 
 ## Major bug fix updates produced after the final release of the
 ## distribution.
-deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates main restricted
-deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates main restricted
+deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates main
+deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates main
 
 ## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
 ## team. Also, please note that software in universe WILL NOT receive any
@@ -30,10 +31,10 @@
 ## your rights to use the software. Also, please note that software in
 ## multiverse WILL NOT receive any review or updates from the Ubuntu
 ## security team.
-deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty multiverse
-deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty multiverse
-deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates multiverse
-deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates multiverse
+# deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty multiverse
+# deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty multiverse
+# deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates multiverse
+# deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates multiverse
 
 ## Uncomment the following two lines to add software from the 'backports'
 ## repository.
@@ -42,8 +43,8 @@
 ## newer versions of some applications which may provide useful features.
 ## Also, please note that software in backports WILL NOT receive any review
 ## or updates from the Ubuntu security team.
-deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse
-deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse
+# deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse
+# deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse

so basically, the working AMI has slightly *older* versions of just about everything, since it is based on an Ubuntu AMI from 2015-03-25, whereas the non-working AMI is based on an Ubuntu AMI from 2015-12-08.  One of those "updates" -- either the kernel or one of the packages -- breaks docker.

I think we can live with that -- it's no different than our hold at Docker-1.6.1, really, except we don't know exactly which upgrade breaks it.  When we come to wanting to upgrade the AMI, we can either do some further experimentation to figure out exactly which package hurts, or maybe we can just leave the m1.medium instances running this way.
(Assignee)

Comment 31

2 years ago
I'm going to try this one more time, without the sources.list hackery.  Barring that, I think we just need to remember that March 2015 was a good month for AMIs, and maybe AMIs after that just didn't taste the same.

base:
--> hvm-builder: AMIs were created:
us-west-2: ami-b63822d7
--> pv-builder: AMIs were created:
us-west-2: ami-ed051f8c

app:
--> hvm-builder: AMIs were created:
us-east-1: ami-f62d779c
us-west-1: ami-66d8ad06
us-west-2: ami-c83923a9
--> pv-builder: AMIs were created:
us-east-1: ami-dd2379b7
us-west-1: ami-01dfaa61
us-west-2: ami-5e3a203f

I'll wait until the try jobs have finished before swapping that in, though.

Comment 32

2 years ago
Created attachment 8707563 [details]
MozReview Request: Bug 1237663 - Switch TC Linux64 debug jobs from m3.large 1cpu to m1.medium (1cpu). r=jmaher

Review commit: https://reviewboard.mozilla.org/r/30757/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/30757/
Attachment #8707563 - Flags: review?(jmaher)
(Assignee)

Comment 33

2 years ago
That test job was successful too, so I'm going to leave desktop-test-m1 configured with the AMIs in comment 31 and put it up for review.
Comment on attachment 8707563 [details]
MozReview Request: Bug 1237663 - Switch TC Linux64 debug jobs from m3.large 1cpu to m1.medium (1cpu). r=jmaher

https://reviewboard.mozilla.org/r/30757/#review27583

woohoo
Attachment #8707563 - Flags: review?(jmaher) → review+
(Assignee)

Comment 35

2 years ago
Created attachment 8707568 [details] [review]
https://github.com/taskcluster/docker-worker/pull/205
Attachment #8707568 - Flags: review?(garndt)
Comment on attachment 8707568 [details] [review]
https://github.com/taskcluster/docker-worker/pull/205

merged https://github.com/taskcluster/docker-worker/commit/ac91665d4040b3bc05fd56d2d7d0d6dde346a131
Attachment #8707568 - Flags: review?(garndt) → review+
(Assignee)

Comment 37

2 years ago
Comment on attachment 8707563 [details]
MozReview Request: Bug 1237663 - Switch TC Linux64 debug jobs from m3.large 1cpu to m1.medium (1cpu). r=jmaher

This doesn't need to be landed -- I just changed desktop-test to use m1.mediums.
(Assignee)

Updated

2 years ago
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
(In reply to Dustin J. Mitchell [:dustin] from comment #31)
> Barring that, I think we just need to remember that March 2015 was a good
> month for AMIs, and maybe AMIs after that just didn't taste the same.

I literally LOLed at this. :)
You need to log in before you can comment on or make changes to this bug.