Closed
Bug 1237663
Opened 8 years ago
Closed 8 years ago
Create a new desktop-test-m1 workerType that runs on m1.mediums
Categories
(Taskcluster :: Services, defect)
Taskcluster
Services
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: dustin, Assigned: dustin)
References
Details
Attachments
(2 files)
m3.medium apparently causes test timeouts, even under Buildbot: bug 945981
|
||
Updated•8 years ago
|
Blocks: tc-linux64-debug
|
Assignee | |
Comment 1•8 years ago
|
||
OK, that successfully ran a task: https://tools.taskcluster.net/task-inspector/#FI28ABO2SlSB51Jun2af3A/0 Armen, do you want to switch jobs over to that workerType, or should I? On this bug, or another?
Flags: needinfo?(armenzg)
|
|
||
Comment 2•8 years ago
|
||
This bug should be sufficient. If you could while I'm out tomorrow that would be great but I'm happy to give it a shot since I have the flow of all patches. Let me know what works best for you. This is as simple as changing to the worker type, right? "workerType": "desktop-test-m1",
Flags: needinfo?(armenzg)
|
|
Assignee | |
Comment 3•8 years ago
|
||
Yes, it's that simple. We can do it for all tasks, or (by changing other files) only for certain suites. Can you point me to the patches I should layer on top of?
|
|
Assignee | |
Comment 4•8 years ago
|
||
aiui this ended up mixed into a try push on another bug, so I'll consider this one closed.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
|
|
Assignee | |
Comment 5•8 years ago
|
||
The try jobs are failing, and docker is failing to kill unity-2d-shell. We end up with a process list like
> root 831 0.8 0.4 680424 15664 ? Ssl 14:14 2:39 /usr/bin/docker -d -g /mnt/var/lib/docker -s aufs
> root 1718 0.0 0.1 124484 5288 ? Sl 14:15 0:00 \_ docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32768 -container-ip 172.17.0.1 -container-port 60023
> root 1724 0.1 0.1 39320 4216 ? Ssl 14:15 0:19 \_ /livelog
> root 2053 0.0 0.0 0 0 ? Ss 14:18 0:00 \_ [sudo]
> ubuntu 2223 89.8 0.0 0 0 ? R 14:18 270:10 \_ [unity-2d-panel]
and that unity process ignores all signals but still seems to be getting CPU time. We're debugging now.Status: RESOLVED → REOPENED
Resolution: FIXED → ---
|
|
Assignee | |
Comment 6•8 years ago
|
||
Working conjecture is that this is a bug between the PV filesystem drivers and the aufs driver, as aufs seems to be involved in most of the hangs here. I'm rebuilding the AMI with a newer kernel (3.19.0-43, the latest in the 3.x series) to see if that helps.
|
|
Assignee | |
Comment 7•8 years ago
|
||
I just deployed the newer kernel to desktop-test-m1.
|
|
Assignee | |
Comment 8•8 years ago
|
||
Jobs that had failed: https://tools.taskcluster.net/task-inspector/#I9ExiEGkRLCCp_LukbE0Wg/ https://tools.taskcluster.net/task-inspector/#Ia2bqXMZSj6B4dvTrkk9dQ/ https://tools.taskcluster.net/task-inspector/#NYdTDGRLQ5Cwi7GGPFlVzQ/ Re-runs: https://tools.taskcluster.net/task-inspector/#A-q92kbLSjuwumEs7uoyrA/ https://tools.taskcluster.net/task-inspector/#H7EQcFnGTj-kBOvdCOnXtQ/ https://tools.taskcluster.net/task-inspector/#Am4cCDpfTOCRZRt7RR0Ykw/
|
|
Assignee | |
Comment 9•8 years ago
|
||
The instances aren't starting tasks because they don't have any video devices. Yet:
pv-builder: Setting up v4l2loopback-dkms (0.8.0-1) ...
pv-builder: Loading new v4l2loopback-0.8.0 DKMS files...
pv-builder: First Installation: checking all kernels...
pv-builder: Building for 3.13.0-74-generic and 3.19.0-43-generic
pv-builder: Building initial module for 3.13.0-74-generic
pv-builder: Done.
pv-builder:
pv-builder: v4l2loopback:
pv-builder: Running module version sanity check.
pv-builder: - Original module
pv-builder: - No original module exists within this kernel
pv-builder: - Installation
pv-builder: - Installing to /lib/modules/3.13.0-74-generic/updates/dkms/
pv-builder:
pv-builder: depmod.....
pv-builder:
pv-builder: DKMS: install completed.
pv-builder: Module build for the currently running kernel was skipped since the
pv-builder: kernel source for this kernel does not seem to be installed.
pv-builder: Setting up v4l2loopback-utils (0.8.0-1) ...
so it looks like it *tried* to build it for 3.19.0-43-generic, but never actually did? Because there are no linux-headers for that version. However, once those are installed,
DKMS make.log for v4l2loopback-0.8.0 for kernel 3.19.0-43-generic (x86_64)
Fri Jan 8 23:44:49 UTC 2016
Building v4l2-loopback driver...
make -C /lib/modules/3.19.0-43-generic/build M=/var/lib/dkms/v4l2loopback/0.8.0/build modules
make[1]: Entering directory `/usr/src/linux-headers-3.19.0-43-generic'
CC [M] /var/lib/dkms/v4l2loopback/0.8.0/build/v4l2loopback.o
/var/lib/dkms/v4l2loopback/0.8.0/build/v4l2loopback.c: In function ‘attr_store_maxopeners’:
/var/lib/dkms/v4l2loopback/0.8.0/build/v4l2loopback.c:486:2: error: implicit declaration of function ‘strict_strtoul’ [-Werror=implicit-function-declaration]
if (strict_strtoul(buf, 0, &curr))
^
cc1: some warnings being treated as errors
make[2]: *** [/var/lib/dkms/v4l2loopback/0.8.0/build/v4l2loopback.o] Error 1
make[1]: *** [_module_/var/lib/dkms/v4l2loopback/0.8.0/build] Error 2
make[1]: Leaving directory `/usr/src/linux-headers-3.19.0-43-generic'
make: *** [v4l2loopback.ko] Error 2
https://bugs.launchpad.net/ubuntu/+source/v4l2loopback/+bug/1455791|
|
Assignee | |
Comment 10•8 years ago
|
||
Ah, newer versions of v4l2loopback work with newer kernels, but aren't packaged by Ubuntu. Yay. So we'll get v4l2loopback from Git and build it with DKMS. Fun! I've also rolled back the max capacity of this workerType to 2, since otherwise the provisioner was spawning lots of instances.
|
|
Assignee | |
Comment 11•8 years ago
|
||
pv-builder: Some packages could not be installed. This may mean that you have
pv-builder: requested an impossible situation or if you are using the unstable
pv-builder: distribution that some required packages have not yet been created
pv-builder: or been moved out of Incoming.
pv-builder: The following information may help to resolve the situation:
pv-builder:
pv-builder: The following packages have unmet dependencies:
pv-builder: dkms : Depends: gcc but it is not installable
pv-builder: Depends: make but it is not installable or
pv-builder: E: Unable to correct problems, you have held broken packages.
pv-builder: build-essential but it is not installable or
pv-builder: dpkg-dev but it is not going to be installed
pv-builder: Recommends: fakeroot but it is not installable
pv-builder: linux-image-extra-3.19.0-43-generic : Depends: crda (>= 1.1.1-1ubuntu2) but it is not installable or
pv-builder: wireless-crda but it is not installable
so those will need to be solved. Why is nothing ever even remotely simple???!|
|
Assignee | |
Comment 12•8 years ago
|
||
KERNEL_VER=3.19.0-43-generic
sudo apt-get install -y \
linux-image-${KERNEL_VER} \
linux-headers-${KERNEL_VER} \
linux-image-extra-${KERNEL_VER} \
dkms
works by hand in a 14.04 docker image. I wonder what the difference is. Apt's error messages are inscrutible! "..but it is not going to be installed" -- doesn't it seem like apt could install that, then, like it does all the other dependencies?|
|
Assignee | |
Comment 13•8 years ago
|
||
Here's the whole message (not much more):
pv-builder: + sudo apt-get update -y
pv-builder: Get:1 http://security.ubuntu.com trusty-security InRelease [64.4 kB]
pv-builder: Ign http://archive.ubuntu.com trusty InRelease
pv-builder: Get:2 http://archive.ubuntu.com trusty-updates InRelease [64.4 kB]
pv-builder: Hit http://archive.ubuntu.com trusty Release.gpg
pv-builder: Hit http://archive.ubuntu.com trusty Release
pv-builder: Ign http://archive.ubuntu.com trusty Release
pv-builder: Get:3 http://security.ubuntu.com trusty-security/main amd64 Packages [403 kB]
pv-builder: Get:4 http://archive.ubuntu.com trusty-updates/main amd64 Packages [683 kB]
pv-builder: Get:5 http://security.ubuntu.com trusty-security/restricted amd64 Packages [13.0 kB]
pv-builder: Get:6 http://security.ubuntu.com trusty-security/universe amd64 Packages [122 kB]
pv-builder: Get:7 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [15.9 kB]
pv-builder: Get:8 http://security.ubuntu.com trusty-security/multiverse amd64 Packages [4798 B]
pv-builder: Get:9 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [333 kB]
pv-builder: Get:10 http://security.ubuntu.com trusty-security/main Translation-en [220 kB]
pv-builder: Get:11 http://archive.ubuntu.com trusty-updates/multiverse amd64 Packages [13.0 kB]
pv-builder: Get:12 http://security.ubuntu.com trusty-security/multiverse Translation-en [2437 B]
pv-builder: Get:13 https://get.docker.io docker InRelease
pv-builder: Ign https://get.docker.io docker InRelease
pv-builder: Get:14 http://archive.ubuntu.com trusty-updates/main Translation-en [342 kB]
pv-builder: Get:15 http://security.ubuntu.com trusty-security/restricted Translation-en [3206 B]
pv-builder: Get:16 http://security.ubuntu.com trusty-security/universe Translation-en [71.6 kB]
pv-builder: Get:17 http://archive.ubuntu.com trusty-updates/multiverse Translation-en [6832 B]
pv-builder: Get:18 http://archive.ubuntu.com trusty-updates/restricted Translation-en [3699 B]
pv-builder: Get:19 http://archive.ubuntu.com trusty-updates/universe Translation-en [175 kB]
pv-builder: Get:20 http://archive.ubuntu.com trusty/main amd64 Packages [1350 kB]
pv-builder: Get:21 http://archive.ubuntu.com trusty/restricted amd64 Packages [13.0 kB]
pv-builder: Get:22 http://archive.ubuntu.com trusty/universe amd64 Packages [5859 kB]
pv-builder: Get:23 https://get.docker.io docker Release
pv-builder: Get:24 http://archive.ubuntu.com trusty/multiverse amd64 Packages [132 kB]
pv-builder: Get:25 http://archive.ubuntu.com trusty/main Translation-en [762 kB]
pv-builder: Get:26 https://get.docker.io docker/main amd64 Packages
pv-builder: Get:27 http://archive.ubuntu.com trusty/multiverse Translation-en [102 kB]
pv-builder: Get:28 http://archive.ubuntu.com trusty/restricted Translation-en [3457 B]
pv-builder: Get:29 http://archive.ubuntu.com trusty/universe Translation-en [4089 kB]
pv-builder: Ign https://get.docker.io docker/main Translation-en
pv-builder: Fetched 14.9 MB in 10s (1458 kB/s)
pv-builder: Reading package lists...
pv-builder: W: GPG error: http://archive.ubuntu.com trusty Release: Unknown error executing gpgv
pv-builder:
pv-builder: ## upgrade the kernel, along with extra (which adds AUFS support)
pv-builder: KERNEL_VER=3.19.0-43-generic
pv-builder: + KERNEL_VER=3.19.0-43-generic
pv-builder: sudo apt-get install -y \
pv-builder: linux-image-${KERNEL_VER} \
pv-builder: linux-headers-${KERNEL_VER} \
pv-builder: linux-image-extra-${KERNEL_VER} \
pv-builder: dkms
pv-builder: + sudo apt-get install -y linux-image-3.19.0-43-generic linux-headers-3.19.0-43-generic linux-image-extra-3.19.0-43-generic dkms
pv-builder: Reading package lists...
pv-builder: Building dependency tree...
pv-builder: Reading state information...
pv-builder: Some packages could not be installed. This may mean that you have
pv-builder: requested an impossible situation or if you are using the unstable
pv-builder: distribution that some required packages have not yet been created
pv-builder: or been moved out of Incoming.
pv-builder: The following information may help to resolve the situation:
pv-builder:
pv-builder: The following packages have unmet dependencies:
pv-builder: dkms : Depends: gcc but it is not installable
pv-builder: E: Unable to correct problems, you have held broken packages.
pv-builder: Depends: make but it is not installable or
pv-builder: build-essential but it is not installable or
pv-builder: dpkg-dev but it is not going to be installed
pv-builder: Recommends: fakeroot but it is not installable
pv-builder: linux-image-extra-3.19.0-43-generic : Depends: crda (>= 1.1.1-1ubuntu2) but it is not installable or
pv-builder: wireless-crda but it is not installable
I wonder if the gpg error is related? That, or possibly I don't have the same repos (universe?) enabled in docker as in the base Ubuntu AMI.|
|
Assignee | |
Comment 14•8 years ago
|
||
Catting /etc/apt/sources.list shows, *only* on the pv builder,
pv-builder: deb http://archive.ubuntu.com/ubuntu trusty main restricted universe multiverse
pv-builder: deb http://archive.ubuntu.com/ubuntu trusty-updates main restricted universe multiverse
pv-builder: deb http://security.ubuntu.com/ubuntu trusty-security main restricted universe multiverse
I'm not sure where that's coming from -- starting up the same AMI from the launch wizard gives the expected sources.list with more repos and linking to the ec2 mirrors. Rather than try to solve this (packer bug? cloud-tools bug? dunno), I'll just write out the sources.list I want.|
|
Assignee | |
Comment 15•8 years ago
|
||
==> Builds finished. The artifacts of successful builds are: --> hvm-builder: AMIs were created: us-west-2: ami-2cb6ad4d --> hvm-builder: 'aws' provider box: deploy/packer/base.box --> pv-builder: AMIs were created: us-west-2: ami-feb5ae9f --> pv-builder: 'aws' provider box: deploy/packer/base.box In the interim, Greg found that 'apt-get clean' will fix the sources.list, too. Go figure.
|
|
Assignee | |
Comment 16•8 years ago
|
||
us-east-1: ami-721a4118 us-west-1: ami-4a8ffa2a us-west-2: ami-9eaab1ff --> hvm-builder: 'aws' provider box: app.box --> pv-builder: AMIs were created: us-east-1: ami-b91a41d3 us-west-1: ami-d489fcb4 us-west-2: ami-71b7ac10 --> pv-builder: 'aws' provider box: app.box
|
|
Assignee | |
Comment 17•8 years ago
|
||
That version loaded the wrong kernel in the PV AMI, because the debian install scripts edit /boot/grub/grub.cfg, not the /boot/grub/menu.lst that PV-GRUB loads. So! Trying again.. ==> Builds finished. The artifacts of successful builds are: --> hvm-builder: AMIs were created: us-west-2: ami-7aabb01b --> hvm-builder: 'aws' provider box: deploy/packer/base.box --> pv-builder: AMIs were created: us-west-2: ami-86a9b2e7 --> pv-builder: 'aws' provider box: deploy/packer/base.box ==> Builds finished. The artifacts of successful builds are: --> hvm-builder: AMIs were created: us-east-1: ami-41114a2b us-west-1: ami-308aff50 us-west-2: ami-96afb4f7 --> hvm-builder: 'aws' provider box: app.box --> pv-builder: AMIs were created: us-east-1: ami-e8124982 us-west-1: ami-2e8ffa4e us-west-2: ami-93afb4f2 --> pv-builder: 'aws' provider box: app.box And a task using the new AMI: https://tools.taskcluster.net/task-inspector/#L5U1zSnETemjBC3HN3Lu-Q/
|
|
Assignee | |
Comment 18•8 years ago
|
||
That had a typo ("menu.lsg")
==> Builds finished. The artifacts of successful builds are:
--> hvm-builder: AMIs were created:
us-west-2: ami-eaafb48b
--> hvm-builder: 'aws' provider box: deploy/packer/base.box
--> pv-builder: AMIs were created:
us-west-2: ami-44a9b225
--> pv-builder: 'aws' provider box: deploy/packer/base.box
==> Builds finished. The artifacts of successful builds are:
--> pv-builder: AMIs were created:
us-east-1: ami-fd174c97
us-west-1: ami-7b8aff1b
us-west-2: ami-2ca9b24d
--> pv-builder: 'aws' provider box: app.box
--> hvm-builder: AMIs were created:
us-east-1: ami-51154e3b
us-west-1: ami-c88bfea8
us-west-2: ami-33aeb552
--> hvm-builder: 'aws' provider box: app.box
https://tools.taskcluster.net/task-inspector/#S1bJ6Pi_SIigRsmKJnpW7A/|
|
Assignee | |
Comment 19•8 years ago
|
||
Again: root 1568 23.8 0.7 664040 29904 ? Ssl 21:32 2:03 /usr/bin/docker -d -g /mnt/var/lib/docker -s aufs root 1958 0.0 0.3 124492 12208 ? Sl 21:33 0:00 \_ docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32768 -container-ip 172.17.0.1 -container-port 60023 root 1964 0.1 0.1 40416 7388 ? Ssl 21:33 0:00 \_ /livelog root 2383 0.0 0.0 0 0 ? Ss 21:36 0:00 \_ [sudo] ubuntu 2541 24.8 0.0 0 0 ? R 21:37 1:02 \_ [gnome-settings-] ubuntu 2563 25.0 0.0 0 0 ? R 21:37 1:02 \_ [unity-2d-shell] so, if anything, worse now, since gnome settings daemon is appearing. However, looking at the docker logs from one of those runs: 21:39:07 INFO - dir: devtools/client/aboutdebugging/test 21:39:09 INFO - ImportError: No module named pygtk 21:39:09 INFO - ImportError: No module named pygtk 21:39:09 INFO - Setting pipeline to PAUSED ... 21:39:09 INFO - ERROR: Pipeline doesn't want to pause. 21:39:09 INFO - ERROR: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0: Failed to query attributes of input 0 in device /dev/video0 21:39:09 INFO - Additional debug info: 21:39:09 INFO - v4l2_calls.c(142): gst_v4l2_fill_lists (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0: 21:39:09 INFO - Failed to get 0 in input enumeration for /dev/video0. (25 - Inappropriate ioctl for device) 21:39:09 INFO - Setting pipeline to NULL ... 21:39:09 INFO - Freeing pipeline ... 21:39:09 INFO - Traceback (most recent call last): 21:39:09 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2623, in <module> 21:39:09 INFO - sys.exit(cli()) 21:39:09 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2620, in cli 21:39:09 INFO - return run_test_harness(options) 21:39:09 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2597, in run_test_harness 21:39:09 INFO - result = runner.runTests(options) 21:39:09 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2157, in runTests 21:39:09 INFO - result = self.runMochitests(options, tests_in_dir, onLaunch) 21:39:09 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2106, in runMochitests 21:39:09 INFO - result = self.doTests(options, onLaunch, testsToRun) 21:39:09 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2203, in doTests 21:39:09 INFO - devices = findTestMediaDevices(self.log) 21:39:09 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 1448, in findTestMediaDevices 21:39:09 INFO - 'v4l2sink', 'device=%s' % device]) 21:39:09 INFO - File "/usr/lib/python2.7/subprocess.py", line 511, in check_call 21:39:09 INFO - raise CalledProcessError(retcode, cmd) 21:39:09 INFO - subprocess.CalledProcessError: Command '['/usr/bin/gst-launch-0.10', 'videotestsrc', 'pattern=green', 'num-buffers=1', '!', 'v4l2sink', 'device=/dev/video0']' returned non-zero exit status 255 So the bad news is, the kernel upgrade didn't help. The good news is, we might have a lead on why? On an existing (m3.large) desktop-test instance: ubuntu@ip-172-31-2-12:~$ docker run --device /dev/video0:/dev/video0 -ti --rm taskcluster/desktop-test:0.5.3 root@taskcluster-worker:~# gst-launch-0.10 'videotestsrc' pattern=green num-buffers=1 \! v4l2sink device=/dev/video0 Setting pipeline to PAUSED ... Pipeline is PREROLLING ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock Got EOS from element "pipeline0". Execution ended after 33336501 ns. Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ... ubuntu@ip-172-31-2-12:~$ dmesg | grep v4l2 [ 11.761394] v4l2loopback: module verification failed: signature and/or required key missing - tainting kernel [ 11.761666] v4l2loopback: number of devices is limited to: 8 [ 11.762150] v4l2loopback driver version 0.8.0 loaded But on a new (m1.medium) desktop-test-m1 instance: ubuntu@ip-172-31-10-241:~$ docker run --device /dev/video0:/dev/video0 -ti --rm taskcluster/desktop-test:0.5.3 root@taskcluster-worker:~# gst-launch-0.10 'videotestsrc' pattern=green num-buffers=1 \! v4l2sink device=/dev/video0 Setting pipeline to PAUSED ... ERROR: Pipeline doesn't want to pause. ERROR: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0: Failed to query attributes of input 0 in device /dev/video0 Additional debug info: v4l2_calls.c(142): gst_v4l2_fill_lists (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0: Failed to get 0 in input enumeration for /dev/video0. (25 - Inappropriate ioctl for device) Setting pipeline to NULL ... Freeing pipeline ... ubuntu@ip-172-31-10-241:~$ dmesg | grep v4l2 [ 30.997074] v4l2loopback: module verification failed: signature and/or required key missing - tainting kernel [ 30.997442] v4l2loopback: number of devices is limited to: 8 [ 30.999821] v4l2loopback driver version 0.9.1 loaded probably unrelated, but potentially problematic, dmesg has lots of: [ 986.803165] aufs au_opts_verify:1612:docker[2026]: dirperm1 breaks the protection by the permission bits on the lower branch Out of a lack of other ideas, I'll try again with the 4.2.0-23 kernel (the latest available for trusty).
|
|
Assignee | |
Comment 20•8 years ago
|
||
4.2.0-23-generic: ==> Builds finished. The artifacts of successful builds are: --> hvm-builder: AMIs were created: us-west-2: ami-ba5349db --> hvm-builder: 'aws' provider box: deploy/packer/base.box --> pv-builder: AMIs were created: us-west-2: ami-225c4643 --> pv-builder: 'aws' provider box: deploy/packer/base.box ==> Builds finished. The artifacts of successful builds are: --> hvm-builder: AMIs were created: us-east-1: ami-6c0d5606 us-west-1: ami-70f48110 us-west-2: ami-dd5842bc --> hvm-builder: 'aws' provider box: app.box --> pv-builder: AMIs were created: us-east-1: ami-110d567b us-west-1: ami-2bf5804b us-west-2: ami-f7504a96 --> pv-builder: 'aws' provider box: app.box https://tools.taskcluster.net/task-inspector/#MltRSffdSvyUFEGYg8Wxyg/ Greg has suggested trying the HVM AMIs, too, to see if the errors are the same.
|
|
Assignee | |
Comment 21•8 years ago
|
||
Same result on PV with 4.2.0-23-generic: 00:48:22 INFO - Failed to get 0 in input enumeration for /dev/video0. (25 - Inappropriate ioctl for device) root 1567 19.3 0.7 664040 30080 ? Ssl 22:57 1:50 /usr/bin/docker -d -g /mnt/var/lib/docker -s aufs root 1960 0.0 0.3 124492 12020 ? Sl 22:58 0:00 \_ docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32768 -container-ip 172.17.0.1 -container-port 60023 root 1966 0.1 0.1 105952 6904 ? Ssl 22:58 0:00 \_ /livelog root 2375 0.0 0.0 0 0 ? Ss 23:00 0:00 \_ [sudo] ubuntu 2556 33.5 0.0 0 0 ? R 23:01 1:49 \_ [unity-2d-panel] ubuntu 2557 33.8 0.0 0 0 ? R 23:01 1:50 \_ [unity-2d-shell] On HVM with 4.2.0-23-generic: 23:23:25 INFO - Failed to get 0 in input enumeration for /dev/video0. (25 - Inappropriate ioctl for device) however, the docker container exits correctly. So I think upgrading the kernel has made v4l2loopback non-functional, without fixing the apparently-very-PV-related docker failures.
|
|
Assignee | |
Comment 22•8 years ago
|
||
From meeting notes:
[dustin] docker, instance types, paravirtualization, and tests
maybe try on some other instance types?
help from docker? kernel issues (wedged, un-signalable unity processes)
#docker on freenode
try docker update - greg can help dustin
docker support plans - check in #docker on mozilla to see if someone has that
https://www.docker.com/pricing
help from engineering to work on test failures on newer instance types (dougt)
question of whether to do this work with buildbot on m3's or docker on m3's
docker issues to land end of week
remaining non-intermittent issues:
still have issues with reftests - better with gnome-session, worse with unity
maybe 1-2 weeks of image issues
[todo] give selena a list of the tests that we want help with|
|
Assignee | |
Comment 23•8 years ago
|
||
I created workerTypes desktop-test-c3 and desktop-test-c3-1cpu with c3.xlarge instances and for the latter isolatedContainers. Armen's running try pushes on those now. Building an AMI with Docker-1.9.1: base: --> hvm-builder: AMIs were created: us-west-2: ami-006a7061 --> pv-builder: AMIs were created: us-west-2: ami-966973f7 app: --> hvm-builder: AMIs were created: us-east-1: ami-4cbfe426 us-west-1: ami-cde396ad us-west-2: ami-d26b71b3 --> pv-builder: AMIs were created: us-east-1: ami-7fb8e315 us-west-1: ami-3ee2975e us-west-2: ami-e96a7088 firing those up in AWS manager: https://tools.taskcluster.net/task-inspector/#GBElFJFrSya0d1Xs7DopoQ/
|
|
Assignee | |
Comment 24•8 years ago
|
||
ubuntu@ip-172-31-0-19:~$ docker --version Docker version 1.9.1, build a34a1d5 ubuntu@ip-172-31-0-19:~$ uname -r 4.2.0-23-generic ... 20:41:11 INFO - ImportError: No module named pygtk 20:41:12 INFO - Setting pipeline to PAUSED ... 20:41:12 INFO - ERROR: Pipeline doesn't want to pause. 20:41:12 INFO - ERROR: from element /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0: Failed to query attributes of input 0 in device /dev/video0 20:41:12 INFO - Additional debug info: 20:41:12 INFO - v4l2_calls.c(142): gst_v4l2_fill_lists (): /GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0: 20:41:12 INFO - Failed to get 0 in input enumeration for /dev/video0. (25 - Inappropriate ioctl for device) 20:41:12 INFO - Setting pipeline to NULL ... 20:41:12 INFO - Freeing pipeline ... 20:41:12 INFO - Traceback (most recent call last): 20:41:12 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2623, in <module> 20:41:12 INFO - sys.exit(cli()) 20:41:12 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2620, in cli 20:41:12 INFO - return run_test_harness(options) 20:41:12 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2597, in run_test_harness 20:41:12 INFO - result = runner.runTests(options) 20:41:12 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2157, in runTests 20:41:12 INFO - result = self.runMochitests(options, tests_in_dir, onLaunch) 20:41:12 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2106, in runMochitests 20:41:12 INFO - result = self.doTests(options, onLaunch, testsToRun) 20:41:12 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 2203, in doTests 20:41:12 INFO - devices = findTestMediaDevices(self.log) 20:41:12 INFO - File "/home/worker/workspace/build/tests/mochitest/runtests.py", line 1448, in findTestMediaDevices 20:41:12 INFO - 'v4l2sink', 'device=%s' % device]) 20:41:12 INFO - File "/usr/lib/python2.7/subprocess.py", line 511, in check_call 20:41:12 INFO - raise CalledProcessError(retcode, cmd) 20:41:12 INFO - subprocess.CalledProcessError: Command '['/usr/bin/gst-launch-0.10', 'videotestsrc', 'pattern=green', 'num-buffers=1', '!', 'v4l2sink', 'device=/dev/video0']' returned non-zero exit status 255 20:41:12 ERROR - Return code: 1 20:41:12 ERROR - No tests run or test summary not found 20:41:12 INFO - TinderboxPrint: mochitest-mochitest-devtools-chrome<br/><em class="testfail">T-FAIL</em> 20:41:12 WARNING - # TBPL FAILURE # 20:41:12 WARNING - setting return code to 2 20:41:12 WARNING - The mochitest suite: mochitest-devtools-chrome ran with return status: FAILURE 20:41:12 INFO - Running post-action listener: _package_coverage_data 20:41:12 INFO - Running post-action listener: _resource_record_post_action 20:41:12 INFO - Running post-run listener: _resource_record_post_run 20:41:13 INFO - Total resource usage - Wall time: 28s; CPU: 100.0%; Read bytes: 52420608; Write bytes: 84410368; Read time: 5100; Write time: 42324 20:41:13 INFO - install - Wall time: 25s; CPU: 100.0%; Read bytes: 155648; Write bytes: 65605632; Read time: 16; Write time: 20116 20:41:13 INFO - run-tests - Wall time: 3s; CPU: 99.0%; Read bytes: 52183040; Write bytes: 9580544; Read time: 5044; Write time: 2296 20:41:13 INFO - Running post-run listener: _upload_blobber_files 20:41:13 WARNING - Blob upload gear skipped. Missing cmdline options. 20:41:13 INFO - Copying logs to upload dir... 20:41:13 INFO - mkdir: /home/worker/workspace/build/upload/logs 20:41:13 WARNING - returning nonzero exit status 2 cleanup + cleanup + cp /home/worker/.xsession-errors /home/worker/artifacts/public + '[' -n 36 ']' + '[' false == false ']' + kill 36 ... root 942 31.3 2.1 758752 82864 ? Ssl 20:34 2:27 /usr/bin/docker daemon -g /mnt/var/lib/docker -s aufs root 1848 0.0 0.6 152732 24168 ? Sl 20:35 0:00 \_ docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32768 -container-ip 172.17.0.2 -container-port 60023 root 1859 0.2 0.1 56808 7232 ? Ssl 20:35 0:00 \_ /livelog root 2282 0.0 0.0 0 0 ? Ss 20:39 0:00 \_ [sudo] ubuntu 2431 17.5 0.0 0 0 ? R 20:39 0:29 \_ [unity-2d-panel] ubuntu 2432 18.1 0.0 0 0 ? R 20:39 0:30 \_ [unity-2d-shell] in other words, 1.9.1 -> no help. I want to loop back and see if the ioctl issues occur on a plain-vanilla 3.13 PV instance, too -- perhaps that is triggering a bug in v4l2 which is causing the hangs.
|
|
Assignee | |
Comment 25•8 years ago
|
||
Rebuilding that kernel-3.13, docker-1.6.1 PV image since I'd lost the AMI IDs: base: --> hvm-builder: AMIs were created: us-west-2: ami-8c6278ed --> pv-builder: AMIs were created: us-west-2: ami-2f6f754e app: --> pv-builder: AMIs were created: us-east-1: ami-49abf023 us-west-1: ami-5eec993e us-west-2: ami-36607a57 --> hvm-builder: AMIs were created: us-east-1: ami-37abf05d us-west-1: ami-80ed98e0 us-west-2: ami-32637953 https://tools.taskcluster.net/task-inspector/#LLw5sIEeRx2OhM8k2uoVJg/0 which .. succeeded. Whut? I've dialed the desktop-test-m1 workerType up to a max of 50 instances. Armen, can you do a try push to exercise it and post the link here?
Flags: needinfo?(armenzg)
|
|
Assignee | |
Comment 26•8 years ago
|
||
To be clear, the job succeeded, the processes exited, all good. ubuntu@ip-172-31-11-201:~$ uname -r 3.13.0-48-generic ubuntu@ip-172-31-11-201:~$ docker --version Docker version 1.6.1, build 97cd073 ubuntu@ip-172-31-11-201:~$ curl 169.254.169.254/latest/meta-data/instance-type ; echo m1.medium /me doesn't believe his eyes, tries another task re-run.. https://tools.taskcluster.net/task-inspector/#ZVjs4LYkQKCj0ZtVzFqxOQ/
|
|
Assignee | |
Comment 27•8 years ago
|
||
That one also succeeded. What am I missing?
|
|
||
Comment 28•8 years ago
|
||
Pushed: https://treeherder.mozilla.org/#/jobs?repo=try&revision=287c21e9f37e
Flags: needinfo?(armenzg)
|
|
||
Comment 29•8 years ago
|
||
This instead: https://treeherder.mozilla.org/#/jobs?repo=try&revision=df0f0e8baa5f
|
|
Assignee | |
Comment 30•8 years ago
|
||
The AMIs originally used to test this configuration, back on Jan 7, were --> pv-builder: AMIs were created: us-east-1: ami-82c09ee8 us-west-1: ami-7f681d1f us-west-2: ami-bf6279de So, in us-west-2, using - for Jan 7 (bad) and + for comment 25 (good), we have - ami-bf6279de + ami-36607a57 and those were built from https://github.com/taskcluster/docker-worker/commit/7ce9448ababc70541d71d39c6b6d4fdd0958a977 (packer has a parent sha1, but the changes in that merge were uncommitted at that time, I think). The differences between 7ce9448ababc70541d71d39c6b6d4fdd0958a977 and the packer config for the AMI IDs in comment 25 are: * different source AMI: - ami-47465826 (ubuntu/images/ebs/ubuntu-trusty-14.04-amd64-server-20151218) + ami-7f89a64f (ubuntu/images/ebs/ubuntu-trusty-14.04-amd64-server-20150325) * + includes a hard-coded sources.list, since the PV image includes a degenerate one * + always runs apt-get update in packages.sh Comparing instances spun up with those AMIs, ubuntu$ uname -r -3.13.0-74-generic +3.13.0-48-generic ubuntu@$ cat /etc/cloud/build.info build_name: server -serial: 20151218 +serial: 20150325 ubuntu$ dpkg -l -ii accountsservice 0.6.35-0ubuntu7.2 amd64 +ii accountsservice 0.6.35-0ubuntu7.1 amd64 -ii apparmor 2.8.95~2430-0ubuntu5.3 amd64 -ii apport 2.14.1-0ubuntu3.19 all +ii apparmor 2.8.95~2430-0ubuntu5.1 amd64 +ii apport 2.14.1-0ubuntu3.8 all -ii apt 1.0.1ubuntu2.10 amd64 -ii apt-transport-https 1.0.1ubuntu2.10 amd64 -ii apt-utils 1.0.1ubuntu2.10 amd64 +ii apt 1.0.1ubuntu2.6 amd64 +ii apt-transport-https 1.0.1ubuntu2.6 amd64 +ii apt-utils 1.0.1ubuntu2.6 amd64 -ii base-files 7.2ubuntu5.3 amd64 +ii base-files 7.2ubuntu5.2 amd64 -ii bash-completion 1:2.1-4ubuntu0.1 all +ii bash-completion 1:2.1-4 all -ii bind9-host 1:9.9.5.dfsg-3ubuntu0.6 amd64 +ii bind9-host 1:9.9.5.dfsg-3ubuntu0.2 amd64 -ii bsdutils 1:2.20.1-5.1ubuntu20.7 amd64 +ii bsdutils 1:2.20.1-5.1ubuntu20.4 amd64 -ii cloud-init 0.7.5-0ubuntu1.15 all +ii cloud-init 0.7.5-0ubuntu1.5 all -ii coreutils 8.21-1ubuntu5.3 amd64 +ii coreutils 8.21-1ubuntu5.1 amd64 -ii dh-python 1.20140128-1ubuntu8.2 all +ii dh-python 1.20140128-1ubuntu8 all -ii dnsutils 1:9.9.5.dfsg-3ubuntu0.6 amd64 +ii dnsutils 1:9.9.5.dfsg-3ubuntu0.2 amd64 -ii dpkg 1.17.5ubuntu5.5 amd64 +ii dpkg 1.17.5ubuntu5.3 amd64 -ii e2fslibs:amd64 1.42.9-3ubuntu1.3 amd64 -ii e2fsprogs 1.42.9-3ubuntu1.3 amd64 +ii e2fslibs:amd64 1.42.9-3ubuntu1.2 amd64 +ii e2fsprogs 1.42.9-3ubuntu1.2 amd64 -ii fuse 2.9.2-4ubuntu4.14.04.1 amd64 +ii fuse 2.9.2-4ubuntu4 amd64 -ii gdisk 0.8.8-1ubuntu0.1 amd64 +ii gdisk 0.8.8-1build1 amd64 -ii gnupg 1.4.16-1ubuntu2.3 amd64 -ii gpgv 1.4.16-1ubuntu2.3 amd64 +ii gnupg 1.4.16-1ubuntu2.1 amd64 +ii gpgv 1.4.16-1ubuntu2.1 amd64 -ii grub-common 2.02~beta2-9ubuntu1.6 amd64 +ii grub-common 2.02~beta2-9ubuntu1 amd64 -ii grub-legacy-ec2 0.7.5-0ubuntu1.15 all -ii grub-pc 2.02~beta2-9ubuntu1.6 amd64 -ii grub-pc-bin 2.02~beta2-9ubuntu1.6 amd64 -ii grub2-common 2.02~beta2-9ubuntu1.6 amd64 +ii grub-legacy-ec2 0.7.5-0ubuntu1.5 all +ii grub-pc 2.02~beta2-9ubuntu1 amd64 +ii grub-pc-bin 2.02~beta2-9ubuntu1 amd64 +ii grub2-common 2.02~beta2-9ubuntu1 amd64 -ii initscripts 2.88dsf-41ubuntu6.2 amd64 +ii initscripts 2.88dsf-41ubuntu6.1 amd64 -ii iproute2 3.12.0-2ubuntu1 amd64 +ii iproute2 3.12.0-2 amd64 -ii irqbalance 1.0.6-2ubuntu0.14.04.4 amd64 -ii isc-dhcp-client 4.2.4-7ubuntu12.3 amd64 -ii isc-dhcp-common 4.2.4-7ubuntu12.3 amd64 +ii irqbalance 1.0.6-2ubuntu0.14.04.1 amd64 +ii isc-dhcp-client 4.2.4-7ubuntu12 amd64 +ii isc-dhcp-common 4.2.4-7ubuntu12 amd64 -ii krb5-locales 1.12+dfsg-2ubuntu5.2 all +ii krb5-locales 1.12+dfsg-2ubuntu5.1 all -ii libaccountsservice0:amd64 0.6.35-0ubuntu7.2 amd64 +ii libaccountsservice0:amd64 0.6.35-0ubuntu7.1 amd64 -ii libapparmor-perl 2.8.95~2430-0ubuntu5.3 amd64 -ii libapparmor1:amd64 2.8.95~2430-0ubuntu5.3 amd64 -ii libapt-inst1.5:amd64 1.0.1ubuntu2.10 amd64 +ii libapparmor-perl 2.8.95~2430-0ubuntu5.1 amd64 +ii libapparmor1:amd64 2.8.95~2430-0ubuntu5.1 amd64 +ii libapt-inst1.5:amd64 1.0.1ubuntu2.6 amd64 -ii libapt-pkg4.12:amd64 1.0.1ubuntu2.10 amd64 +ii libapt-pkg4.12:amd64 1.0.1ubuntu2.6 amd64 -ii libbind9-90 1:9.9.5.dfsg-3ubuntu0.6 amd64 -ii libblkid1:amd64 2.20.1-5.1ubuntu20.7 amd64 +ii libbind9-90 1:9.9.5.dfsg-3ubuntu0.2 amd64 +ii libblkid1:amd64 2.20.1-5.1ubuntu20.4 amd64 -ii libcgmanager0:amd64 0.24-0ubuntu7.5 amd64 +ii libcgmanager0:amd64 0.24-0ubuntu7.3 amd64 -ii libcomerr2:amd64 1.42.9-3ubuntu1.3 amd64 +ii libcomerr2:amd64 1.42.9-3ubuntu1.2 amd64 -ii libcurl3-gnutls:amd64 7.35.0-1ubuntu2.5 amd64 +ii libcurl3-gnutls:amd64 7.35.0-1ubuntu2.3 amd64 -ii libdns100 1:9.9.5.dfsg-3ubuntu0.6 amd64 +ii libdns100 1:9.9.5.dfsg-3ubuntu0.2 amd64 -ii libdrm2:amd64 2.4.60-2~ubuntu14.04.1 amd64 +ii libdrm2:amd64 2.4.56-1~ubuntu2 amd64 -ii libexpat1:amd64 2.1.0-4ubuntu1.1 amd64 +ii libexpat1:amd64 2.1.0-4ubuntu1 amd64 -ii libffi6:amd64 3.1~rc1+r3.0.13-12ubuntu0.1 amd64 +ii libffi6:amd64 3.1~rc1+r3.0.13-12 amd64 -ii libfreetype6:amd64 2.5.2-1ubuntu2.5 amd64 +ii libfreetype6:amd64 2.5.2-1ubuntu2.4 amd64 -ii libfuse2:amd64 2.9.2-4ubuntu4.14.04.1 amd64 +ii libfuse2:amd64 2.9.2-4ubuntu4 amd64 -ii libgcrypt11:amd64 1.5.3-2ubuntu4.2 amd64 +ii libgcrypt11:amd64 1.5.3-2ubuntu4.1 amd64 -ii libgnutls-openssl27:amd64 2.12.23-12ubuntu2.3 amd64 -ii libgnutls26:amd64 2.12.23-12ubuntu2.3 amd64 +ii libgnutls-openssl27:amd64 2.12.23-12ubuntu2.2 amd64 +ii libgnutls26:amd64 2.12.23-12ubuntu2.2 amd64 -ii libgssapi-krb5-2:amd64 1.12+dfsg-2ubuntu5.2 amd64 +ii libgssapi-krb5-2:amd64 1.12+dfsg-2ubuntu5.1 amd64 -ii libicu52:amd64 52.1-3ubuntu0.4 amd64 +ii libicu52:amd64 52.1-3ubuntu0.2 amd64 -ii libisc95 1:9.9.5.dfsg-3ubuntu0.6 amd64 -ii libisccc90 1:9.9.5.dfsg-3ubuntu0.6 amd64 -ii libisccfg90 1:9.9.5.dfsg-3ubuntu0.6 amd64 +ii libisc95 1:9.9.5.dfsg-3ubuntu0.2 amd64 +ii libisccc90 1:9.9.5.dfsg-3ubuntu0.2 amd64 +ii libisccfg90 1:9.9.5.dfsg-3ubuntu0.2 amd64 -ii libk5crypto3:amd64 1.12+dfsg-2ubuntu5.2 amd64 +ii libk5crypto3:amd64 1.12+dfsg-2ubuntu5.1 amd64 -ii libkrb5-3:amd64 1.12+dfsg-2ubuntu5.2 amd64 -ii libkrb5support0:amd64 1.12+dfsg-2ubuntu5.2 amd64 -ii libldap-2.4-2:amd64 2.4.31-1+nmu2ubuntu8.2 amd64 +ii libkrb5-3:amd64 1.12+dfsg-2ubuntu5.1 amd64 +ii libkrb5support0:amd64 1.12+dfsg-2ubuntu5.1 amd64 +ii libldap-2.4-2:amd64 2.4.31-1+nmu2ubuntu8 amd64 -ii liblwres90 1:9.9.5.dfsg-3ubuntu0.6 amd64 +ii liblwres90 1:9.9.5.dfsg-3ubuntu0.2 amd64 -ii libmount1:amd64 2.20.1-5.1ubuntu20.7 amd64 +ii libmount1:amd64 2.20.1-5.1ubuntu20.4 amd64 -ii libnss3:amd64 2:3.19.2.1-0ubuntu0.14.04.1 amd64 -ii libnss3-nssdb 2:3.19.2.1-0ubuntu0.14.04.1 all -ii libnuma1:amd64 2.0.9~rc5-1ubuntu3.14.04.1 amd64 +ii libnss3:amd64 2:3.19.2.1-0ubuntu0.14.04.2 amd64 +ii libnss3-nssdb 2:3.19.2.1-0ubuntu0.14.04.2 all +ii libnuma1:amd64 2.0.9~rc5-1ubuntu3 amd64 -ii libpam-systemd:amd64 204-5ubuntu20.15 amd64 +ii libpam-systemd:amd64 204-5ubuntu20.10 amd64 -ii libparted0debian1:amd64 2.3-19ubuntu1.14.04.1 amd64 +ii libparted0debian1:amd64 2.3-19ubuntu1 amd64 -ii libpcre3:amd64 1:8.31-2ubuntu2.1 amd64 +ii libpcre3:amd64 1:8.31-2ubuntu2 amd64 -ii libpng12-0:amd64 1.2.50-1ubuntu2.14.04.1 amd64 +ii libpng12-0:amd64 1.2.50-1ubuntu2 amd64 -ii libpolkit-agent-1-0:amd64 0.105-4ubuntu3.14.04.1 amd64 -ii libpolkit-backend-1-0:amd64 0.105-4ubuntu3.14.04.1 amd64 -ii libpolkit-gobject-1-0:amd64 0.105-4ubuntu3.14.04.1 amd64 +ii libpolkit-agent-1-0:amd64 0.105-4ubuntu2 amd64 +ii libpolkit-backend-1-0:amd64 0.105-4ubuntu2 amd64 +ii libpolkit-gobject-1-0:amd64 0.105-4ubuntu2 amd64 -ii libpython2.7:amd64 2.7.6-8ubuntu0.2 amd64 -ii libpython2.7-minimal:amd64 2.7.6-8ubuntu0.2 amd64 -ii libpython2.7-stdlib:amd64 2.7.6-8ubuntu0.2 amd64 +ii libpython2.7:amd64 2.7.6-8 amd64 +ii libpython2.7-minimal:amd64 2.7.6-8 amd64 +ii libpython2.7-stdlib:amd64 2.7.6-8 amd64 -ii libpython3.4-minimal:amd64 3.4.3-1ubuntu1~14.04.3 amd64 -ii libpython3.4-stdlib:amd64 3.4.3-1ubuntu1~14.04.3 amd64 +ii libpython3.4-minimal:amd64 3.4.0-2ubuntu1 amd64 +ii libpython3.4-stdlib:amd64 3.4.0-2ubuntu1 amd64 -ii libsqlite3-0:amd64 3.8.2-1ubuntu2.1 amd64 -ii libss2:amd64 1.42.9-3ubuntu1.3 amd64 -ii libssl1.0.0:amd64 1.0.1f-1ubuntu2.16 amd64 +ii libsqlite3-0:amd64 3.8.2-1ubuntu2 amd64 +ii libss2:amd64 1.42.9-3ubuntu1.2 amd64 +ii libssl1.0.0:amd64 1.0.1f-1ubuntu2.11 amd64 -ii libsystemd-daemon0:amd64 204-5ubuntu20.15 amd64 -ii libsystemd-login0:amd64 204-5ubuntu20.15 amd64 -ii libtasn1-6:amd64 3.4-3ubuntu0.3 amd64 +ii libsystemd-daemon0:amd64 204-5ubuntu20.10 amd64 +ii libsystemd-login0:amd64 204-5ubuntu20.10 amd64 +ii libtasn1-6:amd64 3.4-3ubuntu0.1 amd64 -ii libudev1:amd64 204-5ubuntu20.15 amd64 +ii libudev1:amd64 204-5ubuntu20.10 amd64 -ii libuuid1:amd64 2.20.1-5.1ubuntu20.7 amd64 +ii libuuid1:amd64 2.20.1-5.1ubuntu20.4 amd64 -ii libxext6:amd64 2:1.3.2-1ubuntu0.0.14.04.1 amd64 +ii libxext6:amd64 2:1.3.2-1 amd64 -ii libxml2:amd64 2.9.1+dfsg1-3ubuntu4.6 amd64 +ii libxml2:amd64 2.9.1+dfsg1-3ubuntu4.4 amd64 -ii linux-headers-3.13.0-74 3.13.0-74.118 all -ii linux-headers-3.13.0-74-generic 3.13.0-74.118 amd64 -ii linux-headers-generic 3.13.0.74.80 amd64 -ii linux-headers-virtual 3.13.0.74.80 amd64 -ii linux-image-3.13.0-74-generic 3.13.0-74.118 amd64 -ii linux-image-extra-3.13.0-74-generic 3.13.0-74.118 amd64 -ii linux-image-virtual 3.13.0.74.80 amd64 +ii linux-headers-3.13.0-48 3.13.0-48.80 all +ii linux-headers-3.13.0-48-generic 3.13.0-48.80 amd64 +ii linux-headers-generic 3.13.0.48.55 amd64 +ii linux-headers-virtual 3.13.0.48.55 amd64 +ii linux-image-3.13.0-48-generic 3.13.0-48.80 amd64 +ii linux-image-extra-3.13.0-48-generic 3.13.0-48.80 amd64 +ii linux-image-virtual 3.13.0.48.55 amd64 -ii linux-virtual 3.13.0.74.80 amd64 +ii linux-virtual 3.13.0.48.55 amd64 -ii login 1:4.1.5.1-1ubuntu9.1 amd64 +ii login 1:4.1.5.1-1ubuntu9 amd64 -ii lshw 02.16-2ubuntu1.3 amd64 +ii lshw 02.16-2ubuntu1.2 amd64 -ii mount 2.20.1-5.1ubuntu20.7 amd64 +ii mount 2.20.1-5.1ubuntu20.4 amd64 -ii ntpdate 1:4.2.6.p5+dfsg-3ubuntu2.14.04.6 amd64 +ii ntpdate 1:4.2.6.p5+dfsg-3ubuntu2.14.04.2 amd64 -ii openssh-client 1:6.6p1-2ubuntu2.3 amd64 -ii openssh-server 1:6.6p1-2ubuntu2.3 amd64 -ii openssh-sftp-server 1:6.6p1-2ubuntu2.3 amd64 -ii openssl 1.0.1f-1ubuntu2.16 amd64 +ii openssh-client 1:6.6p1-2ubuntu2 amd64 +ii openssh-server 1:6.6p1-2ubuntu2 amd64 +ii openssh-sftp-server 1:6.6p1-2ubuntu2 amd64 +ii openssl 1.0.1f-1ubuntu2.11 amd64 -ii overlayroot 0.25ubuntu1.14.04.1 all -ii parted 2.3-19ubuntu1.14.04.1 amd64 -ii passwd 1:4.1.5.1-1ubuntu9.1 amd64 -ii patch 2.7.1-4ubuntu2.3 amd64 +ii overlayroot 0.25ubuntu1 all +ii parted 2.3-19ubuntu1 amd64 +ii passwd 1:4.1.5.1-1ubuntu9 amd64 +ii patch 2.7.1-4ubuntu1 amd64 -ii policykit-1 0.105-4ubuntu3.14.04.1 amd64 -ii pollinate 4.7-0ubuntu1.4 all +ii policykit-1 0.105-4ubuntu2 amd64 +ii pollinate 4.7-0ubuntu1.2 all -ii ppp 2.4.5-5.1ubuntu2.2 amd64 +ii ppp 2.4.5-5.1ubuntu2.1 amd64 -ii python-apt 0.9.3.5ubuntu2 amd64 -ii python-apt-common 0.9.3.5ubuntu2 all +ii python-apt 0.9.3.5ubuntu1 amd64 +ii python-apt-common 0.9.3.5ubuntu1 all -ii python-requests 2.2.1-1ubuntu0.3 all +ii python-requests 2.2.1-1ubuntu0.2 all -ii python-six 1.5.2-1ubuntu1 all +ii python-six 1.5.2-1 all -ii python-urllib3 1.7.1-1ubuntu4 all +ii python-urllib3 1.7.1-1ubuntu0.1 all -ii python2.7 2.7.6-8ubuntu0.2 amd64 -ii python2.7-minimal 2.7.6-8ubuntu0.2 amd64 +ii python2.7 2.7.6-8 amd64 +ii python2.7-minimal 2.7.6-8 amd64 -ii python3-apport 2.14.1-0ubuntu3.19 all -ii python3-apt 0.9.3.5ubuntu2 amd64 +ii python3-apport 2.14.1-0ubuntu3.8 all +ii python3-apt 0.9.3.5ubuntu1 amd64 -ii python3-distupgrade 1:0.220.8 all -ii python3-gdbm:amd64 3.4.3-1~14.04.2 amd64 +ii python3-distupgrade 1:0.220.7 all +ii python3-gdbm:amd64 3.4.0-0ubuntu1 amd64 -ii python3-problem-report 2.14.1-0ubuntu3.19 all +ii python3-problem-report 2.14.1-0ubuntu3.8 all -ii python3-software-properties 0.92.37.6 all -ii python3-update-manager 1:0.196.14 all -ii python3.4 3.4.3-1ubuntu1~14.04.3 amd64 -ii python3.4-minimal 3.4.3-1ubuntu1~14.04.3 amd64 +ii python3-software-properties 0.92.37.3 all +ii python3-update-manager 1:0.196.12 all +ii python3.4 3.4.0-2ubuntu1 amd64 +ii python3.4-minimal 3.4.0-2ubuntu1 amd64 -ii software-properties-common 0.92.37.6 all +ii software-properties-common 0.92.37.3 all -ii sudo 1.8.9p5-1ubuntu1.2 amd64 -ii systemd-services 204-5ubuntu20.15 amd64 +ii sudo 1.8.9p5-1ubuntu1.1 amd64 +ii systemd-services 204-5ubuntu20.10 amd64 -ii sysv-rc 2.88dsf-41ubuntu6.2 all -ii sysvinit-utils 2.88dsf-41ubuntu6.2 amd64 +ii sysv-rc 2.88dsf-41ubuntu6.1 all +ii sysvinit-utils 2.88dsf-41ubuntu6.1 amd64 -ii tcpdump 4.5.1-2ubuntu1.2 amd64 +ii tcpdump 4.5.1-2ubuntu1.1 amd64 -ii tzdata 2015g-0ubuntu0.14.04 all +ii tzdata 2015b-0ubuntu0.14.04 all -ii ubuntu-release-upgrader-core 1:0.220.8 all +ii ubuntu-release-upgrader-core 1:0.220.7 all -ii udev 204-5ubuntu20.15 amd64 +ii udev 204-5ubuntu20.10 amd64 -ii unattended-upgrades 0.82.1ubuntu2.4 all +ii unattended-upgrades 0.82.1ubuntu2.2 all -ii update-manager-core 1:0.196.14 all +ii update-manager-core 1:0.196.12 all -ii util-linux 2.20.1-5.1ubuntu20.7 amd64 -ii uuid-runtime 2.20.1-5.1ubuntu20.7 amd64 +ii util-linux 2.20.1-5.1ubuntu20.4 amd64 +ii uuid-runtime 2.20.1-5.1ubuntu20.4 amd64 ubuntu$ cat /etc/apt/sources.list # See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to # newer versions of the distribution. -deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty main restricted -deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty main restricted +deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty main +deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty main ## Major bug fix updates produced after the final release of the ## distribution. -deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates main restricted -deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates main restricted +deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates main +deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates main ## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu ## team. Also, please note that software in universe WILL NOT receive any @@ -30,10 +31,10 @@ ## your rights to use the software. Also, please note that software in ## multiverse WILL NOT receive any review or updates from the Ubuntu ## security team. -deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty multiverse -deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty multiverse -deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates multiverse -deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates multiverse +# deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty multiverse +# deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty multiverse +# deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates multiverse +# deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates multiverse ## Uncomment the following two lines to add software from the 'backports' ## repository. @@ -42,8 +43,8 @@ ## newer versions of some applications which may provide useful features. ## Also, please note that software in backports WILL NOT receive any review ## or updates from the Ubuntu security team. -deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse -deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse +# deb http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse +# deb-src http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse so basically, the working AMI has slightly *older* versions of just about everything, since it is based on an Ubuntu AMI from 2015-03-25, whereas the non-working AMI is based on an Ubuntu AMI from 2015-12-08. One of those "updates" -- either the kernel or one of the packages -- breaks docker. I think we can live with that -- it's no different than our hold at Docker-1.6.1, really, except we don't know exactly which upgrade breaks it. When we come to wanting to upgrade the AMI, we can either do some further experimentation to figure out exactly which package hurts, or maybe we can just leave the m1.medium instances running this way.
|
|
Assignee | |
Comment 31•8 years ago
|
||
I'm going to try this one more time, without the sources.list hackery. Barring that, I think we just need to remember that March 2015 was a good month for AMIs, and maybe AMIs after that just didn't taste the same. base: --> hvm-builder: AMIs were created: us-west-2: ami-b63822d7 --> pv-builder: AMIs were created: us-west-2: ami-ed051f8c app: --> hvm-builder: AMIs were created: us-east-1: ami-f62d779c us-west-1: ami-66d8ad06 us-west-2: ami-c83923a9 --> pv-builder: AMIs were created: us-east-1: ami-dd2379b7 us-west-1: ami-01dfaa61 us-west-2: ami-5e3a203f I'll wait until the try jobs have finished before swapping that in, though.
|
|
||
Comment 32•8 years ago
|
||
Review commit: https://reviewboard.mozilla.org/r/30757/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/30757/
Attachment #8707563 -
Flags: review?(jmaher)
|
|
Assignee | |
Comment 33•8 years ago
|
||
That test job was successful too, so I'm going to leave desktop-test-m1 configured with the AMIs in comment 31 and put it up for review.
|
||
Comment 34•8 years ago
|
||
Comment on attachment 8707563 [details] MozReview Request: Bug 1237663 - Switch TC Linux64 debug jobs from m3.large 1cpu to m1.medium (1cpu). r=jmaher https://reviewboard.mozilla.org/r/30757/#review27583 woohoo
Attachment #8707563 -
Flags: review?(jmaher) → review+
|
|
Assignee | |
Comment 35•8 years ago
|
||
Attachment #8707568 -
Flags: review?(garndt)
|
||
Comment 36•8 years ago
|
||
Comment on attachment 8707568 [details] [review] https://github.com/taskcluster/docker-worker/pull/205 merged https://github.com/taskcluster/docker-worker/commit/ac91665d4040b3bc05fd56d2d7d0d6dde346a131
Attachment #8707568 -
Flags: review?(garndt) → review+
|
|
Assignee | |
Comment 37•8 years ago
|
||
Comment on attachment 8707563 [details] MozReview Request: Bug 1237663 - Switch TC Linux64 debug jobs from m3.large 1cpu to m1.medium (1cpu). r=jmaher This doesn't need to be landed -- I just changed desktop-test to use m1.mediums.
|
|
Assignee | |
Updated•8 years ago
|
Status: REOPENED → RESOLVED
Closed: 8 years ago → 8 years ago
Resolution: --- → FIXED
|
||
Comment 38•8 years ago
|
||
(In reply to Dustin J. Mitchell [:dustin] from comment #31) > Barring that, I think we just need to remember that March 2015 was a good > month for AMIs, and maybe AMIs after that just didn't taste the same. I literally LOLed at this. :)
|
||
Updated•5 years ago
|
Component: Integration → Services
You need to log in
before you can comment on or make changes to this bug.
Description
•