Upgrade docker version on NSS worker
Categories
(Infrastructure & Operations :: RelOps: Posix OS, enhancement)
Tracking
(Not tracked)
People
(Reporter: franziskus, Assigned: dhouse)
References
Details
NSS uses a rather old image_builder that needs updating. Unfortunately, the host docker version is too old to do that. Can we upgrade the docker-worker that's used by NSS to get a newer docker version on the host?
+ docker version
Client version: 1.6.1
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 97cd073
OS/Arch (client): linux/amd64
Server version: 1.6.1
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 97cd073
OS/Arch (server): linux/amd64
|
||
Updated•5 years ago
|
|
||
Comment 1•5 years ago
|
||
What's the workerpool used by NSS? Does it use a custom AMI or it is the same used by other workers?
|
Reporter | |
Comment 2•5 years ago
|
||
The image builder used to be like this [1]. Not sure how/if that changed with the TC move.
[1] https://tools.taskcluster.net/groups/AtWSaj_jR1qcHimiVMPdTQ/tasks/btepVhghQ2SKytOzujU0RQ/details
|
|
||
Comment 3•5 years ago
|
||
I suspect the underlying docker version on this worker pool has not changed. In the old deployment this was "hg-worker' which was a standard docker-worker workerType.
Relops: I think the first thing to do here is to use a monopacker-based docker-worker image for this project, which is based on a newer Ubuntu. I think that ships with a newer docker, but if not then the next step will be to upgrade docker in that image.
I'm not certain what needs to be done for this. I'll collect some notes:
the workerTypes using this are nss-1/linux, nss-3/linux
https://firefox-ci-tc.services.mozilla.com/worker-manager/nss-1%2Flinux from:
https://hg.mozilla.org/ci/ci-configuration/file/tip/worker-pools.yml#l1464
image: docker-worker-hvm-builder-current
If I understand correctly, the earlier link https://tools.taskcluster.net/groups/AtWSaj_jR1qcHimiVMPdTQ/tasks/btepVhghQ2SKytOzujU0RQ/details, shows how the image for these workers was built previously. And according to Dustin's note, we could switch the nss worker to use an image created from current mono-packer if it has a new enough docker in it.
So, It looks like we can do this:
- check the docker version in the current docker-worker image (we'll have to find that. maybe it is in a repo)
2a. If it is new enough, change the configuration of the workertype to use this image (I think the ci-config workertype entries are where we'd set the image).
2b. If not new enough, we need to create a new docker-worker image with a newer version of docker (need to find where the reqs for that image are, change them, build a new image, and test+switch nss to use that).
Hi Wander, what version of docker is on the docker-worker ami and where can I find that in a repo? I see the docker-worker repo, but I don't see a mono-packer yaml there and I don't know where to look next.
|
|
Reporter | |
Comment 7•5 years ago
|
||
If I understand correctly, the earlier link https://tools.taskcluster.net/groups/AtWSaj_jR1qcHimiVMPdTQ/tasks/btepVhghQ2SKytOzujU0RQ/details, shows how the image for these workers was built previously.
This is how currently images for NSS are built. The docker image used for this (the image builder image) is pretty old and requires a newer docker version (which needs to be supported by the host).
|
|
||
Comment 8•5 years ago
|
||
(In reply to Dave House [:dhouse] from comment #6)
Hi Wander, what version of docker is on the docker-worker ami and where can I find that in a repo? I see the docker-worker repo, but I don't see a mono-packer yaml there and I don't know where to look next.
It is here. Notice that it is the latest version available to Ubuntu 14.04.
(In reply to Wander Lairson Costa [:wcosta] from comment #8)
(In reply to Dave House [:dhouse] from comment #6)
Hi Wander, what version of docker is on the docker-worker ami and where can I find that in a repo? I see the docker-worker repo, but I don't see a mono-packer yaml there and I don't know where to look next.
It is here. Notice that it is the latest version available to Ubuntu 14.04.
Thank you Wander!
|
Assignee | |
Comment 10•5 years ago
|
||
(In reply to Franziskus Kiefer [:franziskus] from comment #0)
NSS uses a rather old
image_builderthat needs updating. Unfortunately, the host docker version is too old to do that. Can we upgrade the docker-worker that's used by NSS to get a newer docker version on the host?+ docker version Client version: 1.6.1
:franziskus, might this be a new enough version?
DOCKER_VERSION=18.06.3ce3-0~ubuntu (https://github.com/taskcluster/docker-worker/blob/master/deploy/packer/base/scripts/packages.sh#L5)
This version looks newer than above (1.6.1).
|
|
Reporter | |
Comment 11•5 years ago
|
||
I think that's new enough. I need --build-args on docker builds in particular, which was added in 1.9.
|
|
Assignee | |
Comment 12•5 years ago
|
||
(In reply to Dave House [:dhouse] from comment #5)
2a. If it is new enough, change the configuration of the workertype to use this image (I think the ci-config workertype entries are where we'd set the image).
Great. So the next step is for us to change the workerType to use this docker-worker image.
|
|
Assignee | |
Comment 13•5 years ago
|
||
The current nss-1/linux is using:
https://hg.mozilla.org/ci/ci-configuration/file/tip/worker-images.yml#l55
docker-worker-v201911141441-hvm-builder:
aws:
eu-central-1: ami-00e8bd2e0ef11bc1d
us-east-1: ami-0ad7b98255befdcbf
us-west-1: ami-0ee7ee2a00fafe982
us-west-2: ami-06d68591fefdc678b
The monopacker docker-worker entry in ci-configuration worker-image.yml is:
https://hg.mozilla.org/ci/ci-configuration/file/tip/worker-images.yml#l68
# monopacker docker-worker images
# docker-worker version v201911141441
monopacker-docker-2019-11-18t20-32-23z:
aws:
us-east-1: ami-0175f02c7d8d75500
us-west-1: ami-0a8d4a07bbb687d8d
us-west-2: ami-0501a72ca38da7e80
fxci-level1-gcp:
projects/taskcluster-imaging/global/images/docker-worker-gcp-googlecompute-2019-11-18t20-32-23z
fxci-level3-gcp:
projects/taskcluster-imaging/global/images/docker-worker-gcp-googlecompute-2019-11-18t20-32-23z
fxci-levelt-gcp:
projects/taskcluster-imaging/global/images/docker-worker-gcp-googlecompute-2019-11-18t20-32-23z
|
|
Assignee | |
Comment 14•5 years ago
•
|
||
This looks like it is the same version of docker-worker that is currently running:
# docker-worker version v201911141441
|
|
Assignee | |
Comment 15•5 years ago
|
||
(In reply to Dave House [:dhouse] from comment #14)
This looks like it is the same version of docker-worker that is currently running:
# docker-worker version v201911141441
Wander, is the ami for "docker-worker-v201911141441-hvm-builder" the same as the monopacker one, or does it include the same version of docker?
:franziskus, it looks like nss-*/linux images were updated since this bug was created. Could you check if they currently have the right version of docker?
|
|
Reporter | |
Comment 16•5 years ago
|
||
:franziskus, it looks like nss-*/linux images were updated since this bug was created. Could you check if they currently have the right version of docker?
It appears that nothing changed for the NSS image builder: https://firefox-ci-tc.services.mozilla.com/tasks/PizZOK0GT_CBEpaIWQuo3g
|
|
||
Comment 17•5 years ago
|
||
(In reply to Dave House [:dhouse] from comment #15)
(In reply to Dave House [:dhouse] from comment #14)
This looks like it is the same version of docker-worker that is currently running:
# docker-worker version v201911141441Wander, is the ami for "docker-worker-v201911141441-hvm-builder" the same as the monopacker one, or does it include the same version of docker?
Nope, it is built using docker-worker scripts for now. :miles is working on migrating to monopacker.
|
|
Assignee | |
Comment 18•5 years ago
|
||
(In reply to Franziskus Kiefer [:franziskus] from comment #16)
:franziskus, it looks like nss-*/linux images were updated since this bug was created. Could you check if they currently have the right version of docker?
It appears that nothing changed for the NSS image builder: https://firefox-ci-tc.services.mozilla.com/tasks/PizZOK0GT_CBEpaIWQuo3g
It isn't a new enough version after all then.
+ docker version
Client version: 1.6.1
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 97cd073
OS/Arch (client): linux/amd64
Server version: 1.6.1
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 97cd073
OS/Arch (server): linux/amd64
|
|
Assignee | |
Comment 19•5 years ago
|
||
Miles, is the docker-worker image from monopacker ready for use (https://hg.mozilla.org/ci/ci-configuration/file/tip/worker-images.yml#l68 "monopacker-docker-2019-11-18t20-32-23z")?
It looks like it is running ubuntu 18.04/bionic and would have a new enough docker-ce cli for NSS's needed '--build-arg' argument usage:
https://github.com/taskcluster/monopacker/blob/master/builders/docker_worker_aws.yaml
|
|
Assignee | |
Comment 20•5 years ago
|
||
(In reply to Franziskus Kiefer [:franziskus] from comment #16)
:franziskus, it looks like nss-*/linux images were updated since this bug was created. Could you check if they currently have the right version of docker?
It appears that nothing changed for the NSS image builder: https://firefox-ci-tc.services.mozilla.com/tasks/PizZOK0GT_CBEpaIWQuo3g
:franziskus, could you work-around this with ENV entries or a file instead of ARG+build-args until we get the image updated? There may be other changes to the image that need to be tested, and so I do not know how long this update will take.
(I looked in the nss and nss-try repos and I don't see ARG usage in the Dockerfiles yet: https://hg.mozilla.org/projects/nss-try/file/acb6a0f742321e33669c46875f8f590b3f68aab2/automation/taskcluster/image_builder/Dockerfile)
|
|
Reporter | |
Comment 21•5 years ago
|
||
:franziskus, could you work-around this with ENV entries or a file instead of ARG+build-args until we get the image updated? There may be other changes to the image that need to be tested, and so I do not know how long this update will take.
This is an external docker image. I could try to get something changed upstream for it but was hoping I don't have to go that route...
(I looked in the nss and nss-try repos and I don't see ARG usage in the Dockerfiles yet: https://hg.mozilla.org/projects/nss-try/file/acb6a0f742321e33669c46875f8f590b3f68aab2/automation/taskcluster/image_builder/Dockerfile)
Note that this is the image builder and not the one that requires --build-arg. That's required when building the actual docker image (on the try runs hard-coded here https://hg.mozilla.org/projects/nss-try/file/02837eea37c51d02b5e226a13184238cf379dcfa/automation/taskcluster/scripts/build_image.sh)
|
|
Assignee | |
Comment 22•5 years ago
•
|
||
(In reply to Franziskus Kiefer [:franziskus] from comment #21)
:franziskus, could you work-around this with ENV entries or a file instead of ARG+build-args until we get the image updated? There may be other changes to the image that need to be tested, and so I do not know how long this update will take.
This is an external docker image. I could try to get something changed upstream for it but was hoping I don't have to go that route...
(I looked in the nss and nss-try repos and I don't see ARG usage in the Dockerfiles yet: https://hg.mozilla.org/projects/nss-try/file/acb6a0f742321e33669c46875f8f590b3f68aab2/automation/taskcluster/image_builder/Dockerfile)
Note that this is the image builder and not the one that requires
--build-arg. That's required when building the actual docker image (on the try runs hard-coded here https://hg.mozilla.org/projects/nss-try/file/02837eea37c51d02b5e226a13184238cf379dcfa/automation/taskcluster/scripts/build_image.sh)
I understand. The development branch Dockerfile in hacl-star has a number of ARG entries, and looks like it could be replaced/hard-coded for a task run but that might be fragile unless you're pinning to a specific version (https://github.com/project-everest/hacl-star/blob/fstar-master/.docker/build/linux/Dockerfile).
|
|
Assignee | |
Comment 23•5 years ago
|
||
(In reply to Wander Lairson Costa [:wcosta] from comment #17)
(In reply to Dave House [:dhouse] from comment #15)
(In reply to Dave House [:dhouse] from comment #14)
This looks like it is the same version of docker-worker that is currently running:
# docker-worker version v201911141441Wander, is the ami for "docker-worker-v201911141441-hvm-builder" the same as the monopacker one, or does it include the same version of docker?
Nope, it is built using docker-worker scripts for now. :miles is working on migrating to monopacker.
Hi Wander, could we use the docker-worker scripts to make a new version of docker-worker or do we need to wait and build it on monopacker? (thinking I could back-port a docker release if there isn't one available for trusty)
|
|
||
Comment 24•5 years ago
|
||
(In reply to Dave House [:dhouse] from comment #23)
(In reply to Wander Lairson Costa [:wcosta] from comment #17)
(In reply to Dave House [:dhouse] from comment #15)
(In reply to Dave House [:dhouse] from comment #14)
This looks like it is the same version of docker-worker that is currently running:
# docker-worker version v201911141441Wander, is the ami for "docker-worker-v201911141441-hvm-builder" the same as the monopacker one, or does it include the same version of docker?
Nope, it is built using docker-worker scripts for now. :miles is working on migrating to monopacker.
Hi Wander, could we use the docker-worker scripts to make a new version of docker-worker or do we need to wait and build it on monopacker? (thinking I could back-port a docker release if there isn't one available for trusty)
I believe you can use monopacker, which is far easier to hack on. The only reason monopacker isn't being used yet is because some tasks only work with Ubuntu 14.04, and monopacker currently only provide Ubuntu 18.04 images. NI :miles to confirm it.
|
||
Comment 25•5 years ago
|
||
Sorry for missing this bug!
(In reply to Dave House [:dhouse] from comment #19)
Miles, is the docker-worker image from monopacker ready for use (https://hg.mozilla.org/ci/ci-configuration/file/tip/worker-images.yml#l68 "monopacker-docker-2019-11-18t20-32-23z")?
It looks like it is running ubuntu 18.04/bionic and would have a new enough docker-ce cli for NSS's needed '--build-arg' argument usage:
https://github.com/taskcluster/monopacker/blob/master/builders/docker_worker_aws.yaml
That's correct re: docker-ce version. That image isn't properly using aws-provider, so it shouldn't be used. We're testing an updated AMI for metal instances.
I've baked a new AMI that should work:
us-east-1: ami-0d8feb7ada717584e
us-west-1: ami-03f559692ea59788c
us-west-2: ami-01fae62d6a7b97e67
I've been waiting to test before landing those in ci-config but have been sidetracked on other things. I've tested a similarly baked image in a dev environment, though.
re: baking yourself, the blockers there would be access to secrets and to the production AWS account - we have the relevant secrets in the taskcluster password store and could share them if you want to trying hacking on images yourself.
|
|
Assignee | |
Comment 26•5 years ago
|
||
(In reply to Miles Crabill [:miles] [also mcrabill@mozilla.com] from comment #25)
Sorry for missing this bug!
(In reply to Dave House [:dhouse] from comment #19)
Miles, is the docker-worker image from monopacker ready for use (https://hg.mozilla.org/ci/ci-configuration/file/tip/worker-images.yml#l68 "monopacker-docker-2019-11-18t20-32-23z")?
It looks like it is running ubuntu 18.04/bionic and would have a new enough docker-ce cli for NSS's needed '--build-arg' argument usage:
https://github.com/taskcluster/monopacker/blob/master/builders/docker_worker_aws.yamlThat's correct re:
docker-ceversion. That image isn't properly usingaws-provider, so it shouldn't be used. We're testing an updated AMI for metal instances.I've baked a new AMI that should work:
us-east-1: ami-0d8feb7ada717584e us-west-1: ami-03f559692ea59788c us-west-2: ami-01fae62d6a7b97e67I've been waiting to test before landing those in ci-config but have been sidetracked on other things. I've tested a similarly baked image in a dev environment, though.
re: baking yourself, the blockers there would be access to secrets and to the production AWS account - we have the relevant secrets in the taskcluster password store and could share them if you want to trying hacking on images yourself.
Miles/Wander, could you set this up for NSS and show me how? (Or if it takes longer to show me, then maybe show me next time to get this one completed at high priority?)
|
|
||
Comment 27•5 years ago
|
||
We are working on a new release of monopacker images. After that, we coordinate for updating nss.
|
|
Assignee | |
Comment 28•5 years ago
|
||
(In reply to Wander Lairson Costa [:wcosta] from comment #27)
We are working on a new release of monopacker images. After that, we coordinate for updating nss.
Thanks! When can we plan for it to be ready?
|
|
Assignee | |
Comment 29•5 years ago
|
||
:franziskus, is this blocking your work currently? Do you have a workaround for now if the new release of the new image(monopacker) is not soon (like in January)?
|
|
Reporter | |
Comment 30•5 years ago
|
||
I work around this for now and update NSS once the images are upgraded.
|
|
Assignee | |
Comment 31•5 years ago
|
||
(In reply to Franziskus Kiefer [:franziskus] from comment #30)
I work around this for now and update NSS once the images are upgraded.
Sounds good. Thank you
|
|
||
Updated•5 years ago
|
|
||
Comment 32•3 years ago
|
||
Marking inactive per bugscrub
Description
•