1529304 - Provide a worker with GPU (CUDA) support

Status: RESOLVED FIXED

(Taskcluster :: Workers, enhancement)

Description

[Marco Castelluccio [:marco]]

It'd be great if it was possible to run tasks for ML workloads using CUDA.

Amazon provides pre-build AMIs, e.g. https://aws.amazon.com/marketplace/pp/B077GCH38C, but looking at the nvidia-docker documentation, it looks like it would be possible to do this without a custom AMI too: https://github.com/NVIDIA/nvidia-docker/wiki/Deploy-on-Amazon-EC2.

Comment 1

[Marco Castelluccio [:marco]]

:coop, could someone from your team investigate if this is easily/not easily feasible?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I think Alexandre is working on this very thing -- perhaps he can help?

Comment 3

[:gerard-majax]

I can't help, but I'm interested. In my case, this is needed to run under linux, and the last time we worked on that with :wcosta, the issue was not being able to get NVIDIA-Docker to work on the AMI to expose the device in the docker-worker container. This was maybe one and a half year ago?

Comment 4

[Chris Cooper [:coop] (he/him)]

The Taskcluster team is extremely constrained for resources right now. Part of that is due to a migration of some AWS workloads to GCE. We can commit a small amount of time to reviews if someone else wants to take a stab at this, either in AWS or GCE (https://cloud.google.com/compute/docs/gpus/add-gpus).

We can help, but likely not until Q3.

Comment 5

[Marco Castelluccio [:marco]]

(In reply to Chris Cooper [:coop] pronoun: he from comment #4)

The Taskcluster team is extremely constrained for resources right now. Part of that is due to a migration of some AWS workloads to GCE. We can commit a small amount of time to reviews if someone else wants to take a stab at this, either in AWS or GCE (https://cloud.google.com/compute/docs/gpus/add-gpus).

We can help, but likely not until Q3.

If we wanted to help, do you have an idea of what this would entail? How long would it take for someone outside of the Taskcluster team?

Comment 6

[Marco Castelluccio [:marco]]

Needinfo for comment 5. We have a new contractor starting soon that might help with this, depending on the amount of work that would be required.

Comment 7

[Chris Cooper [:coop] (he/him)]

(In reply to Marco Castelluccio [:marco] from comment #6)

Needinfo for comment 5. We have a new contractor starting soon that might help with this, depending on the amount of work that would be required.

Redirecting the NI to :wcosta, but my quick take is that it will take someone to play with the docker image to get something a) working, and b) stable enough that it won't be a support nightmare. Per comment #3, maybe NVIDIA-Docker is in a more stable place after 18 months?

Comment 8

[:gerard-majax]

As much as I recall from the state back then, problem was just that NVIDIA-Docker required a more recent version of docker, and that it was unsure what could be the fallout / regressions related for docker-worker. Plus the need to produce a specific AMI to integrate NVIDIA-Docker.

Comment 9

[Wander Lairson Costa]

(In reply to Alexandre LISSY :gerard-majax from comment #8)

As much as I recall from the state back then, problem was just that
NVIDIA-Docker required a more recent version of docker, and that it was
unsure what could be the fallout / regressions related for docker-worker.
Plus the need to produce a specific AMI to integrate NVIDIA-Docker.

I am currently working on upgrading EC2 image to Linux 18.04 and docker 18.09.3. Hopefully this would help to get nvidia-docker running.

Comment 10

[Chris Cooper [:coop] (he/him)]

We're making some improvements to the docker-worker deployment process, including being able to update the docker version. This may be possible soon.

Comment 11

[:gerard-majax]

Is it something we could have now ?

Comment 12

[Wander Lairson Costa]

(In reply to Alexandre LISSY :gerard-majax from comment #11)

Is it something we could have now ?

It is possible to do it now in monorepo. I can do it after baremetal migration.

Comment 13

[:gerard-majax]

(In reply to Wander Lairson Costa from comment #12)

(In reply to Alexandre LISSY :gerard-majax from comment #11)

Is it something we could have now ?

It is possible to do it now in monorepo. I can do it after baremetal migration.

Cool, do you have any idea of the ETA for that ? I have no idea of how much work this requires.

Comment 14

[Wander Lairson Costa]

(In reply to Alexandre LISSY :gerard-majax from comment #13)

(In reply to Wander Lairson Costa from comment #12)

(In reply to Alexandre LISSY :gerard-majax from comment #11)

Is it something we could have now ?

It is possible to do it now in monorepo. I can do it after baremetal migration.

Cool, do you have any idea of the ETA for that ? I have no idea of how much work this requires.

ETA is end of second quarter.

Comment 15

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1529304 - Support translating 'guestAccelerators.acceleratorType' in ci-admin, r?#releng-reviewers!

We want to start supporting GPU enabled pools in GCP for machine learning. This
will allow us to attach the GPUs from the same zone as the instance type.

Comment 16

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1529304 - Add an experimental pool with 4 nvidia v100 GPUs, r?masterwayz!

Depends on D172451

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/e536e2a60c8c
Support translating 'guestAccelerators.acceleratorType' in ci-admin, r=releng-reviewers,gabriel

Comment 18

[Andrew Erickson [:aerickson]]

Attachment: Bug 1529304: add translations image, provider, and workerType

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/cf3796a301f0
add translations image, provider, and workerType r=releng-reviewers,bhearsum

Comment 20

[Andrew Erickson [:aerickson]]

Attachment: Bug 1529304: add translations image, provider, and workerType. round 2 r=bhearsum

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/ace1c740c517
add translations image, provider, and workerType. round 2 r=bhearsum,releng-reviewers

Comment 22

[Andrew Erickson [:aerickson]]

Attachment: Bug 1529304: translations gpu workers: relax cpu platform restriction r=bhearsum

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/085da960ddde
translations gpu workers: relax cpu platform restriction r=bhearsum,releng-reviewers

Comment 24

[Andrew Erickson [:aerickson]]

Attachment: Bug 1529304: reduce instance size to work with v100 gpu r=bhearsum

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/9d5290f431b3
reduce instance size to work with v100 gpu r=bhearsum,releng-reviewers

Comment 26

[Andrew Erickson [:aerickson]]

Attachment: Bug 1529304: add g-w linux implementation, use for translations gpu workers r=bhearsum

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/ee2e4aa3687f
add g-w linux implementation, use for translations gpu workers r=bhearsum,releng-reviewers

Comment 28

[Andrew Erickson [:aerickson]]

Attachment: Bug 1529304: use newer translations image with fixes, r=bhearsum

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/e5364c707d28
use newer translations image with fixes, r=bhearsum

Comment 30

[Andrew Erickson [:aerickson]]

Attachment: Bug 1529304: use newer translation image round 2, r=bhearsum

Fixes missing zstandard (zstd != zstandard).

See https://github.com/taskcluster/monopacker/pull/106 for details about the changes to the image.

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/77ba0dc95bca
use newer translation image round 2, r=bhearsum,releng-reviewers

Comment 32

[Andrew Erickson [:aerickson]]

Attachment: Bug 1529304: update translations image r3, r=bhearsum

See details of change at https://github.com/taskcluster/monopacker/pull/106.

Comment 33

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/93dfe3a710dd
update translations image r3, r=bhearsum,releng-reviewers

Comment 34

[[e-mail] (:bhearsum)]

Attachment: Bug 1529304: Overhaul Firefox Translations workers r?aerickson!,gabriel!

A handful of changes here:

• Drop the unused b-linux-gcp and t-linux-gcp worker pools.
• Rename GPU workers to b-linux style; drop experimental comment
• Drop the seemingly unused scratch disks from GPU workers
• Add permanent 300gb variant for CPU & GPU workers
• Add permanent 1tb variant for GPU workers
• Add some temporary pools to experiment with other GPU types: p100, a100, l4
  • Note that the a100 and l4 GPUs are only available on certain instance types (per https://cloud.google.com/compute/docs/gpus/#l4-gpus). My understanding is that they should be there without being requested explicitly (unlike the v100 and p100s).

The extra disk space variants are required for certain steps that consume and/or output huge amounts of data in full training runs.

Eventually we'll settle on 4 GPU worker pool:

• A fairly low powered one for the steps aren't terribly GPU intensive (this will probably be the existing v100 x 1 - but we'll see)
• 75gb, 300gb, and 1TB variants of a higher powered one that gives us the best bang for our buck, determined by the results from the temporary pools above.

For now, I'm adding 300gb/1tb versions of the v100 x 4 pools to unblock production training while experiments happen.

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/f00cffd1db70
Overhaul Firefox Translations workers r=aerickson

Comment 36

[[e-mail] (:bhearsum)]

Attachment: Bug 1529304: use translations images in the taskcluster-imaging project, to work around permissions issues with the translations-sandbox project. r?aerickson!

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/8996b09fb238
use translations images in the taskcluster-imaging project, to work around permissions issues with the translations-sandbox project. r=aerickson

Comment 38

[[e-mail] (:bhearsum)]

Attachment: Bug 1529304: use a very short idleTimeout for GPU workers r?aerickson!

Comment 39

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/b5cd715b922c
use a very short idleTimeout for GPU workers r=aerickson

Comment 40

[[e-mail] (:bhearsum)]

For lack of a better place to put it, here's a quick write-up on some experimentation I did with different instance/GPU types. I tried running the same training with the following configurations:

• v100 x 1
• p100 x 4
• a100 x 1
• l4 x 1

The a100 was by far the most expensive option, and also seems to have the most limited availability.

The l4 was slightly cheaper than v100s and p100s, but also took nearly double the amount of time to train.

The v100s and p100s similar in price - with the v100s coming out slightly lower.

With all of this is mind, I'm going to stick with v100s for now, and do some additional experiments to see the relative efficiency of 4, 8, and 16 GPUs attached to the same instance.

Comment 41

[[e-mail] (:bhearsum)]

Attachment: Bug 1529304: update translations worker pools for second round of GPU experiements r?#releng-reviewers!

First pass showed that v100s are the best bet. This round will test training with 4, 8, and 16 of these GPUs.

(This diff doesn't read terribly well - this patch is simply removing the a100/p100/l4 instances, and adding a 8 & 16 GPU v100 instances.)

Comment 42

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/46f7dbec4258
update translations worker pools for second round of GPU experiements r=releng-reviewers,gbrown

Comment 43

[[e-mail] (:bhearsum)]

(In reply to [e-mail] (:bhearsum) from comment #40)

For lack of a better place to put it, here's a quick write-up on some experimentation I did with different instance/GPU types. I tried running the same training with the following configurations:

• v100 x 1
• p100 x 4
• a100 x 1
• l4 x 1

The a100 was by far the most expensive option, and also seems to have the most limited availability.

The l4 was slightly cheaper than v100s and p100s, but also took nearly double the amount of time to train.

The v100s and p100s similar in price - with the v100s coming out slightly lower.

With all of this is mind, I'm going to stick with v100s for now, and do some additional experiments to see the relative efficiency of 4, 8, and 16 GPUs attached to the same instance.

It turns out that the max GPUs for v100s is 8. If we're not satisfied with this, the A100s are the next thing to try. These are both faster per GPU, and we have the possibility of using 16 GPUs/instance.

Comment 44

[[e-mail] (:bhearsum)]

(In reply to [e-mail] (:bhearsum) from comment #43)

(In reply to [e-mail] (:bhearsum) from comment #40)

For lack of a better place to put it, here's a quick write-up on some experimentation I did with different instance/GPU types. I tried running the same training with the following configurations:

• v100 x 1
• p100 x 4
• a100 x 1
• l4 x 1

The a100 was by far the most expensive option, and also seems to have the most limited availability.

The l4 was slightly cheaper than v100s and p100s, but also took nearly double the amount of time to train.

The v100s and p100s similar in price - with the v100s coming out slightly lower.

With all of this is mind, I'm going to stick with v100s for now, and do some additional experiments to see the relative efficiency of 4, 8, and 16 GPUs attached to the same instance.

It turns out that the max GPUs for v100s is 8. If we're not satisfied with this, the A100s are the next thing to try. These are both faster per GPU, and we have the possibility of using 16 GPUs/instance.

I did some comparisons with 4 & 8 v100s. Somewhat surprisingly, the 4 GPU instances completed their training tasks nearly as fast as the 8 GPU instances. I did this test four times with the exact same inputs, with these results:

• 4 x v100: 23m, 28m, 23m, 26m
• 8 x v100: 22m, 20m, 21m, 23m

The best 8 GPU run is obviously much faster than the worst 4 GPU run - but on average it was only ~15% faster. Given this, I'm going to stick with 4 GPU instances for the moment, as it's difficult to justify doubling or GPU cost for a relative minor gain.

I would bet that there are things we can tweak in the training config or pipeline that will make better use of the available GPUs (or perhaps updating to the latest marian version?) - so we should revisit this in the future.

Comment 45

[[e-mail] (:bhearsum)]

Attachment: Bug 1529304: remove 8 and 16 GPU instances that we will not be using r?#releng-reviewers!

Comment 46

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/b9b18418387d
remove 8 and 16 GPU instances that we will not be using r=releng-reviewers,gbrown

Comment 47

[[e-mail] (:bhearsum)]

We're now at a point where we can run the entire training pipeline in Taskcluster, and have settled on our initial types of instances. I think this bug has outlive its usefullness - we can file specific follow-ups if we need future changes to the VM image or workers.

Thanks to ahal for adding GPU support to ci-config, and aerickson for all the work on the new VM image.