1435058 - Add cltbld to kvm group on linux talos machines

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Geoff Brown [:gbrown]]

On an experimental basis, I'm trying to run the android emulator on the taskcluster worker type "gecko-t-linux-talos", the new linux talos "moonshot" machines.

I'd like to use kvm acceleration for the emulator, but the emulator complains that it cannot read /dev/kvm, when running as use cltbld.

I see:

crw-rw---- 1 root kvm 10, 232 Jan 29 11:31 /dev/kvm

and 

uid=1000(cltbld) gid=1000(cltbld) groups=1000(cltbld),29(audio),44(video)

so I suspect this will work if cltbld is added to the kvm group.

Comment 1

[:dhouse]

The group kvm owns /dev/kvm on the ubuntu16 moonshots:
```
crw-rw---- 1 root kvm 10, 232 Jan 25 03:08 /dev/kvm
```

Testing on Joel's loaner, #003:
```
[root@t-linux64-ms-003 ~]# id cltbld
uid=1000(cltbld) gid=1000(cltbld) groups=1000(cltbld),29(audio),44(video),114(kvm)
```

We still have the same groups as on ix:
```
[root@talos-linux64-ix-085 ~]# id cltbld
uid=1000(cltbld) gid=1000(cltbld) groups=1000(cltbld),29(audio),44(video)
```

Comment 2

[:dhouse]

Geoff, please test and let me know if it works. There are a couple of machines in the set that didn't respond, but I think you can test with this.

If this fixes the problem, I'll make a patch to apply the group to the cltbld user during build.

details:
`adduser cltbld kvm` applied to t-linux64-ms-
003..012 (013 timed out)
014..015
046..053 (054, 055 timed-out)
056..060
091..105

I tested and made sure that puppet will not change the group.

Comment 3

[:dhouse]

Attachment: add cltbld to group 'kvm'

Comment 4

[Geoff Brown [:gbrown]]

I tried to test with

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b686f929827b98c21fcbdcfddb4968c2e18a3905

but I got machines outside of the set you noted in comment 2, every time. Am I unlucky, or are we looking at different machine sets?

I got:

t-linux64-ms-237.test.releng.mdc1.mozilla.com
t-linux64-ms-136.test.releng.mdc1.mozilla.com
t-linux64-ms-276.test.releng.mdc1.mozilla.com
t-linux64-ms-145.test.releng.mdc1.mozilla.com
t-linux64-ms-194.test.releng.mdc1.mozilla.com


None of those had your change.

https://public-artifacts.taskcluster.net/SqFfqC7sRxiHWi9F52hjCg/0/public/logs/live_backing.log

[taskcluster]  hostBootTime: 2018-01-29T13:05:46-08:00
[taskcluster]  hostname: t-linux64-ms-237.test.releng.mdc1.mozilla.com
[taskcluster]  runId: 0
[taskcluster]  taskId: SqFfqC7sRxiHWi9F52hjCg
[taskcluster]  tasksSinceStartup: 208
[taskcluster]  workerGroup: mdc1
[taskcluster]  workerType: gecko-t-linux-talos
+ set -x -e
++ id
+ echo 'running as' 'uid=1000(cltbld)' 'gid=1000(cltbld)' 'groups=1000(cltbld),29(audio),44(video)'
running as uid=1000(cltbld) gid=1000(cltbld) groups=1000(cltbld),29(audio),44(video)

Comment 5

[:dhouse]

I'm sorry I missed these; I was thinking that was all that were running for talos right now. I applied the kvm group to cltbld on the higher ranges also:
136..150
181..195
226..240
271..280

Comment 6

[Dragos Crisan [:dragrom]]

Comment on attachment 8947637 [details] [diff] [review]
add cltbld to group 'kvm'

LGTM

Comment 7

[Geoff Brown [:gbrown]]

Thanks Dave, but I'm afraid I'm still not seeing a change.

https://treeherder.mozilla.org/logviewer.html#?job_id=160100743&repo=try&lineNumber=1-11

[taskcluster]  hostBootTime: 2018-01-29T10:50:16-08:00
[taskcluster]  hostname: t-linux64-ms-052.test.releng.mdc1.mozilla.com
[taskcluster]  runId: 0
[taskcluster]  taskId: Db2eWjT2RKSDs9WTarWyaw
[taskcluster]  tasksSinceStartup: 232
[taskcluster]  workerGroup: mdc1
[taskcluster]  workerType: gecko-t-linux-talos
+ set -x -e
++ id
+ echo 'running as' 'uid=1000(cltbld)' 'gid=1000(cltbld)' 'groups=1000(cltbld),29(audio),44(video)'
running as uid=1000(cltbld) gid=1000(cltbld) groups=1000(cltbld),29(audio),44(video)

Comment 8

[Geoff Brown [:gbrown]]

Maybe you need to check-in your patch before I'll see the change in try pushes?

Comment 9

[:dhouse]

I checked just now and I see the kvm group is still listed in the cltbld user's groups on the t-linux64-ms-{002..300} machines:
```
uid=1000(cltbld) gid=1000(cltbld) groups=1000(cltbld),29(audio),44(video),114(kvm)
```

I'll apply the patch and we can see if there is a change.

Could you give me a url to see the warning or logs for when the android emulator is failing? Then I could try restarting the worker on a few machines and see if I can catch a run to troubleshoot.

Comment 10

[:dhouse]

Comment on attachment 8947637 [details] [diff] [review]
add cltbld to group 'kvm'

Applied and pushed to default:
remote:   https://hg.mozilla.org/build/puppet/rev/e0b8edd536eeda08e8257e5d20df35ff03839118
Travis passed. Pushing to production:
remote:   https://hg.mozilla.org/build/puppet/rev/ddf48ed338e732e6ff8acd1f50015bd809e47b8a

Comment 11

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a761012d3263867efd3ef26d4db113baf55ae20a&selectedJob=160100743

has this test log for "c1":

https://treeherder.mozilla.org/logviewer.html#?job_id=160100743&repo=try&lineNumber=10

...and I just retried that job, so there's a new one running now.

Comment 12

[Geoff Brown [:gbrown]]

The retry worked! (has cltbld in the kvm group)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a761012d3263867efd3ef26d4db113baf55ae20a&selectedJob=160503260

Great!!

Comment 13

[Geoff Brown [:gbrown]]

I kicked off a bunch of new tasks and this is working great -- thanks much.

Comment 14

[:dhouse]

Thankyou for persisting with this! I'm happy to help if you need any more adjustments for setting it up with the emulator.

Comment 15

[:dhouse]

Attachment: add user to kvm group only on 64bit ubuntu

32bit ubuntu does not have qemu-kvm package installed, and no kvm group

Tested puppet with this patch on a moonshot and yosemite (sane).
Applied, travis passed. Pushed to production:
remote:   https://hg.mozilla.org/build/puppet/rev/0c8485a6cf85f4f8755eb035d945d503b2554c0b

Comment 16

[:dhouse]

Comment on attachment 8948695 [details] [diff] [review]
add user to kvm group only on 64bit ubuntu

The group does not exist yet (chicken before the egg, user before qemu-kvm) on the hosts that will have it.

So I need to change the adding of the kvm group to happen after the qemu-kvm package is installed or as part of the package.

Comment 17

[:dhouse]

Attachment: ensure kvm group exists

Comment 18

[:dhouse]

Comment on attachment 8948721 [details] [diff] [review]
ensure kvm group exists

Applied and pushed to default:
https://hg.mozilla.org/build/puppet/rev/dc0e2a121b1b6a8147200713e53d73a258851ec2
Travis passed. Pushed to production:
remote:   https://hg.mozilla.org/build/puppet/rev/db4a657bd169f6854aaa43f42d8f72d1757f8308

Comment 19

[:dhouse]

With the changed conditions for the kvm group, not on 32bit and ensuring the group exists before user creation, the cltbld user has stayed a member of the kvm group owning /dev/kvm on the moonshots:
```
crw-rw---- 1 root kvm 10, 232 Feb  5 20:40 /dev/kvm
uid=1000(cltbld) gid=1000(cltbld) groups=1000(cltbld),29(audio),44(video),114(kvm)
```