Bug 1499054 Comment 12 Edit History

Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.

Original comment by
on 
The above issue has been resolved, and turned out to be a problem when using the ENI network resolver. After changing to netplan network resolver, this problem has gone away.

The next problem I am hitting is a failing CI test:

```
[taskcluster 2019-09-04T10:34:51.870Z] Worker Type (test-provisioner/test-mdqa8ms3sl6rc9qdjb7ivq-a) settings:
[taskcluster 2019-09-04T10:34:51.871Z]   {
[taskcluster 2019-09-04T10:34:51.871Z]     "aws": {
[taskcluster 2019-09-04T10:34:51.871Z]       "ami-id": "test-ami",
[taskcluster 2019-09-04T10:34:51.871Z]       "availability-zone": "outer-space",
[taskcluster 2019-09-04T10:34:51.871Z]       "instance-id": "test-instance-id",
[taskcluster 2019-09-04T10:34:51.871Z]       "instance-type": "p3.enormous",
[taskcluster 2019-09-04T10:34:51.871Z]       "local-ipv4": "87.65.43.21",
[taskcluster 2019-09-04T10:34:51.871Z]       "public-ipv4": "12.34.56.78"
[taskcluster 2019-09-04T10:34:51.871Z]     },
[taskcluster 2019-09-04T10:34:51.871Z]     "generic-worker": {
[taskcluster 2019-09-04T10:34:51.871Z]       "go-arch": "amd64",
[taskcluster 2019-09-04T10:34:51.871Z]       "go-os": "linux",
[taskcluster 2019-09-04T10:34:51.871Z]       "go-version": "go1.10.8",
[taskcluster 2019-09-04T10:34:51.871Z]       "release": "test-release-url",
[taskcluster 2019-09-04T10:34:51.871Z]       "version": "15.1.4"
[taskcluster 2019-09-04T10:34:51.871Z]     },
[taskcluster 2019-09-04T10:34:51.871Z]     "machine-setup": {
[taskcluster 2019-09-04T10:34:51.871Z]       "maintainer": "pmoore@mozilla.com",
[taskcluster 2019-09-04T10:34:51.871Z]       "script": "test-script-url"
[taskcluster 2019-09-04T10:34:51.871Z]     }
[taskcluster 2019-09-04T10:34:51.871Z]   }
[taskcluster 2019-09-04T10:34:51.871Z] Task ID: bNxVCQwzRqC48SILduxmiA
[taskcluster 2019-09-04T10:34:51.871Z] === Task Starting ===
[taskcluster 2019-09-04T10:34:52.586Z] Uploading redirect artifact public/logs/live.log to URL http://12.34.56.78:46695/log/bF_yM-ILSISTcp022ej-XA with mime type "text/plain; charset=utf-8" and expiry 2019-09-04T10:51:51.985Z
[taskcluster 2019-09-04T10:34:52.586Z] Executing command 0: go get 'github.com/taskcluster/taskcluster-client-go'
go: disabling cache (/home/task_1567592780/.cache/go-build) due to initialization failure: open /home/task_1567592780/.cache/go-build/log.txt: permission denied
go get github.com/taskcluster/taskcluster-client-go: open /home/task_1567592780/gopath1.10.8/pkg/linux_amd64/github.com/taskcluster/taskcluster-client-go.a: permission denied
[taskcluster 2019-09-04T10:34:52.923Z]    Exit Code: 1
[taskcluster 2019-09-04T10:34:52.923Z]    User Time: 854.891ms
[taskcluster 2019-09-04T10:34:52.923Z]  Kernel Time: 230.757ms
[taskcluster 2019-09-04T10:34:52.923Z]    Wall Time: 336.363485ms
[taskcluster 2019-09-04T10:34:52.923Z]       Result: FAILED
[taskcluster 2019-09-04T10:34:52.923Z] === Task Finished ===
[taskcluster 2019-09-04T10:34:52.923Z] Task Duration: 336.885965ms
[taskcluster:error] Uploading error artifact resolvetask.go from file resolvetask.go with message "Could not read file '/home/task_1567592780/gopath1.10.8/src/github.com/taskcluster/generic-worker/testdata/TestResolveResolvedTask/task_1567592780/resolvetask.go'", reason "file-missing-on-worker" and expiry 2019-09-04T11:34:49.270Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read file '/home/task_1567592780/gopath1.10.8/src/github.com/taskcluster/generic-worker/testdata/TestResolveResolvedTask/task_1567592780/resolvetask.go'
[taskcluster 2019-09-04T10:34:53.800Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/bNxVCQwzRqC48SILduxmiA/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2019-09-18T10:34:49.000Z
[taskcluster:error] exit status 1
[taskcluster:error] file-missing-on-worker: Could not read file '/home/task_1567592780/gopath1.10.8/src/github.com/taskcluster/generic-worker/testdata/TestResolveResolvedTask/task_1567592780/resolvetask.go'
```

Troubleshooting this issue now.
Revision 1 by
on 
The above issue has been resolved, and turned out to be a problem when using the ENI network resolver. After changing to netplan network resolver, this problem has gone away. The solution was to add this as the last machine setup step before shutting down and snapshotting the ec2 instance:

```
cat > /etc/cloud/cloud.cfg.d/01_network_renderer_policy.cfg << EOF
system_info:
    network:
      renderers: [ 'netplan', 'eni', 'sysconfig' ]
EOF
```

The next problem I am hitting is a failing CI test:

```
[taskcluster 2019-09-04T10:34:51.870Z] Worker Type (test-provisioner/test-mdqa8ms3sl6rc9qdjb7ivq-a) settings:
[taskcluster 2019-09-04T10:34:51.871Z]   {
[taskcluster 2019-09-04T10:34:51.871Z]     "aws": {
[taskcluster 2019-09-04T10:34:51.871Z]       "ami-id": "test-ami",
[taskcluster 2019-09-04T10:34:51.871Z]       "availability-zone": "outer-space",
[taskcluster 2019-09-04T10:34:51.871Z]       "instance-id": "test-instance-id",
[taskcluster 2019-09-04T10:34:51.871Z]       "instance-type": "p3.enormous",
[taskcluster 2019-09-04T10:34:51.871Z]       "local-ipv4": "87.65.43.21",
[taskcluster 2019-09-04T10:34:51.871Z]       "public-ipv4": "12.34.56.78"
[taskcluster 2019-09-04T10:34:51.871Z]     },
[taskcluster 2019-09-04T10:34:51.871Z]     "generic-worker": {
[taskcluster 2019-09-04T10:34:51.871Z]       "go-arch": "amd64",
[taskcluster 2019-09-04T10:34:51.871Z]       "go-os": "linux",
[taskcluster 2019-09-04T10:34:51.871Z]       "go-version": "go1.10.8",
[taskcluster 2019-09-04T10:34:51.871Z]       "release": "test-release-url",
[taskcluster 2019-09-04T10:34:51.871Z]       "version": "15.1.4"
[taskcluster 2019-09-04T10:34:51.871Z]     },
[taskcluster 2019-09-04T10:34:51.871Z]     "machine-setup": {
[taskcluster 2019-09-04T10:34:51.871Z]       "maintainer": "pmoore@mozilla.com",
[taskcluster 2019-09-04T10:34:51.871Z]       "script": "test-script-url"
[taskcluster 2019-09-04T10:34:51.871Z]     }
[taskcluster 2019-09-04T10:34:51.871Z]   }
[taskcluster 2019-09-04T10:34:51.871Z] Task ID: bNxVCQwzRqC48SILduxmiA
[taskcluster 2019-09-04T10:34:51.871Z] === Task Starting ===
[taskcluster 2019-09-04T10:34:52.586Z] Uploading redirect artifact public/logs/live.log to URL http://12.34.56.78:46695/log/bF_yM-ILSISTcp022ej-XA with mime type "text/plain; charset=utf-8" and expiry 2019-09-04T10:51:51.985Z
[taskcluster 2019-09-04T10:34:52.586Z] Executing command 0: go get 'github.com/taskcluster/taskcluster-client-go'
go: disabling cache (/home/task_1567592780/.cache/go-build) due to initialization failure: open /home/task_1567592780/.cache/go-build/log.txt: permission denied
go get github.com/taskcluster/taskcluster-client-go: open /home/task_1567592780/gopath1.10.8/pkg/linux_amd64/github.com/taskcluster/taskcluster-client-go.a: permission denied
[taskcluster 2019-09-04T10:34:52.923Z]    Exit Code: 1
[taskcluster 2019-09-04T10:34:52.923Z]    User Time: 854.891ms
[taskcluster 2019-09-04T10:34:52.923Z]  Kernel Time: 230.757ms
[taskcluster 2019-09-04T10:34:52.923Z]    Wall Time: 336.363485ms
[taskcluster 2019-09-04T10:34:52.923Z]       Result: FAILED
[taskcluster 2019-09-04T10:34:52.923Z] === Task Finished ===
[taskcluster 2019-09-04T10:34:52.923Z] Task Duration: 336.885965ms
[taskcluster:error] Uploading error artifact resolvetask.go from file resolvetask.go with message "Could not read file '/home/task_1567592780/gopath1.10.8/src/github.com/taskcluster/generic-worker/testdata/TestResolveResolvedTask/task_1567592780/resolvetask.go'", reason "file-missing-on-worker" and expiry 2019-09-04T11:34:49.270Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read file '/home/task_1567592780/gopath1.10.8/src/github.com/taskcluster/generic-worker/testdata/TestResolveResolvedTask/task_1567592780/resolvetask.go'
[taskcluster 2019-09-04T10:34:53.800Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/bNxVCQwzRqC48SILduxmiA/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2019-09-18T10:34:49.000Z
[taskcluster:error] exit status 1
[taskcluster:error] file-missing-on-worker: Could not read file '/home/task_1567592780/gopath1.10.8/src/github.com/taskcluster/generic-worker/testdata/TestResolveResolvedTask/task_1567592780/resolvetask.go'
```

Troubleshooting this issue now.

Back to Bug 1499054 Comment 12