1530682 - docker-worker: Level 3 workers run out of (tmpfs?) space

Status: RESOLVED DUPLICATE of bug 1447695

(Taskcluster :: Workers, defect)

Description

[Johan Lorenzo [:jlorenzo]]

In [1], I added the possibility for github-repos to build their own docker images in taskcluster. Everything was working well[2] on level-1 workers[3]. When it landed, it used the level-3 workers and it started to fail[5] because the worker doesn't have enough disk space apparently. I wonder if I misconfigured something, but both worker types have very similar configuration.
I even beefed up EC2 instance type to c5.9xlarge, but it didn't change anything.

df -a -h doesn't output anything on neither level 3[6] not level 1[7]. Therefore, I can't find if the worker is really lacking disk space. Per [8][9] and [10], gradle uses /tmp.

:jhford, :wcosta, would you guys have ideas of what difference there may be?

[1] https://github.com/mozilla-mobile/android-components/pull/1874
[2] https://tools.taskcluster.net/groups/Efs2E626Qqm5TM1pM1aYCA/tasks/MFv5XI4mTs-KlHL93CUijw/runs/1/logs/public%2Flogs%2Flive_backing.log
[3] https://tools.taskcluster.net/aws-provisioner/mobile-1-images/view
[4] https://tools.taskcluster.net/aws-provisioner/mobile-3-images/view
[5] https://tools.taskcluster.net/groups/HNRILlwrSFuhvS9cPf4Qfg/tasks/KtK5-mBtQ3uMuHsCgDpnGQ/runs/4/logs/public%2Flogs%2Flive_backing.log#L57635
[6] https://tools.taskcluster.net/groups/GZP5IotJTjGaiX-UvR8G_Q/tasks/GZP5IotJTjGaiX-UvR8G_Q/runs/0/logs/public%2Flogs%2Flive_backing.log#L15
[7] https://tools.taskcluster.net/groups/CJ9Sa2aHQ-OVq4Gu1vCx6Q/tasks/CJ9Sa2aHQ-OVq4Gu1vCx6Q/runs/0/logs/public%2Flogs%2Flive.log#L35
[8] https://stackoverflow.com/questions/35949698/android-studio-2-1-preview1-gives-no-space-left-on-device-error
[9] https://stackoverflow.com/questions/44154964/errorfailed-to-complete-gradle-execution-cause-no-space-left-on-device
[10] https://stackoverflow.com/questions/41259024/i-am-getting-a-no-space-left-on-device-error

Comment 1

[Johan Lorenzo [:jlorenzo]]

:mhentges reproduces the same problem:

• [WORKED] I ran docker build ... on a level 1 worker (my fork) normally (no live shell): https://tools.taskcluster.net/groups/Wkq-Mb8lRUW5ctmrGbkEYA/tasks/Wkq-Mb8lRUW5ctmrGbkEYA/runs/0/logs/public%2Flogs%2Flive.log
• [FAILED] I ran docker build ... on a level 3 worker (my fork) normally (no live shell): https://tools.taskcluster.net/groups/dEbo2RBwQHKmVP1jvDCODw/tasks/dEbo2RBwQHKmVP1jvDCODw/runs/0/logs/public%2Flogs%2Flive.log
• [FAILED] I ran docker build ... on a level 3 worker (source repository) normally (no live shell): https://tools.taskcluster.net/groups/LwbLTKRHRuKJcXiMlnl7hA/tasks/LwbLTKRHRuKJcXiMlnl7hA/runs/0/logs/public%2Flogs%2Flive.log

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

I'm going to have to defer to Wander here.