like https://public-artifacts.taskcluster.net/IrvP6mJgTv6RFe9m9hxgxA/0/public/logs/live_backing.log Uploading symbol file "target.crashreporter-symbols-full.zip" to "https://crash-stats.mozilla.com/symbols/upload" Attempt 1 of 5... [taskcluster:error] Task timeout after 600 seconds. Force killing container. [taskcluster 2017-07-07 11:06:29.834Z] === Task Finished === [taskcluster 2017-07-07 11:06:29.834Z] Unsuccessful task run with exit code: -1 completed in 603.179 seconds not sure why 1/5 and what happened to 2/3/4/5 but we should investigate
also hit linux on beta
FWIW, this is the error we get on crash-stats: > UnreadablePostError > timeout during read(65536) on wsgi.input CC'ing Peter who's in charge of Symbols on our side.
:garndt noticed that all these occurrences happened on the same machine (which got terminated meanwhile). More details can be seen here once data is updated in OrangeFactor: https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1379155&entireHistory=true&tree=trunk The re-triggers are green though. In the near future, we should have monitoring tools able to tell whether a worker is consistently failing jobs and disable it. Per IRC: "<•garndt> well the good news is that the machine is gone and retriggers are ok, so I think the issue was isoalted to that machine. Longer term we're working on ideas of how to disable machines on the taskcluster side that pehaps would allow us to automatically throttle or disable/kill a machine that is consistently failing tasks."
7 failures in 656 pushes (0.011 failures/push) were associated with this bug in the last 7 days. Repository breakdown: * mozilla-beta: 5 * mozilla-central: 2 Platform breakdown: * linux64: 2 * android-4-0-armv7-api15-old-id: 2 * android-4-0-armv7-api15: 2 * android-4-2-x86-old-id: 1 For more details, see: https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1379155&startday=2017-07-03&endday=2017-07-09&tree=all