Balrog scriptworker task ran past the 2 hour task timeout and had to be cancelled manually
Categories
(Release Engineering :: Release Automation, defect)
Tracking
(Not tracked)
People
(Reporter: dmeehan, Assigned: hneiva)
Details
Attachments
(2 external files)
A beta balrog task ran for over two hours without being timed out, and RelEng had to cancel it by hand and rerun it.
Task: https://firefox-ci-tc.services.mozilla.com/tasks/U9WeL1IaTeKiqqjrslvYag
- Name:
balrog-nb-NO-linux64-aarch64-shippable/opt - Worker pool:
scriptworker-k8s/gecko-3-balrog - Branch: mozilla-beta, revision
579b0acf3f0a - Task group:
XOTKnyScSZK7mS0-Q7mh5g
What happened:
- Run 0, on worker
gecko-3-balrog-pa1ebm1pqjq69ygubecmw, started at 2026-11-02 13:23:29 UTC. - The worker kept reclaiming the task the whole time;
takenUntilwas still being extended (last value 16:03:34 UTC). - RelEng cancelled it manually at 15:46:37 UTC, after 2 hours 23 minutes.
- Run 0 uploaded no artifacts, not even
public/logs/live_backing.log, so there is no log of what it was stuck on. - The rerun (run 1, on a different worker) completed in 12 seconds.
Expected:
The balrog task payload has no maxRunTime (build_balrog_payload only sets behavior, upstreamArtifacts and suffixes), so the limit should come from scriptworker. In scriptworker-scripts, docker.d/scriptworker.yml sets task_max_timeout: 7200, so scriptworker should have killed the task around 15:23 UTC and resolved it as a failure, letting it be retried automatically.
Actual:
The task never timed out, the worker stayed alive and kept reclaiming, and the release automation was blocked until someone noticed and cancelled it.
Questions:
- Why did
task_max_timeoutnot fire? For example, scriptworker blocked somewhere outside the timed part of the task (claiming, downloading upstream artifacts, or a hung Balrog API request without its own timeout), or the timeout not reaching the balrog pool config. - Should balrogscript set an explicit request timeout on Balrog API calls, so a hung connection cannot stall a task indefinitely?
| Assignee | ||
Comment 1•14 days ago
|
||
A few details that caused this task to hang:
- It hung during the CoT verification process; specifically resolving a DNS lookup.
- The CoT process doesn't have a timeout. The task timeout (max runtime) only comes into effect after CoT has run and the task has started.
- Cancelling the task doesn't stop the CoT process, only the task process. So it lingers around on the worker.
Comment 2•13 days ago
|
||
(via GitHub webhook, authenticated as shared-secret)
Updated•11 days ago
|
Comment 3•11 days ago
|
||
Description
•