Open Bug 2088128 Opened 14 days ago Updated 11 days ago

Balrog scriptworker task ran past the 2 hour task timeout and had to be cancelled manually

Categories

(Release Engineering :: Release Automation, defect)

defect

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: dmeehan, Assigned: hneiva)

Details

Attachments

(2 external files)

A beta balrog task ran for over two hours without being timed out, and RelEng had to cancel it by hand and rerun it.

Task: https://firefox-ci-tc.services.mozilla.com/tasks/U9WeL1IaTeKiqqjrslvYag

  • Name: balrog-nb-NO-linux64-aarch64-shippable/opt
  • Worker pool: scriptworker-k8s/gecko-3-balrog
  • Branch: mozilla-beta, revision 579b0acf3f0a
  • Task group: XOTKnyScSZK7mS0-Q7mh5g

What happened:

  • Run 0, on worker gecko-3-balrog-pa1ebm1pqjq69ygubecmw, started at 2026-11-02 13:23:29 UTC.
  • The worker kept reclaiming the task the whole time; takenUntil was still being extended (last value 16:03:34 UTC).
  • RelEng cancelled it manually at 15:46:37 UTC, after 2 hours 23 minutes.
  • Run 0 uploaded no artifacts, not even public/logs/live_backing.log, so there is no log of what it was stuck on.
  • The rerun (run 1, on a different worker) completed in 12 seconds.

Expected:
The balrog task payload has no maxRunTime (build_balrog_payload only sets behavior, upstreamArtifacts and suffixes), so the limit should come from scriptworker. In scriptworker-scripts, docker.d/scriptworker.yml sets task_max_timeout: 7200, so scriptworker should have killed the task around 15:23 UTC and resolved it as a failure, letting it be retried automatically.

Actual:
The task never timed out, the worker stayed alive and kept reclaiming, and the release automation was blocked until someone noticed and cancelled it.

Questions:

  • Why did task_max_timeout not fire? For example, scriptworker blocked somewhere outside the timed part of the task (claiming, downloading upstream artifacts, or a hung Balrog API request without its own timeout), or the timeout not reaching the balrog pool config.
  • Should balrogscript set an explicit request timeout on Balrog API calls, so a hung connection cannot stall a task indefinitely?

A few details that caused this task to hang:

  • It hung during the CoT verification process; specifically resolving a DNS lookup.
  • The CoT process doesn't have a timeout. The task timeout (max runtime) only comes into effect after CoT has run and the task has started.
  • Cancelling the task doesn't stop the CoT process, only the task process. So it lingers around on the worker.
Assignee: nobody → hneiva
Status: NEW → ASSIGNED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: