1825929 - Intermittent Linux TSAN ba(browser-a11y) [taskcluster:error] Task timeout after 2700 seconds. Force killing container.

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: imoraru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=411045161&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AzzNnD_IRd6qZHwkXFK_KQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-04-01T10:01:10.047Z] 10:01:10     INFO - TEST-START | accessible/tests/browser/telemetry/browser_HCM_telemetry.js
[task 2023-04-01T10:01:12.468Z] 10:01:12     INFO - GECKO(10576) | console.error: (new Error("Polling for changes failed: Unexpected content-type \"text/plain;charset=US-ASCII\".", "resource://services-settings/remote-settings.sys.mjs", 324))
[task 2023-04-01T10:01:32.160Z] 10:01:32     INFO - GECKO(10576) | ALSA lib seq_hw.c:466:(snd_seq_hw_open) open /dev/snd/seq failed: No such file or directory
[taskcluster:error] Task timeout after 2700 seconds. Force killing container.
[taskcluster 2023-04-01 10:01:46.452Z] === Task Finished ===
[taskcluster 2023-04-01 10:01:46.454Z] Unsuccessful task run with exit code: -1 completed in 2823.625 seconds

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Looks like the runtime of this job has gone up quite a bit. Not sure if that's because execution just got slower or if we added more tests or what. Just a quick scan of autoland tells me that 35-45min runtime is pretty typical nowadays, so it isn't surprising that we're occasionally hitting the 45min timeout.

We could increase the timeout to 60min, but I'm wondering if it would be better to split it into 2 chunks instead.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Attachment: Bug 1825929 - Split mochitest-browser-a11y into 2 chunks on TSAN. r=jmaher

Comment 3

[Ryan VanderMeulen [:RyanVM]]

I looked at 3 and 4 chunks as well, but it didn't help as much as I'd hoped. The accessible/tests/browser/e10s/ directory in particular seems to take quite awhile to run, and no amount of extra chunks helps with that. It might be useful to see if we could split those tests up into different directories at some point to improve parallelization. But for now, going to 2 chunks at least gets us a lot more breathing room.

Comment 4

[Intermittent Failures Robot]

5 failures in 3863 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 3

Platform and build breakdown:

• linux1804-64-tsan-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1825929&startday=2023-03-27&endday=2023-04-02&tree=all

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3e6f37c645a7
Split mochitest-browser-a11y into 2 chunks on TSAN. r=jmaher

Comment 6

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/3e6f37c645a7