Closed Bug 1619060 Opened 6 years ago Closed 2 years ago

Intermittent fenix ui-test-x86 [taskcluster:error] Task timeout after 7200 seconds. Force killing container.

Categories

(Firefox Build System :: Task Configuration, defect)

Product:
Firefox Build System
Component:
Task Configuration
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: aryx, Unassigned)

References

Details

(Keywords: intermittent-failure)

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=291035155&repo=fenix&lineNumber=2696

First (?) failure: https://treeherder.mozilla.org/#/jobs?repo=fenix&searchStr=46a5410c08a9b8291d0eff57922fc84123c418e0&tochange=30f351910a16c98aabe6d8a0aceacda41e52b775&fromchange=342ad1bfcaf51e7985c3a4668471428e744ec5df&selectedJob=291036159

Some Fenix ui-test-x86 tasks time out and they had gaps of inactivity earlier during the execution - the same issue might affect them more than once per run:

[task 2020-02-28T21:55:41.911Z] 17m 27s matrix-3nl613vccer8v Nexus6-25 FINISHED
[task 2020-02-28T22:39:01.230Z] 60m 46s matrix-3nl613vccer8v Nexus6-25 Started logcat recording.

...

[task 2020-02-28T22:44:25.789Z] 66m 11s matrix-29arbxhz88wz1 Nexus6-25 FINISHED
[taskcluster:error] Task timeout after 7200 seconds. Force killing container.

Sebastian, can you try to reproduce it or enable logcat to get more insight?

Flags: needinfo?(s.kaspari)

This would have likely been a failure to communicate with the Firebase API (temporary issue) or something stuck on the taskcluster VM.
:jlorenzo can the job retry that we abandoned as a 1 size fits all approach, be used conditionally in a case like this? i.e. could we do a retry if the job doesn't finish after say 30 minutes?

Flags: needinfo?(s.kaspari) → needinfo?(jlorenzo)

In m-c, we definitely have a few places where we detect specific kinds of error, that we have a high confidence are intermittent, and exit with a specific exit code to trigger a retry. This is definitely something we could do in fenix as well.

I do think we would want to have code that wraps the command that is failing or running to long, and detects the failure mode we care about, to trigger an exit.

For example, for firebase, have the wrapper be a python script that detects the hang, and responds to it. I haven't looked at the details of the failure to see exactly what that would entail here.

I agree with Tom. The logic needs to be done in the ui-test script[1]. To be more precise, I see 2 ways:

  1. The simplest one: ui-test.sh should detect the hang, then exit with a specific error code, then we can tell Taskcluster to retry on this specific code.
  2. The most effective one: ui-test.sh detects the hang and what test is actually pending and just reruns this one. The taskcluster retry mechanism isn't used. This way we don't rerun all tests, just the right one.

[1] https://github.com/mozilla-mobile/fenix/blob/f6870564732ad0e2cbe459425577486b6af9acf2/automation/taskcluster/androidTest/ui-test.sh

Flags: needinfo?(jlorenzo)

thanks for the input guys! :jlorenzo, 2. would be ideal but not possible. once we hand off the test execution it's like we hand the API a bill-of-sale for all the tests we want executed and they all get executed asynchronously at the same time. The do have a per-test "re-try" parameter but we tried it and it only seems to work very seldom. Perhaps because the retry happens to fast one after another? (unsure)

In regards to the first situation, that's something we haven't explored yet. Adding :aaronmt to the loop... do you think there's a way we could determine when the firebase API is hanging? I wonder if the test-lab folks may have some insight on that. If we could determine when the Firebase API is unresponsive

Flags: needinfo?(aaron.train)

I've asked the Firebase Community. Flank also exposes a global timeout on test-execution if we want some kind of cut-off under an hour at some threshold time-out value we can agree on as well. This hasn't been defined our x86 yml.

The max time this test run can execute before it is cancelled (default: unlimited).
run-timeout: 60m

Flags: needinfo?(aaron.train)

:aaronmt this is a good option! but then we'd still need info from firebase to know when it's giving up due to timeout vs. job failure. Firebase may treat both of those as a job failure, but we'd want to retry in the case of a timeout, I'd assume

Flank, as of 10 days ago on snapshot (we would need to update again) decreased their retry count from 10 to 4, added exponential backoff and fails fast on error codes (e.g, 429). Maybe that is suffice?

https://github.com/Flank/flank/pull/700

Bootstraponline also mentioned "Flank snapshot uses FTL server side sharding which is more stable. We're working on endpoint instrumentation to collect data on the instability of the FTL APIs"

(In reply to Aaron Train [:aaronmt] from comment #12)

Flank, as of 10 days ago on snapshot (we would need to update again) decreased their retry count from 10 to 4, added exponential backoff and fails fast on error codes (e.g, 429). Maybe that is suffice?

https://github.com/Flank/flank/pull/700

Bootstraponline also mentioned "Flank snapshot uses FTL server side sharding which is more stable. We're working on endpoint instrumentation to collect data on the instability of the FTL APIs"

We can monitor it and decide later if we still want to optimized. I still would be curious how we handle their API when things are taking too long on their end. Do we know that yet? Or, if investigation required, we can wait and see how the changes with the new API will be mitigation enough

Flags: needinfo?(aaron.train)
Flags: needinfo?(aaron.train)

I'm not sure there's much more we can do or if this is still a problem or not. I've asked the Flank team on the state on getting endpoint instrumentation built into Flank.

What's with other platforms reporting here?

Severity: normal → S3

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.