1226581 - [tc-gip] No report generated if the tests take more than 2 hours to be executed (aka the container is killed)

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Johan Lorenzo [:jlorenzo]]

Like many jobs from tc-gip, [1] failed because too many tests timed out. After 7200 seconds, the container was killed even though the tests were still runnin. As a result, nothing has been archived under private/logs.

The report is only generated at the end of the test run. I don't know if generating on the fly is doable.

I see 2 ways of easily workaround the problem:
* Splitting the jobs smaller ones, so the odds of reaching 2 hours are lower
* Increasing the time out to 3 or 4 hours

Greg, what approach sounds best to you? Do you see any other?

[1] https://public-artifacts.taskcluster.net/BpWNAXomQf23pE6_lfEKXA/0/public/logs/live_backing.log

Comment 1

[Greg Arndt [:garndt]]

I think that the longer these run, the harder it is to rerun them, troubleshoot issues, and opens up the possibility of additional problems happening.

Ideally test suites are < 30 minutes in my opinion, but I know when flashing a device and running tests this is probably not realistic :)

Definitely think making them smaller is better in the long run.  Trying to rerun something because one or two tests are failing would really stink if we had to wait 3-4 hours for the results.

Comment 2

[Johan Lorenzo [:jlorenzo]]

Fine by me. I propose to have 6 "f" jobs[1] instead of 3. Since the 3 jobs were created, we doubled the number of devices with 1 SIM card. 

[1] No pun intended

Comment 3

[Johan Lorenzo [:jlorenzo]]

Attachment: 1226581.patch

Here's a simple change. Let's hope it help to get reports every time.

Comment 4

[Carsten Book [:Tomcat]]

Hi, this failed to apply:

applying file_1226581.txt
patching file testing/taskcluster/tasks/tests/flame_kk_gaia_ui_test_functional.yml
Hunk #1 FAILED at 44
1 out of 1 hunks FAILED -- saving rejects to file testing/taskcluster/tasks/tests/flame_kk_gaia_ui_test_functional.yml.rej
patch failed, unable to continue (try -v)
patch failed, rejects left in working directory
errors during apply, please fix and refresh file_1226581.txt

Comment 5

[Johan Lorenzo [:jlorenzo]]

Attachment: 1226581.patch

Sorry, I tried to generate a HG commit from git[1], but it didn't work well apparently. Here's a real HG patch. Carrying r+ from garndt

[1] https://github.com/mozilla/moz-git-tools/blob/master/git-to-hg-commit

Comment 6

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c23c1d8c548f

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/c23c1d8c548f