Closed Bug 1226581 Opened 4 years ago Closed 4 years ago

[tc-gip] No report generated if the tests take more than 2 hours to be executed (aka the container is killed)

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla45

People

(Reporter: jlorenzo, Assigned: jlorenzo)

References

Details

Attachments

(1 file, 1 obsolete file)

Like many jobs from tc-gip, [1] failed because too many tests timed out. After 7200 seconds, the container was killed even though the tests were still runnin. As a result, nothing has been archived under private/logs.

The report is only generated at the end of the test run. I don't know if generating on the fly is doable.

I see 2 ways of easily workaround the problem:
* Splitting the jobs smaller ones, so the odds of reaching 2 hours are lower
* Increasing the time out to 3 or 4 hours

Greg, what approach sounds best to you? Do you see any other?

[1] https://public-artifacts.taskcluster.net/BpWNAXomQf23pE6_lfEKXA/0/public/logs/live_backing.log
Flags: needinfo?(garndt)
I think that the longer these run, the harder it is to rerun them, troubleshoot issues, and opens up the possibility of additional problems happening.

Ideally test suites are < 30 minutes in my opinion, but I know when flashing a device and running tests this is probably not realistic :)

Definitely think making them smaller is better in the long run.  Trying to rerun something because one or two tests are failing would really stink if we had to wait 3-4 hours for the results.
Flags: needinfo?(garndt)
Fine by me. I propose to have 6 "f" jobs[1] instead of 3. Since the 3 jobs were created, we doubled the number of devices with 1 SIM card. 

[1] No pun intended
Attached patch 1226581.patch (obsolete) — Splinter Review
Here's a simple change. Let's hope it help to get reports every time.
Attachment #8690111 - Flags: review?(garndt)
Attachment #8690111 - Flags: review?(garndt) → review+
Keywords: checkin-needed
Attachment #8690111 - Attachment is patch: true
Hi, this failed to apply:

applying file_1226581.txt
patching file testing/taskcluster/tasks/tests/flame_kk_gaia_ui_test_functional.yml
Hunk #1 FAILED at 44
1 out of 1 hunks FAILED -- saving rejects to file testing/taskcluster/tasks/tests/flame_kk_gaia_ui_test_functional.yml.rej
patch failed, unable to continue (try -v)
patch failed, rejects left in working directory
errors during apply, please fix and refresh file_1226581.txt
Flags: needinfo?(jlorenzo)
Attached patch 1226581.patchSplinter Review
Sorry, I tried to generate a HG commit from git[1], but it didn't work well apparently. Here's a real HG patch. Carrying r+ from garndt

[1] https://github.com/mozilla/moz-git-tools/blob/master/git-to-hg-commit
Attachment #8690111 - Attachment is obsolete: true
Flags: needinfo?(jlorenzo)
Attachment #8691296 - Flags: review+
https://hg.mozilla.org/mozilla-central/rev/c23c1d8c548f
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla45
Assignee: nobody → jlorenzo
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.