Closed Bug 918142 Opened 11 years ago Closed 11 years ago

Several 25.0b1 ondemand update testruns are hung for ~40 minutes now causing backlog

Categories

(Mozilla QA Graveyard :: Infrastructure, defect)

defect
Not set
critical

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: u279076, Unassigned)

Details

Nearly all of the ondemand update testruns for Firefox 25.0b1 on betatest are hung and there is a considerable backlog. They all seem to be hung on the final stage of the process:

> Report document created at 'http://mozauto.iriscouch.com/mozmill-release/1039ea48a9d69a5a1cc4fd228c31586f'
> *** Removing updates staging folder '/var/folders/Hq/HqUlSZIYFum1tKZJFYKimE+++TI/-Tmp-/tmplqAOjW.binary/Firefox.app/Contents/MacOS/updates/0'
> *** Removing old installation at /var/folders/Hq/HqUlSZIYFum1tKZJFYKimE+++TI/-Tmp-/tmplqAOjW.binary/Firefox.app
> *** Removing repository '/var/folders/Hq/HqUlSZIYFum1tKZJFYKimE+++TI/-Tmp-/tmpRtbqmV.mozmill-tests'
> 
> Results:
> ========
> * 22.0 => 25.0, minor, ak, complete, betatest, 2013-09-18, '''PASS'''
> ** Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:22.0) Gecko/20100101 Firefox/22.0 ID:20130617145905
> ** Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:25.0) Gecko/20100101 Firefox/25.0 ID:20130917123208
> ** Passed 8 :: Failed 0 :: Skipped 0
> 
> Recording test results
> Email was triggered for: Success
> Sending email for trigger: Success
> 
> (animating throbber)

I have no idea what is happening but this needs to be investigated ASAP. Resolution of this issue blocks shipping Firefox 25.0b1.

Please let me know what information I can provide.
Most of these have cleared out now after an hour, successfully submitting. However I think this should still be investigated. Therefore I'm reducing this in severity from blocker to critical. We can probably still release tomorrow if this problem persists but it will take all day I suspect.
Severity: blocker → critical
Note that I started the ondemand_update testrun around 3pm today; now 5:13pm and I still have half of the testruns in queue.
If a single job does not finish and hangs all the remaining jobs have to wait for its completeness. As of now we have a build timeout of 60 minutes for jobs. There is nothing we can do about except lowering this timeout for update jobs to somewhat 20 minutes. A single job should really never take longer. I think that will help a lot. Anthony, please get an issue filed for mozmill-ci if that is important for you, and I will make sure we will get it fixed.

The cause of this might be bug 860249 where the process most likely hung and was killed after 60 minutes.

I'm closing this bug in favor of the possible mozmill-ci issue and an upgrade to mozmill 2.0 hopefully soon.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WORKSFORME
I reported https://github.com/mozilla/mozmill-ci/issues/301 to deal with the timeout. I will follow up on bug 860249.
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.