813699 - ondemand_update scripts taking a long time to execute for Firefox 17 updates on release channel

Status: RESOLVED INVALID

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[u279076]

Here is an example node:
http://10.250.73.243:8080/job/ondemand_update/4620/

It's been running for 45 minutes at this point and is creating a backlog of test runs. I seem to recall this happening the last release but I can't remember what we did to rectify it.

Comment 1

[u279076]

All nodes seem to be active and hung at the same point right now so no more tests are getting through.

Comment 2

[u279076]

Looks like they finally timed out and aborted. 

From mozmill-ci email:
> Mozmill ondemand_update testrun for  en-US on  (2012-11-20_11-42-45) was aborted.
> 
> View the build in Jenkins:
> http://10.250.73.243:8080/job/ondemand_update/4620/

Console output:
> Started by user anonymous
> Building remotely on mm-ub-1204-64-1 in workspace jenkins/workspace/ondemand_update
>
> Deleting project workspace... done
>
> No emails were triggered.
> Copied 556 artifacts from "get_mozmill-environments » linux,master" build number 3
> Copied 1,176 artifacts from "get_mozmill-tests" build number 95
> Copied 86 artifacts from "get_mozmill-automation" build number 16
> [ondemand_update] $ mozmill-env/run mozdownload --type=$BUILD_TYPE --platform=$PLATFORM  --version=$VERSION --locale=$LOCALE --build-number=$BUILD_NUMBER --directory=builds
> Downloading build: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/14.0.1/linux-x86_64/en-US/firefox-14.0.1.tar.bz2
> [ondemand_update] $ mozmill-env/run mozmill-automation/testrun_update.py --target-buildid=$TARGET_BUILD_ID --channel=$CHANNEL --repository=mozmill-tests --junit=report.xml --report=$REPORT_URL builds/
> Build timed out (after 60 minutes). Marking the build as aborted.
> Build was aborted
> Recording test results
> No test report files were found. Configuration error?
> Email was triggered for: Aborted
> Sending email for trigger: Aborted
> Sending email to: [e-mail]
> mozmill-env/run: line 23: 17759 Terminated              $@
> Finished: ABORTED

I'm also still seeing those intermittent Windows 8 update failures that Dave Hunt was helping with earlier.

The next set of queued runs have been sitting there for nearly 40 minutes. I expect them to abort as well.

Comment 3

[u279076]

It would seem that these are all aborting on Ubuntu. Thanks to Dave for pointing that out.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It's failing in recording test results. And that is the following issue for CI:

https://github.com/mozilla/mozmill-ci/issues/31

Not sure why it has been filed in infrastructure through.