Last Comment Bug 757794 - Constant "command timed out: 1200 seconds without output" in test_0200_app_launch_apply_update.js
: Constant "command timed out: 1200 seconds without output" in test_0200_app_la...
Status: RESOLVED FIXED
:
Product: Toolkit
Classification: Components
Component: Application Update (show other bugs)
: Trunk
: All All
: -- normal (vote)
: mozilla16
Assigned To: Mark Banner (:standard8)
:
Mentors:
Depends on:
Blocks: bgupdates 766264
  Show dependency treegraph
 
Reported: 2012-05-23 04:09 PDT by Mark Banner (:standard8)
Modified: 2012-06-30 08:03 PDT (History)
3 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
fixed


Attachments
Change timeouts to failures (7.47 KB, patch)
2012-06-15 01:43 PDT, Mark Banner (:standard8)
ehsan: review-
Details | Diff | Review
Change timeouts to failures v2 (151 bytes, patch)
2012-06-19 04:16 PDT, Mark Banner (:standard8)
no flags Details | Diff | Review
Change timeouts to failures v2a (16.71 KB, patch)
2012-06-19 06:22 PDT, Mark Banner (:standard8)
ehsan: review+
akeybl: approval‑mozilla‑aurora+
Details | Diff | Review

Description Mark Banner (:standard8) 2012-05-23 04:09:51 PDT
Seen on the Thunderbird builders - all Windows xpcshell tests since bug 307181 landed:

TEST-INFO | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0200_app_launch_apply_update.js | running test ...

command timed out: 1200 seconds without output, attempting to kill
SIGKILL failed to kill process
using fake rc=-1
program finished with exit code -1

A brief look at SeaMonkey had this test passing and 0201 failing...
Comment 1 :Ehsan Akhgari (busy, don't ask for review please) 2012-05-23 06:56:47 PDT
How does one run these tests against Thunderbird?
Comment 2 Mark Banner (:standard8) 2012-05-23 07:02:42 PDT
You can download packaged builds and their unit tests from http://ftp.mozilla.org/pub/mozilla.org/thunderbird/tinderbox-builds/ and run the commands manually (copy from buildbot logs).
Comment 3 :Ehsan Akhgari (busy, don't ask for review please) 2012-06-14 11:18:59 PDT
Is this still an issue with builds off of https://hg.mozilla.org/mozilla-central/rev/983b91e5aa17 or later?
Comment 4 Mark Banner (:standard8) 2012-06-14 14:16:58 PDT
(In reply to Ehsan Akhgari [:ehsan] from comment #3)
> Is this still an issue with builds off of
> https://hg.mozilla.org/mozilla-central/rev/983b91e5aa17 or later?

Yes

https://tbpl.mozilla.org/php/getParsedLog.php?id=12667671&tree=Thunderbird-Trunk
Comment 5 Mark Banner (:standard8) 2012-06-15 01:33:54 PDT
To reproduce with tinderbox-builds, first some setup:

# mkdir xpcshelltest && cd xpcshelltest
# wget /http/path/to/thunderbird-16.0a1.en-US.win32.zip
# unzip -o thunderbird-16.0a1.en-US.win32.zip
# wget /http/path/to/thunderbird-16.0a1.en-US.win32.tests.zip
# unzip -o thunderbird-16.0a1.en-US.win32.tests.zip bin* certs* modules* xpcshell*
# mkdir thunderbird/plugins
# cp bin/xpcshell.exe thunderbird
# cp -R bin/components/* thunderbird/components
# cp -R bin/plugins/* thunderbird/plugins

To run:

# python -u xpcshell/runxpcshelltests.py --manifest=xpcshell/tests/toolkit/mozapps/update/test/unit/xpcshell.ini thunderbird/xpcshell.exe

or:

# python -u xpcshell/runxpcshelltests.py --test-path=test_0201_app_launch_apply_update.js --profile-name=thunderbird --interactive --verbose thunderbird/xpcshell.exe xpcshell/tests/toolkit/mozapps/update/test/unit/
Comment 6 Mark Banner (:standard8) 2012-06-15 01:43:18 PDT
Created attachment 633434 [details] [diff] [review]
Change timeouts to failures

This doesn't fix the issue, but it changes the timeouts into test failures which is somewhat better than timeouts.

The issue appears to be that updates aren't applying, proceeding or something, but the do_timeouts are just repeatedly called. The patch gets us out of that loop.

I've gone for a 5 minute maximum for the whole test, although tinderbox times out at 20, I don't think there's much point in letting the whole 20 if I've understood right that the updates are just testing a small patch/app.
Comment 7 Mark Banner (:standard8) 2012-06-15 01:45:36 PDT
I've pushed this to both Thunderbird try (to check we now fail rather than timeout) and Firefox try (to check the shorter timeouts):

https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=389f41b458c0
https://tbpl.mozilla.org/?tree=Try&rev=fc0e1f5074f1
Comment 8 Mark Banner (:standard8) 2012-06-15 06:11:18 PDT
Thunderbird try server is now giving failures rather than timeouts. 

The logs don't show much extra, except that it is timing out:

https://tbpl.mozilla.org/php/getParsedLog.php?id=12694762&tree=Thunderbird-Try#error0

However, I noticed in the debug logs this one as well:

TEST-PASS | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test/unit/test_0030_general.js | [check_test_helper_pt1_1 : 65] 1 == 1
*** AUS:SVC readStringFromFile - file doesn't exist: C:\Users\cltbld\AppData\Local\\\thunderbird\updates\0\update.status
*** AUS:SVC readStatusFile - status: null, path: C:\Users\cltbld\AppData\Local\\\thunderbird\updates\0\update.status
*** AUS:SVC UpdateManager:_loadXMLFileIntoArray: XML file does not exist
*** AUS:SVC UpdateManager:_loadXMLFileIntoArray: XML file does not exist
*** AUS:SVC UpdateManager:_loadXMLFileIntoArray: XML file does not exist
*** AUS:SVC Downloader:downloadUpdate - downloading from http://localhost:4444/data/simple_no_pib.mar to C:\Users\cltbld\AppData\Local\\\thunderbird\updates\0\update.mar
###!!! ASSERTION: mTempFile not equal to mTargetFile: 'Error', file e:/builds/moz2_slave/tb-try-c-cen-w32-dbg/build/mozilla/netwerk/base/src/nsFileStreams.cpp, line 759
Comment 9 Mark Banner (:standard8) 2012-06-18 00:40:01 PDT
(In reply to Mark Banner (:standard8) from comment #7)
> I've pushed this to both Thunderbird try (to check we now fail rather than
> timeout) and Firefox try (to check the shorter timeouts):
> 
> https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=389f41b458c0
> https://tbpl.mozilla.org/?tree=Try&rev=fc0e1f5074f1

Firefox Try server tests passed, even though they didn't report the fact.
Comment 10 :Ehsan Akhgari (busy, don't ask for review please) 2012-06-18 11:50:41 PDT
Comment on attachment 633434 [details] [diff] [review]
Change timeouts to failures

You should probably apply the same changes to test_0202 and also the test_svc variants.
Comment 11 Mark Banner (:standard8) 2012-06-19 03:55:09 PDT
(In reply to Ehsan Akhgari [:ehsan] from comment #10)
> You should probably apply the same changes to test_0202 and also the
> test_svc variants.

test_0202 doesn't have this repetitive do_timeout call - there is only one. I'll do the others though.
Comment 12 Mark Banner (:standard8) 2012-06-19 04:16:26 PDT
Created attachment 634348 [details] [diff] [review]
Change timeouts to failures v2

Updated for the other tests. Also on try server: https://tbpl.mozilla.org/?tree=Try&rev=1fcc102b248b
Comment 13 :Ehsan Akhgari (busy, don't ask for review please) 2012-06-19 06:22:51 PDT
Comment on attachment 634348 [details] [diff] [review]
Change timeouts to failures v2

Empty patch!
Comment 14 Mark Banner (:standard8) 2012-06-19 06:22:55 PDT
Created attachment 634372 [details] [diff] [review]
Change timeouts to failures v2a

Try server got the actual patch, but then I messed up the refresh before uploading, so here's the real patch.
Comment 15 :Ehsan Akhgari (busy, don't ask for review please) 2012-06-19 06:28:06 PDT
Comment on attachment 634372 [details] [diff] [review]
Change timeouts to failures v2a

r=me
Comment 16 :Ehsan Akhgari (busy, don't ask for review please) 2012-06-19 08:14:57 PDT
Comment on attachment 634372 [details] [diff] [review]
Change timeouts to failures v2a

https://hg.mozilla.org/mozilla-central/rev/d29af708ec3c
Comment 17 Mark Banner (:standard8) 2012-06-19 12:45:31 PDT
I'm going to call this fixed, in that the timeouts are now test failures. The test failures will be fixed in bug 766264.
Comment 18 :Ehsan Akhgari (busy, don't ask for review please) 2012-06-19 13:05:11 PDT
Please nom this for Aurora as well?
Comment 19 Mark Banner (:standard8) 2012-06-19 13:26:22 PDT
Comment on attachment 634372 [details] [diff] [review]
Change timeouts to failures v2a

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Bug 307181
User impact if declined: Test-only patch
Testing completed (on m-c, etc.): Landed on m-c
Risk to taking this patch (and alternatives if risky): None
String or UUID changes made by this patch: None

If we don't take this patch, then Thunderbird tests on Windows will constantly time out and make it difficult to see other failures.
Comment 20 Alex Keybl [:akeybl] 2012-06-19 19:46:25 PDT
Comment on attachment 634372 [details] [diff] [review]
Change timeouts to failures v2a

[Triage Comment]
Test only changes, approved for Aurora 15.
Comment 21 Ed Morley [:emorley] 2012-06-20 02:21:17 PDT
https://hg.mozilla.org/mozilla-central/rev/d3c41f0a8d7c
Comment 22 Mark Banner (:standard8) 2012-06-30 08:03:44 PDT
https://hg.mozilla.org/releases/mozilla-aurora/rev/d47f25160d66

Note You need to log in before you can comment on or make changes to this bug.