757794 - Constant "command timed out: 1200 seconds without output" in test_0200_app_launch_apply_update.js

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect)

Description

[Mark Banner (:standard8)]

Seen on the Thunderbird builders - all Windows xpcshell tests since bug 307181 landed:

TEST-INFO | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0200_app_launch_apply_update.js | running test ...

command timed out: 1200 seconds without output, attempting to kill
SIGKILL failed to kill process
using fake rc=-1
program finished with exit code -1

A brief look at SeaMonkey had this test passing and 0201 failing...

Comment 1

[(no longer active)]

How does one run these tests against Thunderbird?

Comment 2

[Mark Banner (:standard8)]

You can download packaged builds and their unit tests from http://ftp.mozilla.org/pub/mozilla.org/thunderbird/tinderbox-builds/ and run the commands manually (copy from buildbot logs).

Comment 3

[(no longer active)]

Is this still an issue with builds off of https://hg.mozilla.org/mozilla-central/rev/983b91e5aa17 or later?

Comment 4

[Mark Banner (:standard8)]

(In reply to Ehsan Akhgari [:ehsan] from comment #3)
> Is this still an issue with builds off of
> https://hg.mozilla.org/mozilla-central/rev/983b91e5aa17 or later?

Yes

https://tbpl.mozilla.org/php/getParsedLog.php?id=12667671&tree=Thunderbird-Trunk

Comment 5

[Mark Banner (:standard8)]

To reproduce with tinderbox-builds, first some setup:

# mkdir xpcshelltest && cd xpcshelltest
# wget /http/path/to/thunderbird-16.0a1.en-US.win32.zip
# unzip -o thunderbird-16.0a1.en-US.win32.zip
# wget /http/path/to/thunderbird-16.0a1.en-US.win32.tests.zip
# unzip -o thunderbird-16.0a1.en-US.win32.tests.zip bin* certs* modules* xpcshell*
# mkdir thunderbird/plugins
# cp bin/xpcshell.exe thunderbird
# cp -R bin/components/* thunderbird/components
# cp -R bin/plugins/* thunderbird/plugins

To run:

# python -u xpcshell/runxpcshelltests.py --manifest=xpcshell/tests/toolkit/mozapps/update/test/unit/xpcshell.ini thunderbird/xpcshell.exe

or:

# python -u xpcshell/runxpcshelltests.py --test-path=test_0201_app_launch_apply_update.js --profile-name=thunderbird --interactive --verbose thunderbird/xpcshell.exe xpcshell/tests/toolkit/mozapps/update/test/unit/

Comment 6

[Mark Banner (:standard8)]

Attachment: Change timeouts to failures

This doesn't fix the issue, but it changes the timeouts into test failures which is somewhat better than timeouts.

The issue appears to be that updates aren't applying, proceeding or something, but the do_timeouts are just repeatedly called. The patch gets us out of that loop.

I've gone for a 5 minute maximum for the whole test, although tinderbox times out at 20, I don't think there's much point in letting the whole 20 if I've understood right that the updates are just testing a small patch/app.

Comment 7

[Mark Banner (:standard8)]

I've pushed this to both Thunderbird try (to check we now fail rather than timeout) and Firefox try (to check the shorter timeouts):

https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=389f41b458c0
https://tbpl.mozilla.org/?tree=Try&rev=fc0e1f5074f1

Comment 8

[Mark Banner (:standard8)]

Thunderbird try server is now giving failures rather than timeouts. 

The logs don't show much extra, except that it is timing out:

https://tbpl.mozilla.org/php/getParsedLog.php?id=12694762&tree=Thunderbird-Try#error0

However, I noticed in the debug logs this one as well:

TEST-PASS | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test/unit/test_0030_general.js | [check_test_helper_pt1_1 : 65] 1 == 1
*** AUS:SVC readStringFromFile - file doesn't exist: C:\Users\cltbld\AppData\Local\\\thunderbird\updates\0\update.status
*** AUS:SVC readStatusFile - status: null, path: C:\Users\cltbld\AppData\Local\\\thunderbird\updates\0\update.status
*** AUS:SVC UpdateManager:_loadXMLFileIntoArray: XML file does not exist
*** AUS:SVC UpdateManager:_loadXMLFileIntoArray: XML file does not exist
*** AUS:SVC UpdateManager:_loadXMLFileIntoArray: XML file does not exist
*** AUS:SVC Downloader:downloadUpdate - downloading from http://localhost:4444/data/simple_no_pib.mar to C:\Users\cltbld\AppData\Local\\\thunderbird\updates\0\update.mar
###!!! ASSERTION: mTempFile not equal to mTargetFile: 'Error', file e:/builds/moz2_slave/tb-try-c-cen-w32-dbg/build/mozilla/netwerk/base/src/nsFileStreams.cpp, line 759

Comment 9

[Mark Banner (:standard8)]

(In reply to Mark Banner (:standard8) from comment #7)
> I've pushed this to both Thunderbird try (to check we now fail rather than
> timeout) and Firefox try (to check the shorter timeouts):
> 
> https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=389f41b458c0
> https://tbpl.mozilla.org/?tree=Try&rev=fc0e1f5074f1

Firefox Try server tests passed, even though they didn't report the fact.

Comment 10

[(no longer active)]

Comment on attachment 633434 [details] [diff] [review]
Change timeouts to failures

You should probably apply the same changes to test_0202 and also the test_svc variants.

Comment 11

[Mark Banner (:standard8)]

(In reply to Ehsan Akhgari [:ehsan] from comment #10)
> You should probably apply the same changes to test_0202 and also the
> test_svc variants.

test_0202 doesn't have this repetitive do_timeout call - there is only one. I'll do the others though.

Comment 12

[Mark Banner (:standard8)]

Attachment: Change timeouts to failures v2

Updated for the other tests. Also on try server: https://tbpl.mozilla.org/?tree=Try&rev=1fcc102b248b

Comment 13

[(no longer active)]

Comment on attachment 634348 [details] [diff] [review]
Change timeouts to failures v2

Empty patch!

Comment 14

[Mark Banner (:standard8)]

Attachment: Change timeouts to failures v2a

Try server got the actual patch, but then I messed up the refresh before uploading, so here's the real patch.

Comment 15

[(no longer active)]

Comment on attachment 634372 [details] [diff] [review]
Change timeouts to failures v2a

r=me

Comment 16

[(no longer active)]

Comment on attachment 634372 [details] [diff] [review]
Change timeouts to failures v2a

https://hg.mozilla.org/mozilla-central/rev/d29af708ec3c

Comment 17

[Mark Banner (:standard8)]

I'm going to call this fixed, in that the timeouts are now test failures. The test failures will be fixed in bug 766264.

Comment 18

[(no longer active)]

Please nom this for Aurora as well?

Comment 19

[Mark Banner (:standard8)]

Comment on attachment 634372 [details] [diff] [review]
Change timeouts to failures v2a

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Bug 307181
User impact if declined: Test-only patch
Testing completed (on m-c, etc.): Landed on m-c
Risk to taking this patch (and alternatives if risky): None
String or UUID changes made by this patch: None

If we don't take this patch, then Thunderbird tests on Windows will constantly time out and make it difficult to see other failures.

Comment 20

[Alex Keybl [:akeybl]]

Comment on attachment 634372 [details] [diff] [review]
Change timeouts to failures v2a

[Triage Comment]
Test only changes, approved for Aurora 15.

Comment 21

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/d3c41f0a8d7c

Comment 22

[Mark Banner (:standard8)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/d47f25160d66