767968 - Intermittent test_0201_app_launch_apply_update.js, test_0203_app_launch_apply_update.js | WindowsError: [Error 13] The process cannot access the file because it is being used by another process: '...\\maintenanceservice_installer.exe'

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect)

Description

[Ed Morley [:emorley]]

Rev3 WINNT 6.1 mozilla-inbound pgo test xpcshell on 2012-06-23 23:31:51 PDT for push 030728d0d1d7

slave: talos-r3-w7-029

https://tbpl.mozilla.org/php/getParsedLog.php?id=12947115&tree=Mozilla-Inbound

{
TEST-INFO | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0202_app_launch_apply_update_dirlocked.js | running test ...
TEST-PASS | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0202_app_launch_apply_update_dirlocked.js | test passed (time: 1417.000ms)
TEST-INFO | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | running test ...
TEST-PASS | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | test passed (time: 8303.000ms)
Traceback (most recent call last):
  File "xpcshell/runxpcshelltests.py", line 915, in <module>
    main()
  File "xpcshell/runxpcshelltests.py", line 911, in main
    if not xpcsh.runTests(args[0], testdirs=args[1:], **options.__dict__):
  File "xpcshell/runxpcshelltests.py", line 795, in runTests
    self.removeDir(self.profileDir)
  File "xpcshell/runxpcshelltests.py", line 321, in removeDir
    shutil.rmtree(dirname)
  File "c:\mozilla-build\python25\Lib\shutil.py", line 169, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "c:\mozilla-build\python25\Lib\shutil.py", line 169, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "c:\mozilla-build\python25\Lib\shutil.py", line 174, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "c:\mozilla-build\python25\Lib\shutil.py", line 172, in rmtree
    os.remove(fullname)
WindowsError: [Error 13] The process cannot access the file because it is being used by another process: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmpgc8wyz\\ExecutableDir.tmp\\bin\\maintenanceservice_installer.exe'
program finished with exit code 1
elapsedTime=2678.976000
TinderboxPrint: xpcshell<br/><em class="testfail">T-FAIL</em>
Unknown Error: command finished with exit code: 1
}

Comment 1

[(no longer active)]

Robert, seems like we need to increase the helper timeout even more.  Does that sound fine to you?

Comment 2

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=12907737&tree=Firefox

Comment 3

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=13056497&tree=Ionmonkey

Comment 4

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=13088876&tree=Ionmonkey

Comment 5

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=13159461&tree=Mozilla-Inbound

Comment 6

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=13168386&tree=Mozilla-Inbound

Comment 7

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=13201668&tree=Profiling

Comment 8

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=13299091&tree=Firefox

Comment 9

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=13349625&tree=Mozilla-Aurora

Comment 10

[Robert Strong (they/them - no direct email)]

(In reply to Ehsan Akhgari [:ehsan] from comment #1)
> Robert, seems like we need to increase the helper timeout even more.  Does
> that sound fine to you?
I'm ok with this though I wonder what else can be done to make this more deterministic.

Comment 11

[(no longer active)]

(In reply to comment #10)
> (In reply to Ehsan Akhgari [:ehsan] from comment #1)
> > Robert, seems like we need to increase the helper timeout even more.  Does
> > that sound fine to you?
> I'm ok with this though I wonder what else can be done to make this more
> deterministic.

I don't have any other ideas right now...

Comment 12

[(no longer active)]

Attachment: Increase the value by twice

Comment 13

[(no longer active)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/092c73924711

Comment 14

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=13419622&tree=Profiling

Comment 15

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=13414413&tree=Fx-Team

Comment 16

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/092c73924711

Comment 17

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=13637947&tree=Mozilla-Inbound

Comment 18

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=13660744&tree=Mozilla-Inbound

Comment 19

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=13704088&tree=Mozilla-Inbound

Comment 20

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14211383&tree=Ionmonkey

Comment 21

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14430201&tree=Mozilla-Inbound

Comment 22

[Legacy TBPL/Treeherder Robot]

Standard8
https://tbpl.mozilla.org/php/getParsedLog.php?id=14510656&tree=Thunderbird-Trunk
TB Rev3 WINNT 5.1 comm-central debug test xpcshell on 2012-08-19 09:46:26
slave: talos-r3-xp-069

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | C:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test/unit/test_0203_app_launch_apply_update.js | launch application timer expired - See following stack:

Comment 23

[Legacy TBPL/Treeherder Robot]

Standard8
https://tbpl.mozilla.org/php/getParsedLog.php?id=14559602&tree=Thunderbird-Trunk
TB Rev3 WINNT 5.1 comm-central debug test xpcshell on 2012-08-21 04:13:20
slave: talos-r3-xp-054

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test/unit/test_0203_app_launch_apply_update.js | launch application timer expired - See following stack:

Comment 24

[Legacy TBPL/Treeherder Robot]

Standard8
https://tbpl.mozilla.org/php/getParsedLog.php?id=14619478&tree=Thunderbird-Trunk
TB Rev3 WINNT 5.1 comm-central debug test xpcshell on 2012-08-22 20:12:32
slave: talos-r3-xp-017

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test/unit/test_0203_app_launch_apply_update.js | launch application timer expired - See following stack:

Comment 25

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14860350&tree=Mozilla-Beta

Comment 26

[Legacy TBPL/Treeherder Robot]

Standard8
https://tbpl.mozilla.org/php/getParsedLog.php?id=15053977&tree=Thunderbird-Beta
TB Rev3 WINNT 5.1 comm-beta debug test xpcshell on 2012-09-07 10:49:26
slave: talos-r3-xp-054

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test/unit/test_0203_app_launch_apply_update.js | launch application timer expired - See following stack:

Comment 27

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15247109&tree=Mozilla-Aurora

Comment 28

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15400046&tree=Mozilla-Inbound

Comment 29

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15545598&tree=Services-Central

Comment 30

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15720481&tree=Mozilla-Inbound

Comment 31

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15740775&tree=Firefox

Comment 32

[Legacy TBPL/Treeherder Robot]

Standard8
https://tbpl.mozilla.org/php/getParsedLog.php?id=15763131&tree=Thunderbird-Beta
TB Rev3 WINNT 5.1 comm-beta debug test xpcshell on 2012-10-02 20:23:12
slave: talos-r3-xp-006

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test/unit/test_0203_app_launch_apply_update.js | launch application timer expired - See following stack:

Comment 33

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=16400445&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound debug test xpcshell on 2012-10-23 18:21:06
slave: talos-r3-w7-011

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: <type 'exceptions.WindowsError'>

Comment 34

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=16549527&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound opt test xpcshell on 2012-10-29 03:31:31
slave: talos-r3-w7-019

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: <type 'exceptions.WindowsError'>

Comment 35

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=16562053&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound opt test xpcshell on 2012-10-29 13:32:37
slave: talos-r3-w7-036

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: <type 'exceptions.WindowsError'>

Comment 36

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=16568084&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound opt test xpcshell on 2012-10-29 16:37:56
slave: talos-r3-w7-051

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: <type 'exceptions.WindowsError'>

Comment 37

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=16592880&tree=Cedar
Rev3 WINNT 6.1 cedar opt test xpcshell on 2012-10-30 10:23:53
slave: talos-r3-w7-089

11:21:31  WARNING -  TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\tests\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: <type 'exceptions.WindowsError'>

Comment 38

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=16784268&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound pgo test xpcshell on 2012-11-06 03:18:40
slave: talos-r3-w7-066

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: [Error 13] The process cannot access the file because it is being used by another process: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmp08lupx\\ExecutableDir.tmp\\bin\\maintenanceservice_installer.exe'

Comment 39

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=16901977&tree=Mozilla-Esr17
Rev3 WINNT 6.1 mozilla-esr17 pgo test xpcshell on 2012-11-09 10:07:08
slave: talos-r3-w7-098

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: <type 'exceptions.WindowsError'>

Comment 40

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=16973324&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound debug test xpcshell on 2012-11-12 16:46:30
slave: talos-r3-w7-083

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: [Error 13] The process cannot access the file because it is being used by another process: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmp358ao_\\ExecutableDir.tmp\\bin\\uninstall\\helper.exe'

Comment 41

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=17005481&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound debug test xpcshell on 2012-11-13 14:30:48
slave: talos-r3-w7-093

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: [Error 13] The process cannot access the file because it is being used by another process: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmp8xc9vs\\ExecutableDir.tmp\\bin\\uninstall\\helper.exe'
TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\services\sync\tests\unit\test_history_tracker.js | test failed (with xpcshell return code: -2147483645), see following log:
PROCESS-CRASH | C:\talos-slave\test\build\xpcshell\tests\services\sync\tests\unit\test_history_tracker.js | application crashed (minidump found)
Thread 0 (crashed)

Comment 42

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=17231263&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound opt test xpcshell on 2012-11-20 19:44:33
slave: talos-r3-w7-073

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: [Error 13] The process cannot access the file because it is being used by another process: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmpjdf-uc\\ExecutableDir.tmp\\bin\\uninstall\\helper.exe'

Comment 43

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=17351798&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound debug test xpcshell on 2012-11-26 12:31:54
slave: talos-r3-w7-047

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: [Error 13] The process cannot access the file because it is being used by another process: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmpvl_x8l\\ExecutableDir.tmp\\bin\\uninstall\\helper.exe'

Comment 44

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=17535825&tree=Mozilla-Aurora
Rev3 WINNT 6.1 mozilla-aurora pgo test xpcshell on 2012-12-02 04:42:16
slave: talos-r3-w7-008

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: [Error 13] The process cannot access the file because it is being used by another process: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmphrah4d\\ExecutableDir.tmp\\bin\\uninstall\\helper.exe'

Comment 45

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=17713691&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound opt test xpcshell on 2012-12-07 06:23:22
slave: talos-r3-w7-046

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: [Error 13] The process cannot access the file because it is being used by another process: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmpcjmzct\\ExecutableDir.tmp\\bin\\maintenanceservice_installer.exe'

Comment 46

[Legacy TBPL/Treeherder Robot]

Ms2ger%gmail.com
https://tbpl.mozilla.org/php/getParsedLog.php?id=17729921&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound pgo test xpcshell on 2012-12-07 15:50:21
slave: talos-r3-w7-023

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: [Error 13] The process cannot access the file because it is being used by another process: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmphs_pay\\ExecutableDir.tmp\\bin\\maintenanceservice_installer.exe'

Comment 47

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=17767785&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound debug test xpcshell on 2012-12-09 16:34:04
slave: talos-r3-w7-082

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0203_app_launch_apply_update.js | Failed to clean up the test profile directory: [Error 13] The process cannot access the file because it is being used by another process: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmp7x9pje\\ExecutableDir.tmp\\bin\\uninstall\\helper.exe'

Comment 48

[Robert Strong (they/them - no direct email)]

Note: unless I am mistaken, the only time we run the maintenanceservice_installer.exe in the bin directory is from
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/common/updatehelper.cpp#192
and here
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/test/unit/head_update.js.in#656

Comment 49

[Robert Strong (they/them - no direct email)]

Mistakenly posted to bug 762032
ehsan
https://tbpl.mozilla.org/php/getParsedLog.php?id=18519656&tree=Firefox
Rev3 WINNT 6.1 mozilla-central pgo test xpcshell on 2013-01-06 09:33:03
slave: talos-r3-w7-098

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0201_app_launch_apply_update_svc.js | Failed to clean up the test profile directory: [Error 13] The process cannot access the file because it is being used by another process: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmpkol5pf\\ExecutableDir.tmp\\bin\\maintenanceservice_installer.exe'

Comment 50

[Robert Strong (they/them - no direct email)]

Also note that there is bug 823996 for  test_0201_app_launch_apply_update_svc.js and test_0203_app_launch_apply_update_svc.js

Comment 51

[Robert Strong (they/them - no direct email)]

Attachment: patch rev1 - use waitForFilesInUse implemented in bug 762032

Comment 52

[Brian R. Bondy [:bbondy]]

Comment on attachment 709427 [details] [diff] [review]
patch rev1 - use waitForFilesInUse implemented in bug 762032

Review of attachment 709427 [details] [diff] [review]:
-----------------------------------------------------------------

It's somewhat surprising that we have this given: 
http://dxr.mozilla.org/mozilla-central/toolkit/mozapps/update/test/unit/head_update.js.in.html#l804

Comment 53

[Robert Strong (they/them - no direct email)]

Pushed to mozilla-inbound
https://hg.mozilla.org/integration/mozilla-inbound/rev/687a4154ebaf

Comment 54

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/687a4154ebaf

Comment 55

[Robert Strong (they/them - no direct email)]

Pushed to mozilla-aurora
https://hg.mozilla.org/releases/mozilla-aurora/rev/3eeb9c28f829

Pushed to mozilla-beta
https://hg.mozilla.org/releases/mozilla-beta/rev/096747eee2ee

Pushed to mozilla-esr17
https://hg.mozilla.org/releases/mozilla-esr17/rev/893e305a2b2a