Closed Bug 692715 Opened 12 years ago Closed 9 years ago

Windows slaves often get permission denied errors while rm'ing files

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
x86
Windows 7
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: philor, Assigned: coop)

References

Details

(Whiteboard: [buildslaves][purple]) 

I know. I share your belief that it's just the way it is, it's just msys being msys. But. I star these all day, without ever keeping track of what slave that one was, or what slave the last three were. Indulge me for a little while.

https://tbpl.mozilla.org/php/getParsedLog.php?id=6719240&tree=Mozilla-Inbound
talos-r3-w7-015
rm: cannot lstat `tools/breakpad/osx': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=6743710&tree=Mozilla-Inbound
talos-r3-w7-019
Permission denied: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmpf2vzio\\profile\\Cache\\_CACHE_001_'
Since this is w7 I wonder if "Windows Search" is running as a service here.

"C:\Windows\system32\SearchIndexer.exe /Embedding"
https://tbpl.mozilla.org/php/getParsedLog.php?id=6754457&tree=Mozilla-Aurora
talos-r3-w7-019
Permission denied: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmpnniqzt\\profile\\Cache\\_CACHE_001_'
Depends on: 693184
And apparently it will do some good to expand my memory of slave names, filed bug 693184 for talos-r3-w7-019's bustage.
Priority: -- → P3
Whiteboard: [buildslaves][purple]
https://tbpl.mozilla.org/php/getParsedLog.php?id=7077399&tree=Firefox
slave: talos-r3-w7-013
rm: cannot lstat `tools/breakpad/linux': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7217940&tree=Mozilla-Inbound
slave: talos-r3-w7-018
rm: cannot lstat `tools/.hg/store/data/mozilla/tools/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7287626&tree=Firefox
slave: talos-r3-w7-012
rm: cannot lstat `tools/lib/perl/Release/Patcher': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7517294&tree=Fx-Team
talos-r3-w7-040
rm: cannot lstat `tools/lib/python/slavealloc/daemon/ui': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7560553&tree=Mozilla-Inbound
talos-r3-w7-043
rm: cannot lstat `tools/.hg/store/data/mozilla/tools/buildbot/buildbot/test/subdir': Permission denied
Roughly speaking, https://tbpl.mozilla.org/php/getParsedLog.php?id=7742331&tree=Mozilla-Inbound
talos-r3-w7-043
'rm' '-rf' 'build'
rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=7743535&tree=Mozilla-Inbound
talos-r3-w7-043
'rm' '-rf' 'build'
rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
Depends on: 707756
https://tbpl.mozilla.org/php/getParsedLog.php?id=7793479&tree=Mozilla-Inbound
talos-r3-w7-062
rm: cannot lstat `tools/buildfarm/mobile/n900-imaging/rootfs/etc/event.d': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7832069&tree=Firefox
talos-r3-w7-016
rm: cannot lstat `tools/scripts/valgrind': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7931518&tree=Firefox
talos-r3-w7-021
rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=7932735&tree=Firefox
talos-r3-w7-021
rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
Looks like 021 is up to either 11 or 13 failures (2 are jetpack, so they are probably utterly without meaning in any way), so it should be close to the time when it self-heals right as I start calling for it to be dealt with, right?
https://tbpl.mozilla.org/php/getParsedLog.php?id=7964379&tree=Mozilla-Inbound
talos-r3-w7-064
rm: cannot remove `build/bin/ssltunnel.exe': Permission denied
rm: cannot remove `build/bin/xpcshell.exe': Permission denied
rm: cannot remove `build/certs/cert8.db': Permission denied
rm: cannot remove `build/certs/key3.db': Permission denied
rm: cannot remove `build/firefox/components/browsercomps.dll': Permission denied
rm: cannot remove `build/firefox/firefox.exe': Permission denied
rm: cannot remove `build/firefox/freebl3.dll': Permission denied
rm: cannot remove `build/firefox/gkmedias.dll': Permission denied
rm: cannot remove `build/firefox/mozalloc.dll': Permission denied
rm: cannot remove `build/firefox/mozjs.dll': Permission denied
rm: cannot remove `build/firefox/mozsqlite3.dll': Permission denied
rm: cannot remove `build/firefox/mozutils.dll': Permission denied
rm: cannot remove `build/firefox/nspr4.dll': Permission denied
rm: cannot remove `build/firefox/nss3.dll': Permission denied
rm: cannot remove `build/firefox/nssckbi.dll': Permission denied
rm: cannot remove `build/firefox/nssdbm3.dll': Permission denied
rm: cannot remove `build/firefox/nssutil3.dll': Permission denied
rm: cannot remove `build/firefox/omni.ja': Permission denied
rm: cannot remove `build/firefox/plc4.dll': Permission denied
rm: cannot remove `build/firefox/plds4.dll': Permission denied
rm: cannot remove `build/firefox/plugin-container.exe': Permission denied
rm: cannot remove `build/firefox/smime3.dll': Permission denied
rm: cannot remove `build/firefox/softokn3.dll': Permission denied
rm: cannot remove `build/firefox/ssl3.dll': Permission denied
rm: cannot remove `build/firefox/xpcom.dll': Permission denied
rm: cannot remove `build/firefox/xul.dll': Permission denied
rm: cannot remove `build/mochitest/server.js': Permission denied
rm: cannot remove `build/mochitest/websock.log': Permission denied
program finished with exit code 1
https://tbpl.mozilla.org/php/getParsedLog.php?id=7970421&tree=Mozilla-Inbound
talos-r3-w7-005
rm: cannot lstat `tools/.hg/store/data/mozilla/tools/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7989792&tree=Firefox
slave: talos-r3-w7-064
rm: cannot lstat `tools/.hg/store/data/scripts/talos': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=8000923&tree=Mozilla-Inbound
talos-r3-w7-040
rm: cannot lstat `tools/lib/python/slavealloc/daemon/ui': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=8070809&tree=Mozilla-Inbound
talos-r3-w7-049
rm: cannot lstat `tools/scripts/talos': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=8176335&tree=Mozilla-Inbound
talos-r3-w7-027
rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=8177371&tree=Firefox
talos-r3-w7-027
rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=8277196&tree=Firefox
talos-r3-w7-007
rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
Depends on: 714211
https://tbpl.mozilla.org/php/getParsedLog.php?id=8277591&tree=Firefox
talos-r3-w7-007
rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=8279509&tree=Mozilla-Inbound
talos-r3-w7-007
rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=8362943&tree=Mozilla-Inbound
talos-r3-w7-018
rm: cannot lstat `tools/.hg/store/data/release/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=8392325&tree=Mozilla-Inbound
talos-r3-w7-042
rm: cannot lstat `tools/.hg/store/data/scripts/nanojit': Permission denied
So that little lot started out with a fail in 'Rev3 WINNT 6.1 mozilla-aurora debug test xpcshell' on rev 89407d03f983 at 16:23:

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\xpcom\tests\unit\test_nsIProcess.js | test failed (with xpcshell return code: 0), see following log:
...
TEST-UNEXPECTED-FAIL | (xpcshell/head.js) | [Exception... "Component returned failure code: 0x8052000e (NS_ERROR_FILE_IS_LOCKED) [nsILocalFile.moveTo]"  nsresult: "0x8052000e (NS_ERROR_FILE_IS_LOCKED)"  location: "JS frame :: c:/talos-slave/test/build/xpcshell/tests/xpcom/tests/unit/test_nsIProcess.js :: rename_and_test :: line 133"  data: no]

and then it hit this 
rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty

I don't know if it managed to reboot or not. Eight subsequent unit test jobs, which share the same directory, all hit the rm error but two talos were OK. 

Then it does a 'Rev3 WINNT 6.1 try debug test xpcshell' (rev 64509aa1e135 at 20:09), failing the first rm, passing xpcshell, and passing the second rm. Dunno how that happens, but it now appears to be OK now and this comes back to not being able to kill off processes properly on windows test slaves.
bug 493778 comment 9 and bug 582821 comment 16 and bug 582821 comment 17 for previous instances of the NS_ERROR_FILE_IS_LOCKED, but that far in the past it's not likely to be possible to see whether that always starts an episode of the build/xpcshell/tests/xpcom/tests/unit thing, or whether episodes of it (which are sometimes here, sometimes in WFM separate bugs I've filed, and sometimes only in hypothetical logs of #build) have always been started by that. Might be possible for me to remember to look in the future, though, as long as the band of our comments about it here is wide enough for me to spot :)
https://tbpl.mozilla.org/php/getParsedLog.php?id=8537523&tree=Mozilla-Inbound
talos-r3-w7-031
rm: cannot lstat `tools/.hg/store/data/release/l10n': Permission denied
That being

https://tbpl.mozilla.org/php/getParsedLog.php?id=8590063&tree=Try
talos-r3-xp-067
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/0/00/00png0.png': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/0/00/00text0': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/0/00/00text1': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/0/0exe0.exe': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/1/10/10text0': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/searchplugins/searchpluginspng0.png': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/searchplugins/searchpluginspng1.png': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/searchplugins/searchpluginstext0': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0111_svc_applyToDir/a/b/2/20/20png0.png': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0111_svc_applyToDir/a/b/2/20/20text0': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0151_svc_applyToDir/a/b/2/20/20png0.png': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0151_svc_applyToDir/a/b/2/20/20text0': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0181_svc_applyToDir/a/b/2/20/20png0.png': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0181_svc_applyToDir/a/b/2/20/20text0': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0183_svc_applyToDir/a/b/2/20/20png0.png': Permission denied
rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0183_svc_applyToDir/a/b/2/20/20text0': Permission denied

Probably just a coincidence that that's one of the freshly added today repurposed slaves, though, probably. And that it's one that got thrown back out of prod the last time it got added. Probably.
Or perhaps not, bug 718534.
https://tbpl.mozilla.org/php/getParsedLog.php?id=8769791&tree=Mozilla-Aurora
talos-r3-w7-009
rm: cannot lstat `tools/breakpad/linux': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=8793420&tree=Profiling
talos-r3-w7-035
rm: cannot lstat `tools/buildfarm/opsi': Permission denied
Depends on: 582821
https://tbpl.mozilla.org/php/getParsedLog.php?id=8875884&tree=Mozilla-Inbound
talos-r3-w7-068
rm: cannot lstat `tools/.hg/store/data/buildfarm/mobile/n900-imaging-old/notes': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9094439&tree=Try
talos-r3-w7-050
rm: cannot lstat `tools/.hg/store/data/mozilla/testing/tests/l10n/tests': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9165060&tree=Mozilla-Inbound
talos-r3-w7-022
rm: cannot lstat `tools/.hg/store/data/buildfarm/mobile/n900-imaging-old/rootfs/root-skel': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9180298&tree=Fx-Team
talos-r3-w7-015
rm: cannot lstat `tools/breakpad/linux64': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9217125&tree=Mozilla-Inbound
talos-r3-w7-007
mv: cannot move `talos-data' to `t-28353': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9386865&tree=Firefox
talos-r3-w7-005
rm: cannot lstat `tools/lib/python/slavealloc/daemon/ui': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9566706&tree=Profiling
talos-r3-w7-018
rm: cannot lstat `tools/.hg/store/data/buildfarm/mobile/production-sd/rootfs/bin': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9664103&tree=Mozilla-Inbound
talos-r3-w7-012
rm: cannot lstat `tools/.hg/store/data/scripts/release/updates': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9697051&tree=Mozilla-Inbound
talos-r3-w7-063
rm: cannot lstat `tools/scripts/android': Permission denied
Depends on: 732022
Blocks: talos-r3-xp-064
No longer blocks: talos-r3-xp-064
In bug 732022 I did this to recover:
C:\talos-slave\test>rm -rf build      
rm: reading directory `build/xpcshell/tests/xpcom/tests/unit': Invalid argument
rm: cannot remove directory `build/xpcshell/tests/xpcom/tests': Directory not empty
                                    
C:\talos-slave\test>rmdir /s /q build
https://tbpl.mozilla.org/php/getParsedLog.php?id=9763609&tree=Services-Central
talos-r3-w7-076
rm: cannot lstat `tools/.hg/store/data/buildfarm/mobile/n900-imaging-old/notes': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9852082&tree=Fx-Team
talos-r3-w7-067
rm: cannot lstat `tools/lib/python/slavealloc/daemon/ui': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=10142698&tree=Fx-Team
talos-r3-w7-049
rm: cannot lstat `tools/.hg/store/data/mozilla/app/locales/en-_u_s/other': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=10257218&tree=Mozilla-Aurora
talos-r3-xp-056
mv: cannot move `talos-data' to `t-2744': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=10296053&tree=Jetpack
t-r3-w764-001
rm: reading directory `scripts/.hg/store/data/mozilla/toolkit/locales/en-_u_s': Permission denied

(I should probably file a new bug for the astonishingly bad Win64 test slaves, which are astonishingly bad even by Windows slave standards.)
https://tbpl.mozilla.org/php/getParsedLog.php?id=10630587&tree=Mozilla-Inbound
talos-r3-w7-064
rm: cannot lstat `tools/.hg/store/data/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools__choose__boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=10721450&tree=Mozilla-Inbound
talos-r3-w7-030
rm: cannot lstat `tools/.hg/store/data/scripts/talos': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=10759256&tree=Mozilla-Aurora
talos-r3-xp-035
rm: cannot remove directory `build/xpcshell/tests/toolkit/components/downloads/test/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=10825250&tree=Mozilla-Inbound
talos-r3-w7-078
rm: cannot lstat `tools/.hg/store/data/release/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11087368&tree=Mozilla-Inbound
talos-r3-w7-071
rm: cannot remove `build/bin/vmwarerecordinghelper.dll': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11088607&tree=Firefox
talos-r3-w7-054
rm: cannot lstat `tools/.hg/store/data/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools__choose__boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11299910&tree=Mozilla-Inbound
talos-r3-w7-061
rm: cannot lstat `tools/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/ssl_match_hostname': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11383634&tree=Fx-Team
talos-r3-w7-057
rm: cannot lstat `tools/.hg/store/data/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools__choose__boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11461107&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound debug test mochitest-other on 2012-05-04 04:07:30 PDT for push 1f576da2253d
slave: talos-r3-w7-039
rm: cannot lstat `tools/.hg/store/data/release/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11551918&tree=Mozilla-Inbound
Rev3 WINNT 5.1 mozilla-inbound talos chrome.2 on 2012-05-07 18:55:54 PDT for push 4940a331b147
slave: talos-r3-xp-019
mv: cannot move `talos-data' to `t-21658': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11553345&tree=Firefox
mv: cannot move `talos-data' to `t-31690': Permission denied
And while it would make some sort of sense if that had been the same slave, it wasn't, that was talos-r3-xp-023.
WINNT 5.2 mozilla-central leak test build on 2012-05-11 03:12:41 PDT for push 3670d1eb1b31
slave: mw32-ix-slave02
rm: cannot remove `./dist/bin/components/browsercomps.dll': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11671551&tree=Firefox
builder: mozilla-central_w764_test-jetpack
slave: t-r3-w764-005
rm: reading directory `tools/lib/perl/Release': Permission denied
rm: cannot remove directory `tools/lib/perl': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=11740489&tree=Firefox&full=1
Assignee: nobody → coop
Component: Release Engineering → Release Engineering: Platform Support
QA Contact: release → coop
https://tbpl.mozilla.org/php/getParsedLog.php?id=11921101&tree=Mozilla-Aurora
slave: talos-r3-w7-019
rm: cannot lstat `tools/.hg/store/data/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools__choose__boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11987550&tree=Firefox
slave: talos-r3-w7-078
rm: cannot lstat `tools/.hg/store/data/mozilla/tools/buildbot/buildbot/test/subdir': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12023481&tree=Mozilla-Inbound
slave: talos-r3-w7-041
rm: cannot lstat `tools/scripts/android': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12342489&tree=Mozilla-Inbound
slave: talos-r3-w7-039
rm: cannot lstat `tools/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools_choose_boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12470679&tree=Ionmonkey
slave: mw32-ix-slave14
rm: cannot remove directory `scripts/scripts/spidermonkey_builds': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=12475351&tree=Ionmonkey
slave: mw32-ix-slave14
rm: cannot lstat `scripts/scripts/spidermonkey_builds/spidermonkey.sh': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12487883&tree=Mozilla-Inbound
slave: talos-r3-xp-038
rm: cannot remove `build/firefox-16.0a1.en-US.win32.zip': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12491476&tree=Mozilla-Inbound
slave: talos-r3-w7-056
rm: cannot remove `build/firefox-16.0a1.en-US.win32.tests.zip': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12541698&tree=Mozilla-Inbound
slave: talos-r3-w7-065
rm: cannot lstat `tools/buildfarm/mobile/n900-imaging/rootfs/etc/event.d': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12631593&tree=Profiling
slave: talos-r3-w7-048
rm: cannot lstat `tools/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12752450&tree=Mozilla-Beta
slave: talos-r3-w7-061
rm: cannot lstat `tools/.hg/store/data/mozilla/tools/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12873127&tree=Mozilla-Inbound
slave: talos-r3-w7-062
rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12873113&tree=Services-Central
slave: talos-r3-w7-041
rm: cannot lstat `tools/.hg/store/data/mozilla/tools/buildbot/buildbot/test/subdir': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12922859&tree=Ionmonkey
slave: talos-r3-w7-054
rm: cannot lstat `tools/release/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13076706&tree=Firefox
slave: talos-r3-w7-042
rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13084364&tree=Ionmonkey
slave: talos-r3-w7-059
rm: cannot lstat `tools/lib/python/vendor/python-daemon-1.5.5/python_daemon.egg-info': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13122605&tree=Mozilla-Inbound
slave: talos-r3-w7-066
rm: cannot lstat `tools/breakpad/linux64': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13118395&tree=Mozilla-Aurora
slave: talos-r3-w7-020
rm: cannot lstat `tools/.hg/store/data/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools__choose__boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13168998&tree=Mozilla-Inbound
slave: talos-r3-w7-030
rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13202374&tree=Mozilla-Inbound
slave: talos-r3-w7-052
rm: cannot lstat `tools/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13227659&tree=Profiling
slave: talos-r3-w7-042
rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13241316&tree=Fx-Team
slave: talos-r3-w7-045
rm: cannot lstat `tools/scripts/android': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13246436&tree=Mozilla-Inbound
slave: talos-r3-w7-044
rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13254849&tree=Ionmonkey
slave: talos-r3-w7-074
rm: cannot lstat `tools/.hg/store/data/mozilla/toolkit/locales/en-_u_s/dir': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13270793&tree=Mozilla-Inbound
slave: talos-r3-w7-073
rm: cannot lstat `tools/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14193117&tree=Mozilla-Inbound
slave: talos-r3-w7-077
rm: cannot lstat `tools/.hg/store/data/mozilla/tools/buildbot/buildbot/test/subdir': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14216744&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound opt test peptest on 2012-08-08 00:53:45 PDT for push 9428d40e8ead
slave: talos-r3-w7-065
rm: reading directory `scripts/.hg/store/data/buildfarm/mobile/n900-imaging': Permission denied
rm: cannot remove directory `scripts/.hg/store/data/buildfarm/mobile': Directory not empty
Rev3 WINNT 6.1 mozilla-inbound opt test peptest on 2012-08-08 03:44:03 PDT for push 4683a522952d

slave: talos-r3-w7-065

https://tbpl.mozilla.org/php/getParsedLog.php?id=14219803&tree=Mozilla-Inbound

{
03:44:19     INFO - ##### Running clobber step.
03:44:19     INFO - #####
03:44:19     INFO - rmtree: c:\talos-slave\test\build
Traceback (most recent call last):
  File "scripts/scripts/peptest.py", line 228, in <module>
    peptest.run()
  File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 620, in run
    self._possibly_run_method(method_name, error_if_missing=True)
  File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 591, in _possibly_run_method
    return getattr(self, method_name)()
  File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 641, in clobber
    self.rmtree(dirs['abs_work_dir'])
  File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 64, in rmtree
    self._rmdir_recursive(path)
  File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 105, in _rmdir_recursive
    self._rmdir_recursive(full_name)
  File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 105, in _rmdir_recursive
    self._rmdir_recursive(full_name)
  File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 109, in _rmdir_recursive
    os.remove(full_name)
WindowsError: [Error 13] Access is denied: 'c:\\talos-slave\\test\\build\\firefox\\components\\browsercomps.dll'
program finished with exit code 1
}
https://tbpl.mozilla.org/php/getParsedLog.php?id=14243599&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound opt test peptest on 2012-08-08 19:04:26 PDT for push 6c8f8fbe9aee
slave: talos-r3-w7-068

rm: reading directory `scripts/.hg/store/data/mozilla/tools/buildbot': Permission denied
rm: cannot remove directory `scripts/.hg/store/data/mozilla/tools': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=14282844&tree=Firefox
Rev3 WINNT 6.1 mozilla-central opt test peptest on 2012-08-09 22:13:51 PDT for push 4e150530c925
slave: talos-r3-w7-060

rm: reading directory `scripts/buildfarm/mobile/n900-imaging/rootfs/etc': Permission denied
rm: cannot remove directory `scripts/buildfarm/mobile/n900-imaging/rootfs': Directory not empty


Riddle me this: why have the last three instances all been peptest, on slaves in the 060s?
(One possible answer would involve the way that peptest used to be hidden, because it was and is utterly pointless to have it unhidden, since it intentionally stays green even if the browser doesn't even start, so the only thing it will ever show is infra failures like this.)
(In reply to Phil Ringnalda (:philor) from comment #115)
> (One possible answer would involve the way that peptest used to be hidden,
> because it was and is utterly pointless to have it unhidden, since it
> intentionally stays green even if the browser doesn't even start, so the
> only thing it will ever show is infra failures like this.)

Unhidden in bug 774817 comment 13 apparently fwiw.
https://tbpl.mozilla.org/php/getParsedLog.php?id=14300727&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound opt test mochitests-4/5 on 2012-08-10 12:35:27 PDT for push a3f1d6668fcb
slave: talos-r3-w7-024

rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14330987&tree=Mozilla-Aurora
Rev3 WINNT 6.1 mozilla-aurora pgo test reftest-no-accel on 2012-08-12 20:49:14 PDT for push 7119a45e0fea
slave: talos-r3-w7-072

rm: cannot lstat `tools/.hg/store/data/compare-locales/tests': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14346361&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound opt test crashtest on 2012-08-13 11:52:30 PDT for push 356689768e3f
slave: talos-r3-w7-070

rm: cannot lstat `tools/.hg/store/data/scripts/talos': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14354148&tree=Ionmonkey
Rev3 WINNT 6.1 ionmonkey debug test reftest on 2012-08-13 17:47:32 PDT for push a8235a2a29c2
slave: talos-r3-w7-042

rm: cannot lstat `tools/.hg/store/data/mozilla/tools/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14383946&tree=Mozilla-Aurora
Rev3 WINNT 6.1 mozilla-aurora pgo test jsreftest on 2012-08-14 18:16:49 PDT for push d343a687b8ee
slave: talos-r3-w7-046

rm: cannot lstat `tools/.hg/store/data/scripts/tooltool': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14412823&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound opt test mochitests-3/5 on 2012-08-15 13:41:22 PDT for push dad98b444955
slave: talos-r3-w7-059

rm: cannot lstat `tools/buildfarm/mobile/n900-imaging/rootfs/etc/event.d': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14446251&tree=Mozilla-Aurora
Rev3 WINNT 6.1 mozilla-aurora pgo test mochitests-1/5 on 2012-08-16 14:01:42 PDT for push 340db7fd3e4d
slave: talos-r3-w7-076

rm: cannot lstat `tools/breakpad/linux': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14455666&tree=Firefox
Rev3 WINNT 6.1 mozilla-central debug test crashtest on 2012-08-16 20:16:01 PDT for push a79132ac2f05
slave: talos-r3-w7-053

rm: cannot lstat `tools/graphserver_webapp/webapp/static': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14455846&tree=Ionmonkey
Rev3 WINNT 6.1 ionmonkey debug test jsreftest on 2012-08-16 20:21:49 PDT for push f6c3f006b57a
slave: talos-r3-w7-045

rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14585793&tree=Ionmonkey
Rev3 WINNT 6.1 ionmonkey pgo test reftest on 2012-08-21 22:12:37 PDT for push c797118c02a8
slave: talos-r3-w7-039

rm: cannot lstat `tools/.hg/store/data/breakpad/linux': Permission denied
Blocks: 784681
https://tbpl.mozilla.org/php/getParsedLog.php?id=14709124&tree=Mozilla-Aurora
slave: talos-r3-w7-005

rm: cannot lstat `tools/.hg/store/data/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools__choose__boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14984797&tree=Mozilla-Aurora
rm: cannot lstat `tools/breakpad/linux': Permission denied
Depends on: 727551
https://tbpl.mozilla.org/php/getParsedLog.php?id=15209990&tree=Mozilla-Inbound
rm: cannot remove `../../../dist/xpi-stage/specialpowers/chrome.manifest': Permission denied

Nothing like a rm failure deep into a build, to make the failures at the start of test jobs seem nice by comparison.
https://tbpl.mozilla.org/php/getParsedLog.php?id=15411940&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=15470370&tree=Firefox

(well done, win64 - we broke tests there, for the first time ever I think, and you coalesced the job on one possible push, and did this on the next one, and did this on the retrigger on the next one)
Depends on: 794760
Depends on: 794768
No longer blocks: 784681
Working on a patch in staging to use |rmdir /q /s| on Windows where we currently use |rm -rf| for removing directories.
Status: NEW → ASSIGNED
Priority: P3 → P2
Product: mozilla.org → Release Engineering
Blocks: 582821
No longer depends on: 582821
We have a working replacement for msys rm now from bug 727551 (thanks, vlad!). Per https://bugzilla.mozilla.org/show_bug.cgi?id=727551#c77, the new version of rm deletes 3 levels of nested m-c clones in <3min vs ~11min for the default msys version.

I still need to test it on the test slaves, but I'll start the deployment to the build slaves this week.
Depends on: 1011542
I've deployed this to all the Windows build/try slaves now (w64-ix, b-2008-ix, and b-2008-sm), and filed bug 1011542 to get this added to the builder GPO.

I'm getting the various flavours of Window test slaves setup in staging now.
Depends on: 1013610
This appears to have broken windows repacks for 30.0b6.
Blocks: 1013700
Depends on: 1013705
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.