Closed Bug 692715 Opened 13 years ago Closed 10 years ago

Windows slaves often get permission denied errors while rm'ing files

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
x86
Windows 7
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: philor, Assigned: coop)

References

Details

(Whiteboard: [buildslaves][purple])

I know. I share your belief that it's just the way it is, it's just msys being msys. But. I star these all day, without ever keeping track of what slave that one was, or what slave the last three were. Indulge me for a little while. https://tbpl.mozilla.org/php/getParsedLog.php?id=6719240&tree=Mozilla-Inbound talos-r3-w7-015 rm: cannot lstat `tools/breakpad/osx': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=6743710&tree=Mozilla-Inbound talos-r3-w7-019 Permission denied: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmpf2vzio\\profile\\Cache\\_CACHE_001_'
Since this is w7 I wonder if "Windows Search" is running as a service here. "C:\Windows\system32\SearchIndexer.exe /Embedding"
https://tbpl.mozilla.org/php/getParsedLog.php?id=6754457&tree=Mozilla-Aurora talos-r3-w7-019 Permission denied: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmpnniqzt\\profile\\Cache\\_CACHE_001_'
Depends on: 693184
And apparently it will do some good to expand my memory of slave names, filed bug 693184 for talos-r3-w7-019's bustage.
Priority: -- → P3
Whiteboard: [buildslaves][purple]
https://tbpl.mozilla.org/php/getParsedLog.php?id=7077399&tree=Firefox slave: talos-r3-w7-013 rm: cannot lstat `tools/breakpad/linux': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7217940&tree=Mozilla-Inbound slave: talos-r3-w7-018 rm: cannot lstat `tools/.hg/store/data/mozilla/tools/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7287626&tree=Firefox slave: talos-r3-w7-012 rm: cannot lstat `tools/lib/perl/Release/Patcher': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7517294&tree=Fx-Team talos-r3-w7-040 rm: cannot lstat `tools/lib/python/slavealloc/daemon/ui': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7560553&tree=Mozilla-Inbound talos-r3-w7-043 rm: cannot lstat `tools/.hg/store/data/mozilla/tools/buildbot/buildbot/test/subdir': Permission denied
Roughly speaking, https://tbpl.mozilla.org/php/getParsedLog.php?id=7742331&tree=Mozilla-Inbound talos-r3-w7-043 'rm' '-rf' 'build' rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=7743535&tree=Mozilla-Inbound talos-r3-w7-043 'rm' '-rf' 'build' rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
Depends on: 707756
https://tbpl.mozilla.org/php/getParsedLog.php?id=7793479&tree=Mozilla-Inbound talos-r3-w7-062 rm: cannot lstat `tools/buildfarm/mobile/n900-imaging/rootfs/etc/event.d': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7832069&tree=Firefox talos-r3-w7-016 rm: cannot lstat `tools/scripts/valgrind': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7931518&tree=Firefox talos-r3-w7-021 rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=7932735&tree=Firefox talos-r3-w7-021 rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
Looks like 021 is up to either 11 or 13 failures (2 are jetpack, so they are probably utterly without meaning in any way), so it should be close to the time when it self-heals right as I start calling for it to be dealt with, right?
https://tbpl.mozilla.org/php/getParsedLog.php?id=7964379&tree=Mozilla-Inbound talos-r3-w7-064 rm: cannot remove `build/bin/ssltunnel.exe': Permission denied rm: cannot remove `build/bin/xpcshell.exe': Permission denied rm: cannot remove `build/certs/cert8.db': Permission denied rm: cannot remove `build/certs/key3.db': Permission denied rm: cannot remove `build/firefox/components/browsercomps.dll': Permission denied rm: cannot remove `build/firefox/firefox.exe': Permission denied rm: cannot remove `build/firefox/freebl3.dll': Permission denied rm: cannot remove `build/firefox/gkmedias.dll': Permission denied rm: cannot remove `build/firefox/mozalloc.dll': Permission denied rm: cannot remove `build/firefox/mozjs.dll': Permission denied rm: cannot remove `build/firefox/mozsqlite3.dll': Permission denied rm: cannot remove `build/firefox/mozutils.dll': Permission denied rm: cannot remove `build/firefox/nspr4.dll': Permission denied rm: cannot remove `build/firefox/nss3.dll': Permission denied rm: cannot remove `build/firefox/nssckbi.dll': Permission denied rm: cannot remove `build/firefox/nssdbm3.dll': Permission denied rm: cannot remove `build/firefox/nssutil3.dll': Permission denied rm: cannot remove `build/firefox/omni.ja': Permission denied rm: cannot remove `build/firefox/plc4.dll': Permission denied rm: cannot remove `build/firefox/plds4.dll': Permission denied rm: cannot remove `build/firefox/plugin-container.exe': Permission denied rm: cannot remove `build/firefox/smime3.dll': Permission denied rm: cannot remove `build/firefox/softokn3.dll': Permission denied rm: cannot remove `build/firefox/ssl3.dll': Permission denied rm: cannot remove `build/firefox/xpcom.dll': Permission denied rm: cannot remove `build/firefox/xul.dll': Permission denied rm: cannot remove `build/mochitest/server.js': Permission denied rm: cannot remove `build/mochitest/websock.log': Permission denied program finished with exit code 1
https://tbpl.mozilla.org/php/getParsedLog.php?id=7970421&tree=Mozilla-Inbound talos-r3-w7-005 rm: cannot lstat `tools/.hg/store/data/mozilla/tools/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=7989792&tree=Firefox slave: talos-r3-w7-064 rm: cannot lstat `tools/.hg/store/data/scripts/talos': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=8000923&tree=Mozilla-Inbound talos-r3-w7-040 rm: cannot lstat `tools/lib/python/slavealloc/daemon/ui': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=8070809&tree=Mozilla-Inbound talos-r3-w7-049 rm: cannot lstat `tools/scripts/talos': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=8176335&tree=Mozilla-Inbound talos-r3-w7-027 rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=8177371&tree=Firefox talos-r3-w7-027 rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=8277196&tree=Firefox talos-r3-w7-007 rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
Depends on: 714211
https://tbpl.mozilla.org/php/getParsedLog.php?id=8277591&tree=Firefox talos-r3-w7-007 rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=8279509&tree=Mozilla-Inbound talos-r3-w7-007 rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=8362943&tree=Mozilla-Inbound talos-r3-w7-018 rm: cannot lstat `tools/.hg/store/data/release/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=8392325&tree=Mozilla-Inbound talos-r3-w7-042 rm: cannot lstat `tools/.hg/store/data/scripts/nanojit': Permission denied
So that little lot started out with a fail in 'Rev3 WINNT 6.1 mozilla-aurora debug test xpcshell' on rev 89407d03f983 at 16:23: TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\xpcom\tests\unit\test_nsIProcess.js | test failed (with xpcshell return code: 0), see following log: ... TEST-UNEXPECTED-FAIL | (xpcshell/head.js) | [Exception... "Component returned failure code: 0x8052000e (NS_ERROR_FILE_IS_LOCKED) [nsILocalFile.moveTo]" nsresult: "0x8052000e (NS_ERROR_FILE_IS_LOCKED)" location: "JS frame :: c:/talos-slave/test/build/xpcshell/tests/xpcom/tests/unit/test_nsIProcess.js :: rename_and_test :: line 133" data: no] and then it hit this rm: cannot remove directory `build/xpcshell/tests/xpcom/tests/unit': Directory not empty I don't know if it managed to reboot or not. Eight subsequent unit test jobs, which share the same directory, all hit the rm error but two talos were OK. Then it does a 'Rev3 WINNT 6.1 try debug test xpcshell' (rev 64509aa1e135 at 20:09), failing the first rm, passing xpcshell, and passing the second rm. Dunno how that happens, but it now appears to be OK now and this comes back to not being able to kill off processes properly on windows test slaves.
bug 493778 comment 9 and bug 582821 comment 16 and bug 582821 comment 17 for previous instances of the NS_ERROR_FILE_IS_LOCKED, but that far in the past it's not likely to be possible to see whether that always starts an episode of the build/xpcshell/tests/xpcom/tests/unit thing, or whether episodes of it (which are sometimes here, sometimes in WFM separate bugs I've filed, and sometimes only in hypothetical logs of #build) have always been started by that. Might be possible for me to remember to look in the future, though, as long as the band of our comments about it here is wide enough for me to spot :)
https://tbpl.mozilla.org/php/getParsedLog.php?id=8537523&tree=Mozilla-Inbound talos-r3-w7-031 rm: cannot lstat `tools/.hg/store/data/release/l10n': Permission denied
That being https://tbpl.mozilla.org/php/getParsedLog.php?id=8590063&tree=Try talos-r3-xp-067 rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/0/00/00png0.png': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/0/00/00text0': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/0/00/00text1': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/0/0exe0.exe': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/1/10/10text0': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/searchplugins/searchpluginspng0.png': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/searchplugins/searchpluginspng1.png': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/searchplugins/searchpluginstext0': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0111_svc_applyToDir/a/b/2/20/20png0.png': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0111_svc_applyToDir/a/b/2/20/20text0': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0151_svc_applyToDir/a/b/2/20/20png0.png': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0151_svc_applyToDir/a/b/2/20/20text0': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0181_svc_applyToDir/a/b/2/20/20png0.png': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0181_svc_applyToDir/a/b/2/20/20text0': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0183_svc_applyToDir/a/b/2/20/20png0.png': Permission denied rm: cannot remove `build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0183_svc_applyToDir/a/b/2/20/20text0': Permission denied Probably just a coincidence that that's one of the freshly added today repurposed slaves, though, probably. And that it's one that got thrown back out of prod the last time it got added. Probably.
Or perhaps not, bug 718534.
https://tbpl.mozilla.org/php/getParsedLog.php?id=8769791&tree=Mozilla-Aurora talos-r3-w7-009 rm: cannot lstat `tools/breakpad/linux': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=8793420&tree=Profiling talos-r3-w7-035 rm: cannot lstat `tools/buildfarm/opsi': Permission denied
Depends on: 582821
https://tbpl.mozilla.org/php/getParsedLog.php?id=8875884&tree=Mozilla-Inbound talos-r3-w7-068 rm: cannot lstat `tools/.hg/store/data/buildfarm/mobile/n900-imaging-old/notes': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9094439&tree=Try talos-r3-w7-050 rm: cannot lstat `tools/.hg/store/data/mozilla/testing/tests/l10n/tests': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9165060&tree=Mozilla-Inbound talos-r3-w7-022 rm: cannot lstat `tools/.hg/store/data/buildfarm/mobile/n900-imaging-old/rootfs/root-skel': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9180298&tree=Fx-Team talos-r3-w7-015 rm: cannot lstat `tools/breakpad/linux64': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9217125&tree=Mozilla-Inbound talos-r3-w7-007 mv: cannot move `talos-data' to `t-28353': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9386865&tree=Firefox talos-r3-w7-005 rm: cannot lstat `tools/lib/python/slavealloc/daemon/ui': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9566706&tree=Profiling talos-r3-w7-018 rm: cannot lstat `tools/.hg/store/data/buildfarm/mobile/production-sd/rootfs/bin': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9664103&tree=Mozilla-Inbound talos-r3-w7-012 rm: cannot lstat `tools/.hg/store/data/scripts/release/updates': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9697051&tree=Mozilla-Inbound talos-r3-w7-063 rm: cannot lstat `tools/scripts/android': Permission denied
Depends on: 732022
Blocks: talos-r3-xp-064
No longer blocks: talos-r3-xp-064
In bug 732022 I did this to recover: C:\talos-slave\test>rm -rf build rm: reading directory `build/xpcshell/tests/xpcom/tests/unit': Invalid argument rm: cannot remove directory `build/xpcshell/tests/xpcom/tests': Directory not empty C:\talos-slave\test>rmdir /s /q build
https://tbpl.mozilla.org/php/getParsedLog.php?id=9763609&tree=Services-Central talos-r3-w7-076 rm: cannot lstat `tools/.hg/store/data/buildfarm/mobile/n900-imaging-old/notes': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=9852082&tree=Fx-Team talos-r3-w7-067 rm: cannot lstat `tools/lib/python/slavealloc/daemon/ui': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=10142698&tree=Fx-Team talos-r3-w7-049 rm: cannot lstat `tools/.hg/store/data/mozilla/app/locales/en-_u_s/other': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=10257218&tree=Mozilla-Aurora talos-r3-xp-056 mv: cannot move `talos-data' to `t-2744': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=10296053&tree=Jetpack t-r3-w764-001 rm: reading directory `scripts/.hg/store/data/mozilla/toolkit/locales/en-_u_s': Permission denied (I should probably file a new bug for the astonishingly bad Win64 test slaves, which are astonishingly bad even by Windows slave standards.)
https://tbpl.mozilla.org/php/getParsedLog.php?id=10630587&tree=Mozilla-Inbound talos-r3-w7-064 rm: cannot lstat `tools/.hg/store/data/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools__choose__boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=10721450&tree=Mozilla-Inbound talos-r3-w7-030 rm: cannot lstat `tools/.hg/store/data/scripts/talos': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=10759256&tree=Mozilla-Aurora talos-r3-xp-035 rm: cannot remove directory `build/xpcshell/tests/toolkit/components/downloads/test/unit': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=10825250&tree=Mozilla-Inbound talos-r3-w7-078 rm: cannot lstat `tools/.hg/store/data/release/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11087368&tree=Mozilla-Inbound talos-r3-w7-071 rm: cannot remove `build/bin/vmwarerecordinghelper.dll': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11088607&tree=Firefox talos-r3-w7-054 rm: cannot lstat `tools/.hg/store/data/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools__choose__boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11299910&tree=Mozilla-Inbound talos-r3-w7-061 rm: cannot lstat `tools/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/ssl_match_hostname': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11383634&tree=Fx-Team talos-r3-w7-057 rm: cannot lstat `tools/.hg/store/data/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools__choose__boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11461107&tree=Mozilla-Inbound Rev3 WINNT 6.1 mozilla-inbound debug test mochitest-other on 2012-05-04 04:07:30 PDT for push 1f576da2253d slave: talos-r3-w7-039 rm: cannot lstat `tools/.hg/store/data/release/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11551918&tree=Mozilla-Inbound Rev3 WINNT 5.1 mozilla-inbound talos chrome.2 on 2012-05-07 18:55:54 PDT for push 4940a331b147 slave: talos-r3-xp-019 mv: cannot move `talos-data' to `t-21658': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11553345&tree=Firefox mv: cannot move `talos-data' to `t-31690': Permission denied
And while it would make some sort of sense if that had been the same slave, it wasn't, that was talos-r3-xp-023.
WINNT 5.2 mozilla-central leak test build on 2012-05-11 03:12:41 PDT for push 3670d1eb1b31 slave: mw32-ix-slave02 rm: cannot remove `./dist/bin/components/browsercomps.dll': Permission denied https://tbpl.mozilla.org/php/getParsedLog.php?id=11671551&tree=Firefox
builder: mozilla-central_w764_test-jetpack slave: t-r3-w764-005 rm: reading directory `tools/lib/perl/Release': Permission denied rm: cannot remove directory `tools/lib/perl': Directory not empty https://tbpl.mozilla.org/php/getParsedLog.php?id=11740489&tree=Firefox&full=1
Assignee: nobody → coop
Component: Release Engineering → Release Engineering: Platform Support
QA Contact: release → coop
https://tbpl.mozilla.org/php/getParsedLog.php?id=11921101&tree=Mozilla-Aurora slave: talos-r3-w7-019 rm: cannot lstat `tools/.hg/store/data/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools__choose__boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=11987550&tree=Firefox slave: talos-r3-w7-078 rm: cannot lstat `tools/.hg/store/data/mozilla/tools/buildbot/buildbot/test/subdir': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12023481&tree=Mozilla-Inbound slave: talos-r3-w7-041 rm: cannot lstat `tools/scripts/android': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12342489&tree=Mozilla-Inbound slave: talos-r3-w7-039 rm: cannot lstat `tools/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools_choose_boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12470679&tree=Ionmonkey slave: mw32-ix-slave14 rm: cannot remove directory `scripts/scripts/spidermonkey_builds': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=12475351&tree=Ionmonkey slave: mw32-ix-slave14 rm: cannot lstat `scripts/scripts/spidermonkey_builds/spidermonkey.sh': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12487883&tree=Mozilla-Inbound slave: talos-r3-xp-038 rm: cannot remove `build/firefox-16.0a1.en-US.win32.zip': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12491476&tree=Mozilla-Inbound slave: talos-r3-w7-056 rm: cannot remove `build/firefox-16.0a1.en-US.win32.tests.zip': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12541698&tree=Mozilla-Inbound slave: talos-r3-w7-065 rm: cannot lstat `tools/buildfarm/mobile/n900-imaging/rootfs/etc/event.d': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12631593&tree=Profiling slave: talos-r3-w7-048 rm: cannot lstat `tools/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12752450&tree=Mozilla-Beta slave: talos-r3-w7-061 rm: cannot lstat `tools/.hg/store/data/mozilla/tools/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12873127&tree=Mozilla-Inbound slave: talos-r3-w7-062 rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12873113&tree=Services-Central slave: talos-r3-w7-041 rm: cannot lstat `tools/.hg/store/data/mozilla/tools/buildbot/buildbot/test/subdir': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=12922859&tree=Ionmonkey slave: talos-r3-w7-054 rm: cannot lstat `tools/release/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13076706&tree=Firefox slave: talos-r3-w7-042 rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13084364&tree=Ionmonkey slave: talos-r3-w7-059 rm: cannot lstat `tools/lib/python/vendor/python-daemon-1.5.5/python_daemon.egg-info': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13122605&tree=Mozilla-Inbound slave: talos-r3-w7-066 rm: cannot lstat `tools/breakpad/linux64': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13118395&tree=Mozilla-Aurora slave: talos-r3-w7-020 rm: cannot lstat `tools/.hg/store/data/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools__choose__boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13168998&tree=Mozilla-Inbound slave: talos-r3-w7-030 rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13202374&tree=Mozilla-Inbound slave: talos-r3-w7-052 rm: cannot lstat `tools/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13227659&tree=Profiling slave: talos-r3-w7-042 rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13241316&tree=Fx-Team slave: talos-r3-w7-045 rm: cannot lstat `tools/scripts/android': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13246436&tree=Mozilla-Inbound slave: talos-r3-w7-044 rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13254849&tree=Ionmonkey slave: talos-r3-w7-074 rm: cannot lstat `tools/.hg/store/data/mozilla/toolkit/locales/en-_u_s/dir': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=13270793&tree=Mozilla-Inbound slave: talos-r3-w7-073 rm: cannot lstat `tools/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14193117&tree=Mozilla-Inbound slave: talos-r3-w7-077 rm: cannot lstat `tools/.hg/store/data/mozilla/tools/buildbot/buildbot/test/subdir': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14216744&tree=Mozilla-Inbound Rev3 WINNT 6.1 mozilla-inbound opt test peptest on 2012-08-08 00:53:45 PDT for push 9428d40e8ead slave: talos-r3-w7-065 rm: reading directory `scripts/.hg/store/data/buildfarm/mobile/n900-imaging': Permission denied rm: cannot remove directory `scripts/.hg/store/data/buildfarm/mobile': Directory not empty
Rev3 WINNT 6.1 mozilla-inbound opt test peptest on 2012-08-08 03:44:03 PDT for push 4683a522952d slave: talos-r3-w7-065 https://tbpl.mozilla.org/php/getParsedLog.php?id=14219803&tree=Mozilla-Inbound { 03:44:19 INFO - ##### Running clobber step. 03:44:19 INFO - ##### 03:44:19 INFO - rmtree: c:\talos-slave\test\build Traceback (most recent call last): File "scripts/scripts/peptest.py", line 228, in <module> peptest.run() File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 620, in run self._possibly_run_method(method_name, error_if_missing=True) File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 591, in _possibly_run_method return getattr(self, method_name)() File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 641, in clobber self.rmtree(dirs['abs_work_dir']) File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 64, in rmtree self._rmdir_recursive(path) File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 105, in _rmdir_recursive self._rmdir_recursive(full_name) File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 105, in _rmdir_recursive self._rmdir_recursive(full_name) File "c:\talos-slave\test\scripts\mozharness\base\script.py", line 109, in _rmdir_recursive os.remove(full_name) WindowsError: [Error 13] Access is denied: 'c:\\talos-slave\\test\\build\\firefox\\components\\browsercomps.dll' program finished with exit code 1 }
https://tbpl.mozilla.org/php/getParsedLog.php?id=14243599&tree=Mozilla-Inbound Rev3 WINNT 6.1 mozilla-inbound opt test peptest on 2012-08-08 19:04:26 PDT for push 6c8f8fbe9aee slave: talos-r3-w7-068 rm: reading directory `scripts/.hg/store/data/mozilla/tools/buildbot': Permission denied rm: cannot remove directory `scripts/.hg/store/data/mozilla/tools': Directory not empty
https://tbpl.mozilla.org/php/getParsedLog.php?id=14282844&tree=Firefox Rev3 WINNT 6.1 mozilla-central opt test peptest on 2012-08-09 22:13:51 PDT for push 4e150530c925 slave: talos-r3-w7-060 rm: reading directory `scripts/buildfarm/mobile/n900-imaging/rootfs/etc': Permission denied rm: cannot remove directory `scripts/buildfarm/mobile/n900-imaging/rootfs': Directory not empty Riddle me this: why have the last three instances all been peptest, on slaves in the 060s?
(One possible answer would involve the way that peptest used to be hidden, because it was and is utterly pointless to have it unhidden, since it intentionally stays green even if the browser doesn't even start, so the only thing it will ever show is infra failures like this.)
(In reply to Phil Ringnalda (:philor) from comment #115) > (One possible answer would involve the way that peptest used to be hidden, > because it was and is utterly pointless to have it unhidden, since it > intentionally stays green even if the browser doesn't even start, so the > only thing it will ever show is infra failures like this.) Unhidden in bug 774817 comment 13 apparently fwiw.
https://tbpl.mozilla.org/php/getParsedLog.php?id=14300727&tree=Mozilla-Inbound Rev3 WINNT 6.1 mozilla-inbound opt test mochitests-4/5 on 2012-08-10 12:35:27 PDT for push a3f1d6668fcb slave: talos-r3-w7-024 rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14330987&tree=Mozilla-Aurora Rev3 WINNT 6.1 mozilla-aurora pgo test reftest-no-accel on 2012-08-12 20:49:14 PDT for push 7119a45e0fea slave: talos-r3-w7-072 rm: cannot lstat `tools/.hg/store/data/compare-locales/tests': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14346361&tree=Mozilla-Inbound Rev3 WINNT 6.1 mozilla-inbound opt test crashtest on 2012-08-13 11:52:30 PDT for push 356689768e3f slave: talos-r3-w7-070 rm: cannot lstat `tools/.hg/store/data/scripts/talos': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14354148&tree=Ionmonkey Rev3 WINNT 6.1 ionmonkey debug test reftest on 2012-08-13 17:47:32 PDT for push a8235a2a29c2 slave: talos-r3-w7-042 rm: cannot lstat `tools/.hg/store/data/mozilla/tools/l10n': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14383946&tree=Mozilla-Aurora Rev3 WINNT 6.1 mozilla-aurora pgo test jsreftest on 2012-08-14 18:16:49 PDT for push d343a687b8ee slave: talos-r3-w7-046 rm: cannot lstat `tools/.hg/store/data/scripts/tooltool': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14412823&tree=Mozilla-Inbound Rev3 WINNT 6.1 mozilla-inbound opt test mochitests-3/5 on 2012-08-15 13:41:22 PDT for push dad98b444955 slave: talos-r3-w7-059 rm: cannot lstat `tools/buildfarm/mobile/n900-imaging/rootfs/etc/event.d': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14446251&tree=Mozilla-Aurora Rev3 WINNT 6.1 mozilla-aurora pgo test mochitests-1/5 on 2012-08-16 14:01:42 PDT for push 340db7fd3e4d slave: talos-r3-w7-076 rm: cannot lstat `tools/breakpad/linux': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14455666&tree=Firefox Rev3 WINNT 6.1 mozilla-central debug test crashtest on 2012-08-16 20:16:01 PDT for push a79132ac2f05 slave: talos-r3-w7-053 rm: cannot lstat `tools/graphserver_webapp/webapp/static': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14455846&tree=Ionmonkey Rev3 WINNT 6.1 ionmonkey debug test jsreftest on 2012-08-16 20:21:49 PDT for push f6c3f006b57a slave: talos-r3-w7-045 rm: cannot lstat `tools/.hg/store/data/scripts/support': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14585793&tree=Ionmonkey Rev3 WINNT 6.1 ionmonkey pgo test reftest on 2012-08-21 22:12:37 PDT for push c797118c02a8 slave: talos-r3-w7-039 rm: cannot lstat `tools/.hg/store/data/breakpad/linux': Permission denied
Blocks: 784681
https://tbpl.mozilla.org/php/getParsedLog.php?id=14709124&tree=Mozilla-Aurora slave: talos-r3-w7-005 rm: cannot lstat `tools/.hg/store/data/lib/python/vendor/requests-0.10.8/requests/packages/urllib3/packages/mimetools__choose__boundary': Permission denied
https://tbpl.mozilla.org/php/getParsedLog.php?id=14984797&tree=Mozilla-Aurora rm: cannot lstat `tools/breakpad/linux': Permission denied
Depends on: 727551
https://tbpl.mozilla.org/php/getParsedLog.php?id=15209990&tree=Mozilla-Inbound rm: cannot remove `../../../dist/xpi-stage/specialpowers/chrome.manifest': Permission denied Nothing like a rm failure deep into a build, to make the failures at the start of test jobs seem nice by comparison.
https://tbpl.mozilla.org/php/getParsedLog.php?id=15411940&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=15470370&tree=Firefox (well done, win64 - we broke tests there, for the first time ever I think, and you coalesced the job on one possible push, and did this on the next one, and did this on the retrigger on the next one)
Depends on: 794760
Depends on: 794768
No longer blocks: 784681
Working on a patch in staging to use |rmdir /q /s| on Windows where we currently use |rm -rf| for removing directories.
Status: NEW → ASSIGNED
Priority: P3 → P2
Product: mozilla.org → Release Engineering
Blocks: 582821
No longer depends on: 582821
We have a working replacement for msys rm now from bug 727551 (thanks, vlad!). Per https://bugzilla.mozilla.org/show_bug.cgi?id=727551#c77, the new version of rm deletes 3 levels of nested m-c clones in <3min vs ~11min for the default msys version. I still need to test it on the test slaves, but I'll start the deployment to the build slaves this week.
Depends on: 1011542
I've deployed this to all the Windows build/try slaves now (w64-ix, b-2008-ix, and b-2008-sm), and filed bug 1011542 to get this added to the builder GPO. I'm getting the various flavours of Window test slaves setup in staging now.
Depends on: 1013610
This appears to have broken windows repacks for 30.0b6.
Blocks: 1013700
Depends on: 1013705
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.