583129 - Use native windows tools to clobber directories

Status: RESOLVED DUPLICATE of bug 727551

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

We currently rely on msys's rm program to clobber unit test directories.  Instead, we should try using the os native tools to remove a directory (attrib and rmdir).

Comment 1

[Chris AtLee [:catlee]]

so, some simple timings on w32-ix-slave05:

hg clone mozilla-central t1 && time rm -rf t1
184 seconds real time

hg clone mozilla-central t1 && time cmd /c 'rmdir /s /q t1'
66 seconds real time

So rmdir is almost 3x as fast.

What's attrib for?

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

My original intention with this bug was to deal with the issues with msys' rm.  That is quite a nice performance improvement.  The issues with the rm from msys have shown to exist on a build slave (bug 643073).

(In reply to comment #1)
> So rmdir is almost 3x as fast.
> 
> What's attrib for?

It is to remove system, hidden, read-only attributes so that rmdir doesn't fail on system, hidden, read-only files.

Comment 3

[Chris AtLee [:catlee]]

attrib takes 3 minutes to work through 'configs', whereas rmdir takes 3 second to delete it...

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

but if you don't run attrib, you might not be able to run rmdir.  if return codes are set properly we could do something like

if not rmdir(configs) == 0:
    attrib(configs)
    rmdir(configs)

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

This is starting to show up on windows builds.  I think we need to either use attrib+rmdir or implement rm as a python script/c program.

Comment 6

[Armen [:armenzg]]

I'm debugging an issue where I can't change the attributes and I can't remove the files at all.

That 1 showing up for the file means that there is one hard link but not sure how to determine which.

C:\talos-slave\test\build>ls -l xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_svc_applyToDir\a\b\0\00\00png0.png
-rw-rw-rw-  1 cltbld 0 878 2012-01-17 09:52 xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_svc_applyToDir\a\b\0\00\00png0.png

C:\talos-slave\test\build>attrib -r -s -h xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_svc_applyToDir\a\b\0\00/*.* /S
Access denied - C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_svc_applyToDir\a\b\0\00\00png0.png
Access denied - C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_svc_applyToDir\a\b\0\00\00text0
Access denied - C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_svc_applyToDir\a\b\0\00\00text1

C:\talos-slave\test\build>rmdir /s /q xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_svc_applyToDir\a\b\0\00
xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_svc_applyToDir\a\b\0\00\00png0.png - Access is denied.
xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_svc_applyToDir\a\b\0\00\00text0 - Access is denied.
xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_svc_applyToDir\a\b\0\00\00text1 - Access is denied.
Access is denied.

Comment 7

[John Ford [:jhford] CET/CEST Berlin Time]

it looks like you are trying to change an Admin owned file as a non-admin user (or because of UAC, an admin in non-admin context).

Try launching a command prompt as administrator and trying again.

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

Lets look at using my winrm program, available at http://hg.mozilla.org/users/jford_mozilla.com/jhford-native-rm

Comment 9

[Armen [:armenzg]]

Related bugs:
https://bugzilla.mozilla.org/show_bug.cgi?id=583129
https://bugzilla.mozilla.org/show_bug.cgi?id=692715
https://bugzilla.mozilla.org/show_bug.cgi?id=727551

Comment 10

[Chris Cooper [:coop] (he/him)]

jhford: are you going to have time to get this deployed soon?

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Chris Cooper [:coop] from comment #10)
> jhford: are you going to have time to get this deployed soon?

nope.  putting bug back in the pool