test failures on talos-r3-xp-066, talos-r3-xp-067, and talos-r3-xp-009 (possible disk degradation)?

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: armenzg, Assigned: dividehex)

Tracking

Details

(Reporter)

Description

6 years ago
There are some files that cannot be removed in both of these slaves.
I tried "rm" and "rmdir" and even login as the Administrator user.
I can't change attributes either as I mentioned in bug 583129#c6.

Here are the permissions of the file for the curious:
C:\talos-slave\test\build>ls -l xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_svc_applyToDir\a\b\0\00\00png0.png
-rw-rw-rw-  1 cltbld 0 878 2012-01-17 09:52 xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_svc_applyToDir\a\b\0\00\00png0.png

On another hand, I am allowed to move the directory if I want to:
cd C:\talos-slave\
mv build delete

I've done the move for #67 but I have left #66 intact.

Please if you could run chdsk with auto-repair and/or whatever means to diagnose if there is anything bad with the disk I would greatly appreciate it.
We can also DNC if we see this becomes a rat-hole.

C:\Documents and Settings\cltbld>rm -rf C:\talos-slave\test\build
rm: cannot remove `C:\\talos-slave\\test\\build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/0/00/00png0.png': Permission denied
rm: cannot remove `C:\\talos-slave\\test\\build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/0/00/00text0': Permission denied
rm: cannot remove `C:\\talos-slave\\test\\build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/0/00/00text1': Permission denied
rm: cannot remove `C:\\talos-slave\\test\\build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0000_svc_applyToDir/a/b/0/0exe0.exe': Permission denied
0/20png0.png': Permission denied
rm: cannot remove `C:\\talos-slave\\test\\build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0181_svc_applyToDir/a/b/2/20/20text0': Permission denied
rm: cannot remove `C:\\talos-slave\\test\\build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0183_svc_applyToDir/a/b/2/20/20png0.png': Permission denied
rm: cannot remove `C:\\talos-slave\\test\\build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/0183_svc_applyToDir/a/b/2/20/20text0': Permission denied

C:\Documents and Settings\cltbld>rmdir /s /q C:\talos-slave\test\build
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_S~2\a\b\0\00\00png0.png - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_S~2\a\b\0\00\00text0 - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_S~2\a\b\0\00\00text1 - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_S~2\a\b\0\00 - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_S~2\a\b\0\0exe0.exe - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_S~2\a\b\1\10\10text0 - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_S~2\a\b\1\10 - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_S~2\a\b\SEARCH~1\searchpluginspng0.png - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_S~2\a\b\SEARCH~1\searchpluginspng1.png - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0000_S~2\a\b\SEARCH~1\searchpluginstext0 - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0111_S~2\a\b\2\20\20png0.png - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0111_S~2\a\b\2\20\20text0 - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0111_S~2\a\b\2\20 - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0151_S~2\a\b\2\20\20png0.png - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0151_S~2\a\b\2\20\20text0 - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0151_S~2\a\b\2\20 - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0181_S~2\a\b\2\20\20png0.png - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0181_S~2\a\b\2\20\20text0 - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0181_S~2\a\b\2\20 - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0183_S~2\a\b\2\20\20png0.png - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0183_S~2\a\b\2\20\20text0 - Access is denied.
C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\0183_S~2\a\b\2\20 - Access is denied.
Assignee: server-ops-releng → mlarrain
(Reporter)

Updated

6 years ago
Blocks: 683734
All the files in question are Read-Only and I cannot change them. Forced a chkdsk and waitng for it to finish
No longer blocks: 683734
both these systems aren't responding to anything I do. Pretty sure they are broken past repair. Having dividedhex reimage them.
punting to dividedhex to finish up reimaging
Assignee: mlarrain → jwatkins
Status: NEW → ASSIGNED
(Reporter)

Comment 4

6 years ago
Thanks for trying!

If they get re-imaged as XP slaves and that doesn't fix then we might want to re-image them as something else. Hopefully we won't need to.
(Assignee)

Comment 5

6 years ago
talos-r3-xp-066 and talos-r3-xp-067 have been re-imaged. sending back to releng for post re-image work.  If they still show problems, repoen and I will dive deeper.
Assignee: jwatkins → nobody
Component: Server Operations: RelEng → Release Engineering
QA Contact: zandr → release
(In reply to Jake Watkins [:dividehex] from comment #5)
> talos-r3-xp-066 and talos-r3-xp-067 have been re-imaged. sending back to
> releng for post re-image work.  If they still show problems, repoen and I
> will dive deeper.

Thanks Jake!
Duplicate of this bug: 718538
Assignee: nobody → bhearsum
Both of these are still hitting the same problems. Can we check the hardware / get them repaired?
Assignee: bhearsum → server-ops-releng
Component: Release Engineering → Server Operations: RelEng
QA Contact: release → zandr
Assignee: server-ops-releng → jwatkins
Duplicate of this bug: 718534
Also talos-r3-xp-slave009 per bug 715595.
Summary: Disk problems with talos-r3-xp-066 and talos-r3-xp-067? → test failures on talos-r3-xp-066, talos-r3-xp-067, and talos-r3-xp-009 (possible disk degradation)?
Duplicate of this bug: 715595
(Assignee)

Comment 12

6 years ago
I have run smartctl from a linux boot disk on all 3 slaves and all 3 pass short and long test. There have been 0 ata errors logged.
(Assignee)

Comment 13

6 years ago
Armenzg will put these back into testing staging systems in order to reproduce their failure states.  I will dig into this when one of them fails.
(Reporter)

Comment 14

6 years ago
Sorry for the late reply!
I looked at them and I didn't see them getting into the same state.

I will do another full run and keep track in bug 726631 when I put them in production.

Thanks for your debugging and help!
Blocks: 726631
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.