551963 - Talos orange: Rev3 WINNT 6.1 Talos sometimes fails to cleanup symbols, and then fails to mkdir symbols again: "A subdirectory or file symbols already exists."

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6/1268402720.1268404921.12102.gz

Couldn't delete the old symbols dir (permission denied), and then failed to re-create it because it already existed.

Comment 1

[[e-mail] (:bhearsum)]

Log snippet, for posterity:

C:\Windows\system32\cmd.exe /c nohup rm -vrf *
 in dir c:\talos-slave\mozilla-1.9.2-win7-jss\../talos-data (timeout 1200 secs)
 watching logfiles {}
 argv: ['C:\\Windows\\system32\\cmd.exe', '/c', 'nohup rm -vrf *']
<snip>
removed `symbols/rdfpoll.pdb/05F6B46B88D242E89A6027620AA4CB422/rdfpoll.sym'
removed directory: `symbols/rdfpoll.pdb/05F6B46B88D242E89A6027620AA4CB422'
removed directory: `symbols/rdfpoll.pdb'
rm: reading directory `symbols': Permission denied
removed `talos/20100312_0538_config.yml'
<snip>
=== Output ===
C:\Windows\system32\cmd.exe /c mkdir symbols
 in dir c:\talos-slave\mozilla-1.9.2-win7-jss\../talos-data (timeout 1200 secs)
 watching logfiles {}
 argv: ['C:\\Windows\\system32\\cmd.exe', '/c', 'mkdir', 'symbols']
 environment:
  ALLUSERSPROFILE=C:\ProgramData
  APPDATA=C:\Users\cltbld\AppData\Roaming
  COMMONPROGRAMFILES=C:\Program Files\Common Files
  COMPUTERNAME=TALOS-R3-W7-002
  COMSPEC=C:\Windows\system32\cmd.exe
  FP_NO_HOST_CHECK=NO
  HOMEDRIVE=C:
  HOMEPATH=\Users\cltbld
  LOCALAPPDATA=C:\Users\cltbld\AppData\Local
  LOGONSERVER=\\TALOS-R3-W7-002
  NUMBER_OF_PROCESSORS=2
  OS=Windows_NT
  PATH=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;c:\Program Files\gnuwin32\bin;c:\Program Files\Vim\vim72\;c:\Python24\;c:\Python24\Scripts\;c:\Windows\;
  PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
  PROCESSOR_ARCHITECTURE=x86
  PROCESSOR_IDENTIFIER=x86 Family 6 Model 23 Stepping 10, GenuineIntel
  PROCESSOR_LEVEL=6
  PROCESSOR_REVISION=170a
  PROGRAMDATA=C:\ProgramData
  PROGRAMFILES=C:\Program Files
  PROMPT=$P$G
  PSMODULEPATH=C:\Windows\system32\WindowsPowerShell\v1.0\Modules\
  PUBLIC=C:\Users\Public
  SYSTEMDRIVE=C:
  SYSTEMROOT=C:\Windows
  TEMP=C:\Users\cltbld\AppData\Local\Temp
  TMP=C:\Users\cltbld\AppData\Local\Temp
  USERDOMAIN=TALOS-R3-W7-002
  USERNAME=cltbld
  USERPROFILE=C:\Users\cltbld
  WINDIR=C:\Windows
 closing stdin
 using PTY: False
A subdirectory or file symbols already exists.
program finished with exit code 1
elapsedTime=0.102000

Comment 2

[[e-mail] (:bhearsum)]

I'll try to look into this this week.

Comment 3

[[e-mail] (:bhearsum)]

I'm not going to have time to look at this

Comment 4

[Jesse Ruderman]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1269206403.1269209250.15933.gz&fulltext=1

Comment 5

[alice nodelman [:alice] [:anode]]

We've seen similar remove failures when file name length becomes too long - could be the same issue here.

Comment 6

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272204729.1272205133.1176.gz
Rev3 WINNT 6.1 mozilla-central talos scroll on 2010/04/25 07:12:09  
s: talos-r3-w7-031

Comment 7

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272222831.1272225707.16826.gz
Rev3 WINNT 6.1 mozilla-central talos tp4 on 2010/04/25 12:13:51  
s: talos-r3-w7-033

Comment 8

[Mark Banner (:standard8)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272356949.1272357428.21045.gz
Rev3 WINNT 6.1 mozilla-central talos nochrome on 2010/04/27 01:38
s: talos-r3-w7-020

Comment 9

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272370733.1272371276.23242.gz
Rev3 WINNT 6.1 mozilla-central talos on 2010/04/27 05:18:53  
s: talos-r3-w7-025

Comment 10

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272386561.1272389617.20226.gz
Rev3 WINNT 6.1 mozilla-central talos dirty on 2010/04/27 09:42:41  
s: talos-r3-w7-030

Comment 11

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272457188.1272457869.15328.gz
Rev3 WINNT 6.1 mozilla-central talos svg on 2010/04/28 05:19:48  
s: talos-r3-w7-026

Comment 12

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272499684.1272508616.18185.gz
Rev3 WINNT 6.1 mozilla-central talos tp4 on 2010/04/28 17:08:04  
s: talos-r3-w7-005

Comment 13

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272613177.1272616486.25192.gz
Rev3 WINNT 6.1 mozilla-central talos tp4 on 2010/04/30 00:39:37  
s: talos-r3-w7-023

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272613181.1272613700.16699.gz
Rev3 WINNT 6.1 mozilla-central talos nochrome on 2010/04/30 00:39:41  
s: talos-r3-w7-036

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272628652.1272631524.23412.gz
Rev3 WINNT 6.1 mozilla-central talos dirty on 2010/04/30 04:57:32  
s: talos-r3-w7-021

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272628652.1272629115.5769.gz
Rev3 WINNT 6.1 mozilla-central talos nochrome on 2010/04/30 04:57:32  
s: talos-r3-w7-018

Comment 14

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272748854.1272749260.12007.gz
Rev3 WINNT 6.1 mozilla-central talos scroll on 2010/05/01 14:20:54  
s: talos-r3-w7-005

Comment 15

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272845302.1272845855.3913.gz
Rev3 WINNT 6.1 mozilla-central talos nochrome on 2010/05/02 17:08:22  
s: talos-r3-w7-033

Comment 16

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272887938.1272888539.9392.gz
Rev3 WINNT 6.1 mozilla-central talos svg on 2010/05/03 04:58:58  
s: talos-r3-w7-034

Comment 17

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1272943004.1272943525.22444.gz
Rev3 WINNT 6.1 tracemonkey talos svg on 2010/05/03 20:16:44  
s: talos-r3-w7-011

Comment 18

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1273031440.1273033998.29243.gz
Rev3 WINNT 6.1 tracemonkey talos dromaeo on 2010/05/04 20:50:40  
s: talos-r3-w7-029

Comment 19

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1273034621.1273035852.21473.gz
Rev3 WINNT 6.1 tracemonkey talos nochrome on 2010/05/04 21:43:41  
s: talos-r3-w7-028

Comment 21

[Daniel Holbert [:dholbert]]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272971449.1272971950.31353.gz
Rev3 WINNT 6.1 mozilla-central talos scroll on 2010/05/04 04:10:49
s: talos-r3-w7-019

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273042451.1273043103.20631.gz
Rev3 WINNT 6.1 mozilla-central talos on 2010/05/04 23:54:11
s: talos-r3-w7-031

Comment 22

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1273062243.1273062838.18883.gz
Rev3 WINNT 6.1 tracemonkey talos svg on 2010/05/05 05:24:03  
s: talos-r3-w7-004

Comment 23

[Daniel Holbert [:dholbert]]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273089593.1273095730.27149.gz
Rev3 WINNT 6.1 mozilla-central talos tp4 on 2010/05/05 12:59:53
s: talos-r3-w7-011

Comment 24

[alice nodelman [:alice] [:anode]]

Attachment: move symbols out of the way

This solution worked for similar woes for tp4 - at the very least it moves the symbols directory out of the way so that a given talos run can keep going even if it can't fully clean up the directory.

Comment 25

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273557648.1273561583.21825.gz
Rev3 WINNT 6.1 mozilla-central talos tp4 on 2010/05/10 23:00:48  
s: talos-r3-w7-012

Comment 26

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273563943.1273564539.32316.gz
Rev3 WINNT 6.1 mozilla-central talos svg on 2010/05/11 00:45:43  
s: talos-r3-w7-025

Comment 27

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273594825.1273595759.22165.gz
Rev3 WINNT 6.1 mozilla-central talos on 2010/05/11 09:20:25  
s: talos-r3-w7-035

Comment 28

[[e-mail] (:bhearsum)]

Comment on attachment 444503 [details] [diff] [review]
move symbols out of the way

Do these get cleaned up at any point?

Comment 29

[alice nodelman [:alice] [:anode]]

The contents of the whole directory get 'rm -vrf', so it should be removed either this run or the next.  We will have to keep an eye for consistent failures as they could build up if they are never successfully removed.

Comment 30

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273619044.1273619845.18586.gz

Comment 31

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273608014.1273618857.13841.gz

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273605389.1273618779.13300.gz

Comment 32

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273752335.1273755090.17749.gz

Comment 33

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273751216.1273752339.21566.gz
Rev3 WINNT 6.1 mozilla-central talos on 2010/05/13 04:46:56  

"A subdirectory or file talos already exists."

New bug that'll be a clone of this, with a new patch that'll be a clone of this, or just expand this one?

Comment 34

[alice nodelman [:alice] [:anode]]

Does look like pretty much the same issue, but will require a new patch.

Comment 35

[[e-mail] (:bhearsum)]

Comment on attachment 444503 [details] [diff] [review]
move symbols out of the way

changeset:   726:598f0542ccea

Comment 36

[[e-mail] (:bhearsum)]

Restarted tm02 for this.