365989 - Win32 symbols directory overwritten by "Win32" file in Talkback symbols repo (/MozillaOrg/Thunderbird2/Win32)

Status: RESOLVED INVALID

(Release Engineering :: General, defect, P3)

Description

[Jay Patel [:jay]]

In the rsync logs for copying symbols from hal to spike, I noticed this error:

rsync warning: some files vanished before they could be transferred (code 24) at /home/lapo/packaging/tmp/rsync-2.6.6/main.c(791)
Thu Jan 4 06:18:39 PST 2007  - End Hourly SymSync
Thu Jan 4 07:15:00 PST 2007  - Start Hourly SymSync
.
.
.
MozillaOrg/Thunderbird15/MacOSX/2007010405/
MozillaOrg/Thunderbird15/MacOSX/2007010405/thunderbird-bin
MozillaOrg/Thunderbird2/
rsync: delete_file: rmdir "/g/symbols/MozillaOrg/Thunderbird2/Win32" failed: Directory not empty (90)
MozillaOrg/Thunderbird2/MacOSX/
MozillaOrg/Thunderbird2/MacOSX/2007010405/

When I went to check that directory, I found that it has been overwritten by a "Win32" file:

Administrator@hal /e/symbols/MozillaOrg/Thunderbird2
$ ll
total 22744
drwxrwxrwx+  3 symbols None        0 Jan  4 07:13 MacOSX
drwxrwxrwx+  4 symbols None        0 Jan  4 07:12 .
-rw-rw-rw-   1 symbols None 23289590 Jan  4 06:48 Win32
drwxrwxrwx+  3 symbols None        0 Jan  4 03:17 LinuxIntel
drwxrwxrwx+ 13 symbols None        0 Aug 22 07:55 ..

Looks like one of the tinderbox that pushed Talkback symbols to hal around 6:48am today 1/4 did something bad.  Please have a look so that we can start getting symbols for that branch again.

I have removed that file and recreated the directory manually, so I will check tomorrow to see if this problem repeats itself...  might have been a hiccup, but regardless, we need to make sure none of the build stuff overwrites existing directories like that with files... 

How could this have happened?  I cleaned that directory yesterday, so it was empty (or should have been) when it was overwritten, so we didn't lose any data... and I'm guessing a similar operation on a non-empty directory would throw an error and the overwrite would fail...but just in case we should find out what happened.

Comment 1

[Robert Helmer [:rhelmer]]

The time seems to line up with this nightly build on patrocles:

http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla1.8/1167912120.20123.gz&fulltext=1

Comment 2

[Robert Helmer [:rhelmer]]

This looks like the relevant snippet:

Uploading symbols...
cd `cygpath -u /cygdrive/e/builds/tinderbox/Tb-Mozilla1.8/WINNT_5.0_Depend/mozilla/../2007010404`; ./upload-symbols.sh
Making /home/symbols/symbols/MozillaOrg/Thunderbird2/Win32 on the remote host...
daemon() failed: Operation not permitted
ssh: connect to host localhost port 2222: Connection refused
Transferring symbols...
Unpacking symbols on remote host...
bash: line 0: cd: /home/symbols/symbols/MozillaOrg/Thunderbird2/Win32: Not a directory
tar (child): 2007010404.tar.bz2: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error exit delayed from previous errors

In particular:

Making /home/symbols/symbols/MozillaOrg/Thunderbird2/Win32 on the remote host...
daemon() failed: Operation not permitted

Comment 3

[Chris Cooper [:coop] (he/him)]

Invalidated by removing hal from the symbol upload process. We now upload symbols directly to spike (talkback-upload).