631573 - Omit directory times when pushing to mirrors

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

During the firefox 4.0b11 build3 automation run, it was discovered that the en-US installer files weren't showing up for ftp.mozilla.org.  These files are showing as present on stage.mozilla.org.

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b11-candidates/build3/win32/en-US/

vs.

http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b11-candidates/build3/win32/en-US/

The files in question are "Firefox Setup 4.0 Beta 11.exe" and "Firefox Setup 4.0 Beta 11.exe.asc"

I get the same results for FTP and HTTP on ftp.mozilla.org.  I have not checked for other instances of this issue yet.

Comment 1

[juan becerra [:juanb]]

I've only gone through a few more directories, but I've found at least one more missing in Win32: ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b11-candidates/build3/win32/zh-CN/

Comment 2

[Dave Miller [:justdave]]

Verified that both servers have the same NFS share mounted on that mount point.

10.253.0.11:/vol/stage

I changed the mount options to enforce a 30 second cache timeout, and the files still aren't showing up.  Is this something server-side on the netapp?

Comment 3

[Dave Miller [:justdave]]

Aravind's out today.  Who else knows netapp stuff?

Comment 4

[Dave Miller [:justdave]]

Sounds like cshields managed to at least knock it loose...

I added a actimeo=30 to the mount options during my initial fooling around, that may have solved it, but only for newly-added files.  If Corey touched a new file in a directory with missing files, everything in the directory showed up.  Adding new files seem to show up immediately now.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

We've had this sort scenario before, but I can't recall exactly which mounting setup it was for. Basically, someone loads a new directory before builds are published into it, empty state gets cached and doesn't get expired properly. Hopefully we're good now, do let us know if you see it juanb.

Comment 6

[Armen [:armenzg]]

Is this really fixed?

Please compare:
ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build2/win32/en-US/
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build2/win32/en-US/

Comment 7

[Aravind Gottipati [:aravind]]

I added lookupcache=pos and did a remount, that got the directory listing working.  I am not sure that, the change actually improved the situation.  I will leave this bug open for now.

Comment 8

[Corey Shields [:cshields]]

reopen if this becomes a problem again

Comment 9

[Armen [:armenzg]]

See bug 636160.

ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b12-candidates/build1/win32/en-US/
VS
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b12-candidates/build1/win32/en-US/

This is happening pretty much for every release we do.

Comment 11

[Dave Miller [:justdave]]

OK, so... there's a big long thread on the linux-nfs mailing list about this almost identical situation at http://www.spinics.net/lists/linux-nfs/msg08897.html

The executive summary of that thread is at http://www.spinics.net/lists/linux-nfs/msg09052.html in case you don't feel like reading the whole thing.

The short version is it's a bug in the Linux kernel, apparently fixed in 2.6.30 (although nobody in that thread seems to have tested it to ensure that, they were only basing that on the commit message that accompanied it).  RHEL5 of course uses 2.6.18.  RHEL6 is on 2.6.32.

Comment 12

[Aravind Gottipati [:aravind]]

How are these files being copied onto surf/stage?  I am guessing it's some sort of rsync or unzip style thing.  If so, can you please point us to the exact command line used and the steps that create/update these directory trees?

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #11)/
Great to have an idea for a fix. Could we mount the partiton on a RHEL6 box to test it ? And could we do a remount (or similar) on ftp.m.o for now ?

(In reply to comment #12)
> How are these files being copied onto surf/stage?  I am guessing it's some sort
> of rsync or unzip style thing.  If so, can you please point us to the exact
> command line used and the steps that create/update these directory trees?

We use an rsync once all the windows installers are signed, 
 http://hg.mozilla.org/build/tools/file/tip/release/signing/Makefile#l351
STAGE_HOST expands to stage.m.o. 

As for the files in source directory, win32/en-US/*.exe is created first, then update/win32/en-US/*.mar, then repeat for all the locales roughly alphabetically, the create .asc file (gpg signatures) in alphabetical order.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #13)
> And could we do a remount (or similar) on ftp.m.o for now ?

n/m, this must have happened already.

Comment 15

[Aravind Gottipati [:aravind]]

Can we add a -O or --omit-dir-times flag to rsync there?  That skips directory time syncing, which I think will solve this problem.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #11)
AIUI, in our case the ls on the r/o mount is http/ftp requests on the win32/en-US directory.

(In reply to comment #15)
> Can we add a -O or --omit-dir-times flag to rsync there?  That skips directory
> time syncing, which I think will solve this problem.

I think so. bhearsum raises a good point on IRC though, which is that there are other processes which upload lots of data and they might tickle this too. Which I'm taking as 'can we fix the underlying issue ?'. What's the strategy for RHEL upgrades on IT boxes ? Is there a general program or is it being done as needed ?

Comment 17

[Corey Shields [:cshields]]

(In reply to comment #16) 
> I think so. bhearsum raises a good point on IRC though, which is that there are
> other processes which upload lots of data and they might tickle this too. Which
> I'm taking as 'can we fix the underlying issue ?'. What's the strategy for RHEL
> upgrades on IT boxes ? Is there a general program or is it being done as needed
> ?

The entire ftp/stage system needs a bit of re-engineering but the reason it hasn't happened yet is really a lack of time - which would be the biggest problem in upgrading right now as well.

There is no upgrade path for major release of RHEL, you have to start fresh with RHEL6. This would be a big project, but it is on the radar.

Comment 18

[Dave Miller [:justdave]]

FWIW ftp.mozilla.org is already behind a Zeus load balancer on an alternate IP, but we haven't switched DNS to point at it yet.  Someday when we got time and available hardware we'd been planning to spin up an additional node anyway.  Spinning up the additional nodes as RHEL6 when we do so is a no-brainer, then we can just switch to that one entirely and reload the original one.

Comment 19

[Corey Shields [:cshields]]

(In reply to comment #15)
> Can we add a -O or --omit-dir-times flag to rsync there?  That skips directory
> time syncing, which I think will solve this problem.

Has this been done?

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Omit directory times when uploading signed builds

This effectively sets all the timestamps on the directories to the upload time, rather than the time we created the detached signatures. As we discussed this shouldn't cause us any problems.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 515762 [details] [diff] [review]
[checked in] Omit directory times when uploading signed builds

http://hg.mozilla.org/build/tools/rev/2b5ad8671c22

Comment 22

[Corey Shields [:cshields]]

Closing out.  Please reopen if this workaround does not work.

Comment 23

[[e-mail] (:bhearsum)]

Happened again when we pushed to mirrors for rc1.

ftp://ftp.mozilla.org/pub/firefox/releases/4.0rc1/win32/pl/ is empty, but should have files (like http://releases.mozilla.org/pub/mozilla.org/firefox/releases/4.0rc1/win32/pl/)

Comment 24

[[e-mail] (:bhearsum)]

Actually, pulling this into RelEng so we can adjust the push to mirrors rsync, too.

Comment 25

[Dave Miller [:justdave]]

Long term fix: bug 446401.  Get those machines into production and let them take over. :)  (but not till after ff4)

Comment 26

[Chris Cooper [:coop] (he/him)]

What's left for releng to do here?

Comment 27

[[e-mail] (:bhearsum)]

I'm pretty sure we don't care about this anymore.