Every night we get an email saying: /data/ftp-archive/pub/firefox/nightly/2005-08-28-05-trunk already exists! command 'do_push firefox' failed! If we check what is in each directory we see: [root@surf ~]# ls -l /data/ftp-archive/pub/firefox/nightly/2005-08-28-05-trunk total 25228 drwxr-xr-x 2 cltbld firefox 4096 Sep 6 15:16 2005-08-28-05-trunk -rwxrwxr-x 1 cltbld firefox 7868296 Aug 28 05:35 firefox-1.6a1.en-US.linux-i686.complete.mar -rw-r--r-- 1 cltbld mozilla 244886 Sep 1 00:50 firefox-1.6a1.en-US.linux-i686.complete.partial.2005082805-2005082905.mar -rwxrwxr-x 1 cltbld firefox 9043805 Aug 28 05:35 firefox-1.6a1.en-US.linux-i686.installer.tar.gz -rw-r--r-- 1 cltbld mozilla 244886 Sep 6 15:16 firefox-1.6a1.en-US.linux-i686.partial.2005082805-2005082905.mar -rwxrwxr-x 1 cltbld firefox 8370067 Aug 28 05:35 firefox-1.6a1.en-US.linux-i686.tar.gz drwxrwxr-x 2 cltbld firefox 4096 Aug 28 05:35 linux-xpi [root@surf ~]# ls -l /data/ftp/pub/firefox/nightly/2005-08-28-05-trunk total 244 -rw-r--r-- 1 cltbld mozilla 244886 Sep 6 15:16 firefox-1.6a1.en-US.linux-i686.partial.2005082805-2005082905.mar This shows two bugs: 1) the archive scripts are archiving the directory into the directory if it already exists (talked to dbaron about this, simple bash fix). 2) More importantly, this shows that a 2005-08-28-05 .mar was uploaded on Sep 6. What must have happened is since the archive has already been made, once it tried to move this new directory (was it re-added at one point?) it failed. Stage has not been archiving the nightlies since. This presumably is one of the factors of the ever growing rsync module size. 2005-08-28-05 does not seem to be the only nightly affected... so this is not an easy fix. Any ideas why that .mar would be there twice? What is the proper cleaning technique? Thanks for any help. (Editing disabled while spellchecking) Stop spell checking
This is due to the way patches are uploaded to stage. I'll fix this soon.
http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/ and http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/contrib/ should be cleaned out to archives as well... We have builds from July sitting in the first still, and I think it would be easier for FTP mirrors if those would go to archive .m.o :)
When we fix the script it will clean out around 24GB. Seamonkey is another 19GB, bring the total up to ~43GB. Just some quick numbers... should be pretty close though.
Looks like we are still getting errors... but getting a lot closer! /data/ftp-archive/pub/firefox/nightly/2005-11-24-03-mozilla1.8 already exists! command 'do_push firefox' failed! [root@surf ~]# ls -1 /data/ftp/pub/firefox/nightly/2005-11-24-03-mozilla1.8 firefox-1.5.en-US.mac.partial.2005112403-2005112504.mar [root@surf ~]# ls -1 /data/ftp-archive/pub/firefox/nightly/2005-11-24-03-mozilla1.8 firefox-1.5.en-US.mac.dmg firefox-1.5.en-US.mac.mar firefox-1.5.en-US.mac.partial.2005112403-2005112504.mar mac-xpi
Mass reassign of open bugs for email@example.com to firstname.lastname@example.org.
This is still a problem... can anyone help? /data/ftp-archive/pub/firefox/nightly/2005-11-24-03-mozilla1.8 already exists! command 'do_push firefox' failed!
Our mirror has grown to an astonishing 253GB, could somebody please look at this issue?
Schrep: can we get some time assigned for this? We're going to start losing FTP mirrors soon if we can't control the size of the site.
Justdave - yes - losing mirrors is bad - is this something you or Polvi can help with as Preed is a little swamped. Paul how familiar are you iwth the stage mar upload scripts?
Actually - coop can you lend a hand here?
OK, going through the general picture here, it looks like what is happening is the scripts uploading .mar files are re-creating old nightly build directories after this script has already moved them to archive. The script then crashes because it's attempting to move a directory to archive that it's already moved, and it came back, so there's now a duplicate. In comment #1, Chase said: > This is due to the way patches are uploaded to stage. I'll fix this soon. Obviously that didn't get fixed, and something about the way this sounds makes me wonder if we need to... it may be a better idea to fix the script to deal with the situation in some smart way (i.e. merge the directories if it exists in both places? Replace the .mar file with the newer version if it already exists on archive, or something along those lines?) CCing dbaron since from the comments in the script it appears he originally wrote it.
OK, I haven't done anything about duplicate directory resolution, but I made the duplicate directory error be non-fatal (it just skips that directory now and goes on to the next one). First run of the updated script moved 43 GB out of ftp.m.o into archive.m.o. I placed the script in RCS control while I was at it, since I'm sure we'll be making more changes to it. :) We're currently only moving old nightlies for firefox, mozilla, and thunderbird from ftp to archive. We should probably do the same for some other apps as well... (camino, seamonkey, ???)
(In reply to comment #12) > from ftp to archive. We should probably do the same for some other apps as > well... (camino, seamonkey, ???) Yep. Definitely seamonkey, which didn't exist when I wrote the script; perhaps camino as well (not sure if there's enough volume to be worth it).
Thanks Dave for jumping in. I have a meeting with Kveton to take over the mirror relationship so management of this should (hopefully) improve.
[root@surf sbin]# du -sh /pub/mozilla.org/*/nightly 5.6G /pub/mozilla.org/camino/nightly 8.0K /pub/mozilla.org/firebird/nightly 26G /pub/mozilla.org/firefox/nightly 3.0M /pub/mozilla.org/grendel/nightly 29M /pub/mozilla.org/minimo/nightly 5.0G /pub/mozilla.org/mozilla/nightly 8.0K /pub/mozilla.org/phoenix/nightly 53G /pub/mozilla.org/seamonkey/nightly 17G /pub/mozilla.org/thunderbird/nightly 8.6G /pub/mozilla.org/xulrunner/nightly
(In reply to comment #15) > 53G /pub/mozilla.org/seamonkey/nightly Note that a fair share of that is in seamonkey/nightly/contrib/ so you might want to make the script move the dirs from both nightly/ directly and nightly/contrib/ to archive. mozilla/nightly should only have source tarballs nowadays, I think...
Where are we on this now? Have Dave's changes fixed things enough to close this, or do we still need to manage the seamonkey files better?
I think at least the SeaMonkey files (in both nightly/ and nightly/contrib/) should still be taken care of, as the seamonkey/ dir steadily increases and will do even more so soon when we're adding a mac tinderbox doing daily builds on both branches as well...
(In reply to comment #12) > We're currently only moving old nightlies for firefox, mozilla, and thunderbird > from ftp to archive. We should probably do the same for some other apps as > well... (camino, seamonkey, ???) Sam filed a bug some time ago to get all Camino nightlies older than 01-01-2005 moved to archive, and that was done not too long ago. It probably would be useful if we were included in a yearly automatic move of nightlies to archive.m.o to keep us from having 3 years worth of old builds until we come bugging someone ;)
I think those archive scripts run even more often, but they should be updated to include all dirs that need to be archived... This would help bug 332627 a lot as well, as the numbers in comment #15 show...
I talked this over with Mike (Pinkerton) and we'd prefer to keep more than the usual ~3 weeks of Camino builds on stage/mirrors if for the simple reason that it gives us easier access to builds when finding regressions and we have fewer resources to notice those regressions. But no one's opposed to knocking down the amount kept. I'd say if we could keep 07/2005 -> now, that'd do great for our purposes. That's not really this bug, however. For this bug I'm saying *don't* add us to the nightly clean-out scripts, if at all possible, but *do* add us to the script which cleans out 9 month old builds. :)
IIRC, there were plans to rearrange the FTP structure so the nightly builds all go directly to a server that's equivalent to archive (with archive probably being a CNAME for it), and we have mirroring only for releases. Not sure if I got that straight, though, or how that plan is progressing. So I wouldn't stress too much about fixing things in the current setup.
(In reply to comment #22) > IIRC, there were plans to rearrange the FTP structure so the nightly builds all > go directly to a server that's equivalent to archive (with archive probably > being a CNAME for it), and we have mirroring only for releases. Not sure if I > got that straight, though, or how that plan is progressing. So I wouldn't > stress too much about fixing things in the current setup. Yeah, that was the plan. CCing kveton and cshields because I think they may have been vaguely involved in it and might know the status. I think myk might have been working on this before he stopped being a sysadmin, actually (maybe that's why it hasn't progressed recently). If that's the case we can probably get someone else to take it on if we know what the status is.
I don't have access to the messages in which we discussed the plan because of bug 316607, but as I recall the plan was indeed to stop mirroring ftp.mozilla.org (mirroring only releases.mozilla.org), merge ftp.mozilla.org with archive.mozilla.org, and then stop moving files to the archive. Until enough traffic got moved off ftp.mozilla.org, however, osl was going to continue mirroring that server to help us with the load, and they couldn't handle the size of a merged ftp/archive, so we delayed the merge until traffic died down enough on ftp.mozilla.org to make it practical for them to stop mirroring us. Perhaps that time has come.
So the Web page has a whole bunch of links that point to ftp.mozilla.org but could point to releases.mozilla.org. Should those be changed?
(In reply to comment #25) > So the Web page has a whole bunch of links that point to ftp.mozilla.org but > could point to releases.mozilla.org. Should those be changed? Maybe. The plan was to keep releases.mozilla.org small (thus maximizing mirrorability) by regularly retiring old releases to ftp.mozilla.org. So if you point to releases, you may have to update the link at some point. I think the general rule should be: 1. link high-demand downloads to releases.mozilla.org 2. link low-demand downloads to ftp.mozilla.org 3. once demand wanes, update links from one to the other To make #3 easier, perhaps we could profit from some redirect automation like bouncer.
So what I've changed so far was just mozilla-org/html/VARIABLES and mozilla-com/lib/config.tmpl, which just have pointers to source tarballs and contrib/other directories for the current release.
*** Bug 348352 has been marked as a duplicate of this bug. ***
As the recent dupe indicates, the original bug still exists (old files rather than the emailed error messages). The partial update files from 2005 in firefox/nightly and thunderbird/nightly are well past their use-by date. Can't they just be deleted from stage ? The directory modification times in firefox/nightly they haven't changed all year, and only most recent week of builds are having the partial updates freshened, so that appears to be fixed. Also the update system has changed, so we only need the partial update from the previous build (older builds are given the complete update). The SeaMonkey bug of comment #2 seems to still be an issue.
Looks like the old FF and TB nighties are now gone, I'm not sure if this lengthy bug encompasses other issues now, so I'll leave it open. :)
(In reply to comment #30) > Looks like the old FF and TB nighties are now gone, I'm not sure if this > lengthy bug encompasses other issues now, so I'll leave it open. :) I cleaned those up a little while ago, thought I'd commented at the time. I filed bug 384949 for the issue in comment #2 (not all nightly dirs are automatically transferred to archive.m.o).