If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Stage not properly cleaning old nightlies to archives

RESOLVED FIXED in 4.x (triaged)

Status

AUS Graveyard
Administration
P3
normal
RESOLVED FIXED
12 years ago
10 years ago

People

(Reporter: Alex Polvi, Assigned: nthomas)

Tracking

4.x (triaged)
x86
Linux

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

12 years ago
Every night we get an email saying:

/data/ftp-archive/pub/firefox/nightly/2005-08-28-05-trunk already exists!
command 'do_push firefox' failed!

If we check what is in each directory we see:

[root@surf ~]# ls -l /data/ftp-archive/pub/firefox/nightly/2005-08-28-05-trunk
total 25228
drwxr-xr-x  2 cltbld firefox    4096 Sep  6 15:16 2005-08-28-05-trunk
-rwxrwxr-x  1 cltbld firefox 7868296 Aug 28 05:35 firefox-1.6a1.en-US.linux-i686.complete.mar
-rw-r--r--  1 cltbld mozilla  244886 Sep  1 00:50 firefox-1.6a1.en-US.linux-i686.complete.partial.2005082805-2005082905.mar
-rwxrwxr-x  1 cltbld firefox 9043805 Aug 28 05:35 firefox-1.6a1.en-US.linux-i686.installer.tar.gz
-rw-r--r--  1 cltbld mozilla  244886 Sep  6 15:16 firefox-1.6a1.en-US.linux-i686.partial.2005082805-2005082905.mar
-rwxrwxr-x  1 cltbld firefox 8370067 Aug 28 05:35 firefox-1.6a1.en-US.linux-i686.tar.gz
drwxrwxr-x  2 cltbld firefox    4096 Aug 28 05:35 linux-xpi
[root@surf ~]# ls -l /data/ftp/pub/firefox/nightly/2005-08-28-05-trunk
total 244
-rw-r--r--  1 cltbld mozilla 244886 Sep  6 15:16 firefox-1.6a1.en-US.linux-i686.partial.2005082805-2005082905.mar


This shows two bugs:

1) the archive scripts are archiving the directory into the directory if it already exists (talked to dbaron about this, simple bash fix).
2) More importantly, this shows that a 2005-08-28-05 .mar was uploaded on Sep 6.  What must have happened is since the archive has already been made, once it tried to move this new directory (was it re-added at one point?) it failed. Stage has not been archiving the nightlies since. This presumably is one of the factors of the ever growing rsync module size.

2005-08-28-05 does not seem to be the only nightly affected... so this is not an easy fix.

Any ideas why that .mar would be there twice? What is the proper cleaning technique?

Thanks for any help.
(Editing disabled while spellchecking)
Stop spell checking

Comment 1

12 years ago
This is due to the way patches are uploaded to stage.  I'll fix this soon.
Status: NEW → ASSIGNED

Updated

12 years ago
Component: Tinderbox Configuration → Administration
Product: mozilla.org → AUS
Version: other → 2.0

Comment 2

12 years ago
http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/ and http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/contrib/ should be cleaned out to archives as well... We have builds from July sitting in the first still, and I think it would be easier for FTP mirrors if those would go to archive .m.o :)
(Reporter)

Comment 3

12 years ago
When we fix the script it will clean out around 24GB. Seamonkey is another 19GB, bring the total up to ~43GB. 

Just some quick numbers... should be pretty close though.
(Reporter)

Comment 4

12 years ago
Looks like we are still getting errors... but getting a lot closer!

/data/ftp-archive/pub/firefox/nightly/2005-11-24-03-mozilla1.8 already exists!
command 'do_push firefox' failed!

[root@surf ~]# ls -1 /data/ftp/pub/firefox/nightly/2005-11-24-03-mozilla1.8
firefox-1.5.en-US.mac.partial.2005112403-2005112504.mar
[root@surf ~]# ls -1 /data/ftp-archive/pub/firefox/nightly/2005-11-24-03-mozilla1.8
firefox-1.5.en-US.mac.dmg
firefox-1.5.en-US.mac.mar
firefox-1.5.en-US.mac.partial.2005112403-2005112504.mar
mac-xpi

Comment 5

12 years ago
Mass reassign of open bugs for chase@mozilla.org to build@mozilla-org.bugs.
Assignee: chase → build
Status: ASSIGNED → NEW
(Reporter)

Comment 6

12 years ago
This is still a problem... can anyone help?


/data/ftp-archive/pub/firefox/nightly/2005-11-24-03-mozilla1.8 already exists!
command 'do_push firefox' failed!
(Reporter)

Comment 7

12 years ago
Our mirror has grown to an astonishing 253GB, could somebody please look at this issue?
Schrep: can we get some time assigned for this?  We're going to start losing FTP mirrors soon if we can't control the size of the site.

Comment 9

12 years ago
Justdave - yes - losing mirrors is bad - is this something you or Polvi can help with as Preed is a little swamped.  Paul how familiar are you iwth the stage mar upload scripts?

Assignee: build → morgamic
QA Contact: ccooper → nobody

Updated

12 years ago
Assignee: morgamic → build

Comment 10

12 years ago
Actually - coop can you lend a hand here?
OK, going through the general picture here, it looks like what is happening is the scripts uploading .mar files are re-creating old nightly build directories after this script has already moved them to archive.  The script then crashes because it's attempting to move a directory to archive that it's already moved, and it came back, so there's now a duplicate.

In comment #1, Chase said:
> This is due to the way patches are uploaded to stage.  I'll fix this soon.

Obviously that didn't get fixed, and something about the way this sounds makes me wonder if we need to...  it may be a better idea to fix the script to deal with the situation in some smart way (i.e. merge the directories if it exists in both places?  Replace the .mar file with the newer version if it already exists on archive, or something along those lines?)

CCing dbaron since from the comments in the script it appears he originally wrote it.
OK, I haven't done anything about duplicate directory resolution, but I made the duplicate directory error be non-fatal (it just skips that directory now and goes on to the next one).  First run of the updated script moved 43 GB out of ftp.m.o into archive.m.o.

I placed the script in RCS control while I was at it, since I'm sure we'll be making more changes to it. :)

We're currently only moving old nightlies for firefox, mozilla, and thunderbird from ftp to archive.  We should probably do the same for some other apps as well...  (camino, seamonkey, ???)
(In reply to comment #12)
> from ftp to archive.  We should probably do the same for some other apps as
> well...  (camino, seamonkey, ???)

Yep.  Definitely seamonkey, which didn't exist when I wrote the script; perhaps camino as well (not sure if there's enough volume to be worth it).

Comment 14

12 years ago
Thanks Dave for jumping in.  I have a meeting with Kveton to take over the mirror relationship so management of this should (hopefully) improve.
[root@surf sbin]# du -sh /pub/mozilla.org/*/nightly
5.6G    /pub/mozilla.org/camino/nightly
8.0K    /pub/mozilla.org/firebird/nightly
26G     /pub/mozilla.org/firefox/nightly
3.0M    /pub/mozilla.org/grendel/nightly
29M     /pub/mozilla.org/minimo/nightly
5.0G    /pub/mozilla.org/mozilla/nightly
8.0K    /pub/mozilla.org/phoenix/nightly
53G     /pub/mozilla.org/seamonkey/nightly
17G     /pub/mozilla.org/thunderbird/nightly
8.6G    /pub/mozilla.org/xulrunner/nightly

Comment 16

12 years ago
(In reply to comment #15)
> 53G     /pub/mozilla.org/seamonkey/nightly

Note that a fair share of that is in seamonkey/nightly/contrib/ so you might want to make the script move the dirs from both nightly/ directly and nightly/contrib/ to archive.

mozilla/nightly should only have source tarballs nowadays, I think...

Comment 17

12 years ago
Where are we on this now? Have Dave's changes fixed things enough to close this, or do we still need to manage the seamonkey files better?

Comment 18

12 years ago
I think at least the SeaMonkey files (in both nightly/ and nightly/contrib/) should still be taken care of, as the seamonkey/ dir steadily increases and will do even more so soon when we're adding a mac tinderbox doing daily builds on both branches as well...
(In reply to comment #12)
> We're currently only moving old nightlies for firefox, mozilla, and thunderbird
> from ftp to archive.  We should probably do the same for some other apps as
> well...  (camino, seamonkey, ???)

Sam filed a bug some time ago to get all Camino nightlies older than 01-01-2005 moved to archive, and that was done not too long ago.  It probably would be useful if we were included in a yearly automatic move of nightlies to archive.m.o to keep us from having 3 years worth of old builds until we come bugging someone ;)

Comment 20

12 years ago
I think those archive scripts run even more often, but they should be updated to include all dirs that need to be archived...
This would help bug 332627 a lot as well, as the numbers in comment #15 show...
I talked this over with Mike (Pinkerton) and we'd prefer to keep more than the usual ~3 weeks of Camino builds on stage/mirrors if for the simple reason that it gives us easier access to builds when finding regressions and we have fewer resources to notice those regressions.

But no one's opposed to knocking down the amount kept. I'd say if we could keep 07/2005 -> now, that'd do great for our purposes.

That's not really this bug, however. For this bug I'm saying *don't* add us to the nightly clean-out scripts, if at all possible, but *do* add us to the script which cleans out 9 month old builds. :)
IIRC, there were plans to rearrange the FTP structure so the nightly builds all go directly to a server that's equivalent to archive (with archive probably being a CNAME for it), and we have mirroring only for releases.  Not sure if I got that straight, though, or how that plan is progressing.  So I wouldn't stress too much about fixing things in the current setup.
(In reply to comment #22)
> IIRC, there were plans to rearrange the FTP structure so the nightly builds all
> go directly to a server that's equivalent to archive (with archive probably
> being a CNAME for it), and we have mirroring only for releases.  Not sure if I
> got that straight, though, or how that plan is progressing.  So I wouldn't
> stress too much about fixing things in the current setup.

Yeah, that was the plan.  CCing kveton and cshields because I think they may have been vaguely involved in it and might know the status.  I think myk might have been working on this before he stopped being a sysadmin, actually (maybe that's why it hasn't progressed recently).  If that's the case we can probably get someone else to take it on if we know what the status is.
I don't have access to the messages in which we discussed the plan because of bug 316607, but as I recall the plan was indeed to stop mirroring ftp.mozilla.org (mirroring only releases.mozilla.org), merge ftp.mozilla.org with archive.mozilla.org, and then stop moving files to the archive.

Until enough traffic got moved off ftp.mozilla.org, however, osl was going to continue mirroring that server to help us with the load, and they couldn't handle the size of a merged ftp/archive, so we delayed the merge until traffic died down enough on ftp.mozilla.org to make it practical for them to stop mirroring us.

Perhaps that time has come.
So the Web page has a whole bunch of links that point to ftp.mozilla.org but could point to releases.mozilla.org.  Should those be changed?
(In reply to comment #25)
> So the Web page has a whole bunch of links that point to ftp.mozilla.org but
> could point to releases.mozilla.org.  Should those be changed?

Maybe.  The plan was to keep releases.mozilla.org small (thus maximizing mirrorability) by regularly retiring old releases to ftp.mozilla.org.  So if you point to releases, you may have to update the link at some point.  I think the general rule should be:

1. link high-demand downloads to releases.mozilla.org
2. link low-demand downloads to ftp.mozilla.org
3. once demand wanes, update links from one to the other

To make #3 easier, perhaps we could profit from some redirect automation like bouncer.
So what I've changed so far was just mozilla-org/html/VARIABLES and mozilla-com/lib/config.tmpl, which just have pointers to source tarballs and contrib/other directories for the current release.
QA Contact: nobody → administration
*** Bug 348352 has been marked as a duplicate of this bug. ***
(Assignee)

Comment 29

11 years ago
As the recent dupe indicates, the original bug still exists (old files rather than the emailed error messages). 

The partial update files from 2005 in firefox/nightly and thunderbird/nightly are well past their use-by date. Can't they just be deleted from stage ? 
The directory modification times in firefox/nightly they haven't changed all year, and only most recent week of builds are having the partial updates freshened, so that appears to be fixed. Also the update system has changed, so we only need the partial update from the previous build (older builds are given the complete update).

The SeaMonkey bug of comment #2 seems to still be an issue.

(Assignee)

Updated

10 years ago
Assignee: build → nrthomas
Priority: -- → P3
Looks like the old FF and TB nighties are now gone, I'm not sure if this lengthy bug encompasses other issues now, so I'll leave it open. :)
(Assignee)

Comment 31

10 years ago
(In reply to comment #30)
> Looks like the old FF and TB nighties are now gone, I'm not sure if this
> lengthy bug encompasses other issues now, so I'll leave it open. :)

I cleaned those up a little while ago, thought I'd commented at the time. I filed bug 384949 for the issue in comment #2 (not all nightly dirs are automatically transferred to archive.m.o).
Status: NEW → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.