Closed Bug 360397 Opened 18 years ago Closed 17 years ago

Categories

(mozilla.org :: FTP: Staging, task, P2)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dveditz, Assigned: nthomas)

References

Details

Attachments

(3 files)

When you try to get an old release off the archive server it's extremely painful to open http://archive.mozilla.org/pub/firefox/nightly/ -- it's a couple hundred screenfuls long and takes forever to load and/or render.

Ditto http://archive.mozilla.org/pub/mozilla/nightly/ but not as bad since we hadn't gone branch-crazy by time we stopped doing those.

If the nightlies were chunked by year that would help a little and shouldn't be an administrative burden. Given we seem to be archiving 7-10 builds a day, though, that's still ~3k entries.  Could we do it by month? Either

  http://archive.mozilla.org/pub/firefox/nightly/2005/10/
or
  http://archive.mozilla.org/pub/firefox/nightly/2005-10/

I'd prefer the former myself, but it does add an extra directory level.
The final directory name should still be something like "2005-07-01-06-trunk" just for absolute clarity. The redundancy isn't going to kill anyone, and might make reorganizing easier in the future
Per an email discussion between me, preed, and Justin, changing the default QA Contact for the 18 open FTP:Staging bugs to preed.

Unusual string to help you filter bugmail for this change: craptastic!
QA Contact: myk → preed
Nick, is this something you'd be willing to take? IMO, just having the builds split up by years in http://archive.mozilla.org/pub/firefox/nightly dir. would be good enough.
Nick: do we have any automation now that moves old nightlies to archive? Creating directories for the archives is easy, as long as we're not breaking any existing automation. 

If we don't have any automation, we should write some, but that's another bug.
I had a quick look at this, and it seems like something we'd need for when archive.m.o and ftp.m.o get combined. Eg firefox/nightly on archive.m.o is just over 1 MB of HTML. Splitting it into years would give
    2004 - 130KB
    2005 - 361KB
    2006 - 472KB
The last one is still pretty big, so we may need to break it down into quarters or months.

The existing automation would be kinda obsolete once the hardware reorganisation is complete, but for the record it's a script called ftp-push-to-archive.sh. I don't think we'd need to adapt it at all, at least for my preferred layout of 

firefox/nightly
 ...
 2006
   q1
   q2
   q3
   q3
 2007
   q1
   2007-04-01-04-trunk
   2007-04-01-05-mozilla1.8
   ...

It'd just keep moving builds over to firefox/nightly, and we'd need some new tool to corral them into years and quarters.
I'd suggest splitting by year and month...

* Head off the inevitable "quarterly directories are too big" bug :-)
* No need to people's scripts to deal with converting months to quarters, and vice versa.
Yea, monthly is much better than quarterly. Daily directories would probably just be annoying.

Of course, if all you're looking for is a way to reduce the pain of loading and rendering the list of builds, there are a number of scripts floating about. I'll shamelessly plug my own (http://db48x.net/regression-search/), as it has some nifty features.
Sought further feedback from the community by posting to the newsgroups,
  http://groups.google.com/group/mozilla.dev.builds/browse_thread/thread/809cd46168a78896/2bb2369601faa8c5#2bb2369601faa8c5
Assignee: zach → nrthomas
Priority: -- → P2
Attachment #282136 - Flags: review?(preed)
Comment on attachment 282136 [details] [diff] [review]
perl script to rearrange dirs


># The dir can be relative or absolute, permissions of 755 will be enforced
># on the directories created with user:group ownership. Run this script as 
># as root to ensure that can be done.

You might want to put a little check in there to make sure we're running as root, if this script always needs to be run as root.

>  # get list of dirs that need moving before creating new dirs
>  my @dirsToMove = glob($workingDate . "*");
>  print "\n$workingDate: " . ($#dirsToMove + 1) . " dirs to move\n";

"$#dirsToMove + 1" is easier expressed as "scalar(@dirsToMove);"

>  # bail out if there's nothing to do
>  if (! @dirsToMove) {
>    IncrementMonth();
>    next;
>  }

Nit: I would express this more verbosely (but slightly easier to read) as if (0 == scalar(@dirsToMove)) {

>sub IncrementMonth {
>  my ($year, $month) = split /-/, $workingDate;
>  if ($month == 12) {

Nit: I like to write tests like these as if (12 == $month), so the compiler complains at me if I forget/mistype the second '='.

Also... IncrementMonth() modifying stuff outside its scope is kinda sketchy... but I guess this *is* a one-off. :-)

This script looks fine, although I'd like to do a bit of testing on it before we release it on the ftp-mirror at large. Let's talk offline about how we should test it before using it.

Also, please check this into the oneoffs directory at some point. :-)
Attachment #282136 - Flags: review?(preed) → review+
Comment on attachment 282136 [details] [diff] [review]
perl script to rearrange dirs

One other thing:

>GetOptions("dir=s"   => \$dir,
>           "from=s"  => \$from,
>           "to=s"    => \$to,
>           "user=s"  => \$user,
>           "group=s" => \$group
>) or die("ASSERT: Missing arguments\n");

This isn't gonna do what you want; GetOptions()'s definition of success and failure can be... kinda weird, and I ran this script without all the required arguments and it proceeded on just fine. I actually think your validation at the beginning on the script will catch this condition, but I just wanted to point out that you need to specify the options as required to make this barf the way you're intending. :-)
Attached patch v2 scriptSplinter Review
This patch addresses your earlier comments, and also creates symlinks for the most recent 31 days of builds. Seeking review on the changes.

You can see testing output on stage in ~cltbld/cf/divide-dir/. In the firefox dir there is a 
 * nightly.bak/, a bare directory tree from an rsync -av excluding files from the archive.m.o version of firefox/nightly
 * nightly/, a copy of nightly.bak with these command run 
sudo ./divide-dir.pl --dir=firefox/nightly/ --from=2004-01 --to=2007-09 --user=cltbld --group=firefox 
sudo ./divide-dir.pl --dir=firefox/nightly/contrib --from=2004-01 --to=2007-09 --user=cltbld --group=firefox 

There is also a xulrunner dir, which is a copy of the archive version of xulrunner/nightly, then I ran
sudo ./divide-dir.pl --dir=xulrunner/nightly/ --from=2004-01 --to=2007-09 --user=cltbld --group=xulrunner
You can compare that to /mnt/netapp/stage/archive.mozilla.org/pub/xulrunner/nightly/
Attachment #282465 - Flags: review?(preed)
Attachment #282465 - Flags: review?(preed) → review+
(In reply to comment #13)
> Created an attachment (id=282465) [details]
> v2 script

Committed to mofo repository in release/one-offs/.
The script was used on the directories

  calendar/sunbird/nightly
  calendar/lightning/nightly
  camino/nightly
  firebird/nightly
  firebird/nightly/contrib
  firefox/nightly
  firefox/nightly/contrib
  mozilla/nightly
  seamonkey/nightly
  seamonkey/nightly/contrib
  thunderbird/nightly
  thunderbird/nightly/contrib
  xulrunner/nightly

Getting tinderbox to publish to the new locations was bug 397541.
Blocks: 394069
Status: NEW → RESOLVED
Closed: 17 years ago
Resolution: --- → FIXED
So much better now :-)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: