Closed
Bug 360397
Opened 18 years ago
Closed 17 years ago
reorganize http://archive.mozilla.org/
Categories
(mozilla.org :: FTP: Staging, task, P2)
mozilla.org
FTP: Staging
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: dveditz, Assigned: nthomas)
References
Details
Attachments
(3 files)
2.82 KB,
patch
|
preed
:
review+
|
Details | Diff | Splinter Review |
3.78 KB,
patch
|
preed
:
review+
|
Details | Diff | Splinter Review |
4.76 KB,
text/plain
|
Details |
When you try to get an old release off the archive server it's extremely painful to open http://archive.mozilla.org/pub/firefox/nightly/ -- it's a couple hundred screenfuls long and takes forever to load and/or render. Ditto http://archive.mozilla.org/pub/mozilla/nightly/ but not as bad since we hadn't gone branch-crazy by time we stopped doing those. If the nightlies were chunked by year that would help a little and shouldn't be an administrative burden. Given we seem to be archiving 7-10 builds a day, though, that's still ~3k entries. Could we do it by month? Either http://archive.mozilla.org/pub/firefox/nightly/2005/10/ or http://archive.mozilla.org/pub/firefox/nightly/2005-10/ I'd prefer the former myself, but it does add an extra directory level.
Reporter | ||
Comment 1•18 years ago
|
||
The final directory name should still be something like "2005-07-01-06-trunk" just for absolute clarity. The redundancy isn't going to kill anyone, and might make reorganizing easier in the future
Comment 2•17 years ago
|
||
Per an email discussion between me, preed, and Justin, changing the default QA Contact for the 18 open FTP:Staging bugs to preed. Unusual string to help you filter bugmail for this change: craptastic!
QA Contact: myk → preed
Comment 4•17 years ago
|
||
Nick, is this something you'd be willing to take? IMO, just having the builds split up by years in http://archive.mozilla.org/pub/firefox/nightly dir. would be good enough.
Comment 5•17 years ago
|
||
Nick: do we have any automation now that moves old nightlies to archive? Creating directories for the archives is easy, as long as we're not breaking any existing automation. If we don't have any automation, we should write some, but that's another bug.
Assignee | ||
Comment 6•17 years ago
|
||
I had a quick look at this, and it seems like something we'd need for when archive.m.o and ftp.m.o get combined. Eg firefox/nightly on archive.m.o is just over 1 MB of HTML. Splitting it into years would give 2004 - 130KB 2005 - 361KB 2006 - 472KB The last one is still pretty big, so we may need to break it down into quarters or months. The existing automation would be kinda obsolete once the hardware reorganisation is complete, but for the record it's a script called ftp-push-to-archive.sh. I don't think we'd need to adapt it at all, at least for my preferred layout of firefox/nightly ... 2006 q1 q2 q3 q3 2007 q1 2007-04-01-04-trunk 2007-04-01-05-mozilla1.8 ... It'd just keep moving builds over to firefox/nightly, and we'd need some new tool to corral them into years and quarters.
Comment 7•17 years ago
|
||
I'd suggest splitting by year and month... * Head off the inevitable "quarterly directories are too big" bug :-) * No need to people's scripts to deal with converting months to quarters, and vice versa.
Comment 8•17 years ago
|
||
Yea, monthly is much better than quarterly. Daily directories would probably just be annoying. Of course, if all you're looking for is a way to reduce the pain of loading and rendering the list of builds, there are a number of scripts floating about. I'll shamelessly plug my own (http://db48x.net/regression-search/), as it has some nifty features.
Assignee | ||
Comment 9•17 years ago
|
||
Sought further feedback from the community by posting to the newsgroups, http://groups.google.com/group/mozilla.dev.builds/browse_thread/thread/809cd46168a78896/2bb2369601faa8c5#2bb2369601faa8c5
Assignee: zach → nrthomas
Priority: -- → P2
Assignee | ||
Comment 10•17 years ago
|
||
Attachment #282136 -
Flags: review?(preed)
Comment 11•17 years ago
|
||
Comment on attachment 282136 [details] [diff] [review] perl script to rearrange dirs ># The dir can be relative or absolute, permissions of 755 will be enforced ># on the directories created with user:group ownership. Run this script as ># as root to ensure that can be done. You might want to put a little check in there to make sure we're running as root, if this script always needs to be run as root. > # get list of dirs that need moving before creating new dirs > my @dirsToMove = glob($workingDate . "*"); > print "\n$workingDate: " . ($#dirsToMove + 1) . " dirs to move\n"; "$#dirsToMove + 1" is easier expressed as "scalar(@dirsToMove);" > # bail out if there's nothing to do > if (! @dirsToMove) { > IncrementMonth(); > next; > } Nit: I would express this more verbosely (but slightly easier to read) as if (0 == scalar(@dirsToMove)) { >sub IncrementMonth { > my ($year, $month) = split /-/, $workingDate; > if ($month == 12) { Nit: I like to write tests like these as if (12 == $month), so the compiler complains at me if I forget/mistype the second '='. Also... IncrementMonth() modifying stuff outside its scope is kinda sketchy... but I guess this *is* a one-off. :-) This script looks fine, although I'd like to do a bit of testing on it before we release it on the ftp-mirror at large. Let's talk offline about how we should test it before using it. Also, please check this into the oneoffs directory at some point. :-)
Attachment #282136 -
Flags: review?(preed) → review+
Comment 12•17 years ago
|
||
Comment on attachment 282136 [details] [diff] [review] perl script to rearrange dirs One other thing: >GetOptions("dir=s" => \$dir, > "from=s" => \$from, > "to=s" => \$to, > "user=s" => \$user, > "group=s" => \$group >) or die("ASSERT: Missing arguments\n"); This isn't gonna do what you want; GetOptions()'s definition of success and failure can be... kinda weird, and I ran this script without all the required arguments and it proceeded on just fine. I actually think your validation at the beginning on the script will catch this condition, but I just wanted to point out that you need to specify the options as required to make this barf the way you're intending. :-)
Assignee | ||
Comment 13•17 years ago
|
||
This patch addresses your earlier comments, and also creates symlinks for the most recent 31 days of builds. Seeking review on the changes. You can see testing output on stage in ~cltbld/cf/divide-dir/. In the firefox dir there is a * nightly.bak/, a bare directory tree from an rsync -av excluding files from the archive.m.o version of firefox/nightly * nightly/, a copy of nightly.bak with these command run sudo ./divide-dir.pl --dir=firefox/nightly/ --from=2004-01 --to=2007-09 --user=cltbld --group=firefox sudo ./divide-dir.pl --dir=firefox/nightly/contrib --from=2004-01 --to=2007-09 --user=cltbld --group=firefox There is also a xulrunner dir, which is a copy of the archive version of xulrunner/nightly, then I ran sudo ./divide-dir.pl --dir=xulrunner/nightly/ --from=2004-01 --to=2007-09 --user=cltbld --group=xulrunner You can compare that to /mnt/netapp/stage/archive.mozilla.org/pub/xulrunner/nightly/
Attachment #282465 -
Flags: review?(preed)
Assignee | ||
Comment 14•17 years ago
|
||
Updated•17 years ago
|
Attachment #282465 -
Flags: review?(preed) → review+
Assignee | ||
Comment 15•17 years ago
|
||
(In reply to comment #13) > Created an attachment (id=282465) [details] > v2 script Committed to mofo repository in release/one-offs/.
Assignee | ||
Comment 16•17 years ago
|
||
The script was used on the directories calendar/sunbird/nightly calendar/lightning/nightly camino/nightly firebird/nightly firebird/nightly/contrib firefox/nightly firefox/nightly/contrib mozilla/nightly seamonkey/nightly seamonkey/nightly/contrib thunderbird/nightly thunderbird/nightly/contrib xulrunner/nightly Getting tinderbox to publish to the new locations was bug 397541.
Comment 17•17 years ago
|
||
So much better now :-)
You need to log in
before you can comment on or make changes to this bug.
Description
•