store last 24 hours of hourly builds for accurate regression and perf tests

RESOLVED FIXED

Status

defect
P3
major
RESOLVED FIXED
15 years ago
6 years ago

People

(Reporter: Peter6, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: in production)

Attachments

(9 attachments, 4 obsolete attachments)

85 bytes, text/plain
nthomas
: review-
Details
3.06 KB, patch
nthomas
: review+
Details | Diff | Splinter Review
2.44 KB, patch
rcampbell
: review+
Details | Diff | Splinter Review
4.25 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
4.74 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
32.02 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
2.12 KB, patch
nthomas
: review+
Details | Diff | Splinter Review
129 bytes, text/plain
bhearsum
: review+
Details
2.02 KB, patch
rcampbell
: review+
Details | Diff | Splinter Review
To track down regressions more accurate (preferably down to the build hour) it
would be very usefull if all hourly builds of the past 24 h would allways be
available
CC Chase
OS: Windows 2000 → All
Hardware: PC → All
Assignee: nobody → chase
Component: Build Config → Build & Release
Product: Firefox → mozilla.org
QA Contact: build-config → chase
Version: unspecified → other
Mass reassign of open bugs for chase@mozilla.org to build@mozilla-org.bugs.
Assignee: chase → build
I've worked up a script-based approach at
  http://hourly-archive.localgho.st/

It keeps 1 week of Firefox builds from the trunk.
We need this to make test-only machines more accurate.
Assignee: build → rhelmer
QA Contact: chase → preed
One idea I have for this is to have tinderbox include the (unix-style) timestamp that represents the pull date and build start time (which is the only way the server has of matching up build start/stop times) in the hourly directory, such as:

ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/fx-win32-tbox-trunk/1170370500/ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/fx-win32-tbox-trunk/1170370500/firefox-3.0a1.en-US.win32.installer.exe

It'd be up to the FTP server to clear old directories out periodically. Since that number represents the pull date, that would enable us to match up bonsai queries and give the test-only machines the data they need.

Also, I'd like to change this directory to say "hourly" or "depend" or something more apparent, and get rid of the redundant "tinderbox" and "tinderbox-builds" directories, since the names are very ambiguous (nightly builds are tinderbox builds too!).

Thoughts?
(In reply to comment #5)

> It'd be up to the FTP server to clear old directories out periodically.

We'd need to make sure that there's something there to do this before this change goes in.

Maybe bug 342972 should be a blocker to this?
(In reply to comment #6)
> (In reply to comment #5)
> 
> > It'd be up to the FTP server to clear old directories out periodically.
> 
> We'd need to make sure that there's something there to do this before this
> change goes in.
> 
> Maybe bug 342972 should be a blocker to this?

Agreed.

Depends on: 342972
I vote for standardizing on "tinderbox-builds", since "hourly" isn't quite accurate either. Yes, nightly builds are also tinderbox builds, and they should also end up linked here.

I would also like to recommend reformatting the unix-style timestamp part of the path to be human-readable, e.g. YYYYMMDDHHmmSS (I also wish tinderbox history did this, FWIW). This will make it easier for humans to find the builds they need without really influencing any automated components.
Severity: normal → enhancement
Status: NEW → ASSIGNED
Priority: -- → P3
Whiteboard: waiting on stage migration
Nothing to do here until new stage.m.o is up.
Assignee: rhelmer → build
Status: ASSIGNED → NEW
Assignee: build → nobody
QA Contact: mozpreed → build
I'd really like to push this out until *after* we start building more intelligently, i.e. only when we have a code change. 

Maybe this means waiting for buildbot (bug 299909 and/or bug 401936).
Blocks: 410869
(In reply to comment #10)
> I'd really like to push this out until *after* we start building more
> intelligently, i.e. only when we have a code change. 
> 
> Maybe this means waiting for buildbot (bug 299909 and/or bug 401936).

This is probably a ways off; I don't think it's unreasonable to clean the depend dir of everything over 24h is it? 
No longer depends on: 299909
I'd like to remove the blocking bug 342972 and just set this up. What if we make the builders upload to a unique depend upload dir e.g. 
 ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/FX-WIN32-TBOX-trunk/20070122112700/

And have a braindead cleanup script that removes everything older than 24h in /pub/mozilla/org/firefox/tinderbox-builds/

Once stage migration is complete and supersedes this, wonderful, but it is blocking other things that we need such as bug 410869.
Assignee: nobody → rhelmer
(In reply to comment #12)
> I'd like to remove the blocking bug 342972 and just set this up. What if we
> make the builders upload to a unique depend upload dir e.g. 
> 
> ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/FX-WIN32-TBOX-trunk/20070122112700/
That means a lot of extra time needed for users to download stamped builds if they are in their own separate directories.

> And have a braindead cleanup script that removes everything older than 24h in
> /pub/mozilla/org/firefox/tinderbox-builds/

Based on quite some experience with http://hourly-archive.localgho.st/ I'd say make it at least a few days.
Why not use the setup like http://hourly-archive.localgho.st/ ?
It's user friendly for downloading and offers all we (testers) need including a link to the checkins for that particular build.

Just keep "nightly" and "hourly" builds separate.
Hourly build testers are not the same as nightly build testers.
(In reply to comment #13)
> (In reply to comment #12)
> > I'd like to remove the blocking bug 342972 and just set this up. What if we
> > make the builders upload to a unique depend upload dir e.g. 
> > 
> > ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/FX-WIN32-TBOX-trunk/20070122112700/
> That means a lot of extra time needed for users to download stamped builds if
> they are in their own separate directories.
> 
> > And have a braindead cleanup script that removes everything older than 24h in
> > /pub/mozilla/org/firefox/tinderbox-builds/
> 
> Based on quite some experience with http://hourly-archive.localgho.st/ I'd say
> make it at least a few days.
> Why not use the setup like http://hourly-archive.localgho.st/ ?
> It's user friendly for downloading and offers all we (testers) need including a
> link to the checkins for that particular build.
> 
> Just keep "nightly" and "hourly" builds separate.
> Hourly build testers are not the same as nightly build testers.


Nick, what do you think about making hourly-archive a supported mozilla.org service?
Ok, because of the way Talos works we're hitting this problem a lot more often:

1) buildbot TinderboxPoller sees there's a new Tinderbox build, queues it up
2) test machine becomes available, downloads build and tests
3) reports in with the timestamp from #1

In the time between 1 and 2, it's quite likely that another build was created and uploaded. We would like to be performance testing each checkin individually, so we really need to be storing more than one depend build at a time for this to work.

Upping the severity and priority on this. I don't think we should be blocking on stage migration, a simple script just for tinderbox-builds (clear out files over 24h old) and patching tinderbox to upload to a unique directory are the short path to stop the bleeding here, we can move to the ideal solution when it becomes available.
Severity: enhancement → major
Priority: P3 → P2
unix timestamp in here looks ugly, we can reformat if you like. We should definitely have resolution to the second, though.
Attachment #302265 - Flags: review?(nrthomas)
This just does firefox, I guess we'd also need camino, xulrunner, thunderbird as well.
Attachment #302266 - Flags: review?(nrthomas)
Whiteboard: waiting on stage migration → waiting for review
Attachment #302266 - Attachment is patch: false
Do we want/need to keep build for all apps automatically, or should this be opt-in ? The disk space requirements are non-negligble I bet.
(In reply to comment #18)
> Do we want/need to keep build for all apps automatically, or should this be
> opt-in ? The disk space requirements are non-negligble I bet.

I could make this configurable, then other projects would have to opt in, does that sound reasonable?
Status: NEW → ASSIGNED
Sounds good to me.
Attachment #302265 - Attachment is obsolete: true
Attachment #302265 - Flags: review?(nrthomas)
Working on testing this today, looks like it should do the right thing.
Attachment #303341 - Flags: review?(nrthomas)
Comment on attachment 303341 [details] [diff] [review]
only push to dated depend dir if DependToDated is set

>Index: post-mozilla-rel.pl
>===================================================================
>+    } elsif (! $cachebuild && $Settings::DependToDated) {
>+      $datedir = $start_time;

You'll need to append a "/" after $start_time, and default 0 value to tinder-defaults.pl. If the testing goes ok, then please add those on commit.
Attachment #303341 - Flags: review?(nrthomas) → review+
Comment on attachment 302266 [details]
crontab to clear out old tinderbox-builds

You'll want -mtime +1 there for one day, and a second command to remove empty directories.
Attachment #302266 - Flags: review?(nrthomas) → review-
Depends on: 417633
or -ctime +1 for creation time.
Just to confirm: We are using the start time (in unix time format) as the DependToDated directory, right?

If so, there _may_ be an issue with MozillaStageUpload. It retrieves the BuildID from application.ini and as such only has to-the-hour granularity. It'd be easy for Buildbot to assume that the time is 00:00:00 - but what how do we distinguish between different dep builds? I guess Buildbot could pull it's build-start time and use the time from that, but I'm not sure that's exactly what we want (and it may happen that right as the day switches over we get a wacky thing like Feb 15, 2008 @ 23:59:00 when it's _actually_ Feb 15, 2008 @ 00:01:00.

Is there any way to retrieve a down-to-the-hour buildid/start time?
(In reply to comment #25)
> Just to confirm: We are using the start time (in unix time format) as the
> DependToDated directory, right?
> 
> If so, there _may_ be an issue with MozillaStageUpload. It retrieves the
> BuildID from application.ini and as such only has to-the-hour granularity. It'd
> be easy for Buildbot to assume that the time is 00:00:00 - but what how do we
> distinguish between different dep builds? I guess Buildbot could pull it's
> build-start time and use the time from that, but I'm not sure that's exactly
> what we want (and it may happen that right as the day switches over we get a
> wacky thing like Feb 15, 2008 @ 23:59:00 when it's _actually_ Feb 15, 2008 @
> 00:01:00.
> 
> Is there any way to retrieve a down-to-the-hour buildid/start time?

This value is going to be equivalent to the pulldate used, which is what we want for performance testing. 

This is currently only available from one place that I know of, and that's Tinderbox (unix timestamp must be sent in beginning/end email), so the idea is that Talos or anything else that needs to know this info can ask Tinderbox for the latest build, then find the equivalent directory on FTP.

Whiteboard: waiting for review → testing
Blocks: 406896
I think we are ready to go on this end, but Talos and the perf-test-running Tinderboxes (unless Talos replaces them soon) need to support the new directory scheme first.
Whiteboard: testing → waiting for Talos/Tinderbox change
Summary: store last 24 hours of hourly builds for accurate regression tests → store last 24 hours of hourly builds for accurate regression and perf tests
(In reply to comment #27)
> I think we are ready to go on this end, but Talos and the perf-test-running
> Tinderboxes (unless Talos replaces them soon) need to support the new directory
> scheme first.
Well, the perf-test-running Tinderboxes are being mothballed (see bug#413695 for details) so no need to touch those. 

Alice: whats involved is making the matching Talos change?  
(In reply to comment #28)
> (In reply to comment #27)
> > I think we are ready to go on this end, but Talos and the perf-test-running
> > Tinderboxes (unless Talos replaces them soon) need to support the new directory
> > scheme first.
> Well, the perf-test-running Tinderboxes are being mothballed (see bug#413695
> for details) so no need to touch those. 
> 
> Alice: whats involved is making the matching Talos change?  

There's some discussion of this in bug 417633 (marked blocking this bug).

Whiteboard: waiting for Talos/Tinderbox change → need to coordinate Talos change
Nothing to do here until the old perf machines are gone, and the Talos patch is ready.
Assignee: rhelmer → nobody
Status: ASSIGNED → NEW
(In reply to comment #30)
> Nothing to do here until the old perf machines are gone, 
Marked bug#413695 as blocking.

> and the Talos patch is
> ready.


Removed dependency on 342972 because this is covered with the crontab patch in this bug.
No longer depends on: 342972
Priority: P2 → P3
QA Contact: build → release
Can I get an update on this bug state?  From the comments it looks like we have a patch for the builder side of the world (ie, dropping builds into time stamped directories) but are missing the talos patch.

Is the time stamped directories solution up and running anywhere?  I'd like to start working on the talos patch and it would be nice to have something test with.
Are we also missing a script to do the cleanup after 24 hours to stop massive browser build up?
I set this up as a local patch on moz180-linux-tbox, which publishes hourlies to
  http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/moz180-linux-tbox-mozilla1.8.0/

This build is set to run less frequently than it could, about every 80 minutes, but we can speed it up if that'd help.
Do we care about "last 24 hours of hourly builds" or would "last 'n' hourly builds" be just as good?

(I'm trying to separate out whether we care about all builds in a specific calendar day or if we just care about a continuous sequence of recent builds.)
(In reply to comment #36)
> Do we care about "last 24 hours of hourly builds" or would "last 'n' hourly
> builds" be just as good?
> 
> (I'm trying to separate out whether we care about all builds in a specific
> calendar day or if we just care about a continuous sequence of recent builds.)
> 
That very much depends on the amount of builds.
The Win32 box used to spit out 30+ new builds (with changes) a day, in which case you would want a day of builds (minimum), right now it spits out 13-14 a day tops, which would mean you'd have 2.5 days of builds (if you fix N to 30).

I'd rather see a few days archived anyway.
(In reply to comment #36)
> Do we care about "last 24 hours of hourly builds" or would "last 'n' hourly
> builds" be just as good?
> 
> (I'm trying to separate out whether we care about all builds in a specific
> calendar day or if we just care about a continuous sequence of recent builds.)
> 

"Latest n hourly builds" might be better if for some reason no more builds are produced for several days (like SeaMonkey-trunk for Linux at the moment, see bug 429603). Or else, maybe the bug Summary should specify more explicitly that "the last 24 hours of hourlies" doesn't mean "the hourlies from the last 24 hours" but "the 24 hours of hourlies ending with the latest build".

In reply to comment #37
Maybe the number of "builds to be kept" could be parameterized separately for each product/version/platform etc.?
(In reply to comment #38)
> In reply to comment #37
> Maybe the number of "builds to be kept" could be parameterized separately for
> each product/version/platform etc.?

At the time i filed this bug http://hourly-archive.localgho.st/
 did not exist and anything more than just being able to download the latest hourly was a bonus.

7200 Firefox downloads (on my HD) later I'd say that a week of builds is enough, 3 days is minimum.
The 3 days specifically because it would allow you to do something else (away from the computer) during the weekend.
Comment on attachment 303341 [details] [diff] [review]
only push to dated depend dir if DependToDated is set

>Index: build-seamonkey-util.pl
>===================================================================
>         if (((-e $external_build) and ($build_status eq 'success')) || 
>             ($Settings::SkipMozilla)) {
>-            ($build_status, $binary_url) = PostMozilla::main($build_dir);
>+            ($build_status, $binary_url) = 
>+             PostMozilla::main($build_dir, $server_start_time);

I think this should be just $server_time, testing now.
Argh, I meant $start_time.
Ok, looks like it's working, making dirs
  1208895900moz180-linux-tbox-mozilla1.8.0/
  1208897940moz180-linux-tbox-mozilla1.8.0/
in
  http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/

I must have misunderstood how this was supposed to work, as I was expecting
  1208895900/
  1208897940/
in
  http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/moz180-linux-tbox-mozilla1.8.0/
Is this the format that we want to settle on?  I'll assume that if I can get the talos buildbot to interpret this correctly it wouldn't be much of a leap to something else - but it might be a good time to consider what format we would prefer instead of just what we have around.
(In reply to comment #42)
> Ok, looks like it's working, making dirs
>   1208895900moz180-linux-tbox-mozilla1.8.0/
>   1208897940moz180-linux-tbox-mozilla1.8.0/
> in
>   http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/
> 
> I must have misunderstood how this was supposed to work, as I was expecting
>   1208895900/
>   1208897940/
> in
>  
> http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/moz180-linux-tbox-mozilla1.8.0/
> 

My understanding is, e.g.:
fx-linux-tbox/1208855900
fx-linux-tbox/1208855932

This is the way buildbotcustom.steps.transfer.MozillaStageUpload works.
Posted patch Updated patch (obsolete) — Splinter Review
Hopefully this will do the trick, lets see what moz180-linux-tbox creates.
Adds a MozillaWgetLatestDated class to perfrunner.py.  For this code to be exercised the master.cfg's of production/try/stage talos buildbots will have to be patched to use this class instead of the standard MozillaWgetLatest.
Attachment #318701 - Flags: review?(rcampbell)
Comment on attachment 318701 [details] [diff] [review]
patches perfrunner to add ability to pull from a dated directory (for talos testers)

this looks like it should work to my eyes. One thing I noticed is that there's no real failure handling in the start() method if filename comes back empty. An assert fileURL = "more than url" would probably do the trick.

otherwise, all good.
Attachment #318701 - Flags: review?(rcampbell) → review+
if i understand this right the builds are going to be saved into a dated directory and will have their standard name w/o indication of date/time.

I would prefer the filename format as is (on http://hourly-archive.localgho.st/win32.html) like 20080502_0448_N_firefox-3.0pre.en-US.win32 for the latest win32 build.

YYYYMMDD of checkout
HHMM of checkout
N for nightly and clobbered builds
firefox for type
-3.0pre for firefox version

and simply store them into one dir, or 1 dir per day

It sucks if you have to go to another directory for every build and it's even worse if you have to rename each file and/or create a new directory for each file.

Please look at http://hourly-archive.localgho.st/win32.html , which was designed for ease of use and don't try to reinvent this into a system that is horribly user unfriendly.
(In reply to comment #48)
> if i understand this right the builds are going to be saved into a dated
> directory and will have their standard name w/o indication of date/time.

correct.

> I would prefer the filename format as is (on
> http://hourly-archive.localgho.st/win32.html) like
> 20080502_0448_N_firefox-3.0pre.en-US.win32 for the latest win32 build.
> 
[snip]
> 
> and simply store them into one dir, or 1 dir per day
> 
> It sucks if you have to go to another directory for every build and it's even
> worse if you have to rename each file and/or create a new directory for each
> file.
> 
> Please look at http://hourly-archive.localgho.st/win32.html , which was
> designed for ease of use and don't try to reinvent this into a system that is
> horribly user unfriendly.

The purpose of this is not to provide an hourly archive of builds, but to store 24 hours for tracking regressions and providing more fine-grained accuracy for the performance testing toolkit. Since we're only dealing with 24 hours of builds, this is not *that* onerous.
(In reply to comment #49)
> The purpose of this is not to provide an hourly archive of builds, but to store
> 24 hours for tracking regressions and providing more fine-grained accuracy for
> the performance testing toolkit. Since we're only dealing with 24 hours of
> builds, this is not *that* onerous.
> 
That is not what I filed this bug for.
Peter, is there anything that hourly-archive isn't doing for you right now ? I know this got morphed at about comment #5, but sounds like there are two different goals here.
(In reply to comment #51)
> Peter, is there anything that hourly-archive isn't doing for you right now ? I
> know this got morphed at about comment #5, but sounds like there are two
> different goals here.
> 
Nope, the hourly-archive is fine with me. 
I was assuming the current hourly-archive would be transformed into an official mozilla page.
Currently the archive is only linked from my threads and nowhere from e.g. http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/ or any other official mozilla.org page.
I can file a bug for that if needed
Same as attachment 317225 [details] [diff] [review] but with a tinder-defaults.pl line.
Attachment #317225 - Attachment is obsolete: true
Attachment #319566 - Flags: review?(bhearsum)
Attachment #319566 - Attachment description: v3, with tinder-defaults.pl → tinderbox changes v3, with tinder-defaults.pl
Attachment #319567 - Flags: review?(bhearsum) → review+
Attachment #319566 - Flags: review?(bhearsum) → review+
I've tested all three code paths for upload (by disabling DependToDated for a few cycles on moz180-linux-tbox) so the tinderbox code-change is ready to go. The Firefox tree is a mess at the moment so delaying landing that in case of unforeseen bustage. 

Once it does go in, then the tinderbox config change (attach. 319567) should be committed when we want to deploy this. We'll also need a cron job to clean up old builds.
Whiteboard: need to coordinate Talos change → needs scheduled downtime, need to coordinate Talos change
Comment on attachment 319566 [details] [diff] [review]
[checked in] tinderbox changes v3, with tinder-defaults.pl

mozilla/tools/tinderbox/build-seamonkey-util.pl 	1.389
mozilla/tools/tinderbox/tinder-defaults.pl 	1.128
mozilla/tools/tinderbox/post-mozilla-rel.pl 	1.146
Attachment #319566 - Attachment description: tinderbox changes v3, with tinder-defaults.pl → [checked in] tinderbox changes v3, with tinder-defaults.pl
Comment on attachment 320563 [details] [diff] [review]
switch to pull by date for talos

Seems fine
Attachment #320563 - Flags: review?(bhearsum) → review+
Attachment #320562 - Attachment is obsolete: true
Attachment #320566 - Flags: review?(nrthomas)
Attachment #320562 - Flags: review?(nrthomas)
Attachment #320566 - Flags: review?(nrthomas) → review+
Comment on attachment 319567 [details] [diff] [review]
[checked in] Set DependToDated on the builds Talos tests

mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl 	1.1.12.26
mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl 	1.8.4.25
mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl 	1.2.10.27
mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl 	1.27
mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl 	1.45
mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl 	1.36
Attachment #319567 - Attachment description: Set DependToDated on the builds Talos tests → [checked in] Set DependToDated on the builds Talos tests
Comment on attachment 320566 [details] [diff] [review]
[checked in] enable dependToDated for nightlies too.

Landed in cd3463fd0743
Attachment #320566 - Attachment description: enable dependToDated for nightlies too. → [checked in] enable dependToDated for nightlies too.
talos changes to perfrunner and master.cfg's landed

Checking in perf-staging/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perf-staging/master.cfg,v  <--  master.cfg
new revision: 1.12; previous revision: 1.11
done
Checking in perfmaster2/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/master.cfg,v  <--  master.cfg
new revision: 1.56; previous revision: 1.55
done
Checking in perfmaster2/perfrunner.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/perfrunner.py,v  <--  perfrunner.py
new revision: 1.15; previous revision: 1.14
done
A quick an dirty way to remove any epoch-named dirs older than 1 day from 
  /pub/mozilla.org/firefox/tinderbox-builds/*/
Attachment #320601 - Flags: review?(bhearsum)
Whiteboard: needs scheduled downtime, need to coordinate Talos change → in production
Comment on attachment 320601 [details]
[installed] contab mk2

We'll need to update this script on Wed, 18 May 2033 03:33:20 GMT, but this is fine for now :)
Attachment #320601 - Flags: review?(bhearsum) → review+
Comment on attachment 320601 [details]
[installed] contab mk2

Installed to dm-stage01:/etc/cron.d/remove-hourly-builds with @hourly (I realized we'd get two days of builds then chop it down to one at midnight otherwise). Also temporarily made it rm -rfv to verify it's working ok (mail to me).

Also cleaned up all the files that now live in directories (eg bm-xerve08/firefox...).
Attachment #320601 - Attachment description: contab mk2 → [installed] contab mk2
Further tweaked it so that find doesn't complaing, currently is

@hourly find /pub/mozilla.org/firefox/tinderbox-builds/ -mindepth 2 -maxdepth 2 -type d -mtime +1 -name 1????????? | xargs rm -rfv
One thing we didn't remember is that tinderbox doesn't publish nightly builds to the tinderbox-builds directory, but Talos does try to download that build. If I'd not removed the old files we'd have been testing an old build once a day - having removed them we get Talos burning.

Can we make tinderbox poller ignore builds with a url given ? Eg

Build|Mozilla1.8|MacOSX Darwin 8.7.0 bm-xserve02 Depend Tb-Nightly|success|1210673160|http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2008-05-13-03-mozilla1.8/
Build|Mozilla1.8|WINNT 5.0 patrocles Depend Tb-Nightly|success|1210673340|http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2008-05-13-03-mozilla1.8/

bhearsum points out that the buildbot does publish nightlies to the tinderbox-builds dir, so the special case would apply to Firefox an Mozilla1.8 trees only. Unfortunately tinderbox will require major surgery to fix, but we can look into it if the Talos change is too intrusive.
(In reply to comment #69)
+            # ignore this if it is a nightly
+            if self.night_url <> '':
+                continue

The mozilla-central/action-monkey builds don't have the same problem, so you could test for Tree being "Mozilla1.8" or "Firefox".
Attachment #320726 - Flags: review?(rcampbell) → review+
Attachment #320726 - Attachment is obsolete: true
Attachment #320845 - Flags: review?(rcampbell)
Filed bug 433639 to have talos start testing 1.8/1.9 nightlies.
Updated http://hourly-archive.localgho.st/ to pull from the dated dirs.
Comment on attachment 320845 [details] [diff] [review]
have talos ignore nightly/busted builds

the <> operator is a little odd to my eyes, but it works, so...
Attachment #320845 - Flags: review?(rcampbell) → review+
Checking in tinderboxpoller.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/tinderboxpoller.py,v  <--  tinderboxpoller.py
new revision: 1.2; previous revision: 1.1
done
Is there anything left to do here, or can we now close this bug?
Final version of the cleanup job:

@hourly root find /pub/mozilla.org/firefox/tinderbox-builds/ -mindepth 2 -maxdepth 2 -type d -mtime +0 -name 1????????? | xargs rm -rf

(mtime +0 for older than a day instead of +1 for 2; remove the verbose on the rm now that I'm satisfied it's working OK, still mails me on errors).

I think we're all done, as are the dependent bugs.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Adjusted the cleanup job to do thunderbird/ and seamonkey/ directories too, since they just moved over to buildbot.
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.