291167 - store last 24 hours of hourly builds for accurate regression and perf tests

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Peter van der Woude [:Peter6]]

To track down regressions more accurate (preferably down to the build hour) it
would be very usefull if all hourly builds of the past 24 h would allways be
available

Comment 1

[Peter van der Woude [:Peter6]]

CC Chase

Comment 2

[Chase Phillips]

Mass reassign of open bugs for [e-mail] to [e-mail].

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

I've worked up a script-based approach at
  http://hourly-archive.localgho.st/

It keeps 1 week of Firefox builds from the trunk.

Comment 4

[Robert Helmer [:rhelmer]]

We need this to make test-only machines more accurate.

Comment 5

[Robert Helmer [:rhelmer]]

One idea I have for this is to have tinderbox include the (unix-style) timestamp that represents the pull date and build start time (which is the only way the server has of matching up build start/stop times) in the hourly directory, such as:

ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/fx-win32-tbox-trunk/1170370500/ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/fx-win32-tbox-trunk/1170370500/firefox-3.0a1.en-US.win32.installer.exe

It'd be up to the FTP server to clear old directories out periodically. Since that number represents the pull date, that would enable us to match up bonsai queries and give the test-only machines the data they need.

Also, I'd like to change this directory to say "hourly" or "depend" or something more apparent, and get rid of the redundant "tinderbox" and "tinderbox-builds" directories, since the names are very ambiguous (nightly builds are tinderbox builds too!).

Thoughts?

Comment 6

[J. Paul Reed [:preed]]

(In reply to comment #5)

> It'd be up to the FTP server to clear old directories out periodically.

We'd need to make sure that there's something there to do this before this change goes in.

Maybe bug 342972 should be a blocker to this?

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to comment #6)
> (In reply to comment #5)
> 
> > It'd be up to the FTP server to clear old directories out periodically.
> 
> We'd need to make sure that there's something there to do this before this
> change goes in.
> 
> Maybe bug 342972 should be a blocker to this?

Agreed.

Comment 8

[Chris Cooper [:coop] (he/him)]

I vote for standardizing on "tinderbox-builds", since "hourly" isn't quite accurate either. Yes, nightly builds are also tinderbox builds, and they should also end up linked here.

I would also like to recommend reformatting the unix-style timestamp part of the path to be human-readable, e.g. YYYYMMDDHHmmSS (I also wish tinderbox history did this, FWIW). This will make it easier for humans to find the builds they need without really influencing any automated components.

Comment 9

[Robert Helmer [:rhelmer]]

Nothing to do here until new stage.m.o is up.

Comment 10

[Chris Cooper [:coop] (he/him)]

I'd really like to push this out until *after* we start building more intelligently, i.e. only when we have a code change. 

Maybe this means waiting for buildbot (bug 299909 and/or bug 401936).

Comment 11

[Robert Helmer [:rhelmer]]

(In reply to comment #10)
> I'd really like to push this out until *after* we start building more
> intelligently, i.e. only when we have a code change. 
> 
> Maybe this means waiting for buildbot (bug 299909 and/or bug 401936).

This is probably a ways off; I don't think it's unreasonable to clean the depend dir of everything over 24h is it? 

Comment 12

[Robert Helmer [:rhelmer]]

I'd like to remove the blocking bug 342972 and just set this up. What if we make the builders upload to a unique depend upload dir e.g. 
 ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/FX-WIN32-TBOX-trunk/20070122112700/

And have a braindead cleanup script that removes everything older than 24h in /pub/mozilla/org/firefox/tinderbox-builds/

Once stage migration is complete and supersedes this, wonderful, but it is blocking other things that we need such as bug 410869.

Comment 13

[Peter van der Woude [:Peter6]]

(In reply to comment #12)
> I'd like to remove the blocking bug 342972 and just set this up. What if we
> make the builders upload to a unique depend upload dir e.g. 
> 
> ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/FX-WIN32-TBOX-trunk/20070122112700/
That means a lot of extra time needed for users to download stamped builds if they are in their own separate directories.

> And have a braindead cleanup script that removes everything older than 24h in
> /pub/mozilla/org/firefox/tinderbox-builds/

Based on quite some experience with http://hourly-archive.localgho.st/ I'd say make it at least a few days.
Why not use the setup like http://hourly-archive.localgho.st/ ?
It's user friendly for downloading and offers all we (testers) need including a link to the checkins for that particular build.

Just keep "nightly" and "hourly" builds separate.
Hourly build testers are not the same as nightly build testers.

Comment 14

[Robert Helmer [:rhelmer]]

(In reply to comment #13)
> (In reply to comment #12)
> > I'd like to remove the blocking bug 342972 and just set this up. What if we
> > make the builders upload to a unique depend upload dir e.g. 
> > 
> > ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/FX-WIN32-TBOX-trunk/20070122112700/
> That means a lot of extra time needed for users to download stamped builds if
> they are in their own separate directories.
> 
> > And have a braindead cleanup script that removes everything older than 24h in
> > /pub/mozilla/org/firefox/tinderbox-builds/
> 
> Based on quite some experience with http://hourly-archive.localgho.st/ I'd say
> make it at least a few days.
> Why not use the setup like http://hourly-archive.localgho.st/ ?
> It's user friendly for downloading and offers all we (testers) need including a
> link to the checkins for that particular build.
> 
> Just keep "nightly" and "hourly" builds separate.
> Hourly build testers are not the same as nightly build testers.


Nick, what do you think about making hourly-archive a supported mozilla.org service?

Comment 15

[Robert Helmer [:rhelmer]]

Ok, because of the way Talos works we're hitting this problem a lot more often:

1) buildbot TinderboxPoller sees there's a new Tinderbox build, queues it up
2) test machine becomes available, downloads build and tests
3) reports in with the timestamp from #1

In the time between 1 and 2, it's quite likely that another build was created and uploaded. We would like to be performance testing each checkin individually, so we really need to be storing more than one depend build at a time for this to work.

Upping the severity and priority on this. I don't think we should be blocking on stage migration, a simple script just for tinderbox-builds (clear out files over 24h old) and patching tinderbox to upload to a unique directory are the short path to stop the bleeding here, we can move to the ideal solution when it becomes available.

Comment 16

[Robert Helmer [:rhelmer]]

Attachment: (untested) put start time in FTP url for depend builds

unix timestamp in here looks ugly, we can reformat if you like. We should definitely have resolution to the second, though.

Comment 17

[Robert Helmer [:rhelmer]]

Attachment: crontab to clear out old tinderbox-builds

This just does firefox, I guess we'd also need camino, xulrunner, thunderbird as well.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Do we want/need to keep build for all apps automatically, or should this be opt-in ? The disk space requirements are non-negligble I bet.

Comment 19

[Robert Helmer [:rhelmer]]

(In reply to comment #18)
> Do we want/need to keep build for all apps automatically, or should this be
> opt-in ? The disk space requirements are non-negligble I bet.

I could make this configurable, then other projects would have to opt in, does that sound reasonable?

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Sounds good to me.

Comment 21

[Robert Helmer [:rhelmer]]

Attachment: only push to dated depend dir if DependToDated is set

Working on testing this today, looks like it should do the right thing.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 303341 [details] [diff] [review]
only push to dated depend dir if DependToDated is set

>Index: post-mozilla-rel.pl
>===================================================================
>+    } elsif (! $cachebuild && $Settings::DependToDated) {
>+      $datedir = $start_time;

You'll need to append a "/" after $start_time, and default 0 value to tinder-defaults.pl. If the testing goes ok, then please add those on commit.

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 302266 [details]
crontab to clear out old tinderbox-builds

You'll want -mtime +1 there for one day, and a second command to remove empty directories.

Comment 24

[Rob Campbell [:rc] (:robcee)]

or -ctime +1 for creation time.

Comment 25

[[e-mail] (:bhearsum)]

Just to confirm: We are using the start time (in unix time format) as the DependToDated directory, right?

If so, there _may_ be an issue with MozillaStageUpload. It retrieves the BuildID from application.ini and as such only has to-the-hour granularity. It'd be easy for Buildbot to assume that the time is 00:00:00 - but what how do we distinguish between different dep builds? I guess Buildbot could pull it's build-start time and use the time from that, but I'm not sure that's exactly what we want (and it may happen that right as the day switches over we get a wacky thing like Feb 15, 2008 @ 23:59:00 when it's _actually_ Feb 15, 2008 @ 00:01:00.

Is there any way to retrieve a down-to-the-hour buildid/start time?

Comment 26

[Robert Helmer [:rhelmer]]

(In reply to comment #25)
> Just to confirm: We are using the start time (in unix time format) as the
> DependToDated directory, right?
> 
> If so, there _may_ be an issue with MozillaStageUpload. It retrieves the
> BuildID from application.ini and as such only has to-the-hour granularity. It'd
> be easy for Buildbot to assume that the time is 00:00:00 - but what how do we
> distinguish between different dep builds? I guess Buildbot could pull it's
> build-start time and use the time from that, but I'm not sure that's exactly
> what we want (and it may happen that right as the day switches over we get a
> wacky thing like Feb 15, 2008 @ 23:59:00 when it's _actually_ Feb 15, 2008 @
> 00:01:00.
> 
> Is there any way to retrieve a down-to-the-hour buildid/start time?

This value is going to be equivalent to the pulldate used, which is what we want for performance testing. 

This is currently only available from one place that I know of, and that's Tinderbox (unix timestamp must be sent in beginning/end email), so the idea is that Talos or anything else that needs to know this info can ask Tinderbox for the latest build, then find the equivalent directory on FTP.

Comment 27

[Robert Helmer [:rhelmer]]

I think we are ready to go on this end, but Talos and the perf-test-running Tinderboxes (unless Talos replaces them soon) need to support the new directory scheme first.

Comment 28

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #27)
> I think we are ready to go on this end, but Talos and the perf-test-running
> Tinderboxes (unless Talos replaces them soon) need to support the new directory
> scheme first.
Well, the perf-test-running Tinderboxes are being mothballed (see bug#413695 for details) so no need to touch those. 

Alice: whats involved is making the matching Talos change?  

Comment 29

[Robert Helmer [:rhelmer]]

(In reply to comment #28)
> (In reply to comment #27)
> > I think we are ready to go on this end, but Talos and the perf-test-running
> > Tinderboxes (unless Talos replaces them soon) need to support the new directory
> > scheme first.
> Well, the perf-test-running Tinderboxes are being mothballed (see bug#413695
> for details) so no need to touch those. 
> 
> Alice: whats involved is making the matching Talos change?  

There's some discussion of this in bug 417633 (marked blocking this bug).

Comment 30

[Robert Helmer [:rhelmer]]

Nothing to do here until the old perf machines are gone, and the Talos patch is ready.

Comment 31

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #30)
> Nothing to do here until the old perf machines are gone, 
Marked bug#413695 as blocking.

> and the Talos patch is
> ready.

Comment 32

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Removed dependency on 342972 because this is covered with the crontab patch in this bug.

Comment 33

[alice nodelman [:alice] [:anode]]

Can I get an update on this bug state?  From the comments it looks like we have a patch for the builder side of the world (ie, dropping builds into time stamped directories) but are missing the talos patch.

Is the time stamped directories solution up and running anywhere?  I'd like to start working on the talos patch and it would be nice to have something test with.

Comment 34

[alice nodelman [:alice] [:anode]]

Are we also missing a script to do the cleanup after 24 hours to stop massive browser build up?

Comment 35

[Nick Thomas [:nthomas] (UTC+12)]

I set this up as a local patch on moz180-linux-tbox, which publishes hourlies to
  http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/moz180-linux-tbox-mozilla1.8.0/

This build is set to run less frequently than it could, about every 80 minutes, but we can speed it up if that'd help.

Comment 36

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Do we care about "last 24 hours of hourly builds" or would "last 'n' hourly builds" be just as good?

(I'm trying to separate out whether we care about all builds in a specific calendar day or if we just care about a continuous sequence of recent builds.)

Comment 37

[Peter van der Woude [:Peter6]]

(In reply to comment #36)
> Do we care about "last 24 hours of hourly builds" or would "last 'n' hourly
> builds" be just as good?
> 
> (I'm trying to separate out whether we care about all builds in a specific
> calendar day or if we just care about a continuous sequence of recent builds.)
> 
That very much depends on the amount of builds.
The Win32 box used to spit out 30+ new builds (with changes) a day, in which case you would want a day of builds (minimum), right now it spits out 13-14 a day tops, which would mean you'd have 2.5 days of builds (if you fix N to 30).

I'd rather see a few days archived anyway.

Comment 38

[Tony Mechelynck [:tonymec]]

(In reply to comment #36)
> Do we care about "last 24 hours of hourly builds" or would "last 'n' hourly
> builds" be just as good?
> 
> (I'm trying to separate out whether we care about all builds in a specific
> calendar day or if we just care about a continuous sequence of recent builds.)
> 

"Latest n hourly builds" might be better if for some reason no more builds are produced for several days (like SeaMonkey-trunk for Linux at the moment, see bug 429603). Or else, maybe the bug Summary should specify more explicitly that "the last 24 hours of hourlies" doesn't mean "the hourlies from the last 24 hours" but "the 24 hours of hourlies ending with the latest build".

In reply to comment #37
Maybe the number of "builds to be kept" could be parameterized separately for each product/version/platform etc.?

Comment 39

[Peter van der Woude [:Peter6]]

(In reply to comment #38)
> In reply to comment #37
> Maybe the number of "builds to be kept" could be parameterized separately for
> each product/version/platform etc.?

At the time i filed this bug http://hourly-archive.localgho.st/
 did not exist and anything more than just being able to download the latest hourly was a bonus.

7200 Firefox downloads (on my HD) later I'd say that a week of builds is enough, 3 days is minimum.
The 3 days specifically because it would allow you to do something else (away from the computer) during the weekend.

Comment 40

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 303341 [details] [diff] [review]
only push to dated depend dir if DependToDated is set

>Index: build-seamonkey-util.pl
>===================================================================
>         if (((-e $external_build) and ($build_status eq 'success')) || 
>             ($Settings::SkipMozilla)) {
>-            ($build_status, $binary_url) = PostMozilla::main($build_dir);
>+            ($build_status, $binary_url) = 
>+             PostMozilla::main($build_dir, $server_start_time);

I think this should be just $server_time, testing now.

Comment 41

[Nick Thomas [:nthomas] (UTC+12)]

Argh, I meant $start_time.

Comment 42

[Nick Thomas [:nthomas] (UTC+12)]

Ok, looks like it's working, making dirs
  1208895900moz180-linux-tbox-mozilla1.8.0/
  1208897940moz180-linux-tbox-mozilla1.8.0/
in
  http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/

I must have misunderstood how this was supposed to work, as I was expecting
  1208895900/
  1208897940/
in
  http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/moz180-linux-tbox-mozilla1.8.0/

Comment 43

[alice nodelman [:alice] [:anode]]

Is this the format that we want to settle on?  I'll assume that if I can get the talos buildbot to interpret this correctly it wouldn't be much of a leap to something else - but it might be a good time to consider what format we would prefer instead of just what we have around.

Comment 44

[[e-mail] (:bhearsum)]

(In reply to comment #42)
> Ok, looks like it's working, making dirs
>   1208895900moz180-linux-tbox-mozilla1.8.0/
>   1208897940moz180-linux-tbox-mozilla1.8.0/
> in
>   http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/
> 
> I must have misunderstood how this was supposed to work, as I was expecting
>   1208895900/
>   1208897940/
> in
>  
> http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/moz180-linux-tbox-mozilla1.8.0/
> 

My understanding is, e.g.:
fx-linux-tbox/1208855900
fx-linux-tbox/1208855932

This is the way buildbotcustom.steps.transfer.MozillaStageUpload works.

Comment 45

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Updated patch

Hopefully this will do the trick, lets see what moz180-linux-tbox creates.

Comment 46

[alice nodelman [:alice] [:anode]]

Attachment: patches perfrunner to add ability to pull from a dated directory (for talos testers)

Adds a MozillaWgetLatestDated class to perfrunner.py.  For this code to be exercised the master.cfg's of production/try/stage talos buildbots will have to be patched to use this class instead of the standard MozillaWgetLatest.

Comment 47

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 318701 [details] [diff] [review]
patches perfrunner to add ability to pull from a dated directory (for talos testers)

this looks like it should work to my eyes. One thing I noticed is that there's no real failure handling in the start() method if filename comes back empty. An assert fileURL = "more than url" would probably do the trick.

otherwise, all good.

Comment 48

[Peter van der Woude [:Peter6]]

if i understand this right the builds are going to be saved into a dated directory and will have their standard name w/o indication of date/time.

I would prefer the filename format as is (on http://hourly-archive.localgho.st/win32.html) like 20080502_0448_N_firefox-3.0pre.en-US.win32 for the latest win32 build.

YYYYMMDD of checkout
HHMM of checkout
N for nightly and clobbered builds
firefox for type
-3.0pre for firefox version

and simply store them into one dir, or 1 dir per day

It sucks if you have to go to another directory for every build and it's even worse if you have to rename each file and/or create a new directory for each file.

Please look at http://hourly-archive.localgho.st/win32.html , which was designed for ease of use and don't try to reinvent this into a system that is horribly user unfriendly.

Comment 49

[Rob Campbell [:rc] (:robcee)]

(In reply to comment #48)
> if i understand this right the builds are going to be saved into a dated
> directory and will have their standard name w/o indication of date/time.

correct.

> I would prefer the filename format as is (on
> http://hourly-archive.localgho.st/win32.html) like
> 20080502_0448_N_firefox-3.0pre.en-US.win32 for the latest win32 build.
> 
[snip]
> 
> and simply store them into one dir, or 1 dir per day
> 
> It sucks if you have to go to another directory for every build and it's even
> worse if you have to rename each file and/or create a new directory for each
> file.
> 
> Please look at http://hourly-archive.localgho.st/win32.html , which was
> designed for ease of use and don't try to reinvent this into a system that is
> horribly user unfriendly.

The purpose of this is not to provide an hourly archive of builds, but to store 24 hours for tracking regressions and providing more fine-grained accuracy for the performance testing toolkit. Since we're only dealing with 24 hours of builds, this is not *that* onerous.

Comment 50

[Peter van der Woude [:Peter6]]

(In reply to comment #49)
> The purpose of this is not to provide an hourly archive of builds, but to store
> 24 hours for tracking regressions and providing more fine-grained accuracy for
> the performance testing toolkit. Since we're only dealing with 24 hours of
> builds, this is not *that* onerous.
> 
That is not what I filed this bug for.

Comment 51

[Nick Thomas [:nthomas] (UTC+12)]

Peter, is there anything that hourly-archive isn't doing for you right now ? I know this got morphed at about comment #5, but sounds like there are two different goals here.

Comment 52

[Peter van der Woude [:Peter6]]

(In reply to comment #51)
> Peter, is there anything that hourly-archive isn't doing for you right now ? I
> know this got morphed at about comment #5, but sounds like there are two
> different goals here.
> 
Nope, the hourly-archive is fine with me. 
I was assuming the current hourly-archive would be transformed into an official mozilla page.
Currently the archive is only linked from my threads and nowhere from e.g. http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/ or any other official mozilla.org page.
I can file a bug for that if needed

Comment 53

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] tinderbox changes v3, with tinder-defaults.pl

Same as attachment 317225 [details] [diff] [review] but with a tinder-defaults.pl line.

Comment 54

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Set DependToDated on the builds Talos tests

Comment 55

[Nick Thomas [:nthomas] (UTC+12)]

I've tested all three code paths for upload (by disabling DependToDated for a few cycles on moz180-linux-tbox) so the tinderbox code-change is ready to go. The Firefox tree is a mess at the moment so delaying landing that in case of unforeseen bustage. 

Once it does go in, then the tinderbox config change (attach. 319567) should be committed when we want to deploy this. We'll also need a cron job to clean up old builds.

Comment 56

[[e-mail] (:bhearsum)]

Attachment: upload mozilla2 builds in dependToDated format

Comment 57

[alice nodelman [:alice] [:anode]]

Attachment: switch to pull by date for talos

Comment 58

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 319566 [details] [diff] [review]
[checked in] tinderbox changes v3, with tinder-defaults.pl

mozilla/tools/tinderbox/build-seamonkey-util.pl 	1.389
mozilla/tools/tinderbox/tinder-defaults.pl 	1.128
mozilla/tools/tinderbox/post-mozilla-rel.pl 	1.146

Comment 59

[[e-mail] (:bhearsum)]

Comment on attachment 320563 [details] [diff] [review]
switch to pull by date for talos

Seems fine

Comment 60

[[e-mail] (:bhearsum)]

Attachment: [checked in] enable dependToDated for nightlies too.

Comment 61

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 319567 [details] [diff] [review]
[checked in] Set DependToDated on the builds Talos tests

mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl 	1.1.12.26
mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl 	1.8.4.25
mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl 	1.2.10.27
mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl 	1.27
mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl 	1.45
mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl 	1.36

Comment 62

[[e-mail] (:bhearsum)]

Comment on attachment 320566 [details] [diff] [review]
[checked in] enable dependToDated for nightlies too.

Landed in cd3463fd0743

Comment 63

[alice nodelman [:alice] [:anode]]

talos changes to perfrunner and master.cfg's landed

Checking in perf-staging/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perf-staging/master.cfg,v  <--  master.cfg
new revision: 1.12; previous revision: 1.11
done
Checking in perfmaster2/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/master.cfg,v  <--  master.cfg
new revision: 1.56; previous revision: 1.55
done
Checking in perfmaster2/perfrunner.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/perfrunner.py,v  <--  perfrunner.py
new revision: 1.15; previous revision: 1.14
done

Comment 64

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [installed] contab mk2

A quick an dirty way to remove any epoch-named dirs older than 1 day from 
  /pub/mozilla.org/firefox/tinderbox-builds/*/

Comment 65

[[e-mail] (:bhearsum)]

Comment on attachment 320601 [details]
[installed] contab mk2

We'll need to update this script on Wed, 18 May 2033 03:33:20 GMT, but this is fine for now :)

Comment 66

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 320601 [details]
[installed] contab mk2

Installed to dm-stage01:/etc/cron.d/remove-hourly-builds with @hourly (I realized we'd get two days of builds then chop it down to one at midnight otherwise). Also temporarily made it rm -rfv to verify it's working ok (mail to me).

Also cleaned up all the files that now live in directories (eg bm-xerve08/firefox...).

Comment 67

[Nick Thomas [:nthomas] (UTC+12)]

Further tweaked it so that find doesn't complaing, currently is

@hourly find /pub/mozilla.org/firefox/tinderbox-builds/ -mindepth 2 -maxdepth 2 -type d -mtime +1 -name 1????????? | xargs rm -rfv

Comment 68

[Nick Thomas [:nthomas] (UTC+12)]

One thing we didn't remember is that tinderbox doesn't publish nightly builds to the tinderbox-builds directory, but Talos does try to download that build. If I'd not removed the old files we'd have been testing an old build once a day - having removed them we get Talos burning.

Can we make tinderbox poller ignore builds with a url given ? Eg

Build|Mozilla1.8|MacOSX Darwin 8.7.0 bm-xserve02 Depend Tb-Nightly|success|1210673160|http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2008-05-13-03-mozilla1.8/
Build|Mozilla1.8|WINNT 5.0 patrocles Depend Tb-Nightly|success|1210673340|http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2008-05-13-03-mozilla1.8/

bhearsum points out that the buildbot does publish nightlies to the tinderbox-builds dir, so the special case would apply to Firefox an Mozilla1.8 trees only. Unfortunately tinderbox will require major surgery to fix, but we can look into it if the Talos change is too intrusive.

Comment 69

[alice nodelman [:alice] [:anode]]

Attachment: ignore nightly builds (also busted builds)

Comment 70

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #69)
+            # ignore this if it is a nightly
+            if self.night_url <> '':
+                continue

The mozilla-central/action-monkey builds don't have the same problem, so you could test for Tree being "Mozilla1.8" or "Firefox".

Comment 71

[alice nodelman [:alice] [:anode]]

Attachment: have talos ignore nightly/busted builds

Comment 72

[alice nodelman [:alice] [:anode]]

Filed bug 433639 to have talos start testing 1.8/1.9 nightlies.

Comment 73

[Nick Thomas [:nthomas] (UTC+12)]

Updated http://hourly-archive.localgho.st/ to pull from the dated dirs.

Comment 74

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 320845 [details] [diff] [review]
have talos ignore nightly/busted builds

the <> operator is a little odd to my eyes, but it works, so...

Comment 75

[alice nodelman [:alice] [:anode]]

Checking in tinderboxpoller.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/tinderboxpoller.py,v  <--  tinderboxpoller.py
new revision: 1.2; previous revision: 1.1
done

Comment 76

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Is there anything left to do here, or can we now close this bug?

Comment 77

[Nick Thomas [:nthomas] (UTC+12)]

Final version of the cleanup job:

@hourly root find /pub/mozilla.org/firefox/tinderbox-builds/ -mindepth 2 -maxdepth 2 -type d -mtime +0 -name 1????????? | xargs rm -rf

(mtime +0 for older than a day instead of +1 for 2; remove the verbose on the rm now that I'm satisfied it's working OK, still mails me on errors).

I think we're all done, as are the dependent bugs.

Comment 78

[Nick Thomas [:nthomas] (UTC+12)]

Adjusted the cleanup job to do thunderbird/ and seamonkey/ directories too, since they just moved over to buildbot.