Closed Bug 400083 Opened 18 years ago Closed 16 years ago

Publish Nightly trunk builds with Debug Bits

Categories

(Release Engineering :: General, defect, P3)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: tchung, Assigned: lsblakk)

References

Details

Attachments

(2 files, 2 obsolete files)

archive.sh to copy one nightly debug build per platform to the nightly dir for permanent storage
1.15 KB, patch
nthomas
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
change ls -t to ls -r
865 bytes, patch
nthomas
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
Dev and QA would like access to nightly trunk builds with debug bits enabled. The primary purpose would be tracking down additional logging on tracking down leaks, and show asserts. To accompany the filesize factor, there is talk about stripping out symbols and only enabling -DDEBUG,and --enable-trace-malloc). Can this be done on the same tinderbox immediately after the regular nightlies are scheduled? An ongoing discussion below: ---------------------------- On 10/8/07, L. David Baron <dbaron@mozilla.com> wrote: > > On Monday 2007-10-08 17:39 -0700, Robert Helmer wrote: >> > > Are there good reasons not to publish debug builds? The only one I can >> > > think of is that they are quite large. > > > > The reason they're so large (-g) is independent of the reason most > > people want them (-DDEBUG, plus a few other things like > > --enable-trace-malloc). You could do one without the other. (In > > fact, it's possible that the packaging code will automatically strip > > out the debug symbols.) This is correct, since our official nightlies also build with -g to get symbols for Breakpad. > > That said, having the debugging symbols is necessary if you're using > > nsTraceRefcnt or trace-malloc for more than summary statistics or > > leak detection. It might be necessary for distinguishing leak bugs, > > since the stack traces aren't so great without them. If this is the case, then any debug builds we could ship off of a tinderbox wouldn't be so useful, since we'd strip the symbols. Also, FWIW, a non-universal Mac build with symbols is >500Mb uncompressed last time I looked. Win32 is probably ~70Mb or so. -Ted ---------------------------------------------------- On 10/9/07, Tony Chung <tchung@mozilla.com> wrote: > > > > Rhelmer, > > > > in the leaks call today, we had two questions: > > > > 1) how big would the files be approx if we just did the suggested settings > > from dbaron, and strip out the debug symbols? Should be negligible size difference from nightlies, it's just the debugging code that gets compiled in. The symbols are the significant size difference. > > 2) are there licensing issues for debug builds we need to consider? Debug builds on Windows won't run on a system that doesn't have Visual C++ installed, because they link to the debug version of the CRT, which is not distributable. You could install VC2005 Express to make them work, but we can't ship the Debug CRT DLLs with the builds. I don't think there are any problems on Mac or Linux. -Ted -----------------------------
Related Bugs: bug 374822 bug 391189
So, this is the same as bug 367775 but without the symbols part?
these are for internal use only. bc and I were looking at a way to provide them and deliver setup instructions for leak testing. Also, we do need symbols for this to be effective.
Status: NEW → RESOLVED
Closed: 18 years ago
Resolution: --- → DUPLICATE
whups, considered marking this a dupe, then changed my mind but forgot to clear the field. Sorry!
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Assignee: build → nobody
Status: REOPENED → NEW
QA Contact: mozpreed → build
with the symbol server this is no longer really necessary.
Status: NEW → RESOLVED
Closed: 18 years ago → 18 years ago
Resolution: --- → WONTFIX
Doesn't really serve the purpose from comment 0, which is to get extra debug logging/trace malloc, fwiw.
Reopening a year later since Reed insists on duping my request for Nightly Debug builds to this. You know, I don't want to have to hand roll a build every time we need to check for bugs involving asserts, which happens with every security release.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
Adding Dan since he was part of the discussion yesterday on having debug nightlies.
To quote Al from bug 463065: > Dan Veditz, Sam, and I were discussing this yesterday and Dan believes that we > don't do these builds because of resource constraints from the past. Since we > are not as constrained in the number of machines building, we should revisit > this issue and, ideally, begin making debug nightly builds. The resource issue AFAIK was disk space, not machines. I see in comment#0 here that the biggest size increase is symbols in the builds, not the debug bits...are you looking for both of those, or just the debug bits? If you're looking for both I think we have the same problem with disk usage.
Both, ideally. If we were constrained by disk space a year ago (and previous years), when will we have enough disk space in order to address that concern? It seems pretty straightforward to buy more disks if that is the only real bottleneck here instead of looking at this issue annually.
I'll have to defer that one to John/IT, but I will say we're looking at probably close to 1.5GB/2GB per set of builds. It gets to be pretty nontrivial depending on the retention policy. Additionally, are you looking for official debug builds of releases, too?
I know that we have been producing (maybe only on 1.9.0?) Shark-enabled nightlies for OS X with debug symbols. The trick there is that there's no archiving of nightlies, only the latest nightly is available. Would that be good enough for QA's purposes? It should sidestep the disk space issue by not archiving.
(In reply to comment #13) > I know that we have been producing (maybe only on 1.9.0?) Shark-enabled > nightlies for OS X with debug symbols. The trick there is that there's no > archiving of nightlies, only the latest nightly is available. Would that be > good enough for QA's purposes? It should sidestep the disk space issue by not > archiving. Having only latest nightly builds isn't good enough. We refer to past builds all the time to find regression ranges, and would be even more critical to do so with debug builds as we can trace through stacks and leaks. Its possible we can compromise on how far back we want to go though, if that will help mitigate some of the disk space factor.
I think that would fulfill the most immediate need though it would be good to have these for all three platforms. Personally, I don't know if we need the debug symbols or not for daily builds but I know that others, like Tomcat, probably do. I mean, I don't know how many people are going to pull down 1.5 GB of information on a nightly basis. That is a lot.
I mentioned this in IRC, but a temporary solution would be to use the try server to generate debug builds when necessary: https://wiki.mozilla.org/Build:TryServer
(In reply to comment #15) > I think that would fulfill the most immediate need though it would be good to > have these for all three platforms. > > Personally, I don't know if we need the debug symbols or not for daily builds > but I know that others, like Tomcat, probably do. > > I mean, I don't know how many people are going to pull down 1.5 GB of > information on a nightly basis. That is a lot. - Compressed or uncompressed? The one affects bandwidth, the other affects installed disk space, and the criteria are different. - Maybe not every tester every day (though I believe some would), but if _any_ tester may download a debug build at a moment's notice (e.g. to provide info for a crash bug which not everyone can reproduce) they have to be available -- and I believe many testers would make good use of such availability when they would need to. - IMHO having a debug version of only the latest build (preferably for all three tier-1 platforms) would already fulfil some of the needs. Except for debug-build-only bugs, regression ranges for reproducible bugs can be established with non-debug builds (since the question here is "is the bug present yes/no"), while "additional debug info" can be provided with "today's nightly" build.
Its easy to enable producing Debug builds on nightly basis. However, the problem was, and remains still, where to store all these builds. From comments below, it sounds like just *one* most recently nightly build is not sufficient for your needs. If you know how many debug nightlies you want us to store, we can figure out the disk requirements, and then we can both work with IT to figure out if/when/how to bring more disks online for just this. ...or... Have you tried the suggestion below, of using try server to generate debug builds of any nightly that you are interested in? This seems better to me, because you dynamically create debug builds of the specific revisions you are interested in, not forced to some subset of recent nightlies, which might not cover what you want anyway.
Component: Release Engineering → Release Engineering: Future
QA Contact: build → release
This will be done as a side-effect of bug 372581. The debug builds will be uploaded to stage.mozilla.org.
Depends on: 372581
(In reply to comment #20) > This will be done as a side-effect of bug 372581. The debug builds will be > uploaded to stage.mozilla.org. We're uploading every single debug build we do now, calling this one FIXED.
Status: REOPENED → RESOLVED
Closed: 18 years ago → 17 years ago
Resolution: --- → FIXED
How do we download them? Going to http://stage.mozilla.org/ shows the default Red Hat apache page...
http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/ is where I'd look, but I don't see debug builds there.
They're currently buried along side their non-debug counterparts. e.g. debug build: http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1254190660/ optimized build: http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1254187449/ you can usually tell by the file sizes. In any case, we're going to be putting the debug builds into their own directory soon to avoid this confusion.
Moving closed Future bugs into Release Engineering in preparation for removing the Future component.
Component: Release Engineering: Future → Release Engineering
Reopening because the builds aren't saved. QA would like to have a daily debug build (pick an hourly that you like) saved and archived for all supported platforms. This would make it vastly easier for QA to work on debug builds for bug verification and testing since, right now, we all have to individually build our own bits, consuming a lot of time and effort. CC'ing Matt Evans since I've discussed this with him.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
How often are we talking about accessing these? How many should be archived? Can you give us an idea so that we can look into saving some (but hopefully) not every debug nightly.
We will be accessing them unpredictably for regression range finding, verification of bugs, etc. That was the point in having one of these available for every day. How long do you keep a normal nightly? Right now, we don't have debug nightlies for you to save. We have hourlies which are just tossed.
When bug 520227 is fixed, it will be possible to use the tryserver to get a debug build of any revision, so that would be a great way to get the builds you need for a particular verification or range on an as-needed basis and will save us the overhead of turning on and saving nightly debug builds for an unknown amount of regression range finding purposes. Closing as WONTFIX, as the tryserver changes will be hopefully going into place this quarter.
Status: REOPENED → RESOLVED
Closed: 17 years ago → 16 years ago
Resolution: --- → WONTFIX
> How long do you keep a normal nightly? just to answer this, we have nightlies going back to 2004.
I've been asking for this for years so QA doesn't have to build debug anytime we want to do debug testing and, after almost three years, we've moved not an inch. This is highly frustrating. So, after all this, build is doing the same as it did before and refusing to give us nightly debug builds. What if I don't want a revision but just the equivalent of the nightly build of a given day but in debug flavor? Is there a published list of all changesets used for nightly builds? When terrabytes of diskspace are available for under $100 each, you'd think we could afford to archive one debug build a day of each branch we're supporting. Is the overheard really so expensive that no one will do this?
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
I should also point out that if I'm looking at a regression range for a problem, I often have to check ten or twenty builds. I have to have tryserver build me a separate build for each one of these changes, with all of the time and pain involved, for EACH one of these ten or twenty checks? Compare that work to pulling down builds from an FTP server and you can see why QA wants this. If I just wanted to get a build from a given changeset, I would build it myself and wait two hours. Also, once we end support for a given branch, it becomes a lot more difficult to build versus just pulling an old archived build from a server.
I'm going to look into this more next week and try to get a definitive answer about load/space concerns.
Assignee: nobody → lsblakk
http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/ now has the debug builds in their own folder - we could move N builds per day into an archive dir for QA - would this satisfy? Is there any reason to create a nightly debug build instead of just copying any of these days trunk debug builds to a debug archive on ftp?
I think that would be acceptable.
tested on staging-stage: cp -a /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx64-debug/1276629382/firefox-3.7a6pre.en-US.mac64.txt /home/ftp/pub/mozilla.org/firefox/nightly/2010-06-15-03-mozilla-central/firefox-3.7a6pre.en-US.debug-mac64.txt Also I have made this default grep for mozilla in case QA wants to track a regression in a 1.9.{1,2,3} build. If this isn't wanted I can easily put it back to a grep for mozilla-central only.
Attachment #451414 - Flags: review?(nrthomas)
We only seem to publish debug builds for mozilla-central, so that's all we can catch with archive.sh right now. Is that useful ?
In comment #0 the original request was: > Dev and QA would like access to nightly trunk builds with debug bits enabled. So I assume that it would be ok to have only mozilla-central debug builds saved - Al or Tony can you confirm if that will be acceptable for QA? Once we get confirmatoin we can put this script in place and start saving builds for you.
m-c only is okay with me. Al can comment if he needs branch nightlies. Thanks for getting this together!
M-c is nice though for my work I really do need branch nightlies.
Comment on attachment 451414 [details] [diff] [review] archive.sh to move one nightly debug build per platform to nightlies dir >DIRS=$(ls $BASE_PATH | grep "debug" | grep "mozilla") More compact written as grep 'mozilla.*debug$'. Seems we have debug builds for mac and linux on 1.9.1 and 1.9.2, so either I was blind before or we enabled those recently. > files="$(ls $builddir/$archivedir/*.{exe,bz2,txt,dmg})" This throws a bunch of ls: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.1-linux-debug/*.exe: No such file or directory ls: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.1-linux-debug/*.bz2: No such file or directory ls: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.1-linux-debug/*.txt: No such file or directory ls: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.1-linux-debug/*.dmg: No such file or directory errors. > for file in $files > do > backup=$(echo $file | sed s/en-US\./en-US.debug-/ | cut -d '/' -f10) > echo "copying $file contents to $ARCHIVE_PATH/$DATE_BASE-03-$branch/$backup" > cp -a $file $ARCHIVE_PATH/$DATE_BASE-03-$branch/$backup If you're going to hardcode the hour for a nigthly, which might shift later depending on slave availability, you should do a -d test first. And you'll need to cope with the 1.9.2 nightlies being -04-.
Attachment #451414 - Flags: review?(nrthomas) → review-
Comment on attachment 451414 [details] [diff] [review] archive.sh to move one nightly debug build per platform to nightlies dir > backup=$(echo $file | sed s/en-US\./en-US.debug-/ | cut -d '/' -f10) Use 'basename $file' instead of the cut here, it's much less fragile.
Improved directory checking with 'find', also using 'basename $file' instead of 'cut'. Finally, I took out the hardcoded 03 - this script will find any mozilla debug builds and make a backup copy, by branch, in the nightlies directory with a DATE-BRANCH-DEBUG dir naming structure. I hope it could be ok to do this instead of trying to hit the sometimes moving target that is the hour when builds start for a nightly. This will also make them easier for Al or whomever to find when scanning the nightlies dir for regression-checking installers.
Attachment #451414 - Attachment is obsolete: true
Attachment #471685 - Flags: review?(nrthomas)
my output from testing on staging: ./archive.sh FILE == /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/1283463559/firefox-4.0b6pre.en-US.mac.dmg BACKUP == firefox-4.0b6pre.en-US.debug-mac.dmg COPYING == /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/1283463559/firefox-4.0b6pre.en-US.mac.dmg to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-02-16-mozilla-central/firefox-4.0b6pre.en-US.debug-mac.dmg
Sorry, had to do some cleaning up after I pasted that output above, which was WRONG. This is the output of the patch as it stands: ./archive.sh Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/1283463559/firefox-4.0b6pre.en-US.mac.dmg Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/1283463559/firefox-4.0b6pre.en-US.mac.dmg to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-02-16-mozilla-central-debug/firefox-4.0b6pre.en-US.debug-mac.dmg
Attachment #471685 - Attachment is obsolete: true
Attachment #471688 - Flags: review?(nrthomas)
Attachment #471685 - Flags: review?(nrthomas)
Comment on attachment 471688 [details] [diff] [review] archive.sh to copy one nightly debug build per platform to the nightly dir for permanent storage >diff --git a/buildfarm/maintenance/archive.sh b/buildfarm/maintenance/archive.sh Lets call it archive-debug-builds.sh for clarity. >+ if [ -s $archivedir ]; then >+ if [ -d "$builddir/$archivedir" ]; then Not sure what these are supposed to do in succession, they're basically testing the same thing. Perhaps you mean -n the first time ? That would test for non-zero string length for $archivedir. >+ echo "Copying $file to $ARCHIVE_PATH/$DATE_BASE-$branch-debug/$backup" Once we start running this with cron, where will this output go ? r+ with the first two fixes and some sort of plan for the third.
Attachment #471688 - Flags: review?(nrthomas) → review+
Comment on attachment 471688 [details] [diff] [review] archive.sh to copy one nightly debug build per platform to the nightly dir for permanent storage http://hg.mozilla.org/build/tools/rev/56f4a8861f55 Addressed two nits, plan for this script's deployment is that I will monitor the emails for a week then either make it's output silent or direct it to a log file on stage.
Attachment #471688 - Flags: checked-in+
First night of running this I found that win32 nightlies weren't ready to be copied over yet so I've adjusted the time of running this archive script to run at 10:30 am, well after the nightlies have been generated.
Not sure what you mean here. Aren't we just grabbing the latest debug builds when @daily fires for cron ? That's 4 am right ?
I had it as @daily and that fired at 0:00 - at that time the win32 build wasn't present yet (at least that is how I interpreted that I didn't get a build at 0:00, but had one to grab at 11:30am) From the ouput: Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-linux-debug/1283879559/firefox-3.6.10pre.en-US.linux-i686.tar.bz2 Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-linux-debug/1283879559/firefox-3.6.10pre.en-US.linux-i686.tar.bz2 to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-1.9.2-debug/firefox-3.6.10pre.en-US.debug-linux-i686.tar.bz2 Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-linux-debug/1283879559/firefox-3.6.10pre.en-US.linux-i686.txt Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-linux-debug/1283879559/firefox-3.6.10pre.en-US.linux-i686.txt to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-1.9.2-debug/firefox-3.6.10pre.en-US.debug-linux-i686.txt Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-macosx-debug/1283879459/firefox-3.6.10pre.en-US.mac.dmg Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-macosx-debug/1283879459/firefox-3.6.10pre.en-US.mac.dmg to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-1.9.2-debug/firefox-3.6.10pre.en-US.debug-mac.dmg Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-macosx-debug/1283879459/firefox-3.6.10pre.en-US.mac.txt Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-macosx-debug/1283879459/firefox-3.6.10pre.en-US.mac.txt to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-1.9.2-debug/firefox-3.6.10pre.en-US.debug-mac.txt Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-win32-debug/1283879721/firefox-3.6.10pre.en-US.win32.installer.exe Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-win32-debug/1283879721/firefox-3.6.10pre.en-US.win32.installer.exe to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-1.9.2-debug/firefox-3.6.10pre.en-US.debug-win32.installer.exe Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-win32-debug/1283879721/firefox-3.6.10pre.en-US.win32.txt Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-win32-debug/1283879721/firefox-3.6.10pre.en-US.win32.txt to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-1.9.2-debug/firefox-3.6.10pre.en-US.debug-win32.txt Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1283916331/firefox-4.0b6pre.en-US.linux-i686.tar.bz2 Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1283916331/firefox-4.0b6pre.en-US.linux-i686.tar.bz2 to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-central-debug/firefox-4.0b6pre.en-US.debug-linux-i686.tar.bz2 Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1283916331/firefox-4.0b6pre.en-US.linux-i686.txt Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1283916331/firefox-4.0b6pre.en-US.linux-i686.txt to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-central-debug/firefox-4.0b6pre.en-US.debug-linux-i686.txt Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-debug/1283916877/firefox-4.0b6pre.en-US.linux-x86_64.tar.bz2 Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-debug/1283916877/firefox-4.0b6pre.en-US.linux-x86_64.tar.bz2 to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-central-debug/firefox-4.0b6pre.en-US.debug-linux-x86_64.tar.bz2 Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-debug/1283916877/firefox-4.0b6pre.en-US.linux-x86_64.txt Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-debug/1283916877/firefox-4.0b6pre.en-US.linux-x86_64.txt to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-central-debug/firefox-4.0b6pre.en-US.debug-linux-x86_64.txt Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/1283916821/firefox-4.0b6pre.en-US.mac.dmg Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/1283916821/firefox-4.0b6pre.en-US.mac.dmg to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-central-debug/firefox-4.0b6pre.en-US.debug-mac.dmg Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/1283916821/firefox-4.0b6pre.en-US.mac.txt Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/1283916821/firefox-4.0b6pre.en-US.mac.txt to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-central-debug/firefox-4.0b6pre.en-US.debug-mac.txt Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx64-debug/1283916934/firefox-4.0b6pre.en-US.mac64.dmg Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx64-debug/1283916934/firefox-4.0b6pre.en-US.mac64.dmg to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-central-debug/firefox-4.0b6pre.en-US.debug-mac64.dmg Found recent nightly: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx64-debug/1283916934/firefox-4.0b6pre.en-US.mac64.txt Copying /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx64-debug/1283916934/firefox-4.0b6pre.en-US.mac64.txt to /home/ftp/pub/mozilla.org/firefox/nightly/2010-09-08-00-mozilla-central-debug/firefox-4.0b6pre.en-US.debug-mac64.txt find: /home/ftp/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32-debug/1283828215/: No such file or directory
I think the cron job for cleaning tinderbox-builds was running at the same time and caused this, eg the initial ls found that dir and the cron job removed it before the script could look inside it. It runs on the hour, every hour. It's strange that 'ls -t' returned a build from 19:56 on the 6th though. Perhaps that was from a slow slave publishing an older build later than a fast slave publishing a newer one. Switch 'ls -r' would solve that problem. I note that we're publishing the leak test logs into the opt dir and not the debug one, so we should probably figure out a way to exclude those in this script for when we fix that.
the log files all end in .log except one which ends in -tree so it seems like this script will still be fine one that log location is moved to the debug dir. today's email from cron was successful in copying all platforms for both 1.9.2 and m-c debug builds at 10:30am -- is there a reason to move it back later or can it stay as is?
Attachment #473639 - Flags: review?(nrthomas)
Comment on attachment 473639 [details] [diff] [review] change ls -t to ls -r If these are pseudo-nightlies why not run the script at 3am ? It's not hard to imagine this scenario * someone isolates a regression to a particular nightly * they grab the debug build to dive deeper * debug build is from a vastly different revision and is of no use Taking the latest debug build at 3am minimizes the delta.
Attachment #473639 - Flags: review?(nrthomas) → review+
Comment on attachment 473639 [details] [diff] [review] change ls -t to ls -r http://hg.mozilla.org/build/tools/rev/781bb636419a also updated cron to run at 3am each day in order to close the delta Nick refers to. Will continue to monitor logs until next week and if all looks good, send output to a log file and close this bug.
Attachment #473639 - Flags: checked-in+
Two nights running with successful copying of all the platforms' debug builds. Noticed that 1.9.2 debug builds aren't always copied over but that's because they are not built as frequently (one or two day gaps between builds) so when there are debug builds available we will get a nightly dir archive of them but that will have corresponding gaps to the builds themselves. I've moved the output to append to a log file ($HOME/archive_debug_builds.log) instead of an email.
Status: REOPENED → RESOLVED
Closed: 16 years ago → 16 years ago
Resolution: --- → FIXED
Depends on: 602351
Product: mozilla.org → Release Engineering
Flags: needinfo?(muhammad.saadzaheer991)
QA Contact: aki

Restricting comments on this old fixed bug to avoid more spam.

Restrict Comments: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: