Closed Bug 219936 Opened 21 years ago Closed 21 years ago

No new (or sporadic) Win32 builds due to issues around transition to new servers.

Categories

(mozilla.org :: FTP: Staging, task)

Product:
mozilla.org
Component:
FTP: Staging
Platform:
x86
Windows XP
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: iannbugzilla, Assigned: leaf)

References

Details

There has not been any win32 builds either installer or zip since BuildID 2003091704.
From the build log, it appears that some dir/drive is missing. How about a hard boot-to-the-head by a local admin? ... -->end mozconfig<---------------------------------------- Didn't find /cygdrive/c/builds/tinderbox/post-mozilla.pl
*** Bug 220003 has been marked as a duplicate of this bug. ***
-> Leaf, as per Asa's "Assigned To" reassignment of my duplicate bug.
Assignee: endico → leaf
Well, it's not because of any CVS checkin that calls that file. Every file that references post-mozilla.pl hasn't been changed since August 27 at the latest.
There are no 1.5 trunk builds either. And no Firebird builds.
Does "/cygdrive/c/builds/tinderbox/post-mozilla.pl" definitely exist on the build machine?
The tinderboxes have absolutely nothing to do with the nightly build scripts. Btw, post-mozilla.pl is optional; nothing bad will happen if it doesn't exist. When the machines were moved last week, did anyone move the build machine as well or should someone just start up a cron job on a new machine? (Isn't leaf still on sabbatical?)
Does anybody know which machine was doing the builds?
I hesitate to mark this fixed without someone knowing what was going on (and Firebird builds haven't appeared), but a collection of Windows builds has now appeared in http://ftp.mozilla.org/pub/mozilla/nightly/2003-09-23-04-trunk/
I heard (secondhand) that the problem was due to running out of disk space on komodo (which is the bottleneck now that stage has a larger disk, although it won't be for much longer).
ok, well given that someone apparently knows what was going on, and there are Firebird builds, I guess this can be resolved.
Status: NEW → RESOLVED
Closed: 21 years ago
Resolution: --- → WORKSFORME
btw: ftp://ftp.mozilla.org/pub/data/crash-data/ are also out-of-date. Is this machine broken, been removed or what?
No builds again now since the 27th...
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
yep, they're there. marking resolved again. maybe we should just call them "occasional builds" or something instead...
Status: REOPENED → RESOLVED
Closed: 21 years ago → 21 years ago
Resolution: --- → WORKSFORME
There were no new builds there through last evening. They are indeed occasional especially over a weekend.
> They are indeed occasional especially over a weekend. Normally they don't skip weekdays though. Also, has anybody noticed that they're only being built in the morning now? When's the last time you saw an -08 build? Is this symptomatic of something else?
I believe the -04 (i.e. 4am, or sometimes -05 for 5am) builds were being built by a different machine from the 8am builds. I would imagine the lack of the 8am builds is because the machine that was building them isn't around any more (probably because it was a netscape machine, and it either wasn't transferred or hasn't been set up again).
No 10/6 (Monday) or 10/7 (Tuesday) builds now. I won't reopen this (again) yet but I'm pretty suspcious about this actually having been resolved.
The last win32 nightly was BuildID 2003100404 and the last including a talkback zip was BuildID 2003100304. It might all be fixed once things are moved to the new servers on Wednesday but I'll reopen this anyway.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Summary: No win32 builds since buildid 2003091704 → No recent win32 builds (since buildid 2003100404 this time)
Saying that the new servers on ftp-new.mozilla.org have exactly the same problem.
Rather than having this constantly marked WORKSFORME and then reopened again every week or so - is there anybody who can give a more definitive statement in terms of what has actually been going on here the likelihood and a more permanent fix to the problem? (Even if the "solution" is some kind of better QA procedure such that if a build isn't generated, for whatever reason, by automated methods somebody manually does so.)
well dbaron said what the previous problem was in comment 11 - komodo running out of space. on previous occasions the problem has been the staging server running out of space. Not sure what the second occurance was, but I saw people talking about machines not talking to each other (when the server moved). The latest problem is apparently an issue with the talkback server, which has now been "tweaked" and builds should reappear today (the 8th). I can't give a definitive statement, but it sounds to me as if it's a different problem each time, except that they're all related to various machines/services transitioning from Netscape to Mozilla. So I guess we should theoretically be filing a new bug each time ;) I think pretty much everything is moved over now though, so hopefully this won't be ongoing...
Okay. For those reasons, once this is resolved let's mark it fixed (as transitional problems will be complete and something will have actually been done), then file a new bug should it happen again. Tweaking summary. We should keep this bug open for monitoring purposes until the server transition is complete.
Summary: No recent win32 builds (since buildid 2003100404 this time) → No new (or sporadic) Win32 builds due to issues around transition to new servers.
I'd like to mention that there are no win32 Firebird trunk builds since 10/02.
Server transition has been complete for about a week now. Is this still happening?
Still no win32 Firebird nightly since more than two weeks: http://ftp.mozilla.org/pub/mozilla.org/firebird/nightly/latest-trunk/
There are no 10-17 Win32 trunk builds either.
Make that, only on *some* of the servers that ftp.mozilla.org resolves to are there missing 10-17 builds (and now missing 10-18 builds as well). One being ftp://mozilla.ussg.indiana.edu/pub/mozilla.org/mozilla/nightly/latest-trunk/. ftp://mozilla.isc.org/pub/mozilla.org/mozilla/nightly/latest-trunk/ has the 10-17 build, but not 10-18. Should unreliable mirrors fall into this bug, or should another be opened?
yeah, open another bug for that, under FTP: Mirrors
It gets even better. Now files in the "latest" directories are having their modified dates changed so that they appear to be today's builds, but they really could be days old. See bug 223671. Now you have to download the files before you know if they are old or not.
No new builds now since 10/28.
still no windows build since 28/10 BTW, browsing the FTP site with mozilla gives this date for the last build: 12123 KB 28/10/2002 14:51:00 2002 !! Is it a known Mozilla bug ? (my FTP software as well as Opera label the file as 2003)
Leaf posted some explanation of what's happened to the Windows builds (the legacy system died and they haven't got the new one ready - he's away until Thursday, so I don't imagine it's get finished before then...). See bug 224340 comment 14
It says 2003 for me with the 10/28-04 build (the last one available so far) under XP. For a second I was hoping that it it *was* saying 2002 because then, possibly, new builds actually were being released but just being mislabelled...
It depends on which mirror you get. The ftp.isc.org mirror does indeed have 2002 as the year on the builds. Has anyone heard from leaf? It's Thursday.
I emailed leaf about this, and got a reply from him today: "The machine doing release builds (at netscape) has fallen over dead. I'm working on a replacement. Meant to be done weeks ago, might get it done tomorrow. Stay tuned."
Any update? I'm shuddering thinking of all the regressions that could be happening on Win32 over the last two weeks and we'd never hear about it.
According to Asa's blog and a email to leaf, nightlies should be avaible very soon now (1-3 days) :)
http://ftp.mozilla.org/pub/mozilla.org/mozilla/nightly/2003-11-13-09-trunk/ has installer and stub-installer build, but no talkback (947 byte dummy). License screen has the same wrong format as the first 1.6a release. http://ftp.mozilla.org/pub/mozilla.org/mozilla/nightly/latest-trunk/ still shows the build from Oct 28th as the latest.
*** Bug 225688 has been marked as a duplicate of this bug. ***
*** Bug 225774 has been marked as a duplicate of this bug. ***
http://ftp.mozilla.org/pub/mozilla.org/mozilla/nightly/latest/ Builds are back there. Should this bug be marked RESOLVED now?
I don't see all the builds. I see the installers, but not the zip files. Still need mozilla-win32.zip and mozilla-win32-talkback.zip to show up at least.
Not to mention that those builds are already two days old with no new Win32 builds for 11-15 and 11-16.
Is there a separate bug for Linux builds? Because the "latest" nightly Linux builds are from November 10.
FYI, Even though the date on the stub installer is "11/16" when I downloaded and installed it, the Mozilla build I ended up actually having was 11/14-15. So the builds you see in "Latest Builds" are NOT what they claim to be. This could be related to bug 223671. (In any case, this bug is still nowhere close to being resolved.)
Hmm. Just checked again and I now see the correct dates. Perhaps only some of the mirrors are getting this wrong?
The dates are jacked on some servers. Some servers have builds that other servers don't. In short, the mirrors suck. If you do a netstat while you are connected to a mirror you like, you can get the real location and use that instead of ftp.mozilla.org. That way you won't be connected to the slow servers with wrong dates and no reliable updates of the latest builds.
The *-sea builds aren't showing up yet either (obvious I know but necessary to mention that not all builds are there yet).
The equivalent of the -sea builds (the installers) are there, they just have different names now (without the -sea, which was rather redundant - an installer archive that didn't extract files wouldn't be much use, and the stub installer is named as such) It's still true that not all the builds are there yet...
*** Bug 226160 has been marked as a duplicate of this bug. ***
This really needs to be sorted in time for 1.6 final - it would be very embarrassing not to have any zip builds available for a release. Adding asa to cc seeing the product mozilla.org doesn't have blocking flags.
This problem is more serious than that. They ere have been several bug reports about crahses caused by post 10/28 checkins to the trunk. The status of these (accoring to the comments) are that they are NOT being worked on becuase the reporter did not submit any talkback data. There is no way to submit talkback data since the last talkback enabled build is dated 10/28 and the problems were introduced later. We either need to get talkback builds working or get these developers to work on these issues without using the lack of talkback as an excuse. Otherwise 1.6 will be much less stable than 1.5 is. Just my $.02.
Blocks: majorbugs
Apologize for not commenting more frequently with status updates. Here's where we're at: We have a tinderbox producing release builds and putting them in dated directories under the nightly directory once daily (at least). I initially left out zip files, but that has been rectified. The zip files are now created automatically via the build system, and the filename generated fits the more standard format (citing compiler in addition to platform, in this case using the designation "msvc"; i will try adding something that specifies windows, the same way "linux" is specified for linux builds). So, for now we have .zip files being posted alongside installer builds, called: mozilla-i586-pc-msvc.zip The issue with talkback not being enabled is going to remain an issue for a week or two at least, because mozilla.org no longer has access to talkback servers. We are working on getting them set up, but this will probably not happen in the next week or two. I realize this severely damages our ability to do crash testing, but there's not much that we can do until we get the servers up and running. I will leave the bug open (assigned to me) as a place holder for restoring talkback client-side when we get the servers up and running.
Status: REOPENED → ASSIGNED
As of Nov 29 -2003 the win32 installer exe is 11488 bytes the same size for the last 1.5 weeks .... Just the build Id is changing ....
Glad the win32 nigthly builds are back. Thanks. Leaf mentioned the new file naming coventions for the nightly builds. Can someone please add a readme file to the directory that explains the file naming convention? This would help people quickly figure out which file is right for their purposes, rather than spending a LONG time hunting around www.mozilla.org and hopefully eventually stumbling upon this bug report, like I did.
Someone should have marked this FIXED a long time ago :) The win32 ZIP nightlies are now called 'mozilla-i586-pc-msvc.zip', and the most recent build is here: <http://ftp.mozilla.org/pub/mozilla.org/mozilla/nightly/latest/mozilla-i586-pc-msvc.zip>
Status: ASSIGNED → RESOLVED
Closed: 21 years ago → 21 years ago
Resolution: --- → FIXED
Reopening per comment 58 ("I will leave the bug open (assigned to me) as a place holder for restoring talkback client-side when we get the servers up and running.")
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I'm sorry, I didn't remember that comment. I also thought Talkback was handled by bug 216827 and bug 116934.
Since yesterday starting with the 2004010108 Build the Tinderbox-Creature-Builds will all land in: http://ftp.mozilla.org/pub/mozilla.org/mozilla/nightly/latest/ Much better than the Situation in Oktober/November2003, but I thought normaly the /latest/ would only be updated once the day, see Comment #58. In http://ftp.mozilla.org/pub/mozilla.org/mozilla/tinderbox-builds/CREATURE/ there are no new Builds since yesterday.
marking fixed, as we have builds, and another bug is open for restoring talkback functionality (bug 218898).
Status: REOPENED → RESOLVED
Closed: 21 years ago → 21 years ago
Resolution: --- → FIXED
No longer blocks: majorbugs
You need to log in before you can comment on or make changes to this bug.