Closed Bug 219936 Opened 21 years ago Closed 21 years ago

No new (or sporadic) Win32 builds due to issues around transition to new servers.

Categories

(mozilla.org :: FTP: Staging, task)

Product:
mozilla.org
Component:
FTP: Staging
Platform:
x86
Windows XP
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: iannbugzilla, Assigned: leaf)

References

Details

There has not been any win32 builds either installer or zip since BuildID
2003091704.
From the build log, it appears that some dir/drive is missing.
How about a hard boot-to-the-head by a local admin?

...
-->end mozconfig<----------------------------------------
Didn't find /cygdrive/c/builds/tinderbox/post-mozilla.pl
*** Bug 220003 has been marked as a duplicate of this bug. ***
-> Leaf, as per Asa's "Assigned To" reassignment of my duplicate bug.
Assignee: endico → leaf
Well, it's not because of any CVS checkin that calls that file. Every file that
references post-mozilla.pl hasn't been changed since August 27 at the latest.
There are no 1.5 trunk builds either. And no Firebird builds.
Does "/cygdrive/c/builds/tinderbox/post-mozilla.pl" definitely exist on the
build machine?
The tinderboxes have absolutely nothing to do with the nightly build scripts. 
Btw, post-mozilla.pl is optional; nothing bad will happen if it doesn't exist.

When the machines were moved last week, did anyone move the build machine as
well or should someone just start up a cron job on a new machine?

(Isn't leaf still on sabbatical?)

Does anybody know which machine was doing the builds?
I hesitate to mark this fixed without someone knowing what was going on (and
Firebird builds haven't appeared), but a collection of Windows builds has now
appeared in http://ftp.mozilla.org/pub/mozilla/nightly/2003-09-23-04-trunk/

I heard (secondhand) that the problem was due to running out of disk space on
komodo (which is the bottleneck now that stage has a larger disk, although it
won't be for much longer).
ok, well given that someone apparently knows what was going on, and there are
Firebird builds, I guess this can be resolved.
Status: NEW → RESOLVED
Closed: 21 years ago
Resolution: --- → WORKSFORME
btw: ftp://ftp.mozilla.org/pub/data/crash-data/ are also out-of-date. Is this
machine broken, been removed or what?
No builds again now since the 27th...
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
yep, they're there.

marking resolved again.

maybe we should just call them "occasional builds" or something instead...
Status: REOPENED → RESOLVED
Closed: 21 years ago → 21 years ago
Resolution: --- → WORKSFORME
There were no new builds there through last evening.  They are indeed occasional
especially over a weekend.
> They are indeed occasional especially over a weekend.

Normally they don't skip weekdays though.  Also, has anybody noticed that
they're only being built in the morning now?  When's the last time you saw an
-08 build?  Is this symptomatic of something else?
I believe the -04 (i.e. 4am, or sometimes -05 for 5am) builds were being built
by a different machine from the 8am builds.  I would imagine the lack of the 8am
builds is because the machine that was building them isn't around any more
(probably because it was a netscape machine, and it either wasn't transferred or
hasn't been set up again).


No 10/6 (Monday) or 10/7 (Tuesday) builds now. I won't reopen this (again) yet
but I'm pretty suspcious about this actually having been resolved.
The last win32 nightly was BuildID 2003100404 and the last including a talkback
zip was BuildID 2003100304. It might all be fixed once things are moved to the
new servers on Wednesday but I'll reopen this anyway.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Summary: No win32 builds since buildid 2003091704 → No recent win32 builds (since buildid 2003100404 this time)
Saying that the new servers on ftp-new.mozilla.org have exactly the same problem.
Rather than having this constantly marked WORKSFORME and then reopened again
every week or so - is there anybody who can give a more definitive statement in
terms of what has actually been going on here the likelihood and a more
permanent fix to the problem?  (Even if the "solution" is some kind of better QA
procedure such that if a build isn't generated, for whatever reason, by
automated methods somebody manually does so.)
well dbaron said what the previous problem was in comment 11 - komodo running
out of space.  on previous occasions the problem has been the staging server
running out of space.

Not sure what the second occurance was, but I saw people talking about machines
not talking to each other (when the server moved).

The latest problem is apparently an issue with the talkback server, which has
now been "tweaked" and builds should reappear today (the 8th).

I can't give a definitive statement, but it sounds to me as if it's a different
problem each time, except that they're all related to various machines/services
transitioning from Netscape to Mozilla. So I guess we should theoretically be
filing a new bug each time ;)  I think pretty much everything is moved over now
though, so hopefully this won't be ongoing...
Okay.  For those reasons, once this is resolved let's mark it fixed (as
transitional problems will be complete and something will have actually been
done), then file a new bug should it happen again.

Tweaking summary.

We should keep this bug open for monitoring purposes until the server transition
is complete.
Summary: No recent win32 builds (since buildid 2003100404 this time) → No new (or sporadic) Win32 builds due to issues around transition to new servers.
I'd like to mention that there are no win32 Firebird trunk builds since 10/02.
Server transition has been complete for about a week now.  Is this still happening?
Still no win32 Firebird nightly since more than two weeks:
http://ftp.mozilla.org/pub/mozilla.org/firebird/nightly/latest-trunk/
There are no 10-17 Win32 trunk builds either.
Make that, only on *some* of the servers that ftp.mozilla.org resolves to are
there missing 10-17 builds (and now missing 10-18 builds as well). One being
ftp://mozilla.ussg.indiana.edu/pub/mozilla.org/mozilla/nightly/latest-trunk/.
ftp://mozilla.isc.org/pub/mozilla.org/mozilla/nightly/latest-trunk/ has the
10-17 build, but not 10-18.

Should unreliable mirrors fall into this bug, or should another be opened?
yeah, open another bug for that, under FTP: Mirrors
It gets even better. Now files in the "latest" directories are having their
modified dates changed so that they appear to be today's builds, but they really
could be days old. See bug 223671. Now you have to download the files before you
know if they are old or not.
No new builds now since 10/28.
still no windows build since 28/10

BTW, browsing the FTP site with mozilla gives this date for the last build:
12123 KB  	28/10/2002  	14:51:00

2002 !!

Is it a known Mozilla bug ? (my FTP software as well as Opera label the file as
2003)
Leaf posted some explanation of what's happened to the Windows builds (the
legacy system died and they haven't got the new one ready - he's away until
Thursday, so I don't imagine it's get finished before then...). See bug 224340
comment 14
It says 2003 for me with the 10/28-04 build (the last one available so far)
under XP.

For a second I was hoping that it it *was* saying 2002 because then, possibly,
new builds actually were being released but just being mislabelled...
It depends on which mirror you get. The ftp.isc.org mirror does indeed have 2002
as the year on the builds.

Has anyone heard from leaf? It's Thursday.
I emailed leaf about this, and got a reply from him today:

"The machine doing release builds (at netscape) has fallen over dead. I'm
working on a replacement. Meant to be done weeks ago, might get it done
tomorrow. Stay tuned."
Any update? I'm shuddering thinking of all the regressions that could be
happening on Win32 over the last two weeks and we'd never hear about it.
According to Asa's blog and a email to leaf, nightlies should be avaible very
soon now (1-3 days) :)
http://ftp.mozilla.org/pub/mozilla.org/mozilla/nightly/2003-11-13-09-trunk/

has installer and stub-installer build, but no talkback (947 byte dummy).
License screen has the same wrong format as the first 1.6a release.

http://ftp.mozilla.org/pub/mozilla.org/mozilla/nightly/latest-trunk/
still shows the build from Oct 28th as the latest.
*** Bug 225688 has been marked as a duplicate of this bug. ***
*** Bug 225774 has been marked as a duplicate of this bug. ***
http://ftp.mozilla.org/pub/mozilla.org/mozilla/nightly/latest/
Builds are back there.
Should this bug be marked RESOLVED now?
I don't see all the builds. I see the installers, but not the zip files. Still
need mozilla-win32.zip and mozilla-win32-talkback.zip to show up at least.
Not to mention that those builds are already two days old with no new Win32
builds for 11-15 and 11-16.
Is there a separate bug for Linux builds? Because the "latest" nightly Linux
builds are from November 10.
FYI,

Even though the date on the stub installer is "11/16" when I downloaded and
installed it, the Mozilla build I ended up actually having was 11/14-15.  So the
builds you see in "Latest Builds" are NOT what they claim to be.

This could be related to bug 223671.  (In any case, this bug is still nowhere
close to being resolved.)
Hmm.  Just checked again and I now see the correct dates.  Perhaps only some of
the mirrors are getting this wrong?
The dates are jacked on some servers. Some servers have builds that other
servers don't. In short, the mirrors suck. If you do a netstat while you are
connected to a mirror you like, you can get the real location and use that
instead of ftp.mozilla.org. That way you won't be connected to the slow servers
with wrong dates and no reliable updates of the latest builds.
The *-sea builds aren't showing up yet either (obvious I know but necessary to
mention that not all builds are there yet).
The equivalent of the -sea builds (the installers) are there, they just have
different names now (without the -sea, which was rather redundant - an installer
archive that didn't extract files wouldn't be much use, and the stub installer
is named as such)

It's still true that not all the builds are there yet...
*** Bug 226160 has been marked as a duplicate of this bug. ***
This really needs to be sorted in time for 1.6 final - it would be very
embarrassing not to have any zip builds available for a release.
Adding asa to cc seeing the product mozilla.org doesn't have blocking flags.
This problem is more serious than that.  They ere have been several bug reports
about crahses caused by post 10/28 checkins to the trunk.  The status of these
(accoring to the comments) are that they are NOT being worked on becuase the
reporter did not submit any talkback data.  There is no way to submit talkback
data since the last talkback enabled build is dated 10/28 and the problems were
introduced later.  We either need to get talkback builds working or get these
developers to work on these issues without using the lack of talkback as an
excuse.  Otherwise 1.6 will be much less stable than 1.5 is. Just my $.02.
Blocks: majorbugs
Apologize for not commenting more frequently with status updates.

Here's where we're at:

We have a tinderbox producing release builds and putting them in dated
directories under the nightly directory once daily (at least). I initially left
out zip files, but that has been rectified. The zip files are now created
automatically via the build system, and the filename generated fits the more
standard format (citing compiler in addition to platform, in this case using the
designation "msvc"; i will try adding something that specifies windows, the same
way "linux" is specified for linux builds).

So, for now we have .zip files being posted alongside installer builds, called:
mozilla-i586-pc-msvc.zip

The issue with talkback not being enabled is going to remain an issue for a week
or two at least, because mozilla.org no longer has access to talkback servers.
We are working on getting them set up, but this will probably not happen in the
next week or two. I realize this severely damages our ability to do crash
testing, but there's not much that we can do until we get the servers up and
running.

I will leave the bug open (assigned to me) as a place holder for restoring
talkback client-side when we get the servers up and running.
Status: REOPENED → ASSIGNED
As of Nov 29 -2003 the win32 installer exe is 11488 bytes the same size for the
last 1.5 weeks .... Just the build Id is changing ....
Glad the win32 nigthly builds are back.  Thanks.

Leaf mentioned the new file naming coventions for the nightly builds.  Can
someone please add a readme file to the directory that explains the file naming
convention?

This would help people quickly figure out which file is right for their
purposes, rather than spending a LONG time hunting around www.mozilla.org and
hopefully eventually stumbling upon this bug report, like I did.
Someone should have marked this FIXED a long time ago :)
The win32 ZIP nightlies are now called 'mozilla-i586-pc-msvc.zip', and the most
recent build is here:
<http://ftp.mozilla.org/pub/mozilla.org/mozilla/nightly/latest/mozilla-i586-pc-msvc.zip>
Status: ASSIGNED → RESOLVED
Closed: 21 years ago → 21 years ago
Resolution: --- → FIXED
Reopening per comment 58 ("I will leave the bug open (assigned to me) as a place
holder for restoring talkback client-side when we get the servers up and running.")
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I'm sorry, I didn't remember that comment. I also thought Talkback was handled
by bug 216827 and bug 116934.
Since yesterday starting with the 2004010108 Build the Tinderbox-Creature-Builds
will all land in: http://ftp.mozilla.org/pub/mozilla.org/mozilla/nightly/latest/

Much better than the Situation in Oktober/November2003, but I thought normaly
the /latest/ would only be updated once the day, see Comment #58.

In http://ftp.mozilla.org/pub/mozilla.org/mozilla/tinderbox-builds/CREATURE/
there are no new Builds since yesterday.
marking fixed, as we have builds, and another bug is open for restoring talkback
functionality (bug 218898).
Status: REOPENED → RESOLVED
Closed: 21 years ago → 21 years ago
Resolution: --- → FIXED
No longer blocks: majorbugs
You need to log in before you can comment on or make changes to this bug.