Closed Bug 341752 Opened 14 years ago Closed 14 years ago

Change to FTP mirroring and migration of AUS to new hardware has broken nightly updates (none since 20060615)

Categories

(Webtools Graveyard :: Tinderbox, defect, critical)

defect
Not set
critical

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nthomas, Assigned: preed)

References

Details

Attachments

(1 file, 1 obsolete file)

The tinderbox are using "mozilla.osuosl.org" in the snippets they send to AUS for nightly updates. Now that that host is only in the releases.m.o poool it doesn't have nightly builds, so the tinderbox should point to ftp.mozilla.org (like the default in tinder-defaults.pl).
See bug 341757 for the AUS "half" of restoring nightly updates.
Summary: Change to FTP mirroring means nighty update prefs need changing (no updates available since 20060615) → Change to FTP mirroring means nightly update prefs need changing (no updates available since 20060615)
Fx and Tb has been reconfigured to use ftp.mozilla.org for:

trunk
MOZILLA_1_8
MOZILLA_1_8_0

And l10n for each as well. This should be fixed on the client side for the next  batch of nightly builds.
Assignee: build → rhelmer
*** Bug 341757 has been marked as a duplicate of this bug. ***
Assignee: rhelmer → build
Blocks: 306864
> *** Bug 341757 has been marked as a duplicate of this bug. ***
> 
Just to clarify -- that was marked as a dupe of this bug because with the new patch to aus2, latest updates will be offered as completes.  That means when a new update is released, everyone who isn't one build away will receive the latest complete update.

The bug regarding an incorrect domain in build data coming from tinderbox is breaking new completes and partials for nightlies -- and has been for at least one hop.

Therefore, once the updates start working again for _new_ updates, the older builds will also be fixed because they will receive an update pointing to the new (and fixed) complete updates.

Hope that helps/makes sense.
I guess you must be still working on this, but here's what I see at the moment (based on spot checks amongst the Firefox branches) 

* Anything with a build date of 20060613 or earlier gets a complete update to the 20060615 nightly [1], 20060614 gets a partial update [2]. But, 
 ** both use mozilla.osuosl.org rather than ftp.mozilla.org as the filehost
 ** 306864 is working provided you think 20060615 is the most recent build

* 20060615 and later builds are not offered any updates [3,4]

* There are no partials for 20060618 -> 19 (the most recent nightly at time of writing; patcher is still being given ref's to files on mozilla.osuosl.org ?)

Which seems to point to a problem on AUS, either in the snippets or their interpretation.

[1] https://aus2.mozilla.org/update/1/Firefox/2.0a3/2006061303/WINNT_x86-msvc/en-US/nightly/update.xml
[2] https://aus2.mozilla.org/update/1/Firefox/2.0a3/2006061404/WINNT_x86-msvc/en-US/nightly/update.xml
[3]
https://aus2.mozilla.org/update/1/Firefox/2.0a3/2006061503/WINNT_x86-msvc/en-US/nightly/update.xml
[4]
https://aus2.mozilla.org/update/1/Firefox/2.0a3/2006061804/WINNT_x86-msvc/en-US/nightly/update.xml
(In reply to comment #5)
> I guess you must be still working on this, but here's what I see at the moment
> (based on spot checks amongst the Firefox branches) 

Still working on it.. we now need to clear the backlog of snippets so that new ones will be processed, as the partial patch producer was built with the assumption that it must build a complete chain of partial updates.
(In reply to comment #6)
> Still working on it.. we now need to clear the backlog of snippets so that new
> ones will be processed, as the partial patch producer was built with the
> assumption that it must build a complete chain of partial updates.

Clearing the backlog is the cause of the gap from 20060615 ?  So things should be better after the 20060619 round of nightlies ?

Oops, I mean 20060620 rather than 19
-- I guess we never thought we'd change the aus server. We did.
-- Remove completely dead code paths.
Assignee: build → preed
Status: NEW → ASSIGNED
Attachment #226565 - Flags: review?(morgamic)
Attachment #226565 - Flags: review?(rhelmer)
Comment on attachment 226565 [details] [diff] [review]
Variablize the aus host we publish snippets to

Looks good -- just 0 and 1 aren't used anymore.
Attachment #226565 - Flags: review?(morgamic) → review+
-- I guess we never thought we'd change the aus server. We did.
-- Remove completely dead code paths.
-- Remove all the gotos; they're all effectively a return anyway
-- coalece the following:

if (file doens't exists) {
  .. error ...
}

do something

if (file does exist) {
  .. do something else...
}

into:

if (file doesn't exist) {
  .. error..
} else {
  .. do everything...
}
Attachment #226565 - Attachment is obsolete: true
Attachment #226573 - Flags: review?(morgamic)
Attachment #226565 - Flags: review?(rhelmer)
Attachment #226573 - Flags: review?(rhelmer)
Blocks: 340520
Comment on attachment 226573 [details] [diff] [review]
Variablize the aus host we publish snippets to + cleanup

Even better -- I like the return instead of goto (didn't know you could do that?).  The goto would've just returned anyway -- I didn't see anything it woulda missed by just returning $status.  Paths, etc., look good for the xml side of things.
Attachment #226573 - Flags: review?(morgamic) → review+
Attachment #226573 - Flags: review?(rhelmer) → review?(morgamic)
Attachment #226573 - Flags: review?(morgamic) → review+
v2 checked in; let's see how this helps the update situation.
I checked the tinderbox logs for Mozilla1.8 Firefox Windows, Mac Uni & Linux nightlies, and all are trying to push the snippets to aus2-staging.m.o, so success there. From look of wiki.m.o/Build:Farm, both patrocles and beast will need manual updating for attachment 226573 [details] [diff] [review]. Also, crazyhorse (Tb Mozilla1.8.0) didn't pick up the change when the wiki indicates it should have.

There is a ssh key problem though, the error message is "Host key verifcation failed", so none of the snippets make it on to aus2-staging.
Summary: Change to FTP mirroring means nightly update prefs need changing (no updates available since 20060615) → Change to FTP mirroring and migration of AUS to new hardware has broken nightly updates (none since 20060615)
I've worked up some semi-automatic monitoring of the update situation:
  http://users.ox.ac.uk/~clar0239/upwatch/update.html

In particular: 
 20060622  http://users.ox.ac.uk/~clar0239/upwatch/20060622/master.html
 20060623  http://users.ox.ac.uk/~clar0239/upwatch/20060623/master.html

Not a huge difference between them, even though there has been plenty of time to generate partials since the first of todays nightlies. 

The major issue at the moment is that a significant portion of the tinderbox are not able to push snippets to AUS (see previous comment).

Please note the caveats at the bottom of the pages - the 1.8.0 and windows thunderbird information is unreliable.
Severity: major → critical
Depends on: 342549
Much improvement for Firefox on the 1.8 branch
  http://users.ox.ac.uk/~clar0239/upwatch/20060625/master.html
when compared to the 20060623 table. Seems that the snippets are being pushed from tinderbox to AUS there, and that you've done great work getting the wrinkles out of AUS.

Elsewhere there are issues getting the updates to AUS though. See the notes on the second half of the page for what I discovered.
I'm going to resolve this; I went through the entire list of problems that cf so painstakingly put together in http://users.ox.ac.uk/~clar0239/upwatch/20060625/master.html and corrected all the problems I encountered.

There's bug 342682, filed specifically to track update issues with Mac UB, but other builds should be fine (and were reported to be in #build; someone got a Win32 1.8 update today after we fixed the problems).

(The problem, BTW, was that a Thunderbird machine was posting build snippets with osuosl in them, causing the partial patcher script on prometheus to die when it failed to download the non-existant builds).
Status: ASSIGNED → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Component: Tinderbox Configuration → Tinderbox
Product: mozilla.org → Webtools
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.