341752 - Change to FTP mirroring and migration of AUS to new hardware has broken nightly updates (none since 20060615)

Status: RESOLVED FIXED

(Webtools Graveyard :: Tinderbox, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

The tinderbox are using "mozilla.osuosl.org" in the snippets they send to AUS for nightly updates. Now that that host is only in the releases.m.o poool it doesn't have nightly builds, so the tinderbox should point to ftp.mozilla.org (like the default in tinder-defaults.pl).

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

See bug 341757 for the AUS "half" of restoring nightly updates.

Comment 2

[Robert Helmer [:rhelmer]]

Fx and Tb has been reconfigured to use ftp.mozilla.org for:

trunk
MOZILLA_1_8
MOZILLA_1_8_0

And l10n for each as well. This should be fixed on the client side for the next  batch of nightly builds.

Comment 3

[Michael Morgan [:morgamic]]

*** Bug 341757 has been marked as a duplicate of this bug. ***

Comment 4

[Michael Morgan [:morgamic]]

> *** Bug 341757 has been marked as a duplicate of this bug. ***
> 
Just to clarify -- that was marked as a dupe of this bug because with the new patch to aus2, latest updates will be offered as completes.  That means when a new update is released, everyone who isn't one build away will receive the latest complete update.

The bug regarding an incorrect domain in build data coming from tinderbox is breaking new completes and partials for nightlies -- and has been for at least one hop.

Therefore, once the updates start working again for _new_ updates, the older builds will also be fixed because they will receive an update pointing to the new (and fixed) complete updates.

Hope that helps/makes sense.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

I guess you must be still working on this, but here's what I see at the moment (based on spot checks amongst the Firefox branches) 

* Anything with a build date of 20060613 or earlier gets a complete update to the 20060615 nightly [1], 20060614 gets a partial update [2]. But, 
 ** both use mozilla.osuosl.org rather than ftp.mozilla.org as the filehost
 ** 306864 is working provided you think 20060615 is the most recent build

* 20060615 and later builds are not offered any updates [3,4]

* There are no partials for 20060618 -> 19 (the most recent nightly at time of writing; patcher is still being given ref's to files on mozilla.osuosl.org ?)

Which seems to point to a problem on AUS, either in the snippets or their interpretation.

[1] https://aus2.mozilla.org/update/1/Firefox/2.0a3/2006061303/WINNT_x86-msvc/en-US/nightly/update.xml
[2] https://aus2.mozilla.org/update/1/Firefox/2.0a3/2006061404/WINNT_x86-msvc/en-US/nightly/update.xml
[3]
https://aus2.mozilla.org/update/1/Firefox/2.0a3/2006061503/WINNT_x86-msvc/en-US/nightly/update.xml
[4]
https://aus2.mozilla.org/update/1/Firefox/2.0a3/2006061804/WINNT_x86-msvc/en-US/nightly/update.xml

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to comment #5)
> I guess you must be still working on this, but here's what I see at the moment
> (based on spot checks amongst the Firefox branches) 

Still working on it.. we now need to clear the backlog of snippets so that new ones will be processed, as the partial patch producer was built with the assumption that it must build a complete chain of partial updates.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #6)
> Still working on it.. we now need to clear the backlog of snippets so that new
> ones will be processed, as the partial patch producer was built with the
> assumption that it must build a complete chain of partial updates.

Clearing the backlog is the cause of the gap from 20060615 ?  So things should be better after the 20060619 round of nightlies ?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Oops, I mean 20060620 rather than 19

Comment 9

[J. Paul Reed [:preed]]

Attachment: Variablize the aus host we publish snippets to

-- I guess we never thought we'd change the aus server. We did.
-- Remove completely dead code paths.

Comment 10

[Michael Morgan [:morgamic]]

Comment on attachment 226565 [details] [diff] [review]
Variablize the aus host we publish snippets to

Looks good -- just 0 and 1 aren't used anymore.

Comment 11

[J. Paul Reed [:preed]]

Attachment: Variablize the aus host we publish snippets to + cleanup

-- I guess we never thought we'd change the aus server. We did.
-- Remove completely dead code paths.
-- Remove all the gotos; they're all effectively a return anyway
-- coalece the following:

if (file doens't exists) {
  .. error ...
}

do something

if (file does exist) {
  .. do something else...
}

into:

if (file doesn't exist) {
  .. error..
} else {
  .. do everything...
}

Comment 12

[Michael Morgan [:morgamic]]

Comment on attachment 226573 [details] [diff] [review]
Variablize the aus host we publish snippets to + cleanup

Even better -- I like the return instead of goto (didn't know you could do that?).  The goto would've just returned anyway -- I didn't see anything it woulda missed by just returning $status.  Paths, etc., look good for the xml side of things.

Comment 13

[J. Paul Reed [:preed]]

v2 checked in; let's see how this helps the update situation.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

I checked the tinderbox logs for Mozilla1.8 Firefox Windows, Mac Uni & Linux nightlies, and all are trying to push the snippets to aus2-staging.m.o, so success there. From look of wiki.m.o/Build:Farm, both patrocles and beast will need manual updating for attachment 226573 [details] [diff] [review]. Also, crazyhorse (Tb Mozilla1.8.0) didn't pick up the change when the wiki indicates it should have.

There is a ssh key problem though, the error message is "Host key verifcation failed", so none of the snippets make it on to aus2-staging.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

I've worked up some semi-automatic monitoring of the update situation:
  http://users.ox.ac.uk/~clar0239/upwatch/update.html

In particular: 
 20060622  http://users.ox.ac.uk/~clar0239/upwatch/20060622/master.html
 20060623  http://users.ox.ac.uk/~clar0239/upwatch/20060623/master.html

Not a huge difference between them, even though there has been plenty of time to generate partials since the first of todays nightlies. 

The major issue at the moment is that a significant portion of the tinderbox are not able to push snippets to AUS (see previous comment).

Please note the caveats at the bottom of the pages - the 1.8.0 and windows thunderbird information is unreliable.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Much improvement for Firefox on the 1.8 branch
  http://users.ox.ac.uk/~clar0239/upwatch/20060625/master.html
when compared to the 20060623 table. Seems that the snippets are being pushed from tinderbox to AUS there, and that you've done great work getting the wrinkles out of AUS.

Elsewhere there are issues getting the updates to AUS though. See the notes on the second half of the page for what I discovered.

Comment 17

[J. Paul Reed [:preed]]

I'm going to resolve this; I went through the entire list of problems that cf so painstakingly put together in http://users.ox.ac.uk/~clar0239/upwatch/20060625/master.html and corrected all the problems I encountered.

There's bug 342682, filed specifically to track update issues with Mac UB, but other builds should be fine (and were reported to be in #build; someone got a Win32 1.8 update today after we fixed the problems).

(The problem, BTW, was that a Thunderbird machine was posting build snippets with osuosl in them, causing the partial patcher script on prometheus to die when it failed to download the non-existant builds).