Closed Bug 327140 Opened 19 years ago Closed 19 years ago

Partial update generation failing

Categories

(AUS Graveyard :: Systems, defect)

Product:
AUS Graveyard
Type:
defect
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
4.x (triaged)

People

(Reporter: nthomas, Assigned: rhelmer)

References

Details

From the 20060214 nightly, the updates are only 54 bytes in size and seem to produce no changes - the BuildID/UA does not update. The update applies cleanly though (Update History is correct and last-update.log indcates no error). This affects all nightly updates. Trunk Linux http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-02-13-04-trunk/ Trunk Mac http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-02-13-05-trunk/ Trunk Win32 http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-02-13-06-trunk/ 1.8 branch Linux & Win32 http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-02-13-05-mozilla1.8/ 1.8 branch Mac http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-02-13-04-mozilla1.8/ 1.8.0 branch linux http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-02-13-05-mozilla1.8.0/ Fallout from bug 326465 ?
This could be a permissions issue with the build data. Reassigning to Paul.
Assignee: morgamic → preed
I think this is effecting Thunderbird too (not surprisingly).
The only file getting put into the MAR file is the update manifest (update.manifest). I can help debug this further if I had some direction as to what you guys may have been changing in attempts to fix Bug 326465 with AUS (if anything).
Bug 326465 hasn't prompted any changes to AUS that should affect nightly update generation.
Just trying to narrow down what could have caused this. I just went through the checkin log for the branch for February 13th/14th and verified that there were no client side changes to the update system. And we know this seems to have effected all our products on across all of the branches. So it couldn't be a change to a build machine. That makes me think it has to be a problem with a change to AUS. Or is there another piece of the system that sits between the build machine and AUS which is reponsible for generating partial patches?
I can apply the full updates by hand for Feb. 14th / 15th and they work ok. The problem is isolated to the nightly partials only.
Summary: Nightly updates are only 54 bytes in size and make no changes → Nightly partial updates are only 54 bytes in size and make no changes
*** Bug 327394 has been marked as a duplicate of this bug. ***
There's something strange in my firefox since this bug occurred for the first time. Now, I get sometimes a message in firefox, that it has just completed to download an update. But the strange thing is: I didn't click on "Check for updates" under HELP and automatic updates are disabled, too! These messages about completed downloads are very annoying...
I hope this bug is fixed quickly. Otherwise downloading around 6MB in a dial up connection is very painful.
*** Bug 327673 has been marked as a duplicate of this bug. ***
Raising the Severity to major - this issue has a big impact on testers of nightly builds.
Severity: normal → major
*** Bug 327959 has been marked as a duplicate of this bug. ***
I think I may have found the problem. Using the AUS report emails as a guide, I tried following all the steps in the partial update process by hand on prometheus. I make it as far as running mozilla/tools/update-packaging/unwrap_full_update.sh before encountering a script error. Someone (preed?) had added/modified the following stanza near the start: > list=$($MAR -t "$archive" | cut -d' ' -f3 | sed -e 's/^/\(/' | sed -e 's/$/\)/') > eval "files=($list)" > > for f in $files; do > echo file is "$f" > done > > exit ...and the 'exit' was obviously preventing the script from actually unpacking anything, although the bracketization of the filenames was also causing the script to fail. I will continue investigating, but I've removed that stanza and that may be enough by itself to fix the problem.
The automated patch system just generated a Thunderbird linux patch that is larger than 54 bytes and looks legit when unpacked: -rw-r--r-- 1 cltbld cltbld 282482 Feb 21 08:18 thunderbird-1.5.en-US.linux-i686.partial.2006022004-2006022107.mar [cltbld@prometheus testing]$ $MAR -t ../thunderbird-1.5.en-US.linux-i686.partial.2006022004-2006022107.mar SIZE MODE NAME 109 0644 components/libxpinstall.so.patch 972 0644 update.manifest 110 0644 libfreebl3.so.patch 92 0644 xpicleanup.patch 1214 0644 chrome/toolkit.jar.patch 115 0644 libssl3.so.patch 135 0644 libnspr4.so.patch 119 0644 libsoftokn3.so.patch 112 0644 libnssckbi.so.patch 123 0644 libplds4.so.patch 117 0644 libsmime3.so.patch 276063 0644 thunderbird-bin.patch 632 0644 libfreebl3.chk 124 0644 libplc4.so.patch 117 0644 libnss3.so.patch 385 0644 chrome/en-US.jar.patch 631 0644 libsoftokn3.chk 534 0644 components/nsExtensionManager.js.patch 96 0644 extensions/talkback@mozilla.org/components/talkback/master.ini.patch
Updating the summary.
Summary: Nightly partial updates are only 54 bytes in size and make no changes → Separate nightly mar generation system from release mar generation
I can confirm that Coop's fix to the partial update script did indeed fix this problem. The partial mar files on the trunk for Thunderbird Windows from 2/20 to 2/21: http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2006-02-20-11-trunk/ works correctly!
We still need to regenerate the missing MAR files for the past week. preed is going to take a swing at that, I think.
Thunderbird trunk nightly updated correctly (2006022010 -> 2006022108) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060221 Thunderbird/1.6a1 ID:2006022108
(In reply to comment #18) > Thunderbird trunk nightly updated correctly (2006022010 -> 2006022108) > > Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060221 > Thunderbird/1.6a1 ID:2006022108 > My Thunderbird is still stuck at version 1.5 (20060215), though it was at 20060213 for a while, but it is now stuck on the 15th.
The partial updates situation is much improved for the 20060222 round of nightlies, but there are still some major problems. I worked up this table to summarize: Firefox: Win Lin Mac Trunk N . N P N . Mozil1a1.8 N . N P N . Mozil1a1.8.0 N . N P N . Thunderbird: Win Lin Mac Trunk . . N P N P Mozil1a1.8 N P N P N P Mozil1a1.8.0 N . . . N . where N = Nightly, P = partial patch, and . = not created. Some points of note: - Firefox partials for Linux only - No partials for Thunderbird from 1.8.0 - I verified that the 3 Firefox Linux nightlies all update without error from 20060221 to 22. A fully linkified version is available at http://users.ox.ac.uk/~clar0239/update.html and I'll track 20060223 as well.
Well for now I am stuck on Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8) Gecko/20060215 Firefox/1.5 and I am not seeing any nightly updates. This might also be related to: https://bugzilla.mozilla.org/show_bug.cgi?id=327140 Which means that if we are missing some nightly builds (and I have gotten some full updates instead of partial) we might as well just get the latest instead of waiting for the intermediate builds to be populated or having to do a separate full ZIP download and reinstall.
(In reply to comment #21) > This might also be related to: > https://bugzilla.mozilla.org/show_bug.cgi?id=327140 This is bug 327140. Which other bug did you mean to reference?
Oops my bad. Got confused with the tabs... I meant: https://bugzilla.mozilla.org/show_bug.cgi?id=306864
(In reply to comment #23) > Oops my bad. Got confused with the tabs... > > I meant: https://bugzilla.mozilla.org/show_bug.cgi?id=306864 > This problem was raised in the nighties forums a number of time. The response was that as the AUS system was only designed for users to update their 'offical release' build once every couple of months and therefore it will remain the same, ie when updating nightly, it can't find the latest nightly, but instead it has to update to the next intermediate nightly until it gets to the latest one. The partial update only worked for me once since it was first reported, since then, it has not been working at all.
Could we please keep this bug focused on the issue it deals with. Random comments along the lines of "My <app> is still stuck at .." are not particularly helpful. The current situation: * Partial update generation is sort-of working, see comment #20. I got almost identical results when checking the 20060223 nightly * The regeneration of the partial updates between 20060213 and 20060221 nightlies (comment #17) did not actually occur * The build engineers are very busy The workaround is to update the old way - manually downloading builds from the FTP server.
Blocks: 328369
(In reply to comment #25) > The current situation: > * Partial update generation is sort-of working, see comment #20. I got almost > identical results when checking the 20060223 nightly Thanks for doing the grid; we should have a system that autogenerates this stuff, but that really helped me. :-) It looks like the problem is that for Mac and Win32, the $update_pushinfo variable is not set on certain machines (I was just checking the 1.8.0 Tinderboxen for the investigation). Turns out that there's a comment in the tinder-config.pl file that says "# Turning off updates for now. It's not yet firgured out on the 1.5.0.1 branch." Dunno if that's relevant anymore. I've kicked the 1.8.0 branch tinderbox to produce a nightly nowish. We'll see if we get a partial mar out of it. > * The regeneration of the partial updates between 20060213 and 20060221 > nightlies (comment #17) did not actually occur And likely won't occur, given... > * The build engineers are very busy
No longer blocks: 328369
Status: NEW → ASSIGNED
This problem seems to be back even with Coop's fix. The Full .MAR files for Thunderbird Mac 1.8.0 branch started showing up at 54 bytes again today. Firefox Mac builds are 1k in size. The 02/22 full MAR files for 02/22 have valid sizes. So this is new today. Escalating this bug to blocker status, as not having our testers properly migrating to the latest nightly builds really hurts our ability to leverage the community for testing for our releases.
Severity: major → blocker
I filed Bug 328383 to keep track of the new issue with the full MAR files now having a size of 54 bytes for Mac OS X on the 1.8.0 branch.
And I filed Bug #328369 for keeping track of the problem where at least on the 1.8 branch, the partial that we generated for Thunderbird Windows for going from 02/22 to 02/23 is failing with a CRC checksum error which usually means the partial patch was made against the wrong binaries.
Changing the summary again; I can't seem to find any partial updates for today's builds, 20060224...
Summary: Separate nightly mar generation system from release mar generation → Partial update generation failing
Really ? Seems to be doing the same half-job to me. eg Firefox Mozilla1.8 Linux: 0223 nightly and partial update to 0224 are in http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-02-23-05-mozilla1.8/ 0224 nightly and partial update to 0225 are in http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-02-24-05-mozilla1.8/ 0225 nightly in http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-02-25-04-mozilla1.8
Here's the update matrix for 20050227. Firefox Win Lin Mac Trunk N . N P N . Mozilla1.8 N . N P N . Mozilla1.8.0 N P N P N . Thunderbird Win Lin Mac Trunk N P . . N P Mozilla1.8 N P . . N P Mozilla1.8.0 N P . . N . N = Nightly, P = partial patch done, . = not created Notes: * Tb: all the Linux tinderbox are AWOL at the moment, no builds So preed's work to get Fx win32 mozilla1.8.0 updates have been successful. Here are some machine by machine notes from picking over the nightly logs: Fx Trunk Win - gaius - not pushing 3rd generation update information: |Gathering complete update info... |Got build ID 2006022704. | |Not pushing first-gen update info... | |Completed pushing update info... | |Update build completed. http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1141044360.21957.gz&fulltext=1 Fx Mozilla1.8 Win - not pushing 3rd generation update info (as above for gaius) http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla1.8/1141041360.30402.gz&fulltext=1 Fx Trunk Mac - atlantia - fails pushing 3rd gen update info: |Pushing third-gen update info... |ssh -i /Users/cltbld/.ssh/aus cltbld@aus-staging.mozilla.org mkdir -p /opt/aus2/build/0/Firefox/trunk/Darwin_ppc-gcc3/2006022704/en-US |ssh_askpass: exec(/usr/libexec/ssh-askpass): No such file or directory |Host key verification failed. |scp -i /Users/cltbld/.ssh/aus /builds/tinderbox/Fx-Trunk/Darwin_7.9.0_Depend/mozilla/dist/update/update.snippet.1 cltbld@aus-staging.mozilla.org:/opt/aus2/build/0/Firefox/trunk/Darwin_ppc-gcc3/2006022704/en-US/complete.txt |ssh_askpass: exec(/usr/libexec/ssh-askpass): No such file or directory |Host key verification failed. |lost connection http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1141044000.2079.gz&fulltext=1 Fx Mozilla1.8 Mac - atlantia - fails ssh as above Fx Mozilla1.8.0 Mac - fireball - ssh error |Pushing third-gen update info... |ssh -i /Users/cltbld/.ssh/aus cltbld@aus-staging.mozilla.org mkdir -p /opt/aus2/build/0/Firefox/1.5.0.1/Darwin_ppc-gcc3/2006022704/en-US |Warning: Identity file /Users/cltbld/.ssh/aus does not exist. |Permission denied (publickey). |scp -i /Users/cltbld/.ssh/aus /builds/tinderbox/Fx-Mozilla1.8.0/Darwin_7.9.0_Depend/mozilla/dist/update/update.snippet.1 cltbld@aus-staging.mozilla.org:/opt/aus2/build/0/Firefox/1.5.0.1/Darwin_ppc-gcc3/2006022704/en-US/complete.txt |Warning: Identity file /Users/cltbld/.ssh/aus does not exist. |Permission denied (publickey). |lost connection http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla1.8.0/1141043220.5917.gz&fulltext=1 Tb Mozilla1.8.0 Mac - fireball - ssh error as above Tb Linux - probably ok Trunk & Mozilla1.8 - was working 20060223 Mozilla1.8.0 - esx-test-vm1 - made partials to update to 0225 and 0226. Finally, morgamic said a few days ago (on irc) that the Thunderbird 1.8.0 builds were not getting having update.xml generated by aus2. So the app will find no updates even if there are partial mars generated.
(In reply to comment #32) > Here's the update matrix for 20050227. > Notes: > * Tb: all the Linux tinderbox are AWOL at the moment, no builds The machine that builds 1.8 and the trunk for Linux Thunderbird was hung. We just rebooted the machine and I have builds getting generated as I type this. In a fluke coincidence, the 1.8.0 linux build (esx-test-vm1) was no longer running a tinderbox script, I'm not sure why. But I've fixed that and it just generated a 1.8.0 build. > > Finally, morgamic said a few days ago (on irc) that the Thunderbird 1.8.0 > builds were not getting having update.xml generated by aus2. I finally got partial updates working Friday afternoon with some help from morgamic. So any build after that should be working fine for partial updates on the 1.8.0 branch for Thunderbird.
I'd also be inclined to have the updater signal some kind of failure if the update.manifest file is empty. It's not the updater's fault, but it would be useful to fail over to the full update if the partial is obviously malformed. CC'ing Darin for his thoughts on that.
I filed bug 329156 to have updater fail if it doesn't actually do anything.
*** Bug 330296 has been marked as a duplicate of this bug. ***
Update matrix for 2006-03-13 builds: Firefox Win Lin Mac Trunk N . N . N . Mozil1a1.8 N . N . N . Mozil1a1.8.0 N . N P N . Thunderbird Win Lin Mac Trunk N . N P N . Mozil1a1.8 N P N P N . Mozil1a1.8.0 N . N P N . N = Nightly, P = partial patch done, . = not created Linkified version: http://users.ox.ac.uk/~clar0239/update-20060313.html Comparing with comment #32, there seems to be several builds that no longer generate partial updates (I'm out of the loop at the moment). For example, Fx Mozilla1.8 Linux (prometheus): Pushing third-gen update info... ssh -i /home/cltbld/.ssh/aus cltbld@aus-staging.mozilla.org mkdir -p /opt/aus2/build/0/Firefox/1.5/Linux_x86-gcc3/2006031304/en-US ssh: connect to host aus-staging.mozilla.org port 22: Connection timed out scp -i /home/cltbld/.ssh/aus /builds/tinderbox/Fx-Mozilla1.8/Linux_2.4.21-27.0.4.ELsmp_Depend/mozilla/dist/update/update.snippet.1 cltbld@aus-staging.mozilla.org:/opt/aus2/build/0/Firefox/1.5/Linux_x86-gcc3/2006031304/en-US/complete.txt ssh: connect to host aus-staging.mozilla.org port 22: Connection timed out lost connection Source - http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla1.8/1142254020.22162.gz&fulltext=1
(In reply to comment #37) > Pushing third-gen update info... > ssh -i /home/cltbld/.ssh/aus cltbld@aus-staging.mozilla.org mkdir -p > /opt/aus2/build/0/Firefox/1.5/Linux_x86-gcc3/2006031304/en-US > ssh: connect to host aus-staging.mozilla.org port 22: Connection timed out > scp -i /home/cltbld/.ssh/aus In the past, we've seen similar timeouts on the first connection attempt from a given build machine to aus-staging via ssh when the host key is exchanged and the user must manually choose whether or not to accept the key. This could also be happening if the host key on aus-staging has changed for whatever reason. A bit of a PITA to verify whether this is the case on a machine-by-machine basis, but this probably needs to be done. :/
*** Bug 330449 has been marked as a duplicate of this bug. ***
Partial update works for Thunderbird trunk. Updated from 20060314 nightly to the 20060315 successfully. Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060315 Thunderbird/3.0a1 ID:2006031508
Update matrix for 2006-03-15 builds: Firefox | Thunderbird Win Lin Mac | Win Lin Mac Trunk N . N P N P | Trunk N P N P N P Mozil1a1.8 N . N P N P | Mozil1a1.8 N P N P N P Mozil1a1.8.0 N P N P N . | Mozil1a1.8.0 N P N P N . Key - N = Nightly, P = partial patch done, . = not created So much improved over 03-13 - just a temporary network fault contacting AUS or someone fixed something ? The remaining issues are: Fx Trunk Win - gaius - adjust tinderbox config to push update info Fx Mozilla1.8 Win - pacifica - ditto Fx Mozilla1.8.0 Mac - fireball - ssh error pushing updates snippet to aus, see comment #32 Tb Mozilla1.8.0 Mac - fireball - ditto
Urrgh, let's try that again. Sorry about the spam. 2006-03-15: Firefox | Thunderbird Win Lin Mac | Win Lin Mac Trunk N . N P N P | Trunk N P N P N P Mozil1a1.8 N . N P N P | Mozil1a1.8 N P N P N P Mozil1a1.8.0 N P N P N . | Mozil1a1.8.0 N P N P N .
(In reply to comment #41) > Fx Trunk Win - gaius - adjust tinderbox config to push update info > Fx Mozilla1.8 Win - pacifica - ditto > Fx Mozilla1.8.0 Mac - fireball - ssh error pushing updates snippet to aus, > see comment #32 > Tb Mozilla1.8.0 Mac - fireball - ditto I set $update_package = 1 in tinder-config.pl on pacifica, and added the aus ssh keys to fireball. Those updates should start working on the next pass. gaius already looks to be setup correctly, tinder-config.pl-wise. Not sure what's going wrong there.
(In reply to comment #43) > I set $update_package = 1 in tinder-config.pl on pacifica, and added the aus > ssh keys to fireball. Those updates should start working on the next pass. > > gaius already looks to be setup correctly, tinder-config.pl-wise. Not sure > what's going wrong there. Comment #26 suggests the variable $update_pushinfo must also be set.
Fx Trunk Win - gaius: 2006032104 -> 2006032106 partial patch uploaded to 2006-03-21-05-trunk/ Fx Mozilla1.8 Win - pacifica: still no partial patches today Fx Mozilla1.8.0 Mac - fireball: build stuck for 16+ hours; no nightly or partial patch in nightly/... today; but there is a nightly in tinderbox-builds/fireball-mozilla1.8.0/ Tb Mozilla1.8.0 Mac - fireball: no nightly or partial patch uploaded to nightly/... today; but there is a nightly in tinderbox-builds/fireball-mozilla1.8.0/ FWIW, although it looks like the residual issues are all build-related, I can take a look at AUS if there seem to be remaining issues there.
(In reply to comment #45) > Fx Trunk Win - gaius: > 2006032104 -> 2006032106 partial patch uploaded to 2006-03-21-05-trunk/ This is a bit odd because there isn't a build/respin from gaius at 2006032106 as far as I can see, and the update is 3.5MB in size. There is a build from pacifica (the Win32 tinderbox from before Cairo became default) in http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/experimental/non-cairo/2006-03-21-06-trunk/ See also 331250. This confirmed by the update.xml for 2006032104: <updates> <update type="minor" version="1.6a1" extensionVersion="1.6a1" buildID="2006032106"> <patch type="complete" URL="http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/experimental/non-cairo/2006-03-21-06-trunk/firefox-1.6a1.en-US.win32.complete.mar" hashFunction="SHA1" hashValue="ceb03bb39652fb1edd732d28235e1176d1d1347a" size="6525047"/> <patch type="partial" URL="http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-03-21-05-trunk/firefox-1.6a1.en-US.win32.partial.2006032104-2006032106.mar" hashFunction="SHA1" hashValue="1674c699886b7399692a98131f5711c9da040ed8" size="3700580"/> </update> </updates> (https://aus2.mozilla.org/update/1/Firefox/1.6a1/2006032104/WINNT_x86-msvc/en-US/nightly/update.xml) coop, could you have turned updates back on for Pacifica on the trunk instead of the 1.8branch ?
(In reply to comment #46) > coop, could you have turned updates back on for Pacifica on the trunk instead > of the 1.8branch ? Sigh, yes. /me needs to read the chart more carefully next time. I've made the necessary changes now.
sorry for being impatient, but when will this bug be fixed? It's very annoying and time consuming to update firefox nightly builds manually.
I haven't been able to update thunderbird since the 032104 nightly. Automatic update worked for a couple of days but now it stopped working again. Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060321 Thunderbird/3.0a1 ID:2006032104 Firefox update hasn't been working properly for more than a month.
(In reply to comment #49) At present, Thunderbird aren't even built. See bug 331433.
2006-03-26 update matrix: Firefox | Thunderbird Win Lin Mac | Win Lin Mac Trunk N P N . N . | Trunk . . N P N P Mozilla1.8 N . N . N . | Mozilla1.8 N P N P N P Mozilla1.8.0 N P N P N P | Mozilla1.8.0 N P N P N P N = Nightly, P = partial patch done, . = not created Notes: * Firefox Trunk Linux - prometheus: not attempting to push update snippet (REGRESSION, build log: Gathering complete update info... Got build ID 2006032604. Not pushing first-gen update info... Completed pushing update info... Update build completed.) * Firefox Trunk Mac - atlantia: not attempting to push update snippet (REGRESSION) * Firefox Mozilla1.8 Win - pacifica: not attempting to push update snippet * Firefox Mozilla1.8 Linux - prometheus: not attempting to push update snippet (REGRESSION) * Firefox Mozilla1.8 Mac - atlantia: not attempting to push update snippet (REGRESSION) * All these machines are fallout from attachment 215708 [details] [diff] [review] of bug 329573 ? (Auto update of tinderbox config, tools/tinderbox/build-seamonkey-util.pl, checked in 2006-03-20 19:50) * Thunderbird Trunk Windows - patrocles busted (bug 331433) * Fx&Tb Mozilla1.8.0 Mac - fireball: is producing updates after ssh key change (FIXED) Summary: If the 5 Firefox tinderbox have "$update_pushinfo=1" placed in their config then I think we should have a functional update generation system.
All yours, Rob!
Assignee: preed → rhelmer
Status: ASSIGNED → NEW
This should be fixed for any build after 2 PM PDT. The tinderboxes reconfigured for this were: prometheus: Fx-Trunk & Fx-Mozilla1.8 atlantia: Fx-Trunk & Fx-Mozilla1.8 pacifica: Fx-Mozilla1.8
Thanks Rob - definitely a step in the right direction but still a few glitches to iron out. Firefox Thunderbird Win Lin Mac | Win Lin Mac Trunk N P N . N . | Trunk N P N P N P Mozilla1.8 N . N . N . | Mozilla1.8 N P N P N P Mozilla1.8.0 N . N P . . | Mozilla1.8.0 N P N P N P (http://users.ox.ac.uk/~clar0239/update-20060408.html) The Firefox Mozilla1.8 boxes are now pushing update information, but use an app version of 1.5 instead of 2.0a1. Eg Mozilla1.8 Win32 log: Gathering complete update info... Got build ID 2006040803. Not pushing first-gen update info... Pushing third-gen update info... ssh -i /home/cltbld/.ssh/aus cltbld@aus-staging.mozilla.org mkdir -p /opt/aus2/build/0/Firefox/1.5/WINNT_x86-msvc/2006040803/en-US scp -i /home/cltbld/.ssh/aus /cygdrive/c/builds/tinderbox/Fx-Mozilla1.8/WINNT_5.2_Depend/mozilla/dist/update/update.snippet.1 cltbld@aus-staging.mozilla.org:/opt/aus2/build/0/Firefox/1.5/WINNT_x86-msvc/2006040803/en-US/complete.txt Completed pushing update info... Update build completed. (http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla1.8/1144492920.18981.gz&fulltext=1) How does the tinderbox update script determine the app version ? I don't know what happens in the update system when the version is wrong like this, but the most recent "1.5" nightlies have already been culled from ftp.mozilla.org so the update generator may just bail at that point. Certainly it's no good when the app is going to be requesting updates with a version of 2.0a1. The file mozilla/browser/config/version.txt seems to control the version for Firefox. For the trunk builds, on Linux I should have said the tinderbox was argo rather than prometheus. Sorry! Looks like the Mac pushed update info correctly, but there wasn't a partial update. Probably need to wait until tomorrow so that there are two recent builds to generate the update from. Windows was already working. Finally, for the Mozilla1.8.0 builds, Windows (moz180-win32-tbox) needs the $update_pushinfo treatment (this used to be present but reverted at some point). Linux is ok. The Mac tinderbox (fireball) is missing, but pushed update info on 2005-04-05 (the nightly before it dropped from the tinderbox status page).
20060409 nightlies (http://users.ox.ac.uk/~clar0239/update-20060409.html) Firefox | Thunderbird Win Lin Mac | Win Lin Mac Trunk N P N . N P | Trunk N P N P N P Mozilla1.8 N P* N P* N P* | Mozilla1.8 N P N P N P Mozilla1.8.0 N . N P* . . | Mozilla1.8.0 N P N P N P N = Nightly created, P = Partial patch from previous nightly exists, P* = Partial exists but update not available, . = Nightly or partial patch not created Things of note: * Firefox Trunk Mac created a partial update, as did all the Firefox Mozilla1.8 builds. * The Mozilla1.8 updates were not available using Help > Check for Updates because of the version error when pushing the update information to AUS. The update _does_ apply properly if you override the url (tested windows only). * I had a look at the Mozilla1.8.0 build logs. The Linux & Mac machines are using a version 1.5.0.1, where 1.5.0.2 is the current version (at least for the moment). Can't see what the Windows machine is doing ($update_pushinfo not 1) but presumably similar. It seems that the path that the update snippets are published to is set [1,2] by the variable update_version in the tinderbox config [3]. Presumably this is hardwired to 1.5 on the 1.8 branch, but a value of "trunk" is used for the ... wait for it ... trunk. I wonder if something like Mozilla1.8.0 & Mozilla1.8 can be used on the branches, so that the system doesn't need tweaking each time the app version changes. morgamic, could the AUS side of things handle that ? Alternatively, the app version can be read from {browser,mail}/config/version.txt [1] http://lxr.mozilla.org/mozilla/source/tools/tinderbox/post-mozilla-rel.pl#505 [2] http://lxr.mozilla.org/mozilla/source/tools/tinderbox/post-mozilla-rel.pl#628 [3] eg, http://lxr.mozilla.org/mozilla/source/tools/tinderbox/tinder-defaults.pl#218
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060409 Firefox/3.0a1 - Build ID: 2006040917 pacifica-trunk is not updating and can't find any updates when processed to AUS.
Sorry about the spam. (In reply to comment #56) > Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060409 > Firefox/3.0a1 - Build ID: 2006040917 > > pacifica-trunk is not updating and can't find any updates when processed to > AUS. > 1. linux trunk version 3.0a1 .mar is generated in 20060410 folder, but the 1.6a1 is found with 3.0a1 in latest-trunk 2. windows trunk .mar's are not generated in 20060410 folder, but generated in latest-trunk folder (1.6a1 and 3.0a1) 3. mac trunk .mar's: same as linux .mar's
$push_updateinfo is now: * on for pacifica * on for argo * off for prometheus Looks like we have a special Fx-Trunk-Cairo build on argo; I wonder if that can go away now that we have cairo on the trunk.
Ok, pacifica is non-cairo trunk, so I changed $update_pushinfo: * off for pacifica/Fx-Trunk * on for gaius/Fx-Trunk-Cairo We're getting a release ready on moz180-win32-tbox/Fx-Mozilla1.8.0 so I am going to hold off on that for now, and turn it on right after 1.5.0.2 so we'll get pre-1.5.0.3 nightly updates.
(In reply to comment #59) > Ok, pacifica is non-cairo trunk, so I changed $update_pushinfo: > > * off for pacifica/Fx-Trunk > * on for gaius/Fx-Trunk-Cairo > > We're getting a release ready on moz180-win32-tbox/Fx-Mozilla1.8.0 so I am > going to hold off on that for now, and turn it on right after 1.5.0.2 so we'll > get pre-1.5.0.3 nightly updates. > gaius-trunk is not checking for updates at all. Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060410 Firefox/3.0a1 - Build ID: 2006041003
2006-04-11 update matrix http://users.ox.ac.uk/~clar0239/update-20060411.html Firefox Win Lin Mac Trunk N P . . N P Mozilla1.8 N P* N P* N P* Mozilla1.8.0 . . N P . . Thunderbird Win Lin Mac Trunk . . N . N . Mozilla1.8 . . N P N P Mozilla1.8.0 N . N P N P N= Nightly created, P= Partial patch from previous nightly exists, P*= partial exists but update not available, .= not created * Firefox Trunk Linux - argo: needs to push update info, prometheus had this set after I put the admins wrong :( * Firefox Trunk Mac - atlantia: created an update! * Firefox Mozilla1.8 Win Linux & Mac: now pushing update snippets but using version of 1.5 instead of 2.0a1, all produced partials * Firefox Mozilla1.8.0 Windows - moz180-win32-tbox: not pushing update info * Firefox Mozilla1.8.0 Mac - bm-xserve01 (Universal): tinderbox is broken in configure, OR - fireball: not producing builds anymore and has dropped from status page (was pushing update info on 2005-04-05 though)
Charli, thanks for your enthusiasm but that update matrix is not finished yet - there are still nightlies to come out. Please don't jump the gun. Also, I don't understand your previous comments. Some of them seem to be referring to complete.mar files (latest-trunk in #57) which are not relevant to the partials; pacifica-trunk shouldn't be creating updates because gaius is the box that can produce windows Cairo builds; and yesterdays gaius nightly was 2006041010, so 2006041003 is probably an hourly (which doesn't get updates).
1.8.0 branch here. Linux machine. I have nice nightly updates for thunderbird, but firefox hasn't find any nightly update for weeks. Table in comment #61 seems to indicate that my firefox should be updating nightly, just fine. Not the case :-( And yes, I'm "suscribed" to the nightly channel.
(In reply to comment #63) > 1.8.0 branch here. Linux machine. > > I have nice nightly updates for thunderbird, but firefox hasn't find any > nightly update for weeks. Table in comment #61 seems to indicate that my > firefox should be updating nightly, just fine. Not the case :-( > > And yes, I'm "suscribed" to the nightly channel. > 1. That table was from earlier in the day. Check http://users.ox.ac.uk/~clar0239/update-20060411.html for an updated update matrix. 2. Like you said, you are on the nightly channel, but somehow it isn't updating. This is also my case with the different tinderboxes that I have on my machine. 3. What tinderbox are you using?
(In reply to comment #63) This is a result of some versions set in the tinderbox configuration. After speaking to Mike Morgan, AUS suprermo, a short term fix would be to change: 1, $update_version from 1.5 to 2 on the three Firefox Mozilla1.8 tinderbox 2, $update_version from 1.5.0.1 to 1.5.0.2 on the Firefox Mozilla1.8.0 tinderbox 3, $update_appv & $update_extv from 1.6a1 to 3.0a1 on the Firefox Trunk tinderbox 4, $update_appv & $update_extv from 1.5 to 2.0a1 on the Firefox Mozilla1.8 tinderbox 5, $update_appv & $update_extv from 1.5.0.1 to 1.5.0.2 on the Firefox Mozilla1.8.0 tinderbox Items 1 & 2 make the tinderbox publish the Firefox build snippets to the analagous place to where Thunderbird does (and AUS is setup for). This makes update notifications occur at all. Both steps are sub-optimal as it creates more AUS entities than there are branches, but never mind. Items 3-5 make the update notifications have the right version on them. They would ideally be kept in sync with app version changes. I'm going to propose a better long-term fix for this and the entities issue elsewhere.
Thnaks. At last the Trunk builds have started updating. For me, it started from today. Just updated from : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060408 Firefox/3.0a1 to : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060411 Firefox/3.0a1 (Minefield)
(In reply to comment #66) > Thnaks. At last the Trunk builds have started updating. For me, it started from > today. > > Just updated > from : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060408 > Firefox/3.0a1 > > to : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060411 > Firefox/3.0a1 (Minefield) > What update channel are you using? (To check, go to about:config and check app.update.channel)
(In reply to comment #67) > What update channel are you using? (To check, go to about:config and check > app.update.channel) > Default value ---- nightly
(In reply to comment #68) > (In reply to comment #67) > > > What update channel are you using? (To check, go to about:config and check > > app.update.channel) > > > > Default value ---- nightly > My fx trunks-gaius-trunk and pacifica-trunk do not update on app.update.channel: nightly
(In reply to comment #65) I just checked in fixes to the tinderbox configs for 1,3,4. 2,5 are being used for release right now, I'll get these going once we get 1.5.0.2 out the door.
In reply to comment #64 (kbblogger@verizon.net): > 3. What tinderbox are you using? How can I know?. All "about:config" "app.update.*" values are default. My "app.update.channel" is "nightly" (default).
Update finally works for me!!!!!! However it still identified the update as 'Minefield 1.6a1', it should be 'Minefield 3.0a1' Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060412 Firefox/3.0a1 ID:2006041205 [cairo]
Think we've got the 1.6a1 problem nailed down; it was coming from the tinderbox config, that's now been fixed, so trunk should be all sorted out. I'll work on tracking the remaining mozilla1.8 and mozilla1.8.0 problems down.
Woohoo, my Mac 20060413 branch nightly just updated itself to 20060414!
(In reply to comment #74) > Woohoo, my Mac 20060413 branch nightly just updated itself to 20060414! > And my Win32 20060413 branch nightly. Lovely ;-)
Confirming that on the 1.8 branch that Windows & Mac are updating from 20060413 to -0414 nightlies, with offered version of 2.0a1. Can't test Linux at the moment because prometheus has gone walkabout. Trunk are now offering 3.0a1, and once bug 332088 is fixed should work reliably on all platforms. So that only leaves the 1.8.0 branch to do. The tinderbox are Win = moz180-win32-tbox : Fx-Mozilla1.8.0 Lin = moz180-linux-tbox : Fx-Mozilla1.8.0 Mac = fireball : Fx-Mozilla1.8.0 The changes are: 1, Win needs $update_pushinfo=1 2, All three tinderbox need $update_version="1.5.0.2"; $update_appv="1.5.0.2"; $update_extv="1.5.0.2" At some point Firefox & Thunderbird will be re-versioned 1.5.0.3, and _appv and _extv will need bumping again. I've filed bug 334011 for the update system to automaticly follow the app's version.
Nick, my firefox 1.8.0 branch (LINUX) doesn't update nightly yet. I'm running 2006040418 build. All relevant app.update.* configs are default. Doing a "Check for updates" doesn't find any.
(In reply to comment #77) > Nick, my firefox 1.8.0 branch (LINUX) doesn't update nightly yet. I'm running > 2006040418 build. All relevant app.update.* configs are default. Doing a "Check > for updates" doesn't find any. Your settings are fine, there are no updates because the 1.8.0 tinderbox need reconfiguring (comment #76). When the changes are made then the subsequent nightly should be able to update - eg: IF the changes are made during the daytime on 20 April (PDT), then the 20060421 nightly would (hopefully) be updated to 20060422. It's extremely unlikely that a build from April 4th will ever get any updates, you'll need to download a later build yourself.
Thanks, Nick. Just updated manually to 2006041504. Hope "somebody" yells when the linux 1.8.0 tinderbox be patched.
I was trying to update Thunderbird trunk nightly from 041806 to the 0419 nightly. After it downloaded 200kb partial update, the integrity of the update could not be verified, so it downloaded another 7.6mb for the complete package. Upon restarting, nothing changed, the old nightly still pops up and the software update window appears, froze on 'connecting to the update server. The exact problem we had in Firefox trunk not too long ago, took a long time to fix. Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060418 Thunderbird/3.0a1 ID:2006041806
(In reply to comment #80) WFM: partial update was 322KB, integrity ok and build updated. Something interrupted download perhaps, or AUS offered the update before the partial finished mirroring (partial was created on ftp master at 11:22 PDT). Either way not this bug.
(In reply to comment #81) > (In reply to comment #80) > > WFM: partial update was 322KB, integrity ok and build updated. Something > interrupted download perhaps, or AUS offered the update before the partial > finished mirroring (partial was created on ftp master at 11:22 PDT). Either way > not this bug. > Thanks Nick, it WFM now, must have gotten the update too early. Sorry about the spam.
There seems to be a problem with partial patches today. Not all the nightlies are in yet but: Firefox | Thunderbird Win Lin Mac | Win Lin Mac Trunk N . N . N . | Trunk . . N . N . Mozilla1.8 N P N . N . | Mozilla1.8 . . N . N . Mozilla1.8.0 - - - - - - | Mozilla1.8.0 N . N P N . (http://users.ox.ac.uk/~clar0239/update-20060421.html) Only two partials have been created, at 0353 and 0446 PDT, and the rest of the nightlies have been out long enough by now. As far as I can tell everything should be functioning as normal. (There is a problem with most of the mirrors in ftp.mozilla.org not updating due to a full disk on the master rsync, so be sure to look at stage).
I have: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.2) Gecko/20060422 Firefox/1.5.0.2 - Build ID: 2006042207 and the nightly for 20060423 is available for full download but a check for updates does not find anything. Connectivity seems okay as Thunderbird (now at Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.2) Gecko/20060423 Thunderbird/1.5.0.2 - Build ID: 2006042312) prompted me for three updates first thing this morning.
Whiteboard: All branches updating except Firefox 1.5.0.x, see comment #76
20060424 matrix (excluding the slow building Tb Mozilla1.8 Win, which worked yesterday): Firefox | Thunderbird Win Lin Mac | Win Lin Mac Trunk N P N P N P | Trunk N P N . N . Mozilla1.8 N P N P N P | Mozilla1.8 - - N . N . Mozilla1.8.0 N . N P* N P* | Mozilla1.8.0 N P N P N P N = Nightly created; P = Partial patch from previous nightly exists P* = partial exists but update not available; . = not created (http://users.ox.ac.uk/~clar0239/update-20060424.html) Thunderbird has regressed a bit - the Linux & Mac builds are not pushing update information for Trunk and Mozilla1.8. The machine:buildname combo's are crazyhorse:Tb-Mozilla1.8, crazyhorse:thunderbird-trunk triton:Tb-Mozilla1.8, triton:Tb-Trunk No change with Firefox on Mozilla1.8.0, waiting for tinderbox fixes (comment #76) (In reply to comment #83) A bunch of partial patches hit the FTP server between 09:40 and 10:00 PDT, the previous most recent being 0446. Some build engineer magic or did the partial update generator sort itself out ?
I seem to remember Paul telling me that tinder-config defaults for each of the build processes were going into CVS. Could this move have caused us to stop pushing updates for 1.8 thunderbird on crazyhorse and Triton?
Scott, I spoke to Chris Cooper about this on IRC. He said he'd do the move of configs to CVS and fix up the update settings. Not sure why this problem started though.
I added the missing configs for triton and crazyhorse to CVS this morning, making sure that they all had the necessary $update_pushinfo bits set in the process.
The trunk builds for Windows are updating. Did not test Mozilla1.8 update yet. Also--trunk is offering version of 3.0a1.
(In reply to comment #89) Mozilla1.8 is now updating--offering version of 2.0a1. The updates are WFM now.
(In reply to comment #88) > I added the missing configs for triton and crazyhorse to CVS this morning, > making sure that they all had the necessary $update_pushinfo bits set in the > process. (http://users.ox.ac.uk/~clar0239/update-20060428.html) Partial updates are now being generated for Thunderbird Mac & Linux, on Trunk & Mozilla1.8 branch. There is a problem with the the AUS data though: https://aus2.mozilla.org/update/1/Thunderbird/2.0a1/2006042704/Linux_x86-gcc3/en-US/nightly/update.xml https://aus2.mozilla.org/update/1/Thunderbird/2.0a1/2006042703/Darwin_ppc-gcc3/en-US/nightly/update.xml have extension version of "1.5" instead of "2.0a1". Probably $update_extv is incorrect in the tinder-config for triton and crazyhorse's branch builds. Apart from this, 20060427 --> 20060428 should be working fine.
(In reply to comment #91) >https://aus2.mozilla.org/update/1/Thunderbird/2.0a1/2006042704/Linux_x86-gcc3/en-US/nightly/update.xml > https://aus2.mozilla.org/update/1/Thunderbird/2.0a1/2006042703/Darwin_ppc-gcc3/en-US/nightly/update.xml > have extension version of "1.5" instead of "2.0a1". Probably $update_extv is > incorrect in the tinder-config for triton and crazyhorse's branch builds. > > Apart from this, 20060427 --> 20060428 should be working fine. I noticed that when I was looking at the machines the other day to figure out why partials weren't getting pushed and then forgot about it! yes you are indeed correct update_extv is set incorrectly for triton and crazyhorse 1.8.1 builds. Rob, do you mind tweaking those two values to 2.0a1?
(In reply to comment #92) > Rob, do you mind tweaking those two values to 2.0a1? Rob's changes worked - the respun builds produced this update info: https://aus2.mozilla.org/update/1/Thunderbird/2.0a1/2006050403/Darwin_ppc-gcc3/en-US/nightly/update.xml <updates> <update type="minor" version="2.0a1" extensionVersion="2.0a1" buildID="2006050409"> .... https://aus2.mozilla.org/update/1/Thunderbird/2.0a1/2006050404/Linux_x86-gcc3/en-US/nightly/update.xml <updates> <update type="minor" version="2.0a1" extensionVersion="2.0a1" buildID="2006050411"> ....
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9a1) Gecko/20060508 Minefield/3.0a1 - Build ID: 2006050805 This build stopped updating the next day for some reason. Tinderbox broken again?
(In reply to comment #94) > Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9a1) Gecko/20060508 > Minefield/3.0a1 - Build ID: 2006050805 > > This build stopped updating the next day for some reason. Tinderbox broken > again? > The Tree is closed. http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox-Cairo Apparently the new session restore feature interacting badly with the tinderbox test run after building. http://forums.mozillazine.org/viewtopic.php?t=414539
Blocks: 335894
Charli, your bug 335894 doesn't block this one because they are unrelated. Just because bugzilla has lots of knobs and buttons doesn't mean you should push them, at least until you know enough to make better decisions.
(In reply to comment #96) > Charli, your bug 335894 doesn't block this one because they are unrelated. Just > because bugzilla has lots of knobs and buttons doesn't mean you should push > them, at least until you know enough to make better decisions. > Well, if there are no updates to update fx with, then yes, nobody can reproduce it.
Exec. summary: ============== Nightly updates are almost completely functional, so I'm taking the dramatic step of resolving this bug FIXED and farming the small remaining issues out to other bugs. Note to nightly users: The default channel for new installs is not "nightly" for the following builds: Firefox & Thunderbird 1.5.0.x (uses "release") Firefox 2.0a2 (uses "beta") This is likely to change in the future. I suggest using the Update Channel Changer extension to switch to the nightly channel, otherwise you won't get any updates at all. Just install it, restart Firefox/Thunderbird, go Help > Check for Updates and use the Change button. http://users.blueprintit.co.uk/~dave/web/firefox/updatechannel/index.html Details: ======== Update matrix for 20060514 (http://users.ox.ac.uk/~clar0239/update-20060514.html) Firefox Win Lin Mac Trunk N P I V N P I V N P I _ Mozilla1.8 N P I V N P I V N P I _ Mozilla1.8.0 N P I V N P I V N P I _ Thunderbird Win Lin Mac Trunk N P I V N P I V N P I _ Mozilla1.8 N P I V N P I V N P I _ Mozilla1.8.0 N P _ _ N P I V N P _ _ Key: * N = Nightly created * P = Partial patch from previous nightly exists * I = update.xml - update information for client app available * V = verified update applies successfully * _ = not created/available/verifiable Notes: 1, Thunderbird Mozilla1.8 Win - update_extv is "1.5" instead of "2.0a1", see bug 338019 2, Thunderbird Mozilla1.8.0 Win&Mac - using "1.5.0.2" instead of "1.5.0.4" for update_{version,appv,extv} so no updates, see bug 338017 3, * * * - Some machines using "MD5" hash instead of the "SHA1" used for releases, see bug 338038. 4, I can't test Mac, hence the _ instead of V's there. The update info eyeballed ok.
No longer blocks: 335894
Status: NEW → RESOLVED
Closed: 19 years ago
Resolution: --- → FIXED
Whiteboard: All branches updating except Firefox 1.5.0.x, see comment #76
This trunk nightly (2006051706) not checking for updates to the next day, which has already been generated on the ftp server: ftp://ftp.osuosl.org/pub/mozilla.org/firefox/nightly/2006-05-18-07-trunk --- Otherwise, everything is fine.
Branch also seems stuck on Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) Gecko/20060517 Firefox/1.5.0.4 ID:2006051706
I would question whether continuing to use this bug to report issues is actually useful to the build guys, or maybe this should be left alone and any persistent issues with partial updates be filed as new bugs.
This bug is over, finished, done with, and completely kaput. Please don't treat it as an ongoing catch all for nightly updates issues. As Mossop just said, it's much more useful to have individual bugs targeted to particular issues. The lack of updates since the 20060517 builds is filed as bug 338489. Do not spam that one with "me too" or "<branch> doesn't update for me" reports - they don't help towards getting something fixed, and just add a lot of noise to the technical discussion.
QA Contact: nobody → systems
You need to log in before you can comment on or make changes to this bug.