Closed Bug 302348 Opened 19 years ago Closed 18 years ago

Update fails partial update and then hangs when update not in FTP mirrors yet

Categories

(Toolkit :: Application Update, defect)

Product:
Toolkit
Component:
Application Update
Version:
1.8 Branch
Type:
defect
Priority:
Not set
Severity:
major

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla1.8.1beta1

People

(Reporter: bugzilla-mozilla-20000923, Assigned: Waldo)

References

Details

(Keywords: fixed1.8.1, Whiteboard: [swag: 1.5d])

Attachments

(4 files)

Patch for all issues described
16.75 KB, patch
darin.moz
: superreview+
mtschrep
: approval1.8.1+
Details | Diff | Splinter Review
Update tests used to test patch correctness
5.62 KB, application/zip
Details
this did the download after failing, but then Hung Thunderbird
25.30 KB, image/png
Details
Hang report after update failure
38.78 KB, application/octet-stream
Details 
I just got this while trying to update from 20050726 to 2005072707. It first
contacts the server, then complains it could not verify the partial update
(which doesn't seem to be mentioned in the update file anyway!) then hangs showing:

Paused downloading Deer Park 1.0+            _Details_
[                                                    ]
 o Connecting to the update server...       [ Resume ]

The o icon is stopped, [ Resume ] is disable, and I can only click Close.

===== updates.xml =====

<updates xmlns="http://www.mozilla.org/2005/app-update">
  <update type="minor" name="Deer Park 1.0+" version="1.0+"
extensionVersion="1.0+" detailsURL="undefined" licenseURL="undefined"
serviceURL="https://aus-staging.mozilla.org:8711/update2/0/Firefox/1.0%2B/2005072606/WINNT_x86-msvc/en-US/update.xml"
installDate="1122476382259" statusText="Failed (Unknown Reason)"
isCompleteUpdate="false" licenseAccepted="false" foregroundDownload="true">
  <patch type="complete"
URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2005-07-27-07-trunk/firefox-1.0+.en-US.win32.mar"
hashFunction="MD5" hashValue="c7828c4106f08094ae241db84d8fc130" size="6163473"
selected="true" state=""/>
  </update>
</updates>

===== active-update.xml =====

<updates xmlns="http://www.mozilla.org/2005/app-update">
  <update type="minor" name="Deer Park 1.0+" version="1.0+"
extensionVersion="1.0+" detailsURL="undefined" licenseURL="undefined"
serviceURL="https://aus-staging.mozilla.org:8711/update2/0/Firefox/1.0%2B/2005072606/WINNT_x86-msvc/en-US/update.xml"
installDate="1122476382259" statusText="Failed (Unknown Reason)"
isCompleteUpdate="true" licenseAccepted="false" foregroundDownload="true">
    <patch type="complete"
URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2005-07-27-07-trunk/firefox-1.0+.en-US.win32.mar"
hashFunction="MD5" hashValue="c7828c4106f08094ae241db84d8fc130" size="6163473"
selected="true" state=""/>
  </update>
</updates>
I think we definitely want to behave more sanely when this condition occurs. 
That said, we are working to ensure that AUS only issues update manifests when
files are known to exist on the mirror network.
Blocks: 290390
Severity: normal → major
OS: Windows Server 2003 → All
Hardware: PC → All
I think this has been fixed... (yesterday, as part of 300089) you might want to
re-verify. 
I will attempt to do so, but it is a bit hard to check without some luck. :)
Trying to go from 20050810 to 20050811, I got this.
(In reply to comment #2)
> I think this has been fixed... (yesterday, as part of 300089) you might want to
> re-verify. 

Sorry, it is not fixed. I just got the same problem updating from 2005081106 to
the not-quite-ready 2005081208.

The bug has changed a bit now, and it no longer gets stuck /entirely/. To explain:

- When I try to update, it says there is an update. I ask to update.
- Contacts the server, and says it failed to verify the partial update (BUG).
- Does nothing else except have the undetermined progress bar, and the
"Connecting to" message (BUG).

Hiding and re-opening the update window DOES now work, and it offers the
download again - which just does the same as outlined above. So while the entire
system does not get stuck any more, the individual window is still broken in its
reporting and state.
Hello,
I'm new in this bugzilla.  Tried searching most relevant topic and am posting
relavant problem.  Hopefully it gets looked into...  
I've MR Tech Disable XPI Install delay extension.  When I start the updater, it
hangs in there.  If I uninstall that, it hangs somewhere else sometimes.  I've
1.0.6 Firefox... 
Please contact.  
Hi,
I'm facing a similar problem.  It starts to update and then after completing
almost the full scanning proess; just stops there and never says whether it
found updates or not.  Please inform me.  
I'm new to the bugzilla.  Tried to report a few times; but never succeded. 
Regards,
*** Bug 311873 has been marked as a duplicate of this bug. ***
Bug 311873 provides a reliable test case for this bug.
The testcase from bug 311873 isn't valid anymore, as those resourceful server
admins respun the 20051009 nightly, creating valid update information for
20051008 nightly.
The update service should be able to handle the following scenarios gracefully:

* non-existent update files on server (partial/full);
* 0-byte update files on server (partial/full), as seen recently with bug 217303;
* discrepancies between update data stored on the AUS server and the update files on the ftp site.

I'm just not sure where AUS testcases are being stored :/
*** Bug 317303 has been marked as a duplicate of this bug. ***
*** Bug 317618 has been marked as a duplicate of this bug. ***
Depends on: 317623
This issue still appears sometimes on my computer!

So, I must remove files active-update.xml and update.xml that my update-system works again.... that's very time-consuming and annonying :(

But what should users do who don't know this? They think, that firefox or thunderbird are bad!!! Perhaps they'll use other programs...

So please fix that or add a cancel button. If a user clicks on this button, it should DELETE files active-update.xml and update.xml and close that update-dialog...
Setting blocking? for consideration for Firefox 2, as this bug covers important error handling for software update. So far it hasn't hurt end-users doing release updates, but the nightly channel has had problems.
Flags: blocking-firefox2?
Yeah, this needs some love, though if we're able to transition to bouncer 2 I think we avoid this situation completely (still a bug, but not nearly as critical)
Flags: blocking-firefox2? → blocking-firefox2+
*** Bug 323982 has been marked as a duplicate of this bug. ***
Mike, are we going to be at bouncer 2 soonish, or do we need to get resources on this?
Was planning on working on this after addons v2 public was released.  The plan was to talk about it w/ preed and sysadmins during the quarterly -- so sometime in the next three weeks is the ETA.
bouncer2 will help here, but that's no reason to not look at hardening up the update service.
Keywords: helpwanted
Target Milestone: --- → Firefox 2 beta1
Assignee: nobody → jwalden+bmo
Whiteboard: [swag: 1.5d]
Depends on: 342321
What's the deal with this bug now that the mirrors have changed?
- Catch files listed with 0 size in the updates XML as invalid (without trying to download the file) by causing UpdatePatch and Update to throw when a 0-sized patch is encountered during construction or by throwing if an update contains no (valid) patches
- Catch 0-sized downloaded MAR files as invalid
- Handle MARs which 404
- Handle updates containing no patches, one partial patch, one complete patch, one of each, and other configurations where zero or more of the patches are either a priori invalid or invalid after verification fails
- Properly remove the downloading page as a listener in all cases (can be seen if you try an update, get to the end of it, and then try to update again and get "no Components" object assertions)
- Change STATE_NONE from null to "null", because usually if we're comparing states we're reading from a file and thus getting a string (and "null" != null)
- Change _addUpdate to prevent throwing when this._updates contains null or undefined (occurs when I switch through my various testcases and repeatedly attempt updates -- I never tracked down concise steps to reproduce, but there's no reason the change wouldn't always be safe)
- Guard against updates which contain only one (invalid) partial patch and no complete patch
- Properly handle both partial and complete patches failing by using nsIUpdatePrompt.showUpdateError, and make that fallback actually work by adding a missing QueryInterface (the created exception was being swallowed)
- Get rid of the funky #1= and #1# stuff that you only learn about if you know about #developers, ask what that stuff is when brendan's reading IRC, find the one document on the entire Interweb that documents it, and are willing and able to read through a proposed ECMA-262 specification to understand what the heck sharp variables are

By the way, this is the most hideous hack ever.  I'll file a followup bug after this is reviewed and hopefully fix it properly (see the XXX in updates.js) after b1.
Attachment #226859 - Flags: review?(mconnor)
Darin - can we get your eyes on this?
Blocks: 333908
moving to b2, darin should really review this
Target Milestone: Firefox 2 beta1 → Firefox 2 beta2
Hmm - I'd really like to nail down any client-side AUS updates prior to b1 so we can test in b1->b2.  Is that possible?
Comment on attachment 226859 [details] [diff] [review]
Patch for all issues described

r=darin
Attachment #226859 - Flags: superreview+
Comment on attachment 226859 [details] [diff] [review]
Patch for all issues described

This only needs one review to go in on trunk, and since I've gotten that review I'm landing this on trunk momentarily.
Attachment #226859 - Flags: review?(mconnor)
Patch landed on trunk; this needs testing to make sure it didn't break anything.  I'm fairly confident in it given the amount of time I spent diagnosing and solving the problems (and I have a nice folder of testcases which say everything works right), but I wouldn't be surprised to find that something broke, particularly since I couldn't really test updates which *don't* fail.
Status: NEW → ASSIGNED
pushing out non-critical-path bugs to b2
As noted, we need this in b1 to be able to test going b1->b2
Target Milestone: Firefox 2 beta2 → Firefox 2 beta1
Davel - can you help test this out since you did some SU testing for 1.5?

Jeff - should you be landing on Branch/asking for approval?
These are the tests I used to verify patch correctness in the situations whose behavior I wanted to correct (or keep correct).  For the most part the file names should indicate what's supposed to happen when app.update.url.override is set to each of those files, but I wasn't being ridiculously meticulous about keeping track of exactly which error conditions each XML file tested, and there's probably some redundancy among the tests in the tested scenarios.

To use these tests, you'll either need to copy them to http://localhost/updates/ or rewrite the URLs in the XML files that refer to files beneath that location.  Note also that the XML files are downloaded using nsIXMLHttpRequest, so I'm fairly certain they can't be served from a non-HTTP location such as from file:.

(In reply to comment #31)
> Jeff - should you be landing on Branch/asking for approval?

I was waiting for a nightly in which I could test basic update functionality so I could verify that the current update process still works (since I hadn't been able to test update functionality with real, working updates).  I just tested with the new update info preed generated within the last hour or so.  The Mac PPC build from yesterday will successfully download a partial update when served both a partial and a complete update, and when I downloaded the manifest and hacked away the partial patch it also would successfully install the complete patch, so now that I've been able to test basic functionality I think this is good for branch approval.
Comment on attachment 226859 [details] [diff] [review]
Patch for all issues described

This patch makes client-side update functionality substantially more robust in the face of server-side error.  The patch has been tested to work in the error cases in the posted attachment, and it also works with a 20060628->20060629 update from a manifest with both partial and complete updates using the partial and from a manifest with only a complete update, so I think it should be safe to take on branch.  That doesn't mean I don't want testing of it, but I think it's fairly safe.
Attachment #226859 - Flags: approval1.8.1?
Attachment #226859 - Flags: approval1.8.1? → approval1.8.1+
Patch committed on branch.
Status: ASSIGNED → RESOLVED
Closed: 18 years ago
Keywords: helpwanted → fixed1.8.1
Resolution: --- → FIXED
(In reply to comment #22)
> I'll file a followup bug after this is reviewed and hopefully fix it properly
> (see the XXX in updates.js) after b1.

Filed bug 343260.
Version: Trunk → 2.0 Branch
This patch should also be checked in for old Firefox 1.5.0.x  to ensure that ALL user can get their security updates without any trouble.

I know lots of people which update-system is hanging in that curious, broken "paused mode"... and they don't have the expericence how to get it working again.
(In reply to comment #36)
> This patch should also be checked in for old Firefox 1.5.0.x  to ensure that
> ALL user can get their security updates without any trouble.

1.5.0.x patches are security patches or patches which produce marked improvement in behavior with little risk; such patches also tend to be fairly small.  This patch is none of those.  It touches many different places in the update service with a lot of potential for regressions, and tracking all the possible effects of the changes is non-trivial (and frankly, a waste of time if you can get nearly the same level of confidence in the changes from rigorous testing).  Additionally, note that if server-side reliability can be assumed (which is a reasonable assumption 95% of the time), this patch isn't necessary and could even regress already-working behaviors actually encountered during normal update processing.  If I had to argue against this patch being committed to 1.5.0.x code (and thankfully I don't, because those who do make these decisions will agree with me), these would be some of the reasons I would give.

If you have further questions about this, please email me privately.
Product: Firefox → Toolkit
this did the download after failing, but then hung Thunderbird

will attach crash report after this
TB refused to restart after the update failure had been followed by the full update which also failed and hung Thunderbird
This has not been fixed.
Derek, this bug has been fixed. Please file a new bug for your issue.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: