880737 - [OTA] If the download of an update gets aborted due to a network timeout it cannot be continued until the device gets restarted

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::System, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: adb logcat

Gecko  http://hg.mozilla.org/releases/mozilla-b2g18/rev/09dc1ae3b1b5
Gaia   92a6e36957145cdb2ac8867e5cdba8ecf12308fc
BuildID 20130605070207
Version 18.0

When you download an update and the network screws up and you are getting disconnected, the download stops as expected, but you are not able to continue. You will have to restart the phone to make it working again.

In more detail the downloader fails to reach the update server, and can't continue. Trying to access the same URL in Firefox it works fine. So some code in the downloader might have reached a bad state, which blocks us from successfully continuing the download of the update.

Attached you will find the full adb logcat output.

Comment 1

[bhavana bajaj [:bajaj]]

needsinfo on :whimboo to help answer :
a) If this happens over 3G or wifi
b) Are you able to successfully download once you reconnect? Does it pick up from where it was left off ?
c) Reproducibility frequency

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to bhavana bajaj [:bajaj] from comment #1)
> needsinfo on :whimboo to help answer :
> a) If this happens over 3G or wifi

I never tried updating via the mobile connection. It's too expensive. When this happened I was in the MV office and used the flaky Mozilla guest network inside of 10 forward. 

> b) Are you able to successfully download once you reconnect? Does it pick up
> from where it was left off ?

No it doesn't pick up the download because the update server cannot be found. You will have to restart the phone to make it working again. Not sure if re-enabling WIFI also helps, I can't see this behavior in my network. So we might want to have someone from MV testing this with the hope the network is still flaky. Marcia or Tony, mind trying that?

> c) Reproducibility frequency

It happened a couple of times when I was in MV last week and connected to Mozilla Guest.

Comment 3

[Jason Smith [:jsmith]]

I ended up testing this today with a 6/7 build and was able to reproduce exactly what Henrik is describing. Here's what I did:

1. Flash a 6/7 build
2. Connect to Mozilla Guest wifi
3. Start downloading the OTA update
4. Wait until you've downloaded some amount of bytes (e.g. 420 KB downloaded)
5. Kill the network
6. Kill the OTA update download
7. Try downloading the OTA update again

Result - The OTA update gets stuck at 0.00 bytes downloaded forever. Occasionally, you'll see that there was an error downloading updates, but the download will keep going.

Another thing I noticed also after step #5 was that the OTA update didn't fail gracefully when the network was lost. So I have a feeling something regressed in the arena around network timeouts when network is lost and ensuring that we're cleaning up correctly.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Jason, when the device is in such a state is it enough to disable wifi and re-enable it? Or do you really have to restart the device?

Comment 5

[Jason Smith [:jsmith]]

(In reply to Henrik Skupin (:whimboo) from comment #4)
> Jason, when the device is in such a state is it enough to disable wifi and
> re-enable it? Or do you really have to restart the device?

Disabling and re-enabling wifi didn't help for me. I still got stuck at 0.00 bytes downloading forever.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, so my initial observation was correct. Would you mind to test if you can manually reach the download URL via Firefox? In my case that was working when I grabbed the url via 'adb logcat'. So I think it's the downloader from OTA which gets confused with DNS entries?

Comment 7

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

@jsmith, can you help confirm if "adb logcat | grep AUS" stop output onProgress log while you reproduce this bug?

I have a similar STR which shows the downloading is still progressing but UI doesn't get update. I'm wondering if this bug is cause by not invoking onProgress callback. Here is the STR I use:
1. Flash a 6/7 build
2. Connect to Mozilla Guest wifi
3. Start downloading the OTA update
4. Wait until you've downloaded some amount of bytes (e.g. 420 KB downloaded)
5. Disable wifi
6. Cancel the ongoing OTA update download
7. Enable wifi and try downloading the OTA update again

Result: UI display the downloaded size before step 5 and never update.

I also observed from logcat that OTA download is resumed right after enable wifi at step 7, even if user doesn't click the download button.

Comment 8

[Jason Smith [:jsmith]]

Attachment: Logcat

Comment 9

[Jason Smith [:jsmith]]

Not entirely sure what's being asked, although I think comment 7 sounds correct. The logcat indicates downloading is happening, but the UI is indicating it's 0.00 bytes that have been downloaded.

Comment 10

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

Attachment: Stop retry timer when pause OTA download

We need to stop OTA retry timer when user want to pause download. Otherwise, the ongoing download will be canceled by UpdateService._attemptResume() and gaia will consider the download is aborted by nsUpdateService.

Comment 11

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

https://tbpl.mozilla.org/?tree=Try&rev=38c0d26e91b3

Comment 12

[Brian R. Bondy [:bbondy]]

Comment on attachment 767694 [details] [diff] [review]
Stop retry timer when pause OTA download

Review of attachment 767694 [details] [diff] [review]:
-----------------------------------------------------------------

In a followup bug, please add to toolkit\mozapps\update\test\unit\test_0030_general.js or create a new test for this.

Comment 13

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

bug 887603 is created for adding test case.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6c31bd729ed9

Comment 15

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

Attachment: rebase for b2g18

Rebase to b2g18, carry r+.

Comment 16

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/6c31bd729ed9

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/0b98670333ae

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/0b98670333ae

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Whatever has been landed here, the reported problem is not fixed for me. My initial issue is still present and I was able to reproduce it with the following build on the nightly channel:

Gecko  http://hg.mozilla.org/releases/mozilla-b2g18/rev/39607fd11f6b
Gaia   d336288e6cda1a8f974ceea41b9d7860c2367d1f
BuildID 20130706230209
Version 18.0

Comment 20

[Jason Smith [:jsmith]]

Please do not flip flags for landings for patches that have already landed. New bugs are filed for issues in this case.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

If a bug has not been fixed by a landing of code, why it should not be reopened? That's the usual task which is done and we ever did in the past. Filing a new bug is ok if parts of a fix remain. But here the patch clearly didn't fix the problem I have seen. So I do not understand why I should file a new bug for something I have reported and which is indeed not fixed.

Please verify tomorrow if the behavior has been changed at all for you with the Mozilla network in the office.

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Jason Smith [:jsmith] from comment #20)
> Please do not flip flags for landings for patches that have already landed.
> New bugs are filed for issues in this case.

FYI, you should have a look at bug 883770 which handles a situation like that correctly. Means the bug is not fixed, so it got reopened.

Comment 23

[Jason Smith [:jsmith]]

(In reply to Henrik Skupin (:whimboo) [away 06/28 - 07/07] from comment #21)
> If a bug has not been fixed by a landing of code, why it should not be
> reopened? That's the usual task which is done and we ever did in the past.
> Filing a new bug is ok if parts of a fix remain. But here the patch clearly
> didn't fix the problem I have seen. So I do not understand why I should file
> a new bug for something I have reported and which is indeed not fixed.

We do not track bugs this way in bugzilla - we track them by landings. This was discussed on dev-platform a while back when I brought this up with reasoning such as landing management, triage management, avoiding endless bugzilla comments with no resolution in site, and misconceptions of definition of not fixed, as the issue found here might . Otherwise, we end up problems such as inconsistent bug statuses, missing landings, inconsistent triage status, etc. I

> 
> Please verify tomorrow if the behavior has been changed at all for you with
> the Mozilla network in the office.

I'll check this.

(In reply to Henrik Skupin (:whimboo) [away 06/28 - 07/07] from comment #22)
> (In reply to Jason Smith [:jsmith] from comment #20)
> > Please do not flip flags for landings for patches that have already landed.
> > New bugs are filed for issues in this case.
> 
> FYI, you should have a look at bug 883770 which handles a situation like
> that correctly. Means the bug is not fixed, so it got reopened.

That bug was not handled correctly. You should not, at any point, reopen bugs for the definition of not fixed.

Comment 24

[Jason Smith [:jsmith]]

(In reply to Jason Smith [:jsmith] from comment #23)
> (In reply to Henrik Skupin (:whimboo) [away 06/28 - 07/07] from comment #21)
> > If a bug has not been fixed by a landing of code, why it should not be
> > reopened? That's the usual task which is done and we ever did in the past.
> > Filing a new bug is ok if parts of a fix remain. But here the patch clearly
> > didn't fix the problem I have seen. So I do not understand why I should file
> > a new bug for something I have reported and which is indeed not fixed.
> 
> We do not track bugs this way in bugzilla - we track them by landings. This
> was discussed on dev-platform a while back when I brought this up with
> reasoning such as landing management, triage management, avoiding endless
> bugzilla comments with no resolution in site, and misconceptions of
> definition of not fixed, as the issue found here might . Otherwise, we end
> up problems such as inconsistent bug statuses, missing landings,
> inconsistent triage status, etc. I

Ack. Missing some information here.

Misconceptions of definition of not fixed, as the issue found here might actually be a new issue not originally seen by the bug shown here. Or the issue might be fixed months back, tested now, and now doesn't work. Reopening that usually causes confusion. 

I would generally lean towards filing new bugs and marking them dependent on the original bug to show the tree of connecting bugs if issues are filed as followups. There's usually a lot of confusion that can happen otherwise - especially with blocking triage, at least, as our definition of blocking changes overtime (it's more stricter as we move closer to release).

Comment 25

[Jason Smith [:jsmith]]

Here's the associated dev-platform discussion btw - https://groups.google.com/forum/#!searchin/mozilla.dev.platform/reopen/mozilla.dev.platform/UnxndrIUIL4/1trAhVaYM-4J.

Comment 26

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Jason Smith [:jsmith] from comment #25)
> Here's the associated dev-platform discussion btw -
> https://groups.google.com/forum/#!searchin/mozilla.dev.platform/reopen/
> mozilla.dev.platform/UnxndrIUIL4/1trAhVaYM-4J.

If you refer to something a link like that would be always welcome. Not everyone is following various kind of newsgroups or discussions.

So I filed the follow-up bug 891009.