1125034 - [Midori 2.0][o2 Germany] updated third party applications with newer version wrongly replaced by old version after FOTA

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia, defect, P2)

Description

[sync-1]

the latest id: Mozilla build ID: 20141019000201    FFOS: 2.0
 
 DEFECT DESCRIPTION:
 After upgrading the device from 2EA+CF19 to 2EA+CF1A, 3rd party apps which were updated while using SW 2EA+CF19 are back to old versions. Furthermore, it is impossible to upgrade them.
 
 Affected apps are Latch, Hello, Pinterest and others
 
  REPRODUCING PROCEDURES:
 1. Install SW 2EA+CF19 (SW 03 004)on the device
 2. Check for updates and decide to upgrade to 2EA+CF1A (SW 03 005)
 
 Please note that the main code 2EA was changed from the 03 004 to the 03 005 SW.
 
  EXPECTED BEHAVIOUR:
 The version of 3rd party apps is not downgraded by the FOTA.
 
  ASSOCIATE SPECIFICATION:
 
  TEST PLAN REFERENCE:
 
  TOOLS AND PLATFORMS USED:
 
  USER IMPACT:
 
  REPRODUCING RATE: 5/5

Comment 1

[Warren WU]

[Blocking Requested - why for this release]:

Dear Mozilla:
I have merged the patch for Bug 1110119,but it still have this issue,please help to solve,thanks!

Comment 2

[Jack Liu]

Dear Wesly,

This is a serious problem, we have fixed it by a workaround but I think there must has a systematic solution including redefinition of app identity and version rules in marketplace.

You know FOTA is always TOP issue in a project, please mozilla pay attention to it and fix it at least in 2.2 build if time is too late for 2.0.

Thanks.

Comment 3

[Wesly Huang (EPM)]

Hi Jack:

After some discussion internally I would like to clarify again the symptom and STR:

1. have a old 2.0 SW in device, install Latch App (let's call it v2) from Firefox marketplace
2. have another 2.0 SW, which is preloaded with older Latch App (let's call it v1).
3. FOTA upgrade the device from SW in (1) to SW in (2)

And you expect we have Latch App v2 after STR#3, but get actually v1?
And this is same for other Apps from Marketplace?

Comment 4

[Jack Liu]

Hi Wesly,

Yes, it is same for all pre-installed apps despite of the origins of them.

I have two suggestions for app updating while FOTA:
1. always keep new version app. -- this is acceptable
2. add special flags in FOTA package to guide the replacement, the flag may indicates 3 options: keep current, keep new, force replace. -- this is better

What do you think?

Comment 5

[Evelyn Hung]

Hi Fabrice, how do we link a pre-installed app with the one on Marketplace? e.g. If 'Wikipedia' was pre-installed in the phone, could the user download again from our Marketplace, or get app update?

Comment 6

[[:fabrice] Fabrice Desré]

Yes that's possible, and this is exactly what we do to preload the marketplace app. Look at how we do it in apps/marketplace.firefox.com. You probably will need to ask the etag values to the marketplace team. Let me know if you need help for that!

Comment 7

[Josh Cheng [:josh]]

Let's move discussion to bug 1110119 as it seems not fully fix

Comment 8

[Beatriz Rodríguez [:brg]]

How is the progress of this issue? We plan to run some updates soon and this issue will block them. I do not see much progess in bug 1110119.

Comment 9

[Beatriz Rodríguez [:brg]]

How is the progress of this issue? We plan to run some updates soon and this issue will block them. I do not see much progess in bug 1110119.

Comment 10

[Jack Liu]

More specific description of this issue:
1. One 3rd party app named appA of v1.0 contained in build-1(no matter 1.3 or 2.0) running on device.
2. User update appA v1.1 from marketplace by himself or herself.
3. A FOTA package of build-2 contains appA v1.0 is released and user decides to perform FOTA update.
4. After FOTA, user will find the newer version appA v1.1 has been wrongly replaced by appA v1.0.  -- KO1
5. Under this condition, user even can't update this old appA v1.0 to newer version from marketplace. --KO2

Expected behavior:
1. Must always keep the newer version of app in device after FOTA: 
1.1 If app in FOTA package is older than which is already in device, need to keep new on in device.
1.2 If app in FOTA package is newer than which is already in device, need to replace old one with new one.

2. App can be updated from marketplace normally no matter whether it been updated by FOTA or not when there is a newer version available in marketplace.

Comment 11

[Beatriz Rodríguez [:brg]]

This is a very important issue, Wesly could you find someone to help here?

Comment 12

[Qu Yi]

(In reply to Beatriz Rodríguez [:brg] from comment #11)
> This is a very important issue, Wesly could you find someone to help here?

Dear Beatriz:

I have contact Mozilla Wesly to handle this issue in high priority.
And the issue is same with bug1110119 .
We can follow the issue in bug1110119 .
Thanks.

Comment 13

[Wesly Huang (EPM)]

Per https://bugzilla.mozilla.org/show_bug.cgi?id=1110119#c30 we back to this issue and reset the status here.

Comment 14

[Wesly Huang (EPM)]

Hi Evelyn:

As just discussed with you and CS, would you help take this one with priority and keep us updated with progress? Thank you.

Comment 15

[Evelyn Hung]

(In reply to Jack Liu from comment #10)
> More specific description of this issue:
> 1. One 3rd party app named appA of v1.0 contained in build-1(no matter 1.3
> or 2.0) running on device.
> 2. User update appA v1.1 from marketplace by himself or herself.
> 3. A FOTA package of build-2 contains appA v1.0 is released and user decides
> to perform FOTA update.
> 4. After FOTA, user will find the newer version appA v1.1 has been wrongly
> replaced by appA v1.0.  -- KO1

If I didn't get it wrong, we didn't check app version while copy app to destination.

https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#570-584

So it seems an intentional behavior, although it acts not properly in this case.

> 5. Under this condition, user even can't update this old appA v1.0 to newer
> version from marketplace. --KO2

We overwrites many fields, do they include `updateTime` and `lastUpdateCheck`? I'm not quite sure how these two fields affect the app update, but I'm guessing marketplace incorrectly assumes we are in the latest version. 

https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#717-725


Hi Fabrice, it's hard to me to understand the whole logic in webapps.jsm (I always got lost in the thousands lines of code), could you comment on my observation and give me more hints? Thanks you!

I got another finding but may not related to this problem:

In bug 1110119, we will keep the old app id in current registry and use it in package file path.

https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#526-529

Then the app ids in webapps.json and in registry will be mismatch. Not sure if webapps.json will be used on somewhere other than app registration, will this mismatch cause any problem?

Comment 16

[[:fabrice] Fabrice Desré]

(In reply to Evelyn Hung [:evelyn] from comment #15)

> If I didn't get it wrong, we didn't check app version while copy app to
> destination.
> 
> https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#570-584

That's correct. On FOTA updates, we always consider that the one from the update is the best one. Note that nothing in gecko is using the version field from the manifest.

> So it seems an intentional behavior, although it acts not properly in this
> case.
> 
> > 5. Under this condition, user even can't update this old appA v1.0 to newer
> > version from marketplace. --KO2
> 
> We overwrites many fields, do they include `updateTime` and
> `lastUpdateCheck`? I'm not quite sure how these two fields affect the app
> update, but I'm guessing marketplace incorrectly assumes we are in the
> latest version. 
> 
> https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#717-725

So it seems we successfully install the new one, but can't update it - this is what I'd like to understand, since we should definitely be able to update. Can we get the content of webapps.json at the different stages of the steps from comment #10? (I'd like to see both /system/b2g/webapps/webapps.json and /data/local/webapps/webapps.json)

> In bug 1110119, we will keep the old app id in current registry and use it
> in package file path.
> 
> https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#526-529
> 
> Then the app ids in webapps.json and in registry will be mismatch. Not sure
> if webapps.json will be used on somewhere other than app registration, will
> this mismatch cause any problem?

I think the current code is correct, but if you can exhibit a bug (with STR please) please file it!

Comment 17

[Evelyn Hung]

(In reply to Fabrice Desré [:fabrice] from comment #16)
> (In reply to Evelyn Hung [:evelyn] from comment #15)
> 
> > If I didn't get it wrong, we didn't check app version while copy app to
> > destination.
> > 
> > https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#570-584
> 
> That's correct. On FOTA updates, we always consider that the one from the
> update is the best one. Note that nothing in gecko is using the version
> field from the manifest.
> 

Wesly told me from partner's point of view, the assumption is not correct since the FOTA package need time to testing. It's possible that a frequently updated app releases a newer version to marketplace during testing period... I wasn't convinced though.

> > So it seems an intentional behavior, although it acts not properly in this
> > case.
> > 
> > > 5. Under this condition, user even can't update this old appA v1.0 to newer
> > > version from marketplace. --KO2
> > 
> > We overwrites many fields, do they include `updateTime` and
> > `lastUpdateCheck`? I'm not quite sure how these two fields affect the app
> > update, but I'm guessing marketplace incorrectly assumes we are in the
> > latest version. 
> > 
> > https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#717-725
> 
> So it seems we successfully install the new one, but can't update it - this
> is what I'd like to understand, since we should definitely be able to
> update. Can we get the content of webapps.json at the different stages of
> the steps from comment #10? (I'd like to see both
> /system/b2g/webapps/webapps.json and /data/local/webapps/webapps.json)


With SC's help, we also suspect the app update may be failed here

https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#2226

since we didn't overwrite 'etag' field while doing app registration, so it's the latest one and the xhr request won't give us status 200.

Jack, if you still have a broken device in hands, could you please provide the following information?

1. attached /system/b2g/webapps/webapps.json and /data/local/webapps/webapps.json in the phone.

2. turn on logs in webapps.jsm by setting the pref 'dom.mozApps.debug' to true. I'd like to check the log of this line:

https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#2222

Thank you.

Comment 18

[Beatriz Rodríguez [:brg]]

(In reply to Evelyn Hung [:evelyn] from comment #17)
> (In reply to Fabrice Desré [:fabrice] from comment #16)
> > (In reply to Evelyn Hung [:evelyn] from comment #15)
> > 
> > > If I didn't get it wrong, we didn't check app version while copy app to
> > > destination.
> > > 
> > > https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#570-584
> > 
> > That's correct. On FOTA updates, we always consider that the one from the
> > update is the best one. Note that nothing in gecko is using the version
> > field from the manifest.
> > 
> 
> Wesly told me from partner's point of view, the assumption is not correct
> since the FOTA package need time to testing. It's possible that a frequently
> updated app releases a newer version to marketplace during testing period...
> I wasn't convinced though.

Third party apps should be considered as user data and avoid changing them. While we improve this functionality, is there anything that OEM can do to keep current version of that apps for example do not include them in the OTA package, is it feasible?

Comment 19

[Lingling Zhao]

Attachment: json&Log.zip

(In reply to Evelyn Hung [:evelyn] from comment #17)
> 1. attached /system/b2g/webapps/webapps.json and
> /data/local/webapps/webapps.json in the phone.
> 
> 2. turn on logs in webapps.jsm by setting the pref 'dom.mozApps.debug' to
> true. I'd like to check the log of this line:
> 
> https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#2222
> 
> Thank you.

Notes:
1.Take Pinterest for example to get the attachement files.

2.The webapps.json in attachement is for the cases such as before-app-update(before-FOTA)/after-app-update(before-FOTA)/after-FOTA.

3.The steps for beforeFota.log are:
1).Update the Pinterest app
2).Open the app
3).Make the fota-upgrading
The steps for afterFOTA.log are:
1). Make sure the fota-upgrading success.
2). Open the Pinterest app.

If other steps are needed to get useful logs, please give more details information.

Thanks.

Comment 20

[Warren WU]

Hi Evelyn:

For the old app can not be updated to newer version after FOTA, we found the 'cachepath' be changed to '/data/local/webapps' in the content of webapps.json, While normally 'cachePath' should be set as '/system/b2g/webapps'.

The following is my analysis of the possible causes for this issue: the app's 'basePath' will be set as '/data/local/webapps' after it was installed, and this information will not be  changed when do a fota process. The system will goes through the firstRun process after FOTA, but the 'basePath' has already been set as '/data/local/webapps', that makes the 'cachePath' also be set as '/data/local/webapps'.

Comment 21

[Evelyn Hung]

(In reply to [e-mail] from comment #20)
> Hi Evelyn:
> 
> For the old app can not be updated to newer version after FOTA, we found the
> 'cachepath' be changed to '/data/local/webapps' in the content of
> webapps.json, While normally 'cachePath' should be set as
> '/system/b2g/webapps'.
> 
> The following is my analysis of the possible causes for this issue: the
> app's 'basePath' will be set as '/data/local/webapps' after it was
> installed, and this information will not be  changed when do a fota process.
> The system will goes through the firstRun process after FOTA, but the
> 'basePath' has already been set as '/data/local/webapps', that makes the
> 'cachePath' also be set as '/data/local/webapps'.

I didn't get your idea. From attached webapps.json files, I always see 'cachePath' pointing to '/data/local/webapps'. Since 'cachePath' is copied from 'basePath', and 'basPath' won't be changed while FOTA, I don't see any suspicious here.

Comment 22

[Evelyn Hung]

(In reply to Lingling Zhao from comment #19)
> Created attachment 8579212 [details]
> json&Log.zip
> 
> (In reply to Evelyn Hung [:evelyn] from comment #17)
> > 1. attached /system/b2g/webapps/webapps.json and
> > /data/local/webapps/webapps.json in the phone.
> > 
> > 2. turn on logs in webapps.jsm by setting the pref 'dom.mozApps.debug' to
> > true. I'd like to check the log of this line:
> > 
> > https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#2222
> > 
> > Thank you.
> 
> Notes:
> 1.Take Pinterest for example to get the attachement files.
> 
> 2.The webapps.json in attachement is for the cases such as
> before-app-update(before-FOTA)/after-app-update(before-FOTA)/after-FOTA.
> 
> 3.The steps for beforeFota.log are:
> 1).Update the Pinterest app
> 2).Open the app
> 3).Make the fota-upgrading
> The steps for afterFOTA.log are:
> 1). Make sure the fota-upgrading success.
> 2). Open the Pinterest app.
> 
> If other steps are needed to get useful logs, please give more details
> information.
> 
> Thanks.

Hi, thanks for your help. Since the log for webapps.jsm didn't been enabled, so we can't see anything. From webapps.json files, we found the registry for Pinterest app didn't been changed after FOTA - they are identical! It's very weird, we assume at least 'updateTime' should be updated.

Therefore, I need more logs. Please remove the following lines to enable debug log.

http://mxr.mozilla.org/mozilla-b2g32_v2_0/source/dom/apps/src/Webapps.jsm#83
http://mxr.mozilla.org/mozilla-b2g32_v2_0/source/dom/apps/src/Webapps.jsm#85

Then attached the same files (webapps.json in /system and /data partitions before and after update/FOTA), and logs. Thank you so much!

Comment 23

[Lingling Zhao]

Hi Evelyn,

I had enabled the debug log as you said, so you can see many logs started with "-*- Webapps.jsm" in the logs I provided, but I also doesnot see the logs you need.

Please give the detailed operation steps for how to get the needed logs.

Thanks.

Comment 24

[Evelyn Hung]

(In reply to Lingling Zhao from comment #23)
> Hi Evelyn,
> 
> I had enabled the debug log as you said, so you can see many logs started
> with "-*- Webapps.jsm" in the logs I provided, but I also doesnot see the
> logs you need.
> 
> Please give the detailed operation steps for how to get the needed logs.
> 
> Thanks.

Hi, yes, I see Webapps.jsm logs in beforeFota.log, but didn't see in afterFOTA.log. However, the thing I care should be happened in after FOTA. Could you please try again? Thanks!

Comment 25

[[:fabrice] Fabrice Desré]

(In reply to Beatriz Rodríguez [:brg] from comment #18)
> 
> Third party apps should be considered as user data and avoid changing them.
> While we improve this functionality, is there anything that OEM can do to
> keep current version of that apps for example do not include them in the OTA
> package, is it feasible?

Well, we don't change the user *data* like any storage used by the app. If you don't include the app in the OTA, people that didn't install it as a 3rd party from the marketplace will obviously not get it.


(In reply to Evelyn Hung [:evelyn] from comment #21)

> I didn't get your idea. From attached webapps.json files, I always see
> 'cachePath' pointing to '/data/local/webapps'. Since 'cachePath' is copied
> from 'basePath', and 'basPath' won't be changed while FOTA, I don't see any
> suspicious here.

Yep, cachePath is only related to appcache path, but Pinterest is not using it.

I looked quickly at the webapps.json files and there's one thing that looks strange to me:
pinterest is preloaded in both (manifest url is https://marketplace.firefox.com/app/6c9a06a4-f9d2-4a2d-95ca-692ba9306932/manifest.webapp) but they have a different appId ({e526bc1a-d6a5-453c-b8c8-927e17702009} before fota, {88cd4549-62f3-4eb8-92a3-148b15a0f119} after fota). That should be fixed.

Comment 26

[Lingling Zhao]

Attachment: json&Log_2.zip

Hi All,

Please see the new json and log, and ignore the previous one.

And please see comment 20 again with the new json.

Thanks.

Comment 27

[Evelyn Hung]

(In reply to Fabrice Desré [:fabrice] from comment #25)
> I looked quickly at the webapps.json files and there's one thing that looks
> strange to me:
> pinterest is preloaded in both (manifest url is
> https://marketplace.firefox.com/app/6c9a06a4-f9d2-4a2d-95ca-692ba9306932/
> manifest.webapp) but they have a different appId
> ({e526bc1a-d6a5-453c-b8c8-927e17702009} before fota,
> {88cd4549-62f3-4eb8-92a3-148b15a0f119} after fota). That should be fixed.

Fabrice, according to bug 1110119 description, I thought that's because we generate a uuid for app-not-specified-origin. Although the appIds are different, but the fix of bug 1110119 force them to be identical (keep the old one) after FOTA, in the webapps.json of /data/local/.

Comment 28

[Evelyn Hung]

After reading logs, I think bug 993011 has fixed the problem. Could you please try to apply the patch on bug 993011?

https://bug993011.bugzilla.mozilla.org/attachment.cgi?id=8498223

Comment 29

[Evelyn Hung]

(In reply to Evelyn Hung [:evelyn] from comment #28)
> After reading logs, I think bug 993011 has fixed the problem. Could you
> please try to apply the patch on bug 993011?
> 
> https://bug993011.bugzilla.mozilla.org/attachment.cgi?id=8498223

and don't forget to modify this line, 

this.webapps[id][field] = data[id][field];

it should be

this.webapps[oldId][field] = data[id][field];

because of patch in bug 1110119.

Comment 30

[Wesly Huang (EPM)]

Hi QuYi:

Would you help arrange a test to see if it's ok with the instruction above? Thanks!

Comment 31

[Lingling Zhao]

Hi Wesly,

I will update the result asap when I finish the test.

Comment 32

[Lingling Zhao]

Hi Evelyn,

The issue is fixed after I apply this patch!
Thanks for all your help.

Comment 33

[Evelyn Hung]

Wesly, do you think we need to uplift bug 993011 to 2.0 or 2.0m?
If not, please close this issue. Thanks!

Comment 34

[Lingling Zhao]

Attachment: Webapps.jsm

Hi Evelyn,

A new issue happens after the patch applied. 
Take the Pinterest for example, make the FOTA updating from SW1 to SW2, the version for Pinterest in SW1 is 0.2 and in SW2 is 0.3. After the FOTA updating, the version for Pinterest in the device is still 0.2.

I attach the webapps.jsm including the patch in our side, please have a look if we missed other patches?

Thanks.

Comment 35

[Evelyn Hung]

(In reply to Lingling Zhao from comment #34)
> Created attachment 8583643 [details]
> Webapps.jsm
> 
> Hi Evelyn,
> 
> A new issue happens after the patch applied. 
> Take the Pinterest for example, make the FOTA updating from SW1 to SW2, the
> version for Pinterest in SW1 is 0.2 and in SW2 is 0.3. After the FOTA
> updating, the version for Pinterest in the device is still 0.2.
> 
> I attach the webapps.jsm including the patch in our side, please have a look
> if we missed other patches?
> 
> Thanks.

Yes, the following lines were accidentally removed:

          // If the id for the app has changed on the update, keep a pointer to the old one
          // since we'll need this to update the app files.
          if (id !== oldId) {
            this.webapps[id] = {oldId: oldId};
          }

It should look like this: 
https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#713-730

Comment 36

[Lingling Zhao]

Attachment: json&Log_3.zip

The issue in Comment 34 cannot be reproduced after I add these lines. But the original issue in this bug appears again. 
Please see the new json and log files in the attachement.

Comment 37

[Evelyn Hung]

(In reply to Lingling Zhao from comment #36)
> Created attachment 8584264 [details]
> json&Log_3.zip
> 
> The issue in Comment 34 cannot be reproduced after I add these lines. But
> the original issue in this bug appears again. 
> Please see the new json and log files in the attachement.

From the log, the symptom should be different. Its updating request got a http status 200, which means the update was triggered. 

> 03-26 23:08:52.282 I/Gecko   (  220): -*- Webapps.jsm : Got http status=200 for https://marketplace.firefox.com/app/6c9a06a4-f9d2-4a2d-95ca-692ba9306932/manifest.webapp
> 03-26 23:08:52.292 I/Gecko   (  220): -*- Webapps.jsm : Manifest hash = 7e3d63f24977455a8bcfe63a1faa7a5e
> 03-26 23:08:52.292 I/Gecko   (  220): -*- Webapps.jsm : Saving /data/local/webapps/webapps.json

However, the manifest hash kept its latest updated value because the information of webapps.json in FOTA package didn't have this field, so we didn't overwrite it. Therefore, we fallen into this |else| case and thought there is no change of manifest file.

http://mxr.mozilla.org/mozilla-b2g32_v2_0/source/dom/apps/src/Webapps.jsm#1800

I'm not sure what's the correct way to avoid that. To quick fix it, we could clean up the |manifestHash| after we overwrite every field in the for loop, like this:

          for (let field in data[id]) {
            if (fieldsBlacklist.indexOf(field) === -1) {
              this.webapps[id][field] = data[id][field];
            }
          }

+         this.webapps[id].manifestHash = '';

          // If the id for the app has changed on the update, keep a pointer to the old one
          // since we'll need this to update the app files.
          if (id !== oldId) {
            this.webapps[id] = {oldId: oldId};
          }


Lingling, could you try above to verify if my guess is correct? Thanks.

Comment 38

[Lingling Zhao]

Hi Evelyn,

In this condition, the issue in Comment 34 comes again.

Comment 39

[Evelyn Hung]

(In reply to Lingling Zhao from comment #38)
> Hi Evelyn,
> 
> In this condition, the issue in Comment 34 comes again.

could you show me the log? Thanks!

Comment 40

[Evelyn Hung]

(In reply to Evelyn Hung [:evelyn] from comment #39)
> (In reply to Lingling Zhao from comment #38)
> > Hi Evelyn,
> > 
> > In this condition, the issue in Comment 34 comes again.
> 
> could you show me the log? Thanks!

and your latest webapps.jsm file. Thank you!

Comment 41

[Lingling Zhao]

Attachment: json&Log&jsm_4.zip

Please see the new files.

Comment 42

[Evelyn Hung]

From log, it seems the app zip didn't being copied to destination. It's weird!

08-08 13:51:02.829 I/Gecko   (  211): -*- Webapps.jsm : Installing 3rd party app : marketplace.firefox.com from /system/b2g/webapps/marketplace.firefox.com to marketplace.firefox.com
08-08 13:51:02.829 I/Gecko   (  211): -*- Webapps.jsm : Opening /system/b2g/webapps/marketplace.firefox.com/application.zip
08-08 13:51:03.009 I/Gecko   (  211): -*- Webapps.jsm : Error: Core app in /system/b2g/webapps/{88cd4549-62f3-4eb8-92a3-148b15a0f119} is empty
08-08 13:51:03.009 I/Gecko   (  211): -*- Webapps.jsm : Saving /data/local/webapps/webapps.json

I don't know what happened in |installPreinstalledApp| function. Could you insert some logs in this function, and all places it calls |return isPreinstalled;|?

 installPreinstalledApp: function installPreinstalledApp(aId) {
+   debug('installPreinstalledApp aId: ' + aId);
    ...
+   debug('return aId: ' + aId + 'with isPreinstalled=' + isPreinstalled);
    return isPreinstalled;
    ...
 }

Furthermore, the code I provided in comment 37 should use 'oldId', not 'id'. I was trying to clear |manifestHash| original in the original app registry.

          for (let field in data[id]) {
            if (fieldsBlacklist.indexOf(field) === -1) {
              this.webapps[oldId][field] = data[id][field];
            }
          }

+         this.webapps[oldId].manifestHash = '';

          // If the id for the app has changed on the update, keep a pointer to the old one
          // since we'll need this to update the app files.
          if (id !== oldId) {
            this.webapps[id] = {oldId: oldId};
          }

Comment 43

[Lingling Zhao]

Attachment: json&Log&jsm_6.zip

Hi Evelyn,

The issue in Comment 34 cannot be reproduced after I modified the code that line. But the original issue in this bug appears again.

I can see in comment 16：
On FOTA updates, we always consider that the one from the update is the best one. Note that nothing in gecko is using the version field from the manifest.

In order to solve the issue fundamentally, maybe it is a better way that take the version field in account .

Comment 44

[Evelyn Hung]

(In reply to Lingling Zhao from comment #43)
> Created attachment 8585864 [details]
> json&Log&jsm_6.zip
> 
> Hi Evelyn,
> 
> The issue in Comment 34 cannot be reproduced after I modified the code that
> line. But the original issue in this bug appears again.
> 

From log, it looks everything is fine in webapps.jsm. Could you describe in detail that how does the update doesn't success? Do you see a update prompt on UI? Please insert a log below to check if app update process goes well.

in http://mxr.mozilla.org/mozilla-b2g32_v2_0/source/dom/apps/src/Webapps.js#572

  _fireEvent: function(aName) {
+    dump("-- webapps.js fireEvent: " + aName);
     let handler = this["_on" + aName];
     ...
  }

in Gaia, https://github.com/mozilla-b2g/gaia/blob/v2.0/apps/system/js/updatable.js#L61

  AppUpdatable.prototype.availableCallBack = function() {
+   dump("-- system app this.app.downloadSize " + this.app.downloadSize);
    this.size = this.app.downloadSize;
    ...
  }

> I can see in comment 16：
> On FOTA updates, we always consider that the one from the update is the best
> one. Note that nothing in gecko is using the version field from the manifest.
> 
> In order to solve the issue fundamentally, maybe it is a better way that
> take the version field in account .

I don't think it's easier to add version check now, it might cause more regression.
I feel we are pretty closed to solve this issue - registry in webapps.json is correct, app update check passed, and got new etag and manifest file from Marketplace. The only question I'm not sure is above - do you see the update UI prompted? If no, I'd like to know why, so let's add more log for understanding. Thanks.

Comment 45

[Lingling Zhao]

Attachment: json&Log_7.zip

(In reply to Evelyn Hung [:evelyn] from comment #44)
The app icon for Pinterest has changed in the updated version, so I check the icon to determine if it is a updated app or not.
At the same time I can see the update UI prompted after the FOTA upating.

Comment 46

[Evelyn Hung]

(In reply to Lingling Zhao from comment #45)
> At the same time I can see the update UI prompted after the FOTA upating.

What happened after you clicked the prompted UI and pressed the 'update' button?

Comment 47

[Lingling Zhao]

In the update available prompted UI, the button is 'download' but not 'update'.
After I click the 'download' button, it can download updating successfully and there is no error prompt. 
But in fact the update can not work. The icon for the app is abnormal and when I click the app icon to enter the app, it prompts "Restart Download/ Do you want to download Pinterest?", nothing happens when I click the Download button. This app does not work until I restart the device, no prompted UI comes and it is the lower version for the app after I restart the device.

Comment 48

[Evelyn Hung]

(In reply to Lingling Zhao from comment #47)
> In the update available prompted UI, the button is 'download' but not
> 'update'.
> After I click the 'download' button, it can download updating successfully
> and there is no error prompt. 
> But in fact the update can not work. The icon for the app is abnormal and
> when I click the app icon to enter the app, it prompts "Restart Download/ Do
> you want to download Pinterest?", nothing happens when I click the Download
> button. This app does not work until I restart the device, no prompted UI
> comes and it is the lower version for the app after I restart the device.

Alright, so it becomes another issue. 
1. Could you check its zip in /data/local/webapps/?
2. Since it's different from here, I suggest we create another bug, and close here. Could you file one? Thanks!

Comment 49

[Lingling Zhao]

Attachment: pinterest_package.zip

Comment 50

[Lingling Zhao]

Hi Evelyn,

As we talked I create a new Bug 1150442 for the new issue.

Comment 51

[Evelyn Hung]

Attachment: 0001-Bug-1125034-Update-existing-Gaia-apps-fields.patch

Here is the patch that used to fix FOTA issue here. It basically is the patch on bug 993011, with some slight modifications. Therefore I kept the commit message intact.

We found that we have to overwrite most fields in app registry while doing FOTA, to avoid the cases like this bug - Marketplace thought the app zip is updated because its etag was kept in an updated value, however, the zip was actually downgraded to an old version in FOTA package.

Comment 52

[Evelyn Hung]

Close this bug per comment 48, the follow-up bug will be tracked in bug 1150442.

Comment 53

[Beatriz Rodríguez [:brg]]

[Blocking Requested - why for this release]:
Should this code land in v2.0 branch? We are facing this issue with at least two devices and we think user impact is very harmful.

Please Wesly give a comment.

Comment 54

[Wesly Huang (EPM)]

[Triage]

This blocks App update which is an important FxOS experience, solution confirmed thus tag.

Comment 55

[Wesly Huang (EPM)]

(In reply to Evelyn Hung [:evelyn] from comment #33)
> Wesly, do you think we need to uplift bug 993011 to 2.0 or 2.0m?
> If not, please close this issue. Thanks!

Hi Evelyn, 

Yes, 2.0m also need it, thanks! (also NI 2.0m EPM Josh for attention, tks)

Comment 56

[Wesly Huang (EPM)]

(In reply to Wesly Huang from comment #55)
> (In reply to Evelyn Hung [:evelyn] from comment #33)
> > Wesly, do you think we need to uplift bug 993011 to 2.0 or 2.0m?
> > If not, please close this issue. Thanks!
> 
> Hi Evelyn, 
> 
> Yes, 2.0m also need it, thanks! (also NI 2.0m EPM Josh for attention, tks)

update per latest status. 

Since use patch of 1150442 will be sufficient for both 1150442 and 1125034, so for 2.0m we should land the fix of 1150442, instead of solution here.