1155035 - [Flame KK] Can't perform FOTA from 2.0/2.1 to 2.2

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect)

Description

[Gerry Chang [:gchang]]

Trying to upgrade 2.0 to 2.2 using FOTA, but got errors when entering recovery mode.
The errors in recovery mode is "failed to open /sdcard/updates/fota/update.zip (No such file or directory)"
But, the file exists in that directory.
adb shell
root@flame:/ # ls /sdcard/updates/fota/                                        
precomplete
update.zip
 
STR:

1. Build 2.2 fota image locally and put the image in local server.
2. Flash flame with below build information

Build ID               20150415000204
Gaia Revision          84898cadf28b1a1fcd03b726cff658de470282f0
Gaia Date              2015-04-03 21:42:36
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/de92ad41847a
Gecko Version          32.0
Device Name            flame
Firmware(Release)      4.4.2

3. Change ota url to local server by using ./change_ota_url.sh in https://github.com/Mozilla-TWQA/B2G-flash-tool
4. Go to Settings > Device information 
5. Tap "Check now" button
6. Perform upgrade in notification bar

### Expected:
1. The device should be upgraded to 2.2

### Actual:
1. The device failed to upgrade to 2.2

### Reproduce rate
always

Comment 1

[Gerry Chang [:gchang]]

Attachment: recovery.log

Comment 2

[Gerry Chang [:gchang]]

Got the same error while performing FOTA from 2.1 to 2.2.

Comment 3

[Gerry Chang [:gchang]]

Attachment: FOTA of 2.1 to 2.2.log

Comment 4

[Gerry Chang [:gchang]]

2.1 build information is shown below.
Build ID               20150415161202
Gaia Revision          bbe983b4e8bebfec26b3726b79568a22d667223c
Gaia Date              2015-04-09 13:52:48
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/c54aa1be51d6
Gecko Version          34.0
Device Name            flame
Firmware(Release)      4.4.2

Comment 5

[bhavana bajaj [:bajaj]]

dhylands, would you know what's going on here from the attached log?

Comment 6

[Gregor Wagner [:gwagner]]

Maybe alex can take a look as well.

Comment 7

[Dave Hylands [:dhylands]]

The recovery log says:

I:Update location: /sdcard/updates/fota/update.zip
E:failed to open /sdcard/updates/fota/update.zip (No such file or directory)

which usually means that something about librecovery.so is incorrect or misconfigured.

But it could be other reasons as well.

Comment 8

[Eric Tsai (no more review request accepted)]

After sync with Gerry, he flashed base image 18D then pvt build.
However, pvt build will not flash recovery partition, it's difficult to debug using T2M's recovery image for me. Besides, we can't update using T2M's recovery image without same sign key, so I change the base condition.
I use base image 18D, extract images from pvt build, using flash.sh in 18D with our own recovery image. After download update.mar and reboot, there is black screen and recovery hangs. I think we need to find out why recovery mode from Mozilla doesn't work, then we can go to next step.

Comment 9

[:gerard-majax]

(In reply to Eric Tsai from comment #8)
> After sync with Gerry, he flashed base image 18D then pvt build.
> However, pvt build will not flash recovery partition, it's difficult to
> debug using T2M's recovery image for me. Besides, we can't update using
> T2M's recovery image without same sign key, so I change the base condition.
> I use base image 18D, extract images from pvt build, using flash.sh in 18D
> with our own recovery image. After download update.mar and reboot, there is
> black screen and recovery hangs. I think we need to find out why recovery
> mode from Mozilla doesn't work, then we can go to next step.

The last time I played with recovery on my Flame, it was working flawlessly.

Comment 10

[:gerard-majax]

And I just tested on a self-built recovery from two days ago, no problem to get into recovery mode.

Comment 11

[Eric Tsai (no more review request accepted)]

I guess no screen in recovery mode is related with bug 1079318. But now I use recovery.img in v18D_nightly and I can see recovery UI now.
I think we can use v18D_nightly as base image and flash pvt build now to test FOTA.

Comment 12

[:gerard-majax]

> Command: "/sbin/recovery" "--update_package=/storage/sdcard1/updates/fota/update.zip"

The /storage/sdcard1 mountpoint is not what I have when flashing v18D. Using v18D provided by T2M, the recovery command I have is:
> 04-17 13:02:00.718  1690  1690 D librecovery: Rebooting into recovery: --update_package=/storage/sdcard0/updates/fota/update.zip

I don't have any problem applying a FOTA update package for master on top of 2.0, our recovery boots as expected and picksup the update package.

Build target tested was gecko-update-fota.

Comment 13

[:gerard-majax]

And it's fine for me with gecko-update-fota-full too.

Comment 14

[:gerard-majax]

Attachment: working applying fota in recovery

Comment 15

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I think you need to start from your own build of 2.0, flash that on the device so you have our librecover.so 

Then create a full fota : ( https://wiki.mozilla.org/B2G/Updating#Generating_a_complete_FOTA_update_zip_and_target_files_zip ) and then try to flash that.  I don't think an incremental FOTA will work...

Comment 16

[Eric Tsai (no more review request accepted)]

Alex, can you share your configuration for FOTA? I've tried:
1) flash v18d_nightly_v2 base image (System recovery <3e> KOT49H test-keys) + my own build update.zip from adb sideload: assertion failed for getprop("ro.build.product"). Skip the assertion check then fail in format data, error message: expects 4 args, got 5. So I stop test here.

2) flash my own 2.0 build boot/recovery/system/data + my own 2.2 build update.zip from adb sideload: no screen but I can sideload by guessing the key sequence. I can sideload and update 2.2 to my flame. Just curious about how can you build recovery correctly.

Comment 17

[:gerard-majax]

(In reply to Eric Tsai from comment #16)
> Alex, can you share your configuration for FOTA? I've tried:
> 1) flash v18d_nightly_v2 base image (System recovery <3e> KOT49H test-keys)
> + my own build update.zip from adb sideload: assertion failed for
> getprop("ro.build.product"). Skip the assertion check then fail in format
> data, error message: expects 4 args, got 5. So I stop test here.

That's expected

> 
> 2) flash my own 2.0 build boot/recovery/system/data + my own 2.2 build
> update.zip from adb sideload: no screen but I can sideload by guessing the
> key sequence. I can sideload and update 2.2 to my flame. Just curious about
> how can you build recovery correctly.

My recovery was probably built from master, I don't know the status of v2.0 regarding this.

Comment 18

[Eric Tsai (no more review request accepted)]

Update, follow https://developer.mozilla.org/en-US/Firefox_OS/Building#Building_an_official-style_engineering_build the recovery image works for me now.

Comment 19

[Bobby Chien]

eric, could you close case per comment 18?

Comment 20

[Eric Tsai (no more review request accepted)]

I think the problem for flame is in librecovery/librecovery.c, convertExternalStoragePath. I guess it uses RECOVERY_EXTERNAL_STORAGE in B2G/device/t2m/flame/BoardConfig.mk:
  RECOVERY_EXTERNAL_STORAGE := /storage/sdcard1
to build destPath: "/storage/sdcard1/updates/fota/update.zip"
I change the RECOVERY_EXTERNAL_STORAGE to /storage/sdcard0 then the FOTA works.

Comment 21

[:gerard-majax]

(In reply to Eric Tsai from comment #20)
> I think the problem for flame is in librecovery/librecovery.c,
> convertExternalStoragePath. I guess it uses RECOVERY_EXTERNAL_STORAGE in
> B2G/device/t2m/flame/BoardConfig.mk:
>   RECOVERY_EXTERNAL_STORAGE := /storage/sdcard1
> to build destPath: "/storage/sdcard1/updates/fota/update.zip"
> I change the RECOVERY_EXTERNAL_STORAGE to /storage/sdcard0 then the FOTA
> works.

Well, I agree that's something I always found not very cool because it makes you depending on an external sdcard while the device already has internal sdcard memory, but the discusion already took place on this and people wants to stick to /storage/sdcard1/ for this usecase.

Comment 22

[Eric Tsai (no more review request accepted)]

Attachment: BoardConfig.mk PR to kitkat

Change recovery external storage from /storage/sdcard1 to /storage/sdcard0

Comment 23

[Eric Tsai (no more review request accepted)]

Attachment: flame-kk.xml PR to v2.2

Update device-flame revision from hashtag to "kitkat". Will request approval-mozilla-b2g37 after r+

Comment 24

[Kai-Zhen Li [:seinlin][:kli]]

Comment on attachment 8597782 [details] [review]
BoardConfig.mk PR to kitkat

Thanks! LGTM.

Comment 25

[Kai-Zhen Li [:seinlin][:kli]]

Comment on attachment 8597783 [details] [review]
flame-kk.xml PR to v2.2

LGTM. Please request approval to land into v2.2. I think v2.0 and v2.1 would also need this fix.

Comment 26

[Eric Tsai (no more review request accepted)]

Attachment: flame-kk.xml PR to v2.0

flame-kk.xml PR for branch v2.0

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): this bug, 1155035
User impact if declined: Developer is unable to flash FOTA package for branch 2.0
Testing completed: Locally verified on branch 2.0
Risk to taking this patch (and alternatives if risky): Low, easy to backout
String or UUID changes made by this patch: none

Comment 27

[Eric Tsai (no more review request accepted)]

Attachment: flame-kk.xml PR to v2.1

flame-kk.xml PR for branch v2.1

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): this bug, 1155035
User impact if declined: Developer is unable to flash FOTA package for branch 2.1
Testing completed: Locally verified on branch 2.1
Risk to taking this patch (and alternatives if risky): Low, easy to backout
String or UUID changes made by this patch: none

Comment 28

[Eric Tsai (no more review request accepted)]

Comment on attachment 8597783 [details] [review]
flame-kk.xml PR to v2.2

flame-kk.xml PR for branch v2.2

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): this bug, 1155035
User impact if declined: Developer is unable to flash FOTA package for branch 2.2
Testing completed: Locally verified on branch 2.2
Risk to taking this patch (and alternatives if risky): Low, easy to backout
String or UUID changes made by this patch: none

Comment 29

[Keven Kuo [:kkuo]]

Hi! Eric,

Since you are working on this case.
Over to you. Thanks

--
Keven

Comment 30

[Ryan VanderMeulen [:RyanVM]]

kitkat: https://github.com/mozilla-b2g/device-flame/commit/a9f3f8fb8b0844724de32426b7bcc4e6dc4fa2ed

v2.2: https://github.com/mozilla-b2g/b2g-manifest/commit/aee786f8c1c0f6a91aadee41f04737083367ca31
v2.1: https://github.com/mozilla-b2g/b2g-manifest/commit/538526c319248594ab1c8e209d261f488f37a8e2
v2.0: https://github.com/mozilla-b2g/b2g-manifest/commit/68f65c9682d9f40d4b70e55cf741cb798f26b2be

Comment 31

[Kai-Zhen Li [:seinlin][:kli]]

This fix causes a regression as bug 1160671. We need to re-land the patch as bug 1160671 comment 9.

Comment 32

[Kai-Zhen Li [:seinlin][:kli]]

As discussed in bug 1160671, re-land all patches.

kitkat-v2: https://github.com/mozilla-b2g/device-flame/commit/ab265fb203390c70b8f2a054f38cf4b2f2dad70a

v2.2: https://github.com/mozilla-b2g/b2g-manifest/commit/b6d82267f0eabd6c0483c46723554373433d2be5
v2.1: https://github.com/mozilla-b2g/b2g-manifest/commit/5946c7913dcc71c5393bca53ac3c0f7610ba7106
v2.0: https://github.com/mozilla-b2g/b2g-manifest/commit/5d297be7039c9d8b728ffbf8700a0d57c4c4c289