1061132 - [dolphin][OTA update] OTA update failed with vendor build 20140828203058 ( w35.4)

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Vendcom, defect)

Description

[Peipei Cheng (needinfo if you need my action)]

STR
-----------------------------------------------------------------------------
1. Download and install partner pac build 20140828203058 (https://mega.co.nz/#F!Ut1V1ZQJ!gC1iHS29o2DOw26DVBBZHQ)
2. Flash an old mozilla pvt build (only gecko and gaia)
3. Change the OTA URL to point to a xml such as http://update.boot2gecko.org/dolphin/1.4.0/nightly/update_20140901000203.xml
   # To do this, please download B2G_Flash_Tools from 
     "git clone https://github.com/Mozilla-TWQA/B2G-flash-tool.git" 
   # and then run ./change_ota_url -u url
4. After device restarts, wait until the OTA update notification appears, then click download
   --> OTA update failed. The new build actually can be downloaded. But it failed after uncompressing the package.

Note：This issue is only reproducible using sprd build 20140828203058 ( w35.4). Last build 20140822194356 (w34.5) works fine.

Comment 1

[Peipei Cheng (needinfo if you need my action)]

James, could you plase assign someone to take a look?

Comment 2

[Peipei Cheng (needinfo if you need my action)]

Attachment: OTA.log

Please see attached adb log

Comment 3

[Zhou Lianxiang]

We get a wrong update-binary for the update.zip package in out new version after Aug 18. We replace the variable "TARGET_UPDATE_BINARY" with "$(PRODUCT_OUT)/system/bin/updater" in file device/sprd/scx15_sp7715ga/BoardConfig.mk. 
The updater would not be compile until we run "make updater", there is no dependent with it. So, in file "gonk-misc/Android.mk:431" the sentence "$(if $(wildcard $(TARGET_UPDATE_BINARY))..." will always return false, the the variable "FOTA_UPDATE_BIN" will be null and the "--update-bin" option will use default setting, which is a compiled updater-binary without the modify for "ubifs" by sprd.
Maybe we need modify the makefile, use dependent replace file-exist judgement (Bug 1008239).

Such as, in gonk-misc add the patch

diff --git a/Android.mk b/Android.mk
index 03334b9..1e8f099 100644
--- a/Android.mk
+++ b/Android.mk
@@ -427,15 +427,9 @@ define detect-partitions
   $(info Mounting /data   from $(B2G_FOTA_DATA_PARTITION))
 endef
 
-define detect-update-bin
-  $(if $(wildcard $(TARGET_UPDATE_BINARY)),
-    $(eval FOTA_UPDATE_BIN := --update-bin $(TARGET_UPDATE_BINARY)))
-endef
-
 define setup-fs
   $(call detect-fstype)
   $(call detect-partitions,$(recovery_fstab))
-  $(call detect-update-bin)
 endef
 
 B2G_FOTA_FLASH_SCRIPT := tools/update-tools/build-flash-fota.py
@@ -456,6 +450,12 @@ $(B2G_FOTA_SYSTEM_FILES): $(PRODUCT_OUT)/system.img
 # Otherwise, our fake java will be used to run signapk.jar
 B2G_FOTA_ENV_PATH := $(shell echo "$$PATH" | sed -e 's|$(ANDROID_JAVA_TOOLCHAIN)||g')
 
+ifneq ($(TARGET_UPDATE_BINARY),)
+$(PRODUCT_OUT)/$(B2G_FOTA_UPDATE_ZIP): $(TARGET_UPDATE_BINARY)
+$(PRODUCT_OUT)/$(B2G_FOTA_UPDATE_FULL_ZIP): $(TARGET_UPDATE_BINARY)
+FOTA_UPDATE_BIN := --update-bin $(TARGET_UPDATE_BINARY)
+endif
+
 $(PRODUCT_OUT)/$(B2G_FOTA_UPDATE_ZIP): $(B2G_FOTA_SYSTEM_FILES) $(PRODUCT_OUT)/system.img
        mkdir -p `dirname $@` || true
        $(call setup-fs)

Comment 4

[Peipei Cheng (needinfo if you need my action)]

(In reply to Zhou Lianxiang from comment #3)
> We get a wrong update-binary for the update.zip package in out new version
> after Aug 18. We replace the variable "TARGET_UPDATE_BINARY" with
> "$(PRODUCT_OUT)/system/bin/updater" in file
> device/sprd/scx15_sp7715ga/BoardConfig.mk. 
> The updater would not be compile until we run "make updater", there is no
> dependent with it. So, in file "gonk-misc/Android.mk:431" the sentence "$(if
> $(wildcard $(TARGET_UPDATE_BINARY))..." will always return false, the the
> variable "FOTA_UPDATE_BIN" will be null and the "--update-bin" option will
> use default setting, which is a compiled updater-binary without the modify
> for "ubifs" by sprd.
> Maybe we need modify the makefile, use dependent replace file-exist
> judgement (Bug 1008239).
> 
> Such as, in gonk-misc add the patch
> 
> diff --git a/Android.mk b/Android.mk
> index 03334b9..1e8f099 100644
> --- a/Android.mk
> +++ b/Android.mk
> @@ -427,15 +427,9 @@ define detect-partitions
>    $(info Mounting /data   from $(B2G_FOTA_DATA_PARTITION))
>  endef
>  
> -define detect-update-bin
> -  $(if $(wildcard $(TARGET_UPDATE_BINARY)),
> -    $(eval FOTA_UPDATE_BIN := --update-bin $(TARGET_UPDATE_BINARY)))
> -endef
> -
>  define setup-fs
>    $(call detect-fstype)
>    $(call detect-partitions,$(recovery_fstab))
> -  $(call detect-update-bin)
>  endef
>  
>  B2G_FOTA_FLASH_SCRIPT := tools/update-tools/build-flash-fota.py
> @@ -456,6 +450,12 @@ $(B2G_FOTA_SYSTEM_FILES): $(PRODUCT_OUT)/system.img
>  # Otherwise, our fake java will be used to run signapk.jar
>  B2G_FOTA_ENV_PATH := $(shell echo "$$PATH" | sed -e
> 's|$(ANDROID_JAVA_TOOLCHAIN)||g')
>  
> +ifneq ($(TARGET_UPDATE_BINARY),)
> +$(PRODUCT_OUT)/$(B2G_FOTA_UPDATE_ZIP): $(TARGET_UPDATE_BINARY)
> +$(PRODUCT_OUT)/$(B2G_FOTA_UPDATE_FULL_ZIP): $(TARGET_UPDATE_BINARY)
> +FOTA_UPDATE_BIN := --update-bin $(TARGET_UPDATE_BINARY)
> +endif
> +
>  $(PRODUCT_OUT)/$(B2G_FOTA_UPDATE_ZIP): $(B2G_FOTA_SYSTEM_FILES)
> $(PRODUCT_OUT)/system.img
>         mkdir -p `dirname $@` || true
>         $(call setup-fs)

lianxiang, is there any update here?

Comment 5

[Peipei Cheng (needinfo if you need my action)]

[Blocking Requested - why for this release]: block v1.4 since it break smoketest

Comment 6

[James Zhang (Spreadtrum)]

It's gonk-misc issue, not vendor issue.

Comment 7

[James Zhang (Spreadtrum)]

Kaizhen, please review Lianxiang's patch and land it on v1.4.

Comment 8

[James Zhang (Spreadtrum)]

Telenor Andreas need not update binary, we should build it then use it.

Comment 9

[Zhou Lianxiang]

Attachment: gonk-misc-ota-update-binary-patch

Comment 10

[Andreas Pehrson [:pehrsons]]

Peipei, please confirm that I understand this correctly. If so, the minimized steps to reproduce are:

1. Use AOSP updater from source (https://bugzilla.mozilla.org/show_bug.cgi?id=1051191)
2. `rm -rf out/`
3. `./build.sh gecko-update-fota` (or gecko-update-fota-full)

Expected: builds an update package
Actual: build fails because $(PRODUCT_OUT)/system/bin/updater does not exist

---

My make skills are a bit limited so I am not sure what is the best approach here. Should we put the dependency on the updater in gonk-misc or in the device config? Alexandre, ideas?

Comment 11

[Peipei Cheng (needinfo if you need my action)]

lianxiang, could you please help answer andreas' quetsion in comment 10?

Comment 12

[Wayne Chang [:wchang]]

KaiZhen,

Can you check this bug?

Comment 13

[:gerard-majax]

Can someone do:

> make -np > make.txt

So that we can see the make targets?

Comment 14

[Zhou Lianxiang]

(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #10)
> Peipei, please confirm that I understand this correctly. If so, the
> minimized steps to reproduce are:
> 
> 1. Use AOSP updater from source
> (https://bugzilla.mozilla.org/show_bug.cgi?id=1051191)
> 2. `rm -rf out/`
> 3. `./build.sh gecko-update-fota` (or gecko-update-fota-full)
> 
> Expected: builds an update package
> Actual: build fails because $(PRODUCT_OUT)/system/bin/updater does not exist
> 
> ---
> 
> My make skills are a bit limited so I am not sure what is the best approach
> here. Should we put the dependency on the updater in gonk-misc or in the
> device config? Alexandre, ideas?

The patch in comment 9 add the dependent to compile the updater.

Comment 15

[:gerard-majax]

Zhou, I'm not sure your patch is the best way to fix this.

FYI, on devices I hack on, updater make target comes from this makefile bootable/recovery/updater/Android.mk

Do you mind checking why your recovery does not makes this target available before hacking in gonk-misc?

Comment 16

[Kai-Zhen Li [:seinlin][:kli]]

Comment on attachment 8483924 [details] [diff] [review]
gonk-misc-ota-update-binary-patch

James, Lianxiang,

I don't know exactly why TARGET_UPDATE_BINARY is pointed to the one under out dir.
TARGET_UPDATE_BINARY  := $(PRODUCT_OUT)/system/bin/updater

But I think building a binary to out dir should add it to PRODUCT_PACKAGES. With the below patch updater will also be built.


--- a/scx15_sp7715ga/scx15_sp7715gaplus.mk
+++ b/scx15_sp7715ga/scx15_sp7715gaplus.mk
@@ -78,6 +78,9 @@ PRODUCT_VIDEO_WALLPAPERS := none
 PRODUCT_PACKAGES += \
        bluetooth.default
 
+# for fota update-binary TARGET_UPDATE_BINARY
+PRODUCT_PACKAGES += updater
+

Comment 17

[Zhou Lianxiang]

(In reply to Kai-Zhen Li from comment #16)
> Comment on attachment 8483924 [details] [diff] [review]
> gonk-misc-ota-update-binary-patch
> 
> James, Lianxiang,
> 
> I don't know exactly why TARGET_UPDATE_BINARY is pointed to the one under
> out dir.
> TARGET_UPDATE_BINARY  := $(PRODUCT_OUT)/system/bin/updater
> 
> But I think building a binary to out dir should add it to PRODUCT_PACKAGES.
> With the below patch updater will also be built.
> 
> 
> --- a/scx15_sp7715ga/scx15_sp7715gaplus.mk
> +++ b/scx15_sp7715ga/scx15_sp7715gaplus.mk
> @@ -78,6 +78,9 @@ PRODUCT_VIDEO_WALLPAPERS := none
>  PRODUCT_PACKAGES += \
>         bluetooth.default
>  
> +# for fota update-binary TARGET_UPDATE_BINARY
> +PRODUCT_PACKAGES += updater
> +

Sometimes, we need not to make updater if we use a binary which was compiled.

Comment 18

[Kai-Zhen Li [:seinlin][:kli]]

(In reply to Zhou Lianxiang from comment #17)
> 
> Sometimes, we need not to make updater if we use a binary which was compiled.

If we don't need updater in system.img, I think it is not a good idea to compile and put updater at $(PRODUCT_OUT)/system/bin/updater. This will result './build.sh gecko-update-fota-full' include system/bin/updater in update.zip.

Comment 19

[James Zhang (Spreadtrum)]

I'll revert Andreas's commit for scx15_sp7715ga only. Don't block mozilla and telenor borad FOTA test.

commit ebb1ce6af72efe15c6919e2ceb9ee805ce2e5960
Author: james.zhang <[e-mail]>
Date:   Fri Sep 5 10:27:53 2014 +0800

    Bug #329382 revert "[PATCH 3/3] Use source-built update-binary" for scx15_sp7715ga
    
    [bug number  ]
    [root cause  ] https://bugzilla.mozilla.org/show_bug.cgi?id=1061132
    [changes     ] revert
    [side effects]
    [self test   ] yes
    [whether AOB ]
    [reviewers   ]
    
    Change-Id: I57cec734f55c35ebd0e59b341d72c053453a01d2

diff --git a/scx15_sp7715ga/BoardConfig.mk b/scx15_sp7715ga/BoardConfig.mk
index 65a39fa..9d16215 100644
--- a/scx15_sp7715ga/BoardConfig.mk
+++ b/scx15_sp7715ga/BoardConfig.mk
@@ -19,7 +19,7 @@
 
 # nand fstab
 TARGET_RECOVERY_FSTAB := device/sprd/scx15/nand/recovery.fstab
-TARGET_UPDATE_BINARY  := $(PRODUCT_OUT)/system/bin/updater
+TARGET_UPDATE_BINARY  := device/sprd/scx15_sp7715ga/update-binary
 
 # board configs
 
diff --git a/scx15_sp7715ga/update-binary b/scx15_sp7715ga/update-binary
new file mode 100755
index 0000000..4a16640
Binary files /dev/null and b/scx15_sp7715ga/update-binary differ

Comment 20

[Andreas Pehrson [:pehrsons]]

(In reply to Alexandre LISSY :gerard-majax from comment #15)
> Zhou, I'm not sure your patch is the best way to fix this.
> 
> FYI, on devices I hack on, updater make target comes from this makefile
> bootable/recovery/updater/Android.mk
> 
> Do you mind checking why your recovery does not makes this target available
> before hacking in gonk-misc?

Actually, https://android.googlesource.com/platform/bootable/recovery/+/android-4.4.2_r1/updater/Android.mk#14
> # Build only in eng, so we don't end up with a copy of this in /system
> # on user builds.

Comment 21

[Kai-Zhen Li [:seinlin][:kli]]

The change was reverted as in comment 19. Now it should work properly in our build.
Peipei, can you confirm and close this bug? Thanks!

Comment 22

[Peipei Cheng (needinfo if you need my action)]

James, I used sprd build w36.5 but can still reproduce this problem. Could you please help confirm build w36.5 reverted that commit?

BTW, I also tested one latest pvt build (gonk + gecko + gaia). Same issue was reproduced.
------------------------------------------------------------------------------------------------------
Gaia      6018a1c18f0c3eab25aac2ba3064904740591dd2
Gecko     https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/39c9dbc311f7
BuildID   20140911000230
Version   30.0
ro.build.version.incremental=eng.cltbld.20140911.034425
ro.build.date=Thu Sep 11 03:44:36 EDT 2014

Comment 23

[Andreas Pehrson [:pehrsons]]

If I understand Peipei's reproduction steps correctly, this happens for gecko/gaia updates only.

The updater that was patched (see comment 21) is for FOTA updates and is only run in recovery mode. The error here happens while still in gecko, so it is completely unrelated.


Relevant lines in the log are:
09-01 17:08:11.930 I/Gecko   (  108): *** AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
09-01 17:08:11.930 I/Gecko   (  108): UpdatePrompt: Update error, state: failed, errorCode: 7
09-01 17:08:11.980 I/Gecko   (  108): UpdatePrompt: Setting gecko.updateStatus: Install Pending
09-01 17:08:12.020 I/Gecko   (  108): *** AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7

Comment 24

[Andreas Pehrson [:pehrsons]]

Status code 7 is WRITE_ERROR:
http://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#132

Peipei, I have a hunch you're out of space. To get around it you should be able to use an sdcard. Could you confirm?

The limited storage on dolphin is the reason we went for incremental FOTA updates for our production device.

Comment 25

[James Zhang (Spreadtrum)]

(In reply to Peipei Cheng from comment #22)
> James, I used sprd build w36.5 but can still reproduce this problem. Could
> you please help confirm build w36.5 reverted that commit?
> 
> BTW, I also tested one latest pvt build (gonk + gecko + gaia). Same issue
> was reproduced.
> -----------------------------------------------------------------------------
> -------------------------
> Gaia      6018a1c18f0c3eab25aac2ba3064904740591dd2
> Gecko     https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/39c9dbc311f7
> BuildID   20140911000230
> Version   30.0
> ro.build.version.incremental=eng.cltbld.20140911.034425
> ro.build.date=Thu Sep 11 03:44:36 EDT 2014

Please use latest build. 36.5 don't include this patch.

Comment 26

[Peipei Cheng (needinfo if you need my action)]

(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #24)
> Status code 7 is WRITE_ERROR:
> http://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/
> nsUpdateService.js#132
> 
> Peipei, I have a hunch you're out of space. To get around it you should be
> able to use an sdcard. Could you confirm?
> 
> The limited storage on dolphin is the reason we went for incremental FOTA
> updates for our production device.

Yes, I confirmed that I had sdcard inserted during my test. And I can see the download process finished and the uncompressing process started. After uncompressing run for a while, there is one error saying that "download failed".
Since the same steps works on an earlier base image, it should be either povb or gonk problem.

James, could you please help provide a new pac so I can test? I will try to get a regression window here.

Comment 27

[Peipei Cheng (needinfo if you need my action)]

James/lianxiang,
It turns out to be sprd platform problem. I did following testing:
w34.5 + pvt gecko/gaia OTA update  --> Pass
w34.5 + pvt latest gonk + pvt gecko/gaia OTA update --> Pass
w36.5 + pvt gecko/gaia OTA update  --> Fail
w36.5 pvt latest gonk + pvt gecko/gaia OTA update --Fail

We still need sprd engineer to take a look.

Comment 28

[Peipei Cheng (needinfo if you need my action)]

(In reply to Peipei Cheng from comment #27)
> James/lianxiang,
> It turns out to be sprd platform problem. I did following testing:
> w34.5 + pvt gecko/gaia OTA update  --> Pass
> w34.5 + pvt latest gonk + pvt gecko/gaia OTA update --> Pass
> w36.5 + pvt gecko/gaia OTA update  --> Fail
> w36.5 pvt latest gonk + pvt gecko/gaia OTA update --Fail
> 
> We still need sprd engineer to take a look.

Note w34.5/w36.5 are sprd base image build. All tests follow the same steps. It should not be gonk problem since case "w34.5 + pvt latest gonk + pvt gecko/gaia OTA update" is a pass.

Comment 29

[Peipei Cheng (needinfo if you need my action)]

why we get WRITE_ERROR according to comment 24?
http://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#132

Comment 30

[Andreas Pehrson [:pehrsons]]

I still think you run out of disk space. Peipei, can you please run 'watch adb shell df' while it's decompressing to monitor the available space on all partitions?

Comment 31

[Peipei Cheng (needinfo if you need my action)]

Thank you very much Andreas!

Yes, it is true that it run out of space on partition /system. What I saw from 'watch adb shell df' is: the ota package will be downloaded to SD card, and then will be decompressed to /system partition. 

With new base image, /system have too little free space, so the decompressing will fail.

James/Lianxiang, could you please check why thew new base image allocate less space for /system ?


New base image (w36.5) when OTA update fail:
------------------------------------------------------------------------------------------------
Filesystem               Size     Used     Free   Blksize
/dev                   107.5M   164.0K   107.3M   4096
/sys/fs/cgroup         107.5M    12.0K   107.5M   4096
/mnt/asec              107.5M     0.0K   107.5M   4096
/mnt/obb               107.5M     0.0K   107.5M   4096
/system                184.5M   133.3M    51.2M   4096
/data                  122.5M     1.5M   121.0M   4096
/cache                   5.2M     2.2M     3.0M   4096
/productinfo             1.1M    44.0K     1.1M   4096
/mnt/internal-sdcard    56.7M    24.0K    56.7M   4096
/mnt/media_rw/sdcard0     3.6G   591.9M     3.1G   32768
/mnt/secure/asec         3.6G   591.9M     3.1G   32768
/storage/sdcard0         3.6G   591.9M     3.1G   32768


Old base image (w34.5) when OTA update pass:
------------------------------------------------------------------------------------------------
Filesystem               Size     Used     Free   Blksize
/dev                   107.5M   164.0K   107.3M   4096
/sys/fs/cgroup         107.5M    12.0K   107.5M   4096
/mnt/asec              107.5M     0.0K   107.5M   4096
/mnt/obb               107.5M     0.0K   107.5M   4096
/system                231.6M   133.0M    98.6M   4096
/data                   75.4M     1.4M    73.9M   4096
/cache                   5.2M     2.5M     2.8M   4096
/productinfo             1.1M    40.0K     1.1M   4096
/mnt/internal-sdcard    56.7M    24.0K    56.7M   4096
/mnt/media_rw/sdcard0     3.6G   575.2M     3.1G   32768
/mnt/secure/asec         3.6G   575.2M     3.1G   32768
/storage/sdcard0         3.6G   575.2M     3.1G   32768

Comment 32

[:gerard-majax]

Peipei, sorry but according to your df dump it seems obvious the new base image has moved some free space from system partition to userdata partition, which makes sense :)

Comment 33

[Peipei Cheng (needinfo if you need my action)]

(In reply to Alexandre LISSY :gerard-majax from comment #32)
> Peipei, sorry but according to your df dump it seems obvious the new base
> image has moved some free space from system partition to userdata partition,
> which makes sense :)

Yes. I can see that. But is there any reason we do this? 

new base image
/system                184.5M   133.3M    51.2M   4096
/data                  122.5M     1.5M   121.0M   4096

old base image
/system                231.6M   133.0M    98.6M   4096
/data                   75.4M     1.4M    73.9M   4096

Comment 34

[Andreas Pehrson [:pehrsons]]

We did some changes to our device (based on dolphin chipset):
* Reduce size of /system from 250 to 200MB
* Increase size of /cache to some 30MB to enable incremental FOTA updates
* Increase size of /data with whatever free space remained, to give users more app storage

Seems like the reference dolphin got the changes as well, except for the cache.

Comment 35

[Peipei Cheng (needinfo if you need my action)]

Andreas, thank you for your clarifying!

Wayne, do you have any suggestion on such situation? It is a smoke test blocker, but customer will use FOTA updates instead.

Comment 36

[Wayne Chang [:wchang]]

James,

Can you revert the change mentioned in comment 34 on the reference Dolphin platform? We should ensure that OTA still works on the reference platform for your other potential launches.

Comment 37

[Wayne Chang [:wchang]]

However, I dont think we should block here since this is vendor configuration. We should make sure that our OTA test can continue again but if we're clear of the reason here we don't need to block the release.

Comment 38

[Kai-Zhen Li [:seinlin][:kli]]

Wayne, our reference dolphin phone use a different device config dir. And need to keep the change in comment 19. Without this FOTA update will fail. The reason of OTA failed is as Andreas mentioned. May be we should re-partition our reference dolphin for OTA issue.

Comment 39

[James Zhang (Spreadtrum)]

I have reverted the system partition size for dolphin reference phone.


Filesystem               Size     Used     Free   Blksize
/dev                   107.5M   164.0K   107.3M   4096
/sys/fs/cgroup         107.5M    12.0K   107.5M   4096
/mnt/asec              107.5M     0.0K   107.5M   4096
/mnt/obb               107.5M     0.0K   107.5M   4096
/system                231.6M   125.7M   105.9M   4096
/data                  103.6M   912.0K   102.7M   4096
/cache                   3.0M   548.0K     2.4M   4096
/productinfo             1.1M    32.0K     1.1M   4096
/mnt/internal-sdcard    56.7M    32.0K    56.7M   4096
/mnt/media_rw/sdcard0     7.3G     1.6G     5.7G   4096
/mnt/secure/asec         7.3G     1.6G     5.7G   4096
/storage/sdcard0         7.3G     1.6G     5.7G   4096

Comment 40

[Peipei Cheng (needinfo if you need my action)]

I would like to close this since the new base image solved this problem.