Closed Bug 935059 Opened 7 years ago Closed 7 years ago

Implement a FOTA update solution for Buri devices

Categories

(Firefox OS Graveyard :: General, defect)

ARM
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(blocking-b2g:1.3+, b2g-v1.3 fixed, b2g-v1.3T fixed, b2g-v1.4 fixed)

RESOLVED FIXED
1.3 C2/1.4 S2(17jan)
blocking-b2g 1.3+
Tracking Status
b2g-v1.3 --- fixed
b2g-v1.3T --- fixed
b2g-v1.4 --- fixed

People

(Reporter: jsmith, Assigned: gerard-majax)

References

Details

(Whiteboard: [systemsfe])

Attachments

(7 files, 1 obsolete file)

Implement a FOTA update solution for Buri devices to allow for in house testing of FOTA updates for Firefox OS.
Per discussion in triage - this needs to block 1.3.
blocking-b2g: --- → 1.3+
should this be for all devices, not just buri's?
(In reply to Tony Chung [:tchung] from comment #2)
> should this be for all devices, not just buri's?

I would say so. I have to figure out how to generalize the process anyway as different devices will need different partition entries in the update script, so it's better to make the update script generic from day one.
(In reply to Gabriele Svelto [:gsvelto] from comment #3)
> (In reply to Tony Chung [:tchung] from comment #2)
> > should this be for all devices, not just buri's?
> 
> I would say so. I have to figure out how to generalize the process anyway as
> different devices will need different partition entries in the update
> script, so it's better to make the update script generic from day one.

We should probably focus first on getting this working on Buri and later, expand to other devices.
While it's only limited to gecko+gaia, I have a script that partly fills the need in bug 933789.
I discussed with dhylands, I think I get the point here, I'll give it a try.
Assignee: nobody → lissyx+mozillians
Whiteboard: [systemsfe]
(In reply to Alexandre LISSY :gerard-majax from comment #6)
> I discussed with dhylands, I think I get the point here, I'll give it a try.

Thanks for picking this up. See also bug 907444 comment 13, bug 907444 comment 60 and bug 907444 comment 61 for some further discussion on this.
(In reply to Gabriele Svelto [:gsvelto] from comment #7)
> (In reply to Alexandre LISSY :gerard-majax from comment #6)
> > I discussed with dhylands, I think I get the point here, I'll give it a try.
> 
> Thanks for picking this up. See also bug 907444 comment 13, bug 907444
> comment 60 and bug 907444 comment 61 for some further discussion on this.

Thanks for those, it's in fact quite interesting.

After talking to dhylands yesterday and digging in the code in the train this morning, there is a couple of limitations in the current fota update producing python script:
 - we need to know the type of system and data partitions
 - we need to know the emmc block device for both
So, to summarize:
 - from bug 907444, the device partitions are too small to be able to perform an in-place update of Gecko and Gaia using the classical MAR package upader
 - we need to build a gecko+gaia android update package, i.e. a package that will be installed in recovery mode using the update-script
 - this gecko+gaia android update package will be packaged as a MAR package to be made available to devices
The creation of the gecko+gaia android package is described by gsvelto in bug 907444 comment 60:

  « I'm modifying the FlashFotaBuilder class in update_tools.py so that when building a Gecko-only FOTA image it will gather only the files contained under 'system/b2g' instead of 'system' and emit an update script that won't format the '/system' and '/data' partitions when run. Instead the update script mounts the partitions and extracts the archive contents over the target 'system/b2g' directory. This extra functionality is then used to assemble the Gecko-only FOTA image »

We still need to get at least the block device to makje the update-script.
So, in AOSP that I'm looking at (from B2G/Nexus S), the partition is loaded from a ZIP file. This ZIP is built through the target 'target-files-package', producing out/target/product/inari/obj/PACKAGING/target_files_intermediates/full_inari-target_files-eng.alex.zip. 

This target depends on two things we might not have: boot.img and recovery.img.

Especially, the device's blocks are recovered from inside this zip, by looking at the RECOVERY/RAMDISK/etc/recovery.fstab file, which contains for my Nexus S:
# mount point	fstype		device

/sdcard		vfat		/dev/block/platform/s3c-sdhci.0/by-name/media
/system		ext4		/dev/block/platform/s3c-sdhci.0/by-name/system
/cache		yaffs2		cache
/data		ext4		/dev/block/platform/s3c-sdhci.0/by-name/userdata
/misc 		mtd 		misc
/boot		mtd		boot
/recovery	mtd		recovery
/bootloader	mtd		bootloader
/radio		mtd		radio
This recovery fstab is provided by the device repo:
$ cat device/samsung/crespo/recovery.fstab 
# mount point	fstype		device

/sdcard		vfat		/dev/block/platform/s3c-sdhci.0/by-name/media
/system		ext4		/dev/block/platform/s3c-sdhci.0/by-name/system
/cache		yaffs2		cache
/data		ext4		/dev/block/platform/s3c-sdhci.0/by-name/userdata
/misc 		mtd 		misc
/boot		mtd		boot
/recovery	mtd		recovery
/bootloader	mtd		bootloader
/radio		mtd		radio
----

However, for Inari/Buri, we have two different files:

$ cat device/qcom/msm7627a/recovery_mmc.fstab 
# mount point	fstype		device			[device2]

/boot		emmc		/dev/block/mmcblk0p8
/cache		ext4		/dev/block/mmcblk0p15
/data		ext4		/dev/block/mmcblk0p13
/recovery       emmc            /dev/block/mmcblk0p16
/misc		emmc		/dev/block/mmcblk0p17
/sdcard		vfat		/dev/block/mmcblk1p1	/dev/block/mmcblk1
/system		ext4		/dev/block/mmcblk0p12
/sys_boot	vfat		/dev/block/mmcblk0p3
----

And,

$ cat device/qcom/msm7627a/recovery.fstab 
# mount point	fstype		device			[device2]

/boot		mtd		boot
/cache		yaffs2		cache
/data		yaffs2		userdata
/misc		mtd		misc
/recovery	mtd		recovery
/sdcard		vfat		/dev/block/mmcblk0p1	/dev/block/mmcblk0
/system		yaffs2		system
----

As far as I can deduce from my Inari, it the second mtd based solution that is used, and mount is done from init.rc:
out/target/product/inari/root/init.rc:# mount mtd partitions
out/target/product/inari/root/init.rc:    mount yaffs2 mtd@system /system
out/target/product/inari/root/init.rc:    mount yaffs2 mtd@system /system ro remount
out/target/product/inari/root/init.rc:    mount yaffs2 mtd@userdata /data nosuid nodev
out/target/product/inari/root/init.rc:    mount yaffs2 mtd@persist /persist nosuid nodev
out/target/product/inari/root/init.rc:    mount yaffs2 mtd@cache /cache nosuid nodev
out/target/product/inari/root/init.rc:    mount yaffs2 mtd@persist /persist nosuid nodev
Okay so as far as I can find for now, we should be able to use the partitions labels when going in recovery and thus in the update-script. Determining filesystem is trivial when we hack into gonk-misc/Android.mk to add the gecko-update-fota target.

As of now, I should have changes to gonk-misc/Android.mk that adds this target and builds a FOTA package put into a MAR. Next step is to hack the package itself as suggested previously by gsveslto.
So building the package itself should be okay by now, and independant from the filesystem and partitions.

$ ./build.sh gecko-update-fota

============================================
PLATFORM_VERSION_CODENAME=AOSP
PLATFORM_VERSION=4.0.4.0.4.0.4
TARGET_PRODUCT=full_inari
TARGET_BUILD_VARIANT=userdebug
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a-neon
HOST_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-3.9.11-030911-generic-x86_64-with-Ubuntu-13.04-raring
HOST_BUILD_TYPE=release
BUILD_ID=OPENMASTER
OUT_DIR=out
============================================

[...]

FOTA Flash ZIP generated: out/target/product/inari/fota-update.zip

Adding file add instructions to file 'update.manifest'
        add: update.zip
        add: precomplete

Adding type instruction to file 'updatev2.manifest'
       type: complete

Concatenating file 'update.manifest' to file 'updatev2.manifest'

Adding file and directory remove instructions from file 'removed-files'

Finished
FOTA Update MAR generated: out/target/product/inari/fota-update.mar

real	1m2.861s
user	0m59.444s
sys	0m3.864s
Run |./flash.sh| to flash all partitions of your device
One can check the update-script, it contains what is expected:
format("yaffs2", "MTD", "mtd@data", 0);
format("yaffs2", "MTD", "mtd@system", 0);
mount("yaffs2", "MTD", "mtd@data", "/data");
mount("yaffs2", "MTD", "mtd@system", "/system");
Next step is to perform the changes documented by bug 907444 comment 60.
Gabriele, you told me you had some hacked code for update_tools.py, I'd be very glad if you could get me a copy :)
Flags: needinfo?(gsvelto)
Dave, looking at bug 907444, I see you only discuss about /system/b2g/. However, we might have dependencies on other directories, for example /system/fonts/. Do we want to handle this kind of case?
Flags: needinfo?(dhylands)
(In reply to Alexandre LISSY :gerard-majax from comment #18)
> Dave, looking at bug 907444, I see you only discuss about /system/b2g/.
> However, we might have dependencies on other directories, for example
> /system/fonts/. Do we want to handle this kind of case?

So my original goal was to just provide what OTA provides, but wrapped up in a FOTA.

If we're going to switch all of our updates to FOTA, then I think it makes sense to think about stuff outside /system/b2g, but otherwise I think it will lead to confusion (i.e. I updated my leo (using OTA) and the fonts look different than what's on my hamachi (using FOTA)).
Flags: needinfo?(dhylands)
(In reply to Dave Hylands [:dhylands] from comment #19)
> (In reply to Alexandre LISSY :gerard-majax from comment #18)
> > Dave, looking at bug 907444, I see you only discuss about /system/b2g/.
> > However, we might have dependencies on other directories, for example
> > /system/fonts/. Do we want to handle this kind of case?
> 
> So my original goal was to just provide what OTA provides, but wrapped up in
> a FOTA.
> 
> If we're going to switch all of our updates to FOTA, then I think it makes
> sense to think about stuff outside /system/b2g, but otherwise I think it
> will lead to confusion (i.e. I updated my leo (using OTA) and the fonts look
> different than what's on my hamachi (using FOTA)).

Not that I want to switch everything, but I remember we had a number of changes around fonts for example, and we might have others (read: emojis, we hacked it trivially at Oslo by just installing a new font). Hence, I think we can have other cases where we might have to make changes in sync with gecko/gaia outside of /system/b2g/.
> Not that I want to switch everything, but I remember we had a number of
> changes around fonts for example, and we might have others (read: emojis, we
> hacked it trivially at Oslo by just installing a new font). Hence, I think
> we can have other cases where we might have to make changes in sync with
> gecko/gaia outside of /system/b2g/.

I have no issues if the script is capable of grabbing more than just whats in /system/b2g, I just think that we should have the ability to limit ourselves to /system/b2g (i.e. emulate what OTA does)

Then we can push the decision upwards :)
(In reply to Alexandre LISSY :gerard-majax from comment #17)
> Gabriele, you told me you had some hacked code for update_tools.py, I'd be
> very glad if you could get me a copy :)

I can't find the modified script on my laptop right now, I'll leave the needinfo flag so when I'll be back at home next week I'll remember to check on my desktop :)
Do we know (and who might know this) if we have devices using EMMC instead of MTD ?
(In reply to Dave Hylands [:dhylands] from comment #21)
> > Not that I want to switch everything, but I remember we had a number of
> > changes around fonts for example, and we might have others (read: emojis, we
> > hacked it trivially at Oslo by just installing a new font). Hence, I think
> > we can have other cases where we might have to make changes in sync with
> > gecko/gaia outside of /system/b2g/.
> 
> I have no issues if the script is capable of grabbing more than just whats
> in /system/b2g, I just think that we should have the ability to limit
> ourselves to /system/b2g (i.e. emulate what OTA does)
> 
> Then we can push the decision upwards :)

I have a set of changes that are ready and more or less should work for this usecase.
So I'm able to build both now:

alex@portable-alex:~/codaz/Mozilla/b2g/devices/Inari/B2G$ ./build.sh gecko-update-fota && ./build.sh gecko-update-fota-full

[...]

FOTA Flash ZIP generated: out/target/product/inari/fota-update.zip

Adding file add instructions to file 'update.manifest'
        add: update.zip
        add: precomplete

Adding type instruction to file 'updatev2.manifest'
       type: complete

Concatenating file 'update.manifest' to file 'updatev2.manifest'

Adding file and directory remove instructions from file 'removed-files'

Finished
FOTA Update MAR generated: out/target/product/inari/fota-update.mar

real	0m24.233s
user	0m22.336s
sys	0m2.188s
Run |./flash.sh| to flash all partitions of your device

[...]

FOTA Flash ZIP generated: out/target/product/inari/fota-update-full.zip

Adding file add instructions to file 'update.manifest'
        add: update.zip
        add: precomplete

Adding type instruction to file 'updatev2.manifest'
       type: complete

Concatenating file 'update.manifest' to file 'updatev2.manifest'

Adding file and directory remove instructions from file 'removed-files'

Finished
FOTA Update MAR generated: out/target/product/inari/fota-update-full.mar

real	0m45.597s
user	0m43.520s
sys	0m2.612s
Run |./flash.sh| to flash all partitions of your device

alex@portable-alex:~/codaz/Mozilla/b2g/devices/Inari/B2G$ unzip -l out/target/product/inari/fota-update.zip | tail -n1
 68132584                     77 files

alex@portable-alex:~/codaz/Mozilla/b2g/devices/Inari/B2G$ unzip -l out/target/product/inari/fota-update-full.zip | tail -n1
142598964                     644 files

And this can be customized to include others subdirectories, e.g. |B2G_FOTA_DIRS="system/b2g system/fonts" ./build.sh gecko-update-fota| will provide a FOTA update containing /system/b2g and /system/fonts

As far as I could visually check, the update-script was okay. Now need a device to break :)
This is a sample of updater-script produced when only updating /system/b2g and /system/fonts:

mount("yaffs2", "MTD", "mtd@data", "/data");
mount("yaffs2", "MTD", "mtd@system", "/system");
delete_recursive("/system/b2g");
delete_recursive("/system/fonts");
package_extract_dir("system", "/system");
symlink("Roboto-Bold.ttf", "/system/fonts/DroidSans-Bold.ttf");
symlink("Roboto-Regular.ttf", "/system/fonts/DroidSans.ttf");
set_perm_recursive(0, 0, 0755, 0644, "/system");
unmount("/data");
unmount("/system");
FYI it looks like we lack a change to ensure correct permissions:

@@ -174,6 +174,9 @@ static struct fs_path_config android_dirs[] = {
 ** and will allow partial matches.
 */
 static struct fs_path_config android_files[] = {
+    { 00755, AID_ROOT,      AID_ROOT,      "system/b2g/b2g" },
+    { 00755, AID_ROOT,      AID_ROOT,      "system/b2g/updater" },
+    { 00755, AID_ROOT,      AID_ROOT,      "system/b2g/plugin-container" },
     { 00440, AID_ROOT,      AID_SHELL,     "system/etc/init.goldfish.rc" },
     { 00550, AID_ROOT,      AID_SHELL,     "system/etc/init.goldfish.sh" },
     { 00440, AID_ROOT,      AID_SHELL,     "system/etc/init.trout.rc" },

This is present for the Fugu device, not for Inari.
The same fix is available on the system/core repository used by mako: https://github.com/mozilla-b2g/platform_system_core/commit/542d1f59dc331b472307e5bd043101d14d5a3a3e
(In reply to Alexandre LISSY :gerard-majax from comment #28)
> This is a sample of updater-script produced when only updating /system/b2g
> and /system/fonts [...]

This looks great! Just a couple of additions mentioned by :dhylands in bug 907444 comment 61 (we could keep them for a follow-up but I'm writing them down here for completeness):

- Before doing anything else if /system/b2g.bak exists rename it to /system/b2g to fix failed OTA updates. I'm not sure if we should then proceed with the FOTA update or just skip everything else:

if file_exists("/system/b2g.bak") then
  delete_recursive("/system/b2g")
  run_commend("mv", "/system/b2g.bak", /system/b2g")
  abort() # Or shall we proceed?
endif

- Remove /system/b2g/libdmd.so before unpacking /system/b2g to avoid picking up a stale libdmd.so when switching from a dmd-enabled build to a non-dmd one:

delete("/system/b2g/libdmd.so")

- Remove directory tree /syste/b2g/updated (if it exists) This might get created from an OTA update that hasn't been applied yet:

delete_recursive("/system/b2g/updated")

(In reply to Alexandre LISSY :gerard-majax from comment #29)
> FYI it looks like we lack a change to ensure correct permissions:

Nice catch! I think I've actually hit this problem already once. As for testing I was wondering if we could use the emulator instead of a device. Alternatively I could do some testing for you on the score of obsoleted devices I have around; I'll have to give them back anyway so if I brick one it's not a big issue. BTW since you're actively working on it I'm changing the bug status to ASSIGNED.
Status: NEW → ASSIGNED
Flags: needinfo?(gsvelto)
(In reply to Gabriele Svelto [:gsvelto] from comment #31)
> (In reply to Alexandre LISSY :gerard-majax from comment #28)
> > This is a sample of updater-script produced when only updating /system/b2g
> > and /system/fonts [...]
> 
> This looks great! Just a couple of additions mentioned by :dhylands in bug
> 907444 comment 61 (we could keep them for a follow-up but I'm writing them
> down here for completeness):
> 
> - Before doing anything else if /system/b2g.bak exists rename it to
> /system/b2g to fix failed OTA updates. I'm not sure if we should then
> proceed with the FOTA update or just skip everything else:
> 
> if file_exists("/system/b2g.bak") then
>   delete_recursive("/system/b2g")
>   run_commend("mv", "/system/b2g.bak", /system/b2g")
>   abort() # Or shall we proceed?
> endif
> 
> - Remove /system/b2g/libdmd.so before unpacking /system/b2g to avoid picking
> up a stale libdmd.so when switching from a dmd-enabled build to a non-dmd
> one:
> 
> delete("/system/b2g/libdmd.so")
> 
> - Remove directory tree /syste/b2g/updated (if it exists) This might get
> created from an OTA update that hasn't been applied yet:
> 
> delete_recursive("/system/b2g/updated")
> 
> (In reply to Alexandre LISSY :gerard-majax from comment #29)
> > FYI it looks like we lack a change to ensure correct permissions:
> 
> Nice catch! I think I've actually hit this problem already once. As for
> testing I was wondering if we could use the emulator instead of a device.
> Alternatively I could do some testing for you on the score of obsoleted
> devices I have around; I'll have to give them back anyway so if I brick one
> it's not a big issue. BTW since you're actively working on it I'm changing
> the bug status to ASSIGNED.

Thanks for your feedback, the checks you are referring to were my last target before putting those bugs in review. But I want to make sure before that the code behave as expected on a device that uses Ext4 :)
So:
 - works well on Inari tree, which uses yaffs2 and has no recovery fstab, thus using default MTD partitions labels
 - fixed to work on Nexus 4 tree, which uses ext4 and has an Android/Linux fstab file

I'm still ensuring that it correctly detects partitions from recovery fstab (format of the fstab file differs), with the Nexus S case.
I just updated the gonk-misc pull request: there was a race condition with some variables, defined or not depending on the order of the includes. I changed the detection of filesystem and partitions to Makefile macros, called on the .zip generation targets. As far as I can test, this produces valid filesystem/partitions detections for the following cases (scoped output of the changes to makefile):

 - inari (no recovery, yaffs2/mtd):
Using yaffs2 filesystem
Mounting /system from "mtd@system"
Mounting /data   from "mtd@data"
Generating FOTA update package
FOTA Flash ZIP generated: out/target/product/inari/fota-update.zip

 - nexus s (recovery fstab, ext4/emmc)
Using ext4 filesystem
Extracting partitions from recovery fstab
Mounting /system from /dev/block/platform/s3c-sdhci.0/by-name/system
Mounting /data   from /dev/block/platform/s3c-sdhci.0/by-name/userdata
Generating FOTA update package
FOTA Flash ZIP generated: out/target/product/crespo/fota-update.zip

 - nexus 4 (linux fstab, ext4/emmc)
Using ext4 filesystem
Extracting partitions from linux fstab
Mounting /system from /dev/block/platform/msm_sdcc.1/by-name/system
Mounting /data   from /dev/block/platform/msm_sdcc.1/by-name/userdata
Generating FOTA update package
FOTA Flash ZIP generated: out/target/product/mako/fota-update.zip
After fighting a couple of hours with a Buri this morning, I'm able to:
 - build the fota update zip
 - push it on the device
 - trigger recovery update

This can be summarized as the following set of coommands:
$ ./build.sh gecko-update-fota
$ adb push out/target/product/hamachi/fota-update/zip /data/local/tmp/
$ adb shell "echo 'boot-recovery ' > /cache/recovery/command"
$ adb shell "echo '--update_package=/data/local/tmp/fota-update.zip' >> /cache/recovery/command"
$ adb shell "echo 'reboot' >> /cache/recovery/command"

Wait a couple of seconds, then
$ adb reboot recovery

I'm now stuck with an error in the recovery process:
I:Set boot command ""
Finding update package...
I:Update location: /data/local/tmp/fota-update.zip
Opening update package...
E:Failed to load keys
Installation aborted.
After fighting yesterday with Buri, I used a Keon (which has a recovery with testkeys). Apart from the fact that this device has no recovery_fstab file set during the build process (which led me to introduce SYSTEM_PARTION and DATA_PARTITION environment variables as a last resort for this case), we are now able to produce a partial update and apply it correctly.

So for example, for Keon:
$ B2G_FOTA_DIRS="system/b2g system/fonts" SYSTEM_PARTITION="/dev/block/mmcblk0p12" DATA_PARTITION="/dev/block/mmcblk0p13" ./build.sh gecko-update-fota

This will produce both out/target/product/keon/fota/partial/update.zip and out/target/product/keon/fota-update.mar. Then one can issue this to perform the update:
$ adb push out/target/product/keon/fota/partial/update.zip /cache/recovery/update.zip && adb shell "echo 'boot-recovery' > /cache/recovery/command" && adb shell "echo '--update_package=/cache/recovery/update.zip' >> /cache/recovery/command" && adb reboot recovery
(In reply to Gabriele Svelto [:gsvelto] from comment #31)

[...]

> - Before doing anything else if /system/b2g.bak exists rename it to
> /system/b2g to fix failed OTA updates. I'm not sure if we should then
> proceed with the FOTA update or just skip everything else:
> 
> if file_exists("/system/b2g.bak") then
>   delete_recursive("/system/b2g")
>   run_commend("mv", "/system/b2g.bak", /system/b2g")
>   abort() # Or shall we proceed?
> endif

While I do find some reference about a file_exists() in Edify, I cannot find this function on any of the devices we build: hamachi, inari, keon, nexus s, nexus 4.

Maybe we can sort out something based on run_program(), ifthen, and stat.

> 
> - Remove /system/b2g/libdmd.so before unpacking /system/b2g to avoid picking
> up a stale libdmd.so when switching from a dmd-enabled build to a non-dmd
> one:
> 
> delete("/system/b2g/libdmd.so")

This should be okay now.

> 
> - Remove directory tree /syste/b2g/updated (if it exists) This might get
> created from an OTA update that hasn't been applied yet:
> 
> delete_recursive("/system/b2g/updated")

This should be okay also now.


FYI, this is the script obtained for the run documented previously:
> mount("ext4", "EMMC", "/dev/block/mmcblk0p13", "/data");
> mount("ext4", "EMMC", "/dev/block/mmcblk0p12", "/system");
> 
> delete_recursive("/system/b2g");
> delete_recursive("/system/fonts");
> delete("/system/b2g/libdmd.so");
> delete_recursive("/system/b2g/updated");
> 
> package_extract_dir("system", "/system");
> 
> symlink("Roboto-Bold.ttf", "/system/fonts/DroidSans-Bold.ttf");
> symlink("Roboto-Regular.ttf", "/system/fonts/DroidSans.ttf");
> 
> set_perm_recursive(0, 0, 0755, 0644, "/system/b2g");
> set_perm(0, 0, 0755, "/system/b2g/b2g");
> set_perm(0, 0, 0755, "/system/b2g/plugin-container");
> set_perm(0, 0, 0755, "/system/b2g/updater");
> set_perm_recursive(0, 0, 0755, 0644, "/system/fonts");
> 
> unmount("/data");
> unmount("/system");
Confirmed: line 6 col 34: unknown function "file_exists"
line 6 col 293: syntax error, unexpected ENDIF
line 29 col 1: syntax error, unexpected $end
There seems to be no mv/stat available in recovery mode.
(In reply to Alexandre LISSY :gerard-majax from comment #38)
> Confirmed: line 6 col 34: unknown function "file_exists"
> line 6 col 293: syntax error, unexpected ENDIF
> line 29 col 1: syntax error, unexpected $end
>
> There seems to be no mv/stat available in recovery mode.

That's annoying... I went through the update-binary code here http://is.gd/hVqg71 and there's no trace of file_exists() besides within the documentation nor any other way of moving a file around. However the code implementing this seems trivially simple, see starting from line 1306:

https://android.googlesource.com/platform/bootable/recovery/+/master/updater/install.c

We could move the missing parts in a follow up bug and try rolling our own updater-binary fork with the bits we miss. What do you think?
(In reply to Gabriele Svelto [:gsvelto] from comment #40)
> (In reply to Alexandre LISSY :gerard-majax from comment #38)
> > Confirmed: line 6 col 34: unknown function "file_exists"
> > line 6 col 293: syntax error, unexpected ENDIF
> > line 29 col 1: syntax error, unexpected $end
> >
> > There seems to be no mv/stat available in recovery mode.
> 
> That's annoying... I went through the update-binary code here
> http://is.gd/hVqg71 and there's no trace of file_exists() besides within the
> documentation nor any other way of moving a file around. However the code
> implementing this seems trivially simple, see starting from line 1306:
> 
> https://android.googlesource.com/platform/bootable/recovery/+/master/updater/
> install.c
> 
> We could move the missing parts in a follow up bug and try rolling our own
> updater-binary fork with the bits we miss. What do you think?

I found an alternative way, relying only on run_program() and mv (which is not in /sbin of recovery but in /system/bin/). Besides, I'm not a big fan of hacking upstream changes.
As far as I could test, everything seems to be working as expected.
Attachment #8340462 - Flags: review?(gsvelto)
Attachment #8340463 - Flags: review?(gsvelto)
For Inari, it looks like the recovery lacks two things:
 - fstab
 - keys

$ cp device/qcom/msm7627a/recovery.fstab out/target/product/inari/recovery/root/etc/
$ cp out/target/product/inari/obj/PACKAGING/ota_keys_intermediates/keys out/target/product/inari/recovery/root/res/
$ out/host/linux-x86/bin/mkbootimg  --kernel out/target/product/inari/kernel  --ramdisk out/target/product/inari/ramdisk-recovery.img --cmdline "androidboot.hardware=otoro" --base 200000 --pagesize 2048 --output out/target/product/inari/recovery.img
$ out/host/linux-x86/bin/mkbootimg  --kernel out/target/product/inari/kernel  --ramdisk out/target/product/inari/ramdisk-recovery.img --cmdline "androidboot.hardware=otoro" --base 200000 --pagesize 2048 --output out/target/product/inari/recovery.img

This should produce a working recovery image.
Comment on attachment 8340462 [details] [review]
Link to Github https://github.com/mozilla-b2g/B2G/pull/303: adding partial fota support

This looks good to me, I've tried testing this on both the inari and the hamachi/buri. On the hamachi/buri I've hit the same issues and on the inari I had no luck as the vendor image I had flashed before the upgrade was non-rooted :-/ The update image was generated correctly though so that's good enough.

BTW once we land this we should update the documentation at https://wiki.mozilla.org/B2G/Updating
Attachment #8340462 - Flags: review?(gsvelto) → review+
Comment on attachment 8340463 [details] [review]
Link to Github https://github.com/mozilla-b2g/gonk-misc/pull/137: adding partial and full fota make targets

This looks good to me but I don't think I can r+ build-system patches so I'm handing this one over to :mwu.
Attachment #8340463 - Flags: review?(gsvelto) → review?(mwu)
I just had a look at the script in the ZTE OPEN v1.1 update, it does a couple of nice things that maybe we want to add:
 - checking ro.product.device and ro.build.product and others
 - embed a couple of .img to flash at the same time: boot.img, recovery.img

We might want to enable this kind of feature on our side too.
(In reply to Alexandre LISSY :gerard-majax from comment #46)
>  - checking ro.product.device and ro.build.product and others

You mean to prevent accidental flashing to the wrong device? That sounds like a nice-to-have thing.

>  - embed a couple of .img to flash at the same time: boot.img, recovery.img

I wonder if this would be out of scope since we're trying to just reproduce the regular OTA functionality under FOTA.
(In reply to Gabriele Svelto [:gsvelto] from comment #47)
> (In reply to Alexandre LISSY :gerard-majax from comment #46)
> >  - checking ro.product.device and ro.build.product and others
> 
> You mean to prevent accidental flashing to the wrong device? That sounds
> like a nice-to-have thing.

Yep.

> 
> >  - embed a couple of .img to flash at the same time: boot.img, recovery.img
> 
> I wonder if this would be out of scope since we're trying to just reproduce
> the regular OTA functionality under FOTA.

It's more or less the same codepath that produces the update.zip file, so ...
Target Milestone: --- → 1.3 C1/1.4 S1(20dec)
Marshall, do you mind reviewing this? I think you know this code better than I do.
Flags: needinfo?(marshall)
Here's a quick recap of my efforts testing this build. I've tried upgrading my hamachi from 1.2 to master starting either with the vendor v1.2_US_20131115.cfg and IS_V1.2_Dev_20131209.cfg builds or from my own 1.2 build. In all cases the update failed with the "E:Failed to load keys" error.
(In reply to Gabriele Svelto [:gsvelto] from comment #50)
> Here's a quick recap of my efforts testing this build. I've tried upgrading
> my hamachi from 1.2 to master starting either with the vendor
> v1.2_US_20131115.cfg and IS_V1.2_Dev_20131209.cfg builds or from my own 1.2
> build. In all cases the update failed with the "E:Failed to load keys" error.

Thanks for your efforts.

This really sucks a LOT, being stuck because of lack of openess :(
Yup I can take it
Flags: needinfo?(marshall)
Comment on attachment 8340463 [details] [review]
Link to Github https://github.com/mozilla-b2g/gonk-misc/pull/137: adding partial and full fota make targets

Looks mostly good to me. Out of curiosity, is there a reason why there is so much logic in the Makefile, rather than in python?
Attachment #8340463 - Flags: review?(mwu) → review+
(In reply to Marshall Culpepper [:marshall_law] from comment #53)
> Comment on attachment 8340463 [details] [review]
> Link to Github https://github.com/mozilla-b2g/gonk-misc/pull/137: adding
> partial and full fota make targets
> 
> Looks mostly good to me. Out of curiosity, is there a reason why there is so
> much logic in the Makefile, rather than in python?

Because I didn't felt like this logic's place was in the python script but rather at a higher level. If it's too cryptic, I can document/make it more readable maybe?
Maybe I should explain it a bit more: most of this logic actually just deals with Makefile's variables defined at several places, and that might be device-dependant. Accessing those values outside of the Makefile was not the most easy way to get them.
Cool, that works for me!
Here's the recovery log I got while trying to update my buri/hamachi device.
I've unpacked the recovery partition of my hamachi with a v1.2_US_20131115.cfg build on it and found out that /res/keys exists but contains the output from our fake Java: "java fake 1.6.0". The recovery partition is under /dev/mtd/mtd7 and I've unpacked it with unmkbootimg http://whiteboard.ping.se/Android/Unmkbootimg and modified it to use the AOSP test keys instead.

As it turns out the image was generated with a modified mkbootimg so I've attached a patch - attachment 8348774 [details] [diff] [review] - with the necessary changes to rebuild the partition image.

I've then flashed this new image and tried the update process again but it still doesn't work. In fact it doesn't even start the update, it just reboots in fastboot mode and gets stuck there. One thing I noticed while doing this is that the newly generated recovery partition is around half the size of the original one but I have no idea why since they *seem* to contain the same stuff. I'll try extracting the recovery partition from the IS_V1.2_Dev_20131209.cfg build and see if it makes a difference.
Thanks, that's more or less my experience ...
(In reply to Alexandre LISSY :gerard-majax from comment #60)
> Thanks, that's more or less my experience ...

I've inspected the recovery partition image and the version I pull from the device is actually more or less the size I would expect (~5MiB) plus 0xff bytes padding the rest of the image all the way to 10MiB mark. I'm not sure if it's relevant or not but I might try padding my image too just to see if it makes a difference.
Duplicate of this bug: 952196
So we should get a recovery with testkeys soon to be able to test it. I should get a buri in the next days, so I can finish this during the holidays :)
Attached file log (obsolete) —
I did a final test using the recovery partition provided by Vance Chen and this time it worked! I've attached the log. The test was done using the following commands:


$ ./build.sh gecko-update-fota-full
$ adb push out/target/product/hamachi/fota-update/update.zip /data/local/tmp/
$ adb shell "echo 'boot-recovery --update_package=/data/local/tmp/fota-update.zip' > /cache/recovery/command"
$ adb shell "echo 'reboot' >> /cache/recovery/command"
$ adb reboot recovery
(In reply to Gabriele Svelto [:gsvelto] from comment #64)
> Created attachment 8350612 [details]
> log
> 
> I did a final test using the recovery partition provided by Vance Chen and
> this time it worked! I've attached the log. The test was done using the
> following commands:
> 
> 
> $ ./build.sh gecko-update-fota-full
> $ adb push out/target/product/hamachi/fota-update/update.zip /data/local/tmp/
> $ adb shell "echo 'boot-recovery
> --update_package=/data/local/tmp/fota-update.zip' > /cache/recovery/command"
> $ adb shell "echo 'reboot' >> /cache/recovery/command"
> $ adb reboot recovery

Thanks !

Having looked at the log file, we have the issue I expected, i.e., that we are using "mtd@system" and "mtd@data" partition names which are not the good ones. Hence, the mount() fails: 'mount: no mtd partition named "mtd@data"mount: no mtd partition named "mtd@system"'.

This means we need to throw an error in Makefile when we have those values. We also lack some hardening on the format and mount commands.
I've updated the PR to add some assert() on formatting and mounting.
Attached file log_assertion
Tested the new PR, everything seem to work as expected with the update bailing out when we hit the assertions about partition names. Now I'm going to test with the correct partition names.
Attached file log
This is a log when running a full FOTA upgrade built with the following command:

./build.sh -B B2G_FOTA_SYSTEM_PARTITION=system B2G_FOTA_DATA_PARTITION=data gecko-update-fota-full

Unfortunately it still seems to hit an issue when dealing with the data partition. Could this be because the update ZIP is held inside of it?
Attachment #8350612 - Attachment is obsolete: true
I've tried partial updates and they also fail because of the assertion around mounting the data partition. Building the partial updates with

./build.sh -B B2G_FOTA_DATA_PARTITION=data gecko-update-fota

... didn't seem to help.
Attached file last_log_buri
This is the recovery log from a Buri that I could get and flash with a testkeys recovery. I added checks in the updater-script to make sure that:
 - if /system or /data is not mounted, we assert(mount()) it
 - if /system or /data is mounted, we just do nothing but ui_print() it.

As far as I can say and according to the log, it works quite well. I'm checking with some other rounds of updates.
Depends on: 952936
Depends on: 952535
No longer depends on: 952535
As far as I can tell, this is working nice on the Buri I have been experimenting on during the holidays. I pushed some updates to the pull requests.
We can also make use of this tool to test updates:
$ ./build.sh gecko-update-fota

Then:
$ python tools/update-tools/test-update.py --fota-update out/target/product/inari/fota-update.mar

As far as I'm currently testing, it works well:
 0. Retrieves FOTA from local httpd on the device. Please note that you might need to have working network to get the ability to tap the "check updates" button
 1. Uncompress and prepare FOTA on /mnt/sdcard/updates/
 2. Reboot into recovery mode, apply FOTA update package in /mnt/sdcard/updates/fota/update.zip
 3. FOTA applied, rebooting device
I've pushed new changes to the Android.mk, adding dependencies against system.img to make sure the updates matches uptodate code.
Duplicate of this bug: 956237
Comment on attachment 8340462 [details] [review]
Link to Github https://github.com/mozilla-b2g/B2G/pull/303: adding partial fota support

I'm resetting the review flag since the code evolved quite a bit.
Attachment #8340462 - Flags: review+ → review?(gsvelto)
Comment on attachment 8340463 [details] [review]
Link to Github https://github.com/mozilla-b2g/gonk-misc/pull/137: adding partial and full fota make targets

I'm resetting the review flag since the code evolved quite a bit.
Attachment #8340463 - Flags: review+ → review?(marshall)
Comment on attachment 8340462 [details] [review]
Link to Github https://github.com/mozilla-b2g/B2G/pull/303: adding partial fota support

Excellent stuff. I like how you added print-outs for most of the operations, this will make post-update diagnostics much easier.
Attachment #8340462 - Flags: review?(gsvelto) → review+
Target Milestone: 1.3 C1/1.4 S1(20dec) → 1.3 C2/1.4 S2(17jan)
I'm jumping in here from bug 956584 - I have a ZTE Open that no longer responds to adb or fastboot from a Fedora Linux host and no longer accepts ZTE-signed updates from SD cards. I've got the scripts already to build from Mozilla source so I can test much of this out.
Comment on attachment 8340463 [details] [review]
Link to Github https://github.com/mozilla-b2g/gonk-misc/pull/137: adding partial and full fota make targets

looks good!
Attachment #8340463 - Flags: review?(marshall) → review+
(In reply to znmeb from comment #78)
> I'm jumping in here from bug 956584 - I have a ZTE Open that no longer
> responds to adb or fastboot from a Fedora Linux host and no longer accepts
> ZTE-signed updates from SD cards. I've got the scripts already to build from
> Mozilla source so I can test much of this out.

That's very kind of you, but I fear that if you lost adb and fastboot access, this bug is not very helpful for you. You might be able to do by putting update.zip on your sdcard, boot into recovery and apply update from sdcard.
Flags: needinfo?(znmeb)
Target Milestone: 1.3 C2/1.4 S2(17jan) → 1.3 C1/1.4 S1(20dec)
Target Milestone: 1.3 C1/1.4 S1(20dec) → 1.3 C2/1.4 S2(17jan)
(In reply to Alexandre LISSY :gerard-majax from comment #80)
> (In reply to znmeb from comment #78)
> > I'm jumping in here from bug 956584 - I have a ZTE Open that no longer
> > responds to adb or fastboot from a Fedora Linux host and no longer accepts
> > ZTE-signed updates from SD cards. I've got the scripts already to build from
> > Mozilla source so I can test much of this out.
> 
> That's very kind of you, but I fear that if you lost adb and fastboot
> access, this bug is not very helpful for you. You might be able to do by
> putting update.zip on your sdcard, boot into recovery and apply update from
> sdcard.

Yeah ... the build requires adb to grab binary files from the ZTE :-(

How do I get an update.zip that will function from the SD card?
Flags: needinfo?(znmeb)
(In reply to znmeb from comment #82)
> (In reply to Alexandre LISSY :gerard-majax from comment #80)
> > (In reply to znmeb from comment #78)
> > > I'm jumping in here from bug 956584 - I have a ZTE Open that no longer
> > > responds to adb or fastboot from a Fedora Linux host and no longer accepts
> > > ZTE-signed updates from SD cards. I've got the scripts already to build from
> > > Mozilla source so I can test much of this out.
> > 
> > That's very kind of you, but I fear that if you lost adb and fastboot
> > access, this bug is not very helpful for you. You might be able to do by
> > putting update.zip on your sdcard, boot into recovery and apply update from
> > sdcard.
> 
> Yeah ... the build requires adb to grab binary files from the ZTE :-(
> 
> How do I get an update.zip that will function from the SD card?

I fear that you are stuck. Maybe you can extract those blobs from ZTE-provided images ?
(In reply to Alexandre LISSY :gerard-majax from comment #83)
> (In reply to znmeb from comment #82)
> > (In reply to Alexandre LISSY :gerard-majax from comment #80)
> > > (In reply to znmeb from comment #78)
> > > > I'm jumping in here from bug 956584 - I have a ZTE Open that no longer
> > > > responds to adb or fastboot from a Fedora Linux host and no longer accepts
> > > > ZTE-signed updates from SD cards. I've got the scripts already to build from
> > > > Mozilla source so I can test much of this out.
> > > 
> > > That's very kind of you, but I fear that if you lost adb and fastboot
> > > access, this bug is not very helpful for you. You might be able to do by
> > > putting update.zip on your sdcard, boot into recovery and apply update from
> > > sdcard.
> > 
> > Yeah ... the build requires adb to grab binary files from the ZTE :-(
> > 
> > How do I get an update.zip that will function from the SD card?
> 
> I fear that you are stuck. Maybe you can extract those blobs from
> ZTE-provided images ?

If I had a list of the files I need, I could try that. But there's very little if any documentation about that floating around. I can unzip archives, change files and re-zip them, but I don't know which files need to go where. I suspect by the time I track down ZTE documentation my Geeksphone Revolution will have arrived and I'll be back on the air. Incidentally, I did put in a support request at ZTE but haven't heard anything from them.
You have a list as of the extract-files.sh shell script available in the inari's device/ subdir.
(In reply to Alexandre LISSY :gerard-majax from comment #85)
> You have a list as of the extract-files.sh shell script available in the
> inari's device/ subdir.

Ah ,,, got it! I ended up extracting the entire 'system' directory from the ZTE update zip file and copying the whole tree into backup-inari. The build script is now running - I should have image files tomorrow.
(In reply to Alexandre LISSY :gerard-majax from comment #81)
> https://github.com/mozilla-b2g/B2G/commit/
> 4aabed65e0147d7c44e64bce53b2f829a67bfcd1
> https://github.com/mozilla-b2g/gonk-misc/commit/
> 6f0e016f221034af391ea2ca47509ebd2767df67

John, can you please assist with the uplift? :)
Flags: needinfo?(jhford)
Depends on: 958202
There is only one branch of B2G, master :(

gonk-misc:
[v1.3 06294fc] Merge pull request #137 from lissyx/bug935059
Flags: needinfo?(jhford)
Blocks: 958202
No longer depends on: 958202
i was trying to test this on 1.4 build, and it's failing on a system partition mount error.   Can anyone else confirm this?   need to check if this is a bug or a device issue.

Device
------
host-2-36:Scripts tchung$ ./check_versions.sh 
Gaia      ac94739a01d64e86890f54f1795ff8e82b8e7939
Gecko     https://hg.mozilla.org/mozilla-central/rev/1f170f9fead0
BuildID   20140205040203
Version   30.0a1
ro.build.version.incremental=324

Logcat:
-------
02-06 20:27:09.541: D/wpa_supplicant(534): nl80211: survey data missing!
02-06 20:27:10.321: I/GeckoUpdater(2389): Progress [ ===============================================    ]
02-06 20:27:10.331: I/GeckoUpdater(2389): Progress [ ================================================   ]
02-06 20:27:10.361: I/GeckoUpdater(2389): Progress [ =================================================  ]
02-06 20:27:10.371: I/GeckoUpdater(2389): Progress [ ================================================== ]
02-06 20:27:10.381: I/GeckoUpdater(2389): Finished applying update
02-06 20:27:10.571: I/GonkAutoMounter(2389): Mounted /system partition as read-only
02-06 20:27:10.581: I/Gecko(134): *** AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
02-06 20:27:10.581: I/Gecko(134): UpdatePrompt: Update error, state: failed, errorCode: 7
02-06 20:27:10.601: I/Gecko(134): UpdatePrompt: Setting gecko.updateStatus: Install Pending
02-06 20:27:10.641: I/Gecko(134): *** AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7


Repro:
1) install 20140205040203 master nightly on Buri
2) settings > check for updates
3) after applying update, verify it fails to apply.
Looks like this is bug 965874.
I also meet the same issue as Tony mentions in comment 89, and that issue is happened while I want to update the Buri device from v1.1 to v1.3 and v1.2 to v1.3. The BuildID of V1.3 is 20140221164003.

- Device Info.
  - V1.1
    - Gaia: 53e2a70d85fb3748d0768218a5efffe5806073f0
    - Gecko: http://hg.mozilla.org/releases/mozilla-b2g18/rev/c630289d6388
    - Build ID: 20131009041203

  - V1.2
    - Gaia: 539a25e1887b902b8b25038c547048e691bd97f6
    - Gecko: https://hg.mozilla.org/releases/mozilla-b2g26_v1_2/rev/c2b030d87d12
    - Build ID: 20140221164002
(In reply to Hubert Lu[:hlu] <hlu@mozilla.com> from comment #91)
> I also meet the same issue as Tony mentions in comment 89, and that issue is
> happened while I want to update the Buri device from v1.1 to v1.3 and v1.2
> to v1.3. The BuildID of V1.3 is 20140221164003.
> 
> - Device Info.
>   - V1.1
>     - Gaia: 53e2a70d85fb3748d0768218a5efffe5806073f0
>     - Gecko: http://hg.mozilla.org/releases/mozilla-b2g18/rev/c630289d6388
>     - Build ID: 20131009041203
> 
>   - V1.2
>     - Gaia: 539a25e1887b902b8b25038c547048e691bd97f6
>     - Gecko:
> https://hg.mozilla.org/releases/mozilla-b2g26_v1_2/rev/c2b030d87d12
>     - Build ID: 20140221164002

Your librecovery is invalid, this has been documented in bug 965874.
You need to log in before you can comment on or make changes to this bug.