Closed Bug 971204 Opened 11 years ago Closed 7 years ago

Applying a FOTA update from old Buri build to a new Buri build keeps the old Buri build

Categories

(Firefox OS Graveyard :: Vendcom, defect)

Product:
Firefox OS Graveyard
Component:
Vendcom
Platform:
ARM
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: jsmith, Unassigned)

Details

(Whiteboard: [systemsfe][POVB])

Attachments

(2 files)

Last Log
5.37 KB, text/plain
Details
dmesg.log
51.49 KB, text/x-log
Details
Build - 2/10/2014 Buri 1.4 STR 1. Check for updates 2. Apply the system update to 2/11/2014 1.4 3. Compare the old vs. new build ID Expected The build ID should be using the 2/11/2014 build. Actual The build ID is using the 2/10/2014 build.
blocking-b2g: --- → 1.3?
Whiteboard: [systemsfe]
I see buildid 20140211004035 and git commit info 2014-02-10 19:39:30, that seems not that bad
(In reply to Alexandre LISSY :gerard-majax from comment #1) > I see buildid 20140211004035 and git commit info 2014-02-10 19:39:30, that > seems not that bad That isn't what I saw. I saw the same build ID before & after.
Attached file Last Log —
Build for testing: * Version: 1.4 * Gecko: 07739c5c874f * Gaia: 9fc36dde3a4a3c5ca200275b68ffb56b4173bec3 * Build ID: 20140210160201
Alexandre mentions that the problem is here: E:Can't mount /updates/fota/update.zip Installation aborted.
The last log exposes the issue: > I:Set boot command "" > Finding update package... > I:Update location: /updates/fota/update.zip > E:unknown volume for path [/updates/fota/update.zip] > E:Can't mount /updates/fota/update.zip > Installation aborted. > mtd: successfully wrote block at 0 > I:Set boot command "" > I:timed out waiting for key input; rebooting. Update location is wrong, in my case it was /sdcard/updates/fota/update.zip Dale or Gabriele, any idea ?
Dale or Gabriele, any idea when we set the path for the recovery command ?
Flags: needinfo?(gsvelto)
Flags: needinfo?(dhylands)
The build.prop shouldn't change unless the system/gonk changes. The gecko/gaia is taken from a different place. see https://github.com/Mozilla-TWQA/B2G-flash-tool/blob/master/check_versions.sh#L74 and https://github.com/Mozilla-TWQA/B2G-flash-tool/blob/master/check_versions.sh#L78 Aki, what does the FOTA update? If we update the whole system.img, then we would have more issues because we don't have the proper configs for the gonk layer.
Flags: needinfo?(aki)
I have no idea. :gsvelto and :dhylands should know.
Flags: needinfo?(aki)
The update service does a logcat here: http://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#3162 of what it thinks the path is. It then calls into the recovery service which I think winds up here: http://dxr.mozilla.org/mozilla-central/source/b2g/components/RecoveryService.js#75 and that then calls into the installFotaUpdate from librecovery.so librecovery is found in the B2G directory, and IIRC it can translate between the path thats in effect when android is running to be the path which will be in effect when the recovery ROM will be running (which because its a totally different environment, may not be the same). So I think we probably need to add some prints to librecovery to figure out exactly how its transforming the path.
Flags: needinfo?(dhylands)
QA Wanted - let's see someone else can reproduce & get a logcat
Keywords: qawanted
Oh - and I seem to recall being burned by this in the past (but I can't remember on which phone). The librecovery.so that we build only gets installed on the phone when doing a full system flash. A regular gecko flash doesn't install it (since librecovery.so goes into /system/lib)
I found the email thread about librecovery. Some portion of the path is set in librecovery/device/hamachi.mk which has: RECOVERY_EXTERNAL_STORAGE := /sdcard I'm not sure what the path is supposed to be under the recovery ROM (for the unagi, I had to change this to /mnt/sdcard).
I tried to update using FOTA locally and i saw this in logcat: *** AUS:SVC UpdateService:applyOsUpdate - Rebooting into recovery to apply FOTA update: /mnt/sdcard/updates/fota/update.zip A:srcPath = '/mnt/sdcard/updates/fota/update.zip' A:getenv(EXTERNAL_STORAGE) = '/mnt/sdcard' A:RECOVERY_EXTERNAL_STORAGE = '/sdcard' X:srcPath = '/mnt/sdcard/updates/fota/update.zip' X:getenv(EXTERNAL_STORAGE) = '/mnt/sdcard' X:RECOVERY_EXTERNAL_STORAGE = '/sdcard' Check 2: destPath = '/sdcard/updates/fota/update.zip' Rebooting into recovery: --update_package=/sdcard/updates/fota/update.zip Rebooting into recovery: --update_package=/sdcard/updates/fota/update.zip and the update failed in the recovery ROM. I got this from /cache/recovery/last_log: Finding update package... I:Update location: /sdcard/updates/fota/update.zip Opening update package... I:1 key(s) loaded from /res/keys Verifying update package... I:comment is 1738 bytes; signature 1720 bytes from end E:failed to verify whole-file signature I:verify_file returned 1 E:signature verification failed Installation aborted. So I obviously need a different base image on my phone. What's the correct base image to use?
Attached file dmesg.log —
For 1.3, 02-11 19:50:05.499: I/Gecko(136): *** AUS:SVC UpdateService:applyOsUpdate - Rebooting into recovery to apply FOTA update: /mnt/sdcard/updates/fota/update.zip : E/(): Device disconnected It then rebooted the phone and it stopped at the system recovery with a red dot ! Attached is the dmesg.log The base build that we use is v1.2-device.cfg using the Teleweb tool.
The red exclamation mark means that the recovery failed. The results can be found in /cache/recovery/last_log I'll get my phone reflashed with the base image and see what I get.
(In reply to Jason Smith [:jsmith] from comment #12) > QA Wanted - let's see someone else can reproduce & get a logcat Naoki confirmed a reproduction. He mentioned that a logcat won't help though, so clearing flag.
Keywords: qawanted
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #16) > Created attachment 8374539 [details] > dmesg.log > > For 1.3, > 02-11 19:50:05.499: I/Gecko(136): *** AUS:SVC UpdateService:applyOsUpdate - > Rebooting into recovery to apply FOTA update: > /mnt/sdcard/updates/fota/update.zip > : E/(): Device disconnected > > It then rebooted the phone and it stopped at the system recovery with a red > dot ! > > Attached is the dmesg.log > > The base build that we use is v1.2-device.cfg using the Teleweb tool. You need a recovery with testkeys. That is not a bug, you need to use the correct image from Teleweb (don't ask me which, but David used it and it's okay on my Buri).
(In reply to Alexandre LISSY :gerard-majax from comment #19) > (In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from > comment #16) > > Created attachment 8374539 [details] > > dmesg.log > > > > For 1.3, > > 02-11 19:50:05.499: I/Gecko(136): *** AUS:SVC UpdateService:applyOsUpdate - > > Rebooting into recovery to apply FOTA update: > > /mnt/sdcard/updates/fota/update.zip > > : E/(): Device disconnected > > > > It then rebooted the phone and it stopped at the system recovery with a red > > dot ! > > > > Attached is the dmesg.log > > > > The base build that we use is v1.2-device.cfg using the Teleweb tool. > > You need a recovery with testkeys. That is not a bug, you need to use the > correct image from Teleweb (don't ask me which, but David used it and it's > okay on my Buri). Actually - Naoki is using the latest base image here, which makes me think this is vendor regression in the base image itself. Tagging as POVB and adding needinfo on Vance to followup with TCL to get this fixed.
blocking-b2g: 1.3? → ---
Component: General → Vendcom
Flags: needinfo?(vchen)
Whiteboard: [systemsfe] → [systemsfe][POVB]
I'm not convinced it should be tagged a vendor regression either. The FOTA updates we are producing are signed with a key. This key is checked for during recovery prior to applying the update. I suspect his device has the productions keys from TCL. But our tree contains only testkeys from AOSP. So the update appears signed with the wrong key. Vance helped us (gabriele and me) getting a base image which has a recovery with testkeys.
Right so we need to make sure that QA and anybody else using FOTA gets the same base image with test keys.
Seems like in order to test our own FOTA mechanism, we need the base image signed with test key. If that is the case, I will ask TCL to release the base image with test key from now on. Please confirm if that is what you want Thanks Vance
Flags: needinfo?(vchen)
I'll check the TeleWeb images to figure out if they're using the AOSP keys or not. AFAIR when me and Alexandre were testing this we were sent a custom boot.img with AOSP keys baked in but that might not have trickled into the TeleWeb images yet.
Flags: needinfo?(gsvelto)
I've just checked the v1.2-device.cfg image and it's recovery partition seems to contain the AOSP test keys so it should verify the FOTA image correctly. Are there later versions available on TeleWeb we're supposed to use?
(In reply to Gabriele Svelto [:gsvelto] from comment #25) > I've just checked the v1.2-device.cfg image and it's recovery partition > seems to contain the AOSP test keys so it should verify the FOTA image > correctly. Are there later versions available on TeleWeb we're supposed to > use? How would I verify that the base image has these test keys? Trying to figure out how we know on QA's side if we've got a base image that does or doesn't have the right test keys.
(In reply to Jason Smith [:jsmith] from comment #26) > (In reply to Gabriele Svelto [:gsvelto] from comment #25) > > I've just checked the v1.2-device.cfg image and it's recovery partition > > seems to contain the AOSP test keys so it should verify the FOTA image > > correctly. Are there later versions available on TeleWeb we're supposed to > > use? > > How would I verify that the base image has these test keys? Trying to figure > out how we know on QA's side if we've got a base image that does or doesn't > have the right test keys. The best but not obvious way I see is rebooting in recovery and trying to adb shell. As far as I remember, with the testkeys loaded image, you can get a shell. Then look into the /res/keys file, and you should have something starting with: {64,0xc926ad21, You can also try to dump the recovery partition and extract its content and check this file.
(In reply to Alexandre LISSY :gerard-majax from comment #27) > You can also try to dump the recovery partition and extract its content and > check this file. Yes, to do that first dump the recovery partition to the sdcard and pull it out: adb shell cat /dev/mtd/mtd7 > /sdcard/recovery.img adb pull /sdcard/recovery.img Once this is done you can extract the initramfs system and unpack that so you'll be able to check if /res/keys contains the right key. To do this follow the instructions available here (you'll need the unmkbootimg tool for this, it's linked on the page): https://developer.mozilla.org/en-US/Firefox_OS/Porting#Extracting_and_modifying_an_existing_boot_image
(In reply to Gabriele Svelto [:gsvelto] from comment #28) > adb shell cat /dev/mtd/mtd7 > /sdcard/recovery.img Sorry, that's actually: adb shell 'cat /dev/mtd/mtd7 > /sdcard/recovery.img'
I flashed the v1.2-device.cfg, and I did what Alexandre suggested: 1 - power off phone 2 - Hold Volume up key. Press power. Release power and volume up once the splash shows 3 - after 10 or 15 seconds I got the orange screen (with a red exclamation, but that probably depends on what the last thing done was). 4 - adb shell cat /res/keys yielded: {64,0xc926ad21,{1795090719,2141396315,950055447,2581568430,4268923165,1920809988,546586521,3498997798,1776797858,3740060814,1805317999,1429410244,129622599,1422441418,1783893377,1222374759,2563319927,323993566,28517732,609753416,1826472888,215237850,4261642700,4049082591,3228462402,774857746,154822455,2497198897,2758199418,3019015328,2794777644,87251430,2534927978,120774784,571297800,3695899472,2479925187,3811625450,3401832990,2394869647,3267246207,950095497,555058928,414729973,1136544882,3044590084,465547824,4058146728,2731796054,1689838846,3890756939,1048029507,895090649,247140249,178744550,3547885223,3165179243,109881576,3944604415,1044303212,3772373029,2985150306,3737520932,3599964420},{3437017481,3784475129,2800224972,3086222688,251333580,2131931323,512774938,325948880,2657486437,2102694287,3820568226,792812816,1026422502,2053275343,2800889200,3113586810,165549746,4273519969,4065247892,1902789247,772932719,3941848426,3652744109,216871947,3164400649,1942378755,3996765851,1055777370,964047799,629391717,2232744317,3910558992,191868569,2758883837,3682816752,2997714732,2702529250,3570700455,3776873832,3924067546,3555689545,2758825434,1323144535,61311905,1997411085,376844204,213777604,4077323584,9135381,1625809335,2804742137,2952293945,1117190829,4237312782,1825108855,3013147971,1111251351,2568837572,1684324211,2520978805,367251975,810756730,2353784344,1175080310}} So I'll retry the OTA update and see if it works. While I had the recovery shell, I could see that there was an empty /sdcard directory and the mount command showed: rootfs / rootfs rw 0 0 tmpfs /dev tmpfs rw,nosuid,relatime,mode=755 0 0 devpts /dev/pts devpts rw,relatime,mode=600 0 0 proc /proc proc rw,relatime 0 0 sysfs /sys sysfs rw,relatime 0 0 /dev/block/mtdblock1 /system yaffs2 rw,relatime 0 0 /dev/block/mtdblock5 /data yaffs2 rw,nodev,noatime,nodiratime 0 0 Doing a local FOTA update, it failed and I see this in /cache/recovery/last_log: Finding update package... I:Update location: /updates/fota/update.zip E:unknown volume for path [/updates/fota/update.zip] E:unknown volume for path [/updates/fota/update.zip] E:unknown volume for path [/updates/fota/update.zip] E:unknown volume for path [/updates/fota/update.zip] E:unknown volume for path [/updates/fota/update.zip] E:unknown volume for path [/updates/fota/update.zip] E:unknown volume for path [/updates/fota/update.zip] E:unknown volume for path [/updates/fota/update.zip] E:unknown volume for path [/updates/fota/update.zip] E:unknown volume for path [/updates/fota/update.zip] E:unknown volume for path [/updates/fota/update.zip] E:unknown volume for path [/updates/fota/update.zip] E:Can't mount /updates/fota/update.zip Installation aborted. logcat before updating into recovery showed: AUS:SVC UpdateService:applyOsUpdate - Rebooting into recovery to apply FOTA update: /mnt/sdcard/updates/fota/update.zip So it looks like librecovery.so is messing us up. So I then did: adb remount adb push out/target/product/hamachi/system/lib/librecovery.so /system/lib adb reboot and tried the local update again. This time I got some additional output in logcat: AUS:SVC UpdateService:applyOsUpdate - Rebooting into recovery to apply FOTA update: /mnt/sdcard/updates/fota/update.zip A:srcPath = '/mnt/sdcard/updates/fota/update.zip' A:getenv(EXTERNAL_STORAGE) = '/mnt/sdcard' A:RECOVERY_EXTERNAL_STORAGE = '/sdcard' X:srcPath = '/mnt/sdcard/updates/fota/update.zip' X:getenv(EXTERNAL_STORAGE) = '/mnt/sdcard' X:RECOVERY_EXTERNAL_STORAGE = '/sdcard' Check 2: destPath = '/sdcard/updates/fota/update.zip' Rebooting into recovery: --update_package=/sdcard/updates/fota/update.zip Rebooting into recovery: --update_package=/sdcard/updates/fota/update.zip and the recovery ROM was able to successfully apply the update. So, we need to make sure that when we flash the phones that we also update the /system/lib/librecovery.so
(In reply to Dave Hylands [:dhylands] from comment #30) > So, we need to make sure that when we flash the phones that we also update > the /system/lib/librecovery.so Very nice catch! We're building librecovery.so but on devices like the hamachi it's not installed with by a plain ./flash.sh invocation because we're not doing a full flash unless we're forcing it explicitly with the |-f| parameter. Should we special-case librecovery.so so that it's always installed even on devices where we don't do a full flash by default?
I think that we have to. Since there isn't one flash.sh script, we need to make sure that they all get fixed. i.e. QA has this script: https://wiki.mozilla.org/B2G/QA/Tips_And_Tricks#Outdated_scripts (fullflash_gecko_ril_gaia.zip) and then there's the flash.sh that gets inscluded with the nightlies. I think Alexandre said that there were sone fonts that needed to be flashed as well (that are in some different place in /system but outside /system/b2g), so I think that we'll need to have a list of files/dirs that we flash in addition to /system/b2g
I don't believe that /system/lib/librecovery.so is made in our builds. We get the gecko tar ball and gaia.zip which doesn't include the /system/lib/librecovery.so I think we need a new base build with these patches included.
Flags: needinfo?(vchen)
Looks like having the 1.2-devices.cfg from the teleweb tool and replacing the librecovery.so will do the trick. I updated to the correct version. It would be easier though if we had a new OEM build with all the necessary pulls, including fonts and anything else.
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #34) > It would be easier though if we had a new OEM build with all the necessary > pulls, including fonts and anything else. We should open a follow up bug to add the necessary infrastructure to properly flash those bits when we flash a phone and possibly package them too in the updates; AFAIR we haven't opened one yet.
(In reply to Gabriele Svelto [:gsvelto] from comment #35) > (In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from > comment #34) > > It would be easier though if we had a new OEM build with all the necessary > > pulls, including fonts and anything else. > > We should open a follow up bug to add the necessary infrastructure to > properly flash those bits when we flash a phone and possibly package them > too in the updates; AFAIR we haven't opened one yet. I agree, I think we need to improve the current FOTA generation, especially being able to much more highly tune the edify scripts.
Firefox OS is not being worked on
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: