1021839 - [Flame] applying update via OTA is failing: errorCode: 7

Status: RESOLVED FIXED

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[Tony Chung [:tchung]]

Attachment: logcat

Doing an OTA update from 6/5 -> 6/6 build, it is failing to apply due to space errors.  I have plenty of storage space on the Flame.

Note: this is a regression.  i was able to apply 6/4 -> 6/5 build successfullly.  but not 6/5 - 6/6.

Logcat and last-update.log attached

logcat snippet:
06-06 10:53:45.441: I/GeckoUpdater(8837): Progress [ ===============================================    ]
06-06 10:53:45.531: I/GeckoUpdater(8837): Progress [ ================================================   ]
06-06 10:53:45.651: I/GeckoUpdater(8837): Progress [ =================================================  ]
06-06 10:53:45.661: I/GeckoUpdater(8837): Progress [ ================================================== ]
06-06 10:53:45.661: I/GeckoUpdater(8837): Finished applying update
06-06 10:53:45.711: I/GonkAutoMounter(8837): Mounted /system partition as read-only
06-06 10:53:45.721: I/Gecko(288): *** AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
06-06 10:53:45.721: I/Gecko(288): UpdatePrompt: Update error, state: failed, errorCode: 7
06-06 10:53:45.741: I/Gecko(288): UpdatePrompt: Setting gecko.updateStatus: Install Pending
06-06 10:53:45.781: I/Gecko(288): *** AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7

Other data:
root@flame:/data/local/updates/0 # cat update.status                           
pending

I also do not have USB sharing enabled at all.

Repro:
1) install 2.0 nightly on flame (5/6/2014 build)
localhost:B2G-flash-tool mozilla$ ./check_versions.sh 
Gaia      d2cfef555dabab415085e548ed44c48a99be5c32
Gecko     https://hg.mozilla.org/mozilla-central/rev/51b428be6213
BuildID   20140605040202
Version   32.0a1
ro.build.version.incremental=94
ro.build.date=Tue May 20 09:29:20 CST 2014

2) settings > device info > check for updates
3) update is found. pull down notification and apply it
4) Verify update does not get applied and stuck in pending.  ErrorCode 7

Expected:
- update applied

Actual:
- update failed on device

Comment 1

[Dave Hylands [:dhylands]]

Attachment: last-update.log

I got tchung to pull /data/local/updates/last-update.log

Comment 2

[Tony Chung [:tchung]]

Attachment: a 2nd last-update.log that matches the timing of the attached logcat

Comment 3

[Dave Hylands [:dhylands]]

I tried to reproduce, but the update worked for me. I tried both of these:
./auto_flash_from_pvt.sh --version=master --device=flame --buildid=20140605040202 --full --eng
./auto_flash_from_pvt.sh --version=master --device=flame --buildid=20140605040202 --full --usr
and the update got to the "Install update now?" screen in both cases.

What's completely baffling to me is that the end of tchung's last-update.log shows:

> FINISH ADD application.ini
> succeeded
> calling QuitProgressUI
> GonkAutoMounter: Mounted /system partition as read-only

and that "succeeded" is logged here:
http://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/updater/updater.cpp#2331

It can only get there if rv == 0 so WriteStatusFile should have been called with 0. And yet the logcat reports:

> 06-06 10:53:45.661: I/GeckoUpdater(8837): Progress [ ================================================== ]
> 06-06 10:53:45.661: I/GeckoUpdater(8837): Finished applying update
> 06-06 10:53:45.711: I/GonkAutoMounter(8837): Mounted /system partition as read-only
> 06-06 10:53:45.721: I/Gecko(288): *** AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
> 06-06 10:53:45.721: I/Gecko(288): UpdatePrompt: Update error, state: failed, errorCode: 7
> 06-06 10:53:45.741: I/Gecko(288): UpdatePrompt: Setting gecko.updateStatus: Install Pending

My logcat shows this update URLs:
https://aus4.mozilla.org/update/3/B2G/32.0a1/20140605040202/flame/en-US/nightly/Boot2Gecko%202.0.0.0-prerelease/default/default/update.xml?force=1
which is identical to the one from tchung's logcat.

So I'm not sure how tchung was able to see what he saw.

tchung: Can you try:

> adb shell mv /storage/sdcard0/updates/0/update.mar /storage/sdcard0/updates/0/update.mar.failed

and then retry the update?

Comment 4

[Dave Hylands [:dhylands]]

Something else to try:

> adb pull /storage/sdcard0/updates/0/update.mar   (add .failed if you already renamed)
> sha512sum update.mar

You may need to do: "sudo apt-get install hashalot" to get the sha512sum tool.

Mine reported: 0c3485d2b7fc34586936b7bf35afb23a5919dffb3248e442ce5451813287c1929e57f99f0750cf62a35d5e56c3f3b68dbe920596053df82543dddcc5e99981e7

Comment 5

[Tony Chung [:tchung]]

localhost:temp mozilla$ shasum -a 512 update.mar.failed 
0c3485d2b7fc34586936b7bf35afb23a5919dffb3248e442ce5451813287c1929e57f99f0750cf62a35d5e56c3f3b68dbe920596053df82543dddcc5e99981e7  update.mar.failed

Comment 6

[Dave Hylands [:dhylands]]

So your phone is doing something different from mine.

I think that in order to debug this further, I'll need to get you to send me your phone.

Comment 7

[Tony Chung [:tchung]]

Attachment: update.log

another update.log.   this was pulled from /data/local/updates/0/update.log

Comment 8

[Dave Hylands [:dhylands]]

OK - so this line: 
> ### execution failed
seems to be the culprit.

First step will be to grab the output of:

> adb shell ls -ld /system/b2g*
> adb shell ls -lR /system/b2g*

I think that you'll have a /system/b2g/updated directory. Let's move that out of the way

> adb remount
> adb shell mv /system/b2g/updated /system/b2g/updated.failed
> adb shell echo pending > /data/local/updates/0/update.status

and then do "Check Now" for new updates and try reapplying. It should detect that it's already downloaded and re-uncompress and try to reapply.

Hmm. I was just doing these steps manually to verify that they were all correct and I got the same error.

Now that I have something, I'll see if I can figure out more.

Comment 9

[Dave Hylands [:dhylands]]

In my case, I definitely ran out of space.

When I start to apply an update that doesn't fail, my free space on /system drops from 146.0 Mb to 57.8 Mb, which means that 88.2 Mb are needed.

Your pastebin from IRC showed:

> /system                354.2M   270.3M    83.9M   4096

Which may explain why you're hitting the problem.

In order to find out where all of the space is going, let's do:

> adb shell du -m /system

I'm going to guess that you have a /system/b2g.bak directory or something like that.

I tried flashing an eng build (as in comment 3), and I still had 120 Mb free.

If nothing else, we need to make reporting out-of-space a little more obvious.

Comment 10

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Odd. Yesterday's build, I got a different error.  Peter got the same error message as well in the recovery log.  Should we file a separate bug?

I:Got arguments from /cache/recovery/command
E:Can mount but Can't open  /cache/recovery/last_locale
locale is [(null)]
Left Split=0
Right Split=0
framebuffer: fd 4 (480 x 854)
Command: "/sbin/recovery" "--wipe_data"

Comment 11

[Dave Hylands [:dhylands]]

I think that the recovery log stuff is a separate issue.

We don't use any of that for regular OTA (which is what this issue is about).

Comment 12

[Tony Chung [:tchung]]

(In reply to Dave Hylands [:dhylands] from comment #9)
> In my case, I definitely ran out of space.
> 
> When I start to apply an update that doesn't fail, my free space on /system
> drops from 146.0 Mb to 57.8 Mb, which means that 88.2 Mb are needed.
> 
> Your pastebin from IRC showed:
> 
> > /system                354.2M   270.3M    83.9M   4096
> 
> Which may explain why you're hitting the problem.
> 
> In order to find out where all of the space is going, let's do:
> 
> > adb shell du -m /system

Attaching my du /system

> 
> I'm going to guess that you have a /system/b2g.bak directory or something
> like that.

Nothing.  

localhost:temp mozilla$ ls -l /system/b2g.bak
ls: /system/b2g.bak: No such file or directory

> 
> I tried flashing an eng build (as in comment 3), and I still had 120 Mb free.
> 
> If nothing else, we need to make reporting out-of-space a little more
> obvious.

Same error snippet on today's build 6/10:

06-10 08:24:23.048: I/GeckoUpdater(3055): Progress [ =================================================  ]
06-10 08:24:23.058: I/GeckoUpdater(3055): Progress [ ================================================== ]
06-10 08:24:23.058: I/GeckoUpdater(3055): Finished applying update
06-10 08:24:23.128: I/Gecko(291): *** AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
06-10 08:24:23.128: I/Gecko(291): UpdatePrompt: Update error, state: failed, errorCode: 7
06-10 08:24:23.138: I/Gecko(291): UpdatePrompt: Setting gecko.updateStatus: Install Pending
06-10 08:24:23.168: I/Gecko(291): *** AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7

What else can i try?

Comment 13

[Tony Chung [:tchung]]

Attachment: du of /system

Comment 14

[Dave Hylands [:dhylands]]

(In reply to Tony Chung [:tchung] from comment #13)
> Created attachment 8437717 [details]
> du of /system

So I compared your phone to mine, and you're using about 70 Mb more in /system than I am.
And it looks like most of that is from /system/b2g/updated (which is the new copy of b2g)

The only other difference I see is that your phone has 23 Mb of stuff in /system/media, and my phone doesn't even have that directory.

If I had to guess, I'd say that's left over from the base image?

I think that if you did:

> adb remount
> adb shell rm -r /system/media

then your OTA will work.

Comment 15

[Dave Hylands [:dhylands]]

Hmm. I also see that your /system/lib almost 20 Mb bigger than mine as well.

Your phone has the following directories that mine doesn't:
/system/lib/modules/ath6kl-3.5 (12 Mb)
/system/etc/param (1 Mb)
/system/etc/surround_sound (1 Mb)
/system/etc/updatecmds (1 Mb)
/system/lib/crda (1 Mb)
/system/usr/share/bmd (1 Mb)
/system/vendor/lib/rfsa/adsp (1 Mb)
/system/vendor/lib/rfsa/rfsa (1 Mb)
/system/vendor/lib/rfsa/sounfx (1 Mb)

Comment 16

[Dave Hylands [:dhylands]]

I think that the difference here are due to the way your phone and my phone were flashed.

For me:
- I flashed the base image
- did a local build
- did a full flash

For you (I'm guessing here):
- flashed base image
- flashed a pvt build using the B2G-flash-tool
- tried to OTA

The B2G-flash-tool doesn't remove files, where doing a full flash will.

So I think that this is the fundamental difference.

Comment 17

[Tony Chung [:tchung]]

(In reply to Dave Hylands [:dhylands] from comment #16)
> I think that the difference here are due to the way your phone and my phone
> were flashed.
> 
> For me:
> - I flashed the base image
> - did a local build
> - did a full flash
> 
> For you (I'm guessing here):
> - flashed base image
> - flashed a pvt build using the B2G-flash-tool
> - tried to OTA
> 
> The B2G-flash-tool doesn't remove files, where doing a full flash will.

Correct, i flashed it this way.  what happens if i remove /system/media?   does it kill my music and video files?
> 
> So I think that this is the fundamental difference.

Comment 18

[Dave Hylands [:dhylands]]

(In reply to Tony Chung [:tchung] from comment #17)
> (In reply to Dave Hylands [:dhylands] from comment #16)
> > I think that the difference here are due to the way your phone and my phone
> > were flashed.
> > 
> > For me:
> > - I flashed the base image
> > - did a local build
> > - did a full flash
> > 
> > For you (I'm guessing here):
> > - flashed base image
> > - flashed a pvt build using the B2G-flash-tool
> > - tried to OTA
> > 
> > The B2G-flash-tool doesn't remove files, where doing a full flash will.
> 
> Correct, i flashed it this way.  what happens if i remove /system/media?  
> does it kill my music and video files?

/system/media is read-only.

No user data will ever be stored there (so your music and video are stored in device storage, which is elsewhere).

/system/media would presumably contain media that some of the builtin apps might need. But if they do need it, then we have other issues, since we're not including them when we do a build.

It looks like /system/media/bootanimation.zip contains the firefox with the flaming tail animation.

The rest of the files in /system/media appear to be leftover from android, and I don't think that they're used by b2g.

Comment 19

[Jason Smith [:jsmith]]

This might be fixed with the latest base image. Can we retest?

Comment 20

[Brogan Zumwalt [Inactive]]

Issue does NOT occur on latest base for 2.0 Flame

Flashed to 20140611091244 build and checked for updates from settings. Update appeared in notification tray. Performed OTA update and successfully updated to 20140612000201

Environmental Variables:
Device: Flame 2.0
BuildID: 20140611091244
Gaia: a0f9f1f41a436daad8a249ce85df80a81a5ba2d5
Gecko: 0c0effd600c4
Version:  32.0a2
Firmware Version: v121-2

Comment 21

[Joshua Mitchell (Inactive)]

Jason - Quick Question - This issue seems to no longer reproduce, which would prompt me to close it as works-for-me but the fix / change is due to the new base; so would this get a resolved: fixed instead?

Comment 22

[Jason Smith [:jsmith]]

I'll mark it fixed via the latest image.

Comment 23

[Natalya Kot [:nkot]]

Verified Fixed - OTA works on the latest base image (v121-2)

OTA'd from build 20140612000201 to build 20140613000203 successfully.

Comment 24

[Brogan Zumwalt [Inactive]]

Attachment: 2_1 Logcat Fail

OTA update fails with code 7 when uncompressing finished on 2.1 Flame. Was updating from 20140612040203 build.

Attached logcat

Environmental Variables:
Device: Flame 
Build ID: 20140612040203
Gaia: 41db6954a67efc55016744bc8f6591ae9e07a285
Gecko: 9e8e3e903484
Version: 33.0a1  
Firmware Version: v121-2

Comment 25

[Natalya Kot [:nkot]]

Just to clarify comment 23 and comment 24 - so the OTA works OK on 2.0 and fails on 2.1. Updating on 2.1 results in the same error (code 7) as this bug describes. 
**Both tests were performed on the newest base v121-2 

Should we reopen this bug then?

Comment 26

[Dave Hylands [:dhylands]]

I think that if you do a full flash using the B2G-flash-tool then the OTA will work.

Comment 27

[Tony Chung [:tchung]]

I verified OTA update is working again for mozilla-aurora branch.

localhost:B2G-flash-tool mozilla$ ./check_versions.sh 
Gaia      a6988c15b361938bea5976c846c147ecdc1121c0
Gecko     https://hg.mozilla.org/releases/mozilla-aurora/rev/52a276202888
BuildID   20140616000203
Version   32.0a2

nkot, please try comment 26 after a clean v121-2 flame base, and flash an older master/m-c build on it.  if it still fails to connect, attach the log for investigation.   Thanks!

Comment 28

[Natalya Kot [:nkot]]

(In reply to Tony Chung [:tchung] from comment #27)
> 
> nkot, please try comment 26 after a clean v121-2 flame base, and flash an
> older master/m-c build on it.  if it still fails to connect, attach the log
> for investigation.   Thanks!

There is an issue enabling WiFi when we flash using the B2G-flash-tool, so we were unable to OTA over the WiFi. Having said that, we were able to perform OTA using Cell Data, successfully updated to the 20140616040202 build.

I leave 'smoketest' keyword here because we're still unable to perform OTA over the WiFi as the TC suggests. We may want to open another bug for that.

OTA'ing after flashing using regular method (flash_Gg.sh) we still hit the same 'error code 7'

Comment 29

[Jason Smith [:jsmith]]

Are you arguing this should be reopened? I don't follow the above comment.

Comment 30

[Dave Hylands [:dhylands]]

I think that the Wifi problem is a totally separate issue...

Comment 31

[Natalya Kot [:nkot]]

(In reply to Jason Smith [:jsmith] from comment #29)
> Are you arguing this should be reopened? I don't follow the above comment.

Sorry if it wasn't clear. 

- We can close/verify this bug - flashing using 'B2G-flash-tool' allows to perform OTA.
- After flashing using 'B2G-flash-tool' there is an issue enabling WiFi - I'll file a separate bug for that (that would be blocking OTA over the WiFi explicitly.)

Comment 32

[Natalya Kot [:nkot]]

Bug 1026278 is filed for further investigation.

Comment 33

[Hubert Lu[:hlu] <[e-mail]>]

Hi all,
    I did some testing for this issue, and the result is in below.

---------------------------------
|           |  v2.0   |   v2.1  |
---------------------------------
|v121-2 base|  Fail   |   Fail  |
---------------------------------
|v122 base  | Updated |   Fail  |
---------------------------------


As Dave mentions in comment 9, this issue will happen if the system space is not enough. 

 - For OTA update, it will check that system partition free space should be around 1.5 times more than update.zip, because it needs enough space to store the unziped files.

Current flame free system partition is around 70 ~ 90 MB after flash build, and this system free space size is the boundary between pass and fail for current OTA update. That's why some builds could update successfully and some builds fail to update. 

For detail partition information of my testing, please refer to space.txt.

Comment 34

[Hubert Lu[:hlu] <[e-mail]>]

Attachment: space.txt

Comment 35

[Tony Chung [:tchung]]

(In reply to Hubert Lu[:hlu] <[e-mail]> from comment #34)
> Created attachment 8442036 [details]
> space.txt

Hubert, can you attach the logcat errors that is being thrown when it fails for you in the scenarios above?  We're trying to see if this bug needs to be reopened (assuming its the same error log), or needs a new bug to be filed (if its a space constraint on flame issue and would need T2M intervention)

Thanks,
Tony

Comment 36

[Gerry Chang [:gchang]]

Attachment: logs of OTA

Comment 37

[Hubert Lu[:hlu] <[e-mail]>]

Gerry has updated the log, so clear ni.

Comment 38

[Tony Chung [:tchung]]

Looks like this bug is still not resolved.   Reopening.

Gerry's log from coment 36 shows:

06-19 10:36:06.406  1809  1810 I GeckoUpdater: Progress [ ===============================================    ]
06-19 10:36:06.446  1809  1810 I GeckoUpdater: Progress [ ================================================   ]
06-19 10:36:06.536  1809  1810 I GeckoUpdater: Progress [ =================================================  ]
06-19 10:36:06.546  1809  1810 I GeckoUpdater: Progress [ ================================================== ]
06-19 10:36:06.546  1809  1810 I GeckoUpdater: Finished applying update
06-19 10:36:06.616  1809  1809 I GonkAutoMounter: Mounted /system partition as read-only
06-19 10:36:06.636   313   313 I Gecko   : *** AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
06-19 10:36:06.636   313   313 E GeckoConsole: AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
06-19 10:36:06.636   313   313 I Gecko   : UpdatePrompt: Update error, state: failed, errorCode: 7
06-19 10:36:06.636   313   313 E GeckoConsole: [JavaScript Error: "formatURLPref: Couldn't get pref: app.update.url.details" {file: "jar:file:///system/b2g/omni.ja!/components/nsURLFormatter.js" line: 134}]
06-19 10:36:06.646   313   313 I Gecko   : UpdatePrompt: Setting gecko.updateStatus: Install Pending
06-19 10:36:06.656   313   313 E GeckoConsole: [JavaScript Error: "formatURLPref: Couldn't get pref: app.update.url.details" {file: "jar:file:///system/b2g/omni.ja!/components/nsURLFormatter.js" line: 134}]
06-19 10:36:06.656   313   313 E GeckoConsole: [JavaScript Error: "formatURLPref: Couldn't get pref: app.update.url.details" {file: "jar:file:///system/b2g/omni.ja!/components/nsURLFormatter.js" line: 134}]
06-19 10:36:06.666   313   313 I Gecko   : *** AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7
06-19 10:36:06.666   313   313 E GeckoConsole: AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7
06-19 10:36:06.666   313   313 I nsVolumeMountLock: nsVolumeMountLock released for 'sdcard'
06-19 10:36:06.676   313   313 I nsVolume: nsVolume: sdcard state Mounted @ '/storage/sdcard0' gen 1 locked 0 fake 0 media 1 sharing 0 formatting 0

Comment 39

[Tony Chung [:tchung]]

Hold on..  i just tried this myself on a 2.1 build, trying to OTA update from 20140619 -> 20140620, and it worked.   I'll attach my logcat for proof.

Also, you can see my version got updated before and after:

localhost:B2G-flash-tool mozilla$ ./check_versions.sh 
Gaia      82e957160ca017087bd374cd051339e55b641e68
Gecko     https://hg.mozilla.org/mozilla-central/rev/f78e532e8a10
BuildID   20140619040201
Version   33.0a1
ro.build.version.incremental=109
ro.build.date=Mon Jun 16 16:51:29 CST 2014

localhost:B2G-flash-tool mozilla$ ./check_versions.sh 
Gaia      ccd70903544486bea04e85d8a4aacf63f1de2a72
Gecko     https://hg.mozilla.org/mozilla-central/rev/bdac18bd6c74
BuildID   20140620040204
Version   33.0a1
ro.build.version.incremental=109
ro.build.date=Mon Jun 16 16:51:29 CST 2014

logcat snippet:
--------------

06-20 09:14:51.444: I/GeckoUpdater(16816): Progress [ =================================================  ]
06-20 09:14:51.444: I/GeckoUpdater(16816): Progress [ ================================================== ]
06-20 09:14:51.454: I/GeckoUpdater(16816): Finished applying update
06-20 09:14:51.604: D/wpa_supplicant(16473): RX ctrl_iface - hexdump(len=11): 53 49 47 4e 41 4c 5f 50 4f 4c 4c
06-20 09:14:51.604: D/wpa_supplicant(16473): wlan0: Control interface command 'SIGNAL_POLL'
06-20 09:14:51.614: D/wpa_supplicant(16473): nl80211: survey data missing!
06-20 09:14:51.654: I/GonkAutoMounter(16816): Mounted /system partition as read-only
06-20 09:14:51.654: I/Gecko(290): *** AUS:SVC readStatusFile - status: applied, path: /data/local/updates/0/update.status
06-20 09:14:51.674: I/nsVolumeMountLock(290): nsVolumeMountLock released for 'sdcard'
06-20 09:14:51.674: I/Gecko(290): *** AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: applied, status: applied
06-20 09:14:51.684: I/Gonk(290): Setting nice for pid 16590 to 18
06-20 09:14:51.684: I/Gonk(290): Changed nice for pid 16590 from 1 to 18.
06-20 09:14:51.694: I/Gecko(290): UpdatePrompt: Update is ready to apply, registering idle timeout of 600 seconds before prompting.
06-20 09:14:51.964: I/nsVolume(290): nsVolume: sdcard state Mounted @ '/storage/sdcard0' gen 1 locked 0 fake 0 media 1 sharing 0 formatting 0
06-20 09:14:51.964: I/AutoMounter(290): UpdateState: umsAvail:1 umsEnabled:1 mode:0 usbCablePluggedIn:1 tryToShare:0
06-20 09:14:51.964: I/AutoMounter(290): UpdateState: Volume external is NoMedia and missing
06-20 09:14:51.964: I/AutoMounter(290): UpdateState: Volume sdcard is Mounted and inserted @ /storage/sdcard0 gen 1 locked 0 sharing en-n
06-20 09:14:51.964: I/AutoMounter(290): UpdateState: umsAvail:1 umsEnabled:1 mode:0 usbCablePluggedIn:1 tryToShare:0
06-20 09:14:51.964: I/AutoMounter(290): UpdateState: Volume external is NoMedia and missing
06-20 09:14:51.964: I/AutoMounter(290): UpdateState: Volume sdcard is Mounted and inserted @ /storage/sdcard0 gen 1 locked 0 sharing en-n
06-20 09:14:51.964: I/AutoMounter(290): UpdateState: umsAvail:1 umsEnabled:1 mode:0 usbCablePluggedIn:1 tryToShare:0
06-20 09:14:51.964: I/AutoMounter(290): UpdateState: Volume external is NoMedia and missing
06-20 09:14:51.964: I/AutoMounter(290): UpdateState: Volume sdcard is Mounted and inserted @ /storage/sdcard0 gen 1 locked 0 sharing en-n
06-20 09:14:55.954: E/QCALOG(430): [MessageQ] ProcessNewMessage: [XT-CS] unknown deliver target [OS-Agent]
06-20 09:14:56.014: E/QCALOG(430): [MessageQ] ProcessNewMessage: [XTWWAN-PE] unknown deliver target [OS-Agent]
06-20 09:14:56.014: E/QCALOG(430): [MessageQ] ProcessNewMessage: [XTWiFi-PE] unknown deliver target [OS-Agent]
06-20 09:14:56.604: D/wpa_supplicant(16473): RX ctrl_iface - hexdump(len=11): 53 49 47 4e 41 4c 5f 50 4f 4c 4c
06-20 09:14:56.604: D/wpa_supplicant(16473): wlan0: Control interface command 'SIGNAL_POLL'
06-20 09:14:56.624: D/wpa_supplicant(16473): nl80211: survey data missing!
06-20 09:14:58.034: I/Gecko(290): UpdatePrompt: Update downloaded, restarting to apply it
06-20 09:14:58.034: I/GeckoDump(290): XXX FIXME : Got a mozContentEvent: update-prompt-apply-result
06-20 09:14:58.054: I/Gecko(290): UpdatePrompt: Set previous_os to: 2.1.0.0-prerelease

Gerry, Hubert, can you guys retest your scenario again on 2.1?    I have base image 122, and just did shallow flash for gecko/gaia, and performed OTA update.   If you confirm this is working, we can close this bug.  if not, we need to debug further.

Comment 40

[Tony Chung [:tchung]]

Attachment: Tony's successful 2.1 OTA log

Comment 41

[Tony Chung [:tchung]]

Also tried this on 2.0 (mozilla-aurora).   OTA update also works there from 0619 -> 0620

Before: 
Gaia      23e06c3624309db22ad9cb736d89700768b42b36
Gecko     https://hg.mozilla.org/releases/mozilla-aurora/rev/164b61458ca5
BuildID   20140619000200
Version   32.0a2
ro.build.version.incremental=109
ro.build.date=Mon Jun 16 16:51:29 CST 2014

After: 
Gaia      c5dd47e3f9e18872961946735fdbc071a1656ac9
Gecko     https://hg.mozilla.org/releases/mozilla-aurora/rev/71b1b4b0850c
BuildID   20140620000202
Version   32.0a2
ro.build.version.incremental=109

Comment 42

[Natalya Kot [:nkot]]

I've tried OTA from 6/19 2.1 build to 6/20 2.1 on both base images.

v121-2 - Failed to the same 'error code 7'
v122   - Successfully updated!


From:
BuildID: 20140619040201
Gaia: 82e957160ca017087bd374cd051339e55b641e68
Gecko: f78e532e8a10
Version: 33.0a1

To:
BuildID: 20140620040204
Gaia: ccd70903544486bea04e85d8a4aacf63f1de2a72
Gecko: bdac18bd6c74
Version: 33.0a1

Comment 43

[Tony Chung [:tchung]]

Gerry, hubert -- can you try the same tests but with Flame v122 base image?

Comment 44

[Tony Chung [:tchung]]

Please reopen if you can still reproduce with v122.   else i'll mark this as fixed by vendor image.

Comment 45

[Hubert Lu[:hlu] <[e-mail]>]

I have tried the v2.0 and v2.1 on both base image, and result is the same to comment 33. (Error code: 7)
For shallow flash, only the case,v2.0 with base image 122,could update successfully.
For full flash, all of combination could update properly.

=== Result (Shallow flash) ===
------------------------------------
|      |  v121 base  |  v122 base  |
------------------------------------
|v2.0  |     Fail    |   Updated   |
------------------------------------
|v2.1  |     Fail    |     Fail    |
------------------------------------

=== Result (Full flash) ===
------------------------------------
|      |  v121 base  |  v122 base  |
------------------------------------
|v2.0  |   Update    |   Updated   |
------------------------------------
|v2.1  |   Update    |   Update    |
------------------------------------


=== Matrix for /system free space after flash ===

------------------------------------------------------
|      |       v121           |       v122           |
|      |---------------------------------------------- 
|      | shallow  |  Full     |  shallow  |  Full    |
|      |  flash   |  flash    |   flash   |  flash   |
------------------------------------------------------
| v2.0 |  63.3 MB | 122.7 MB  |  83.6 MB  | 122.7 MB |
------------------------------------------------------
| v2.1 |  60.6 MB | 116.8 MB  |  80.9 MB  | 116.8 MB |
------------------------------------------------------
* Package size for v2.0, it is 62.85 MB (BuildID: 20140620160201 -> 20140621000204)
* Package size for v2.1, it is 65.96 MB (BuildID: 20140620160202 -> 20140621040203)

=== ENV ===
- v2.0 - 
Gaia      751617235a44ca9c2a641b751de00ff48136657e
Gecko     https://hg.mozilla.org/releases/mozilla-aurora/rev/e72b00065ebe
BuildID   20140620160201
Version   32.0a2
ro.build.version.incremental=108
ro.build.date=Tue Jun 10 19:40:40 CST 2014
   
- v2.1 -
Gaia      bd5065ced020014df5fd45259fba1ac32d65673b
Gecko     https://hg.mozilla.org/mozilla-central/rev/776f6d341b3f
BuildID   20140620160202
Version   33.0a1
ro.build.version.incremental=109
ro.build.date=Mon Jun 16 16:51:29 CST 2014

Comment 46

[Gerry Chang [:gchang]]

Hi Chris,
Due to comment #33, the free space of /system is not enough for device to perform OTA. So, we want releng to build a FOTA package with gaia/gecko only on PVT build server so that device can update gaia/gecko only by using FOTA flow rather than OTA flow. Can you help to create the packages on PVT?

Comment 47

[Francis Lee [:frlee]]

hi Chris AtLee,

would you please help or loop in your member who can help on comment46?

Comment 48

[Gerry Chang [:gchang]]

Hi Tony, 
According to comment 45, I'll re-open this bug.

Comment 49

[Jason Smith [:jsmith]]

(In reply to Gerry Chang [:cfchang] from comment #48)
> Hi Tony, 
> According to comment 45, I'll re-open this bug.

comment 45 is old information. We've been unblocked testing for weeks.

Comment 50

[aliasbody]

 Hello Everyone

Indeed it is a size problem on the system partition. It needs to have at least 90-95mb. The solution I've found, helped by the comments on thise bug report, was to remove all unecessary fonts and only provide the Fira Fonts. It isn't a final solution it is just a workaround.

I've adapted the script of the "font-fix" and provided a way to clean the fonts directory and then providing the font-fix (good for any version of Firefox OS).

PS: A final solution would be to create a new base image which would make the system partition bigger in order to have something like 150-200Mb available (we never know).

Here is the link: 
https://www.dropbox.com/s/bkwtffoipqsn2wx/update_and_fonts.zip?dl=0

Best Regards
Luis Da Costa