Closed Bug 965874 Opened 10 years ago Closed 10 years ago

Error occurs when trying to FOTA update due to /system partition as read-only

Categories

(Firefox OS Graveyard :: General, defect)

Product:
Firefox OS Graveyard
Component:
General
Platform:
ARM
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(blocking-b2g:1.3+)

Status:
VERIFIED FIXED
Project Flags:
blocking-b2g 1.3+

People

(Reporter: nhirata, Assigned: dhylands)

Details

(Keywords: qablocker, Whiteboard: [systemsfe][NPOTB])

Attachments

(3 files)

logcat_OTA.txt
74.99 KB, text/plain
Details
v1.3_logcat.txt
8.70 KB, text/plain
Details
buri.fota.log
11.58 KB, text/x-log
Details 
1. flash a 1.3 build on the device
2. change the channel to 1.4
3. force update via the settings

expected: FOTA downloads, reboots and updates, then reboots again
actual: error:
01-30 13:05:18.298: I/Gecko(1810): *** AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
01-30 13:05:18.298: I/Gecko(1810): UpdatePrompt: Update error, state: failed, errorCode: 7
01-30 13:05:18.328: I/Gecko(1810): UpdatePrompt: Setting gecko.updateStatus: Install Pending
01-30 13:05:18.358: I/Gecko(1810): *** AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7
blocking-b2g: --- → 1.4?
Summary: Error occurs when trying to update 1.3 to 1.4 on Buri → Error occurs when trying to FOTA update 1.3 to 1.4 on Buri
Whiteboard: [systemsfe]
Attached file logcat_OTA.txt — 
1.4 to 1.4 has the same issue for nkot.  I think I may have a different issue when I try to update mine.  nkot's logcat is in the attachments:

01-31 11:17:19.019: I/GonkAutoMounter(1482): Mounted /system partition as read-only
01-31 11:17:19.039: I/Gecko(136): *** AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
01-31 11:17:19.039: I/Gecko(136): UpdatePrompt: Update error, state: failed, errorCode: 7
01-31 11:17:19.069: I/Gecko(136): UpdatePrompt: Setting gecko.updateStatus: Install Pending
01-31 11:17:19.119: I/Gecko(136): *** AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7
Summary: Error occurs when trying to FOTA update 1.3 to 1.4 on Buri → Error occurs when trying to FOTA update due to /system partition as read-only
Not sure who's working on FOTA?
Flags: needinfo?(dhylands)
Flags: needinfo?(aki)
I have deployed FOTA for buri on update.boot2gecko.org, but that's the extent of my knowledge.
Flags: needinfo?(aki)
Found out that Alexandre is working on FOTA.  Alexandre, could you take a look at this please?
Flags: needinfo?(dhylands) → needinfo?(lissyx+mozillians)
I have no idea about this, I worked on the lower level, producing the update.zip that works in recovery mode and that is embedded in the MAR package.
Flags: needinfo?(lissyx+mozillians)
Dave - Any ideas?
Flags: needinfo?(dhylands)
I suspect that the /system partition is mounted read-only by default and the update script may need to mount /system as writable.

We may need to ask the vendor.

It may also depend on the kernel/boot.img which is flashed onto the phone.
Flags: needinfo?(dhylands)
Reading comment 1, the updater should only be trying to mount the /system partition as writable when applying an OTA update, not when applying a FOTA update.

The attached logcat also indicates that this is an OTA update versus a FOTA update.
This is really strange.

According to the logcat, the update which was downloaded was this one:
http://update.boot2gecko.org/hamachi/1.4.0/nightly/b2g_update_20140131040202.mar?build_id=20140131040202&version=29.0a1

which appears to be a FOTA update containing the following files:
-rw-r--r-- 1 dhylands dhylands       14 Jan 31 23:03 precomplete
-rw-r--r-- 1 dhylands dhylands       67 Jan 31 23:03 update.manifest
-rw-r--r-- 1 dhylands dhylands 50380167 Jan 31 23:03 update.zip
-rw-r--r-- 1 dhylands dhylands       78 Jan 31 23:03 updatev2.manifest

However, the updater prints a progress line (with ===) for each file, and the number of progress lines printed doesn't agree with the .mar file containing 4 files.

The failed 7 also indicates that it was processing an OTA update rather than a FOTA update.
(In reply to Dave Hylands [:dhylands] from comment #9)
> The failed 7 also indicates that it was processing an OTA update rather than
> a FOTA update.

Is it possible that the wrong update file was downloaded? I.e. we dowloaded an OTA update and tried to process it as a FOTA one or something along the lines?
This is still reproducing as of right now, with hamachi/1.4.0/nightly channel:
I/GonkAutoMounter( 1467): Mounted /system partition as read-only
I/Gecko   ( 1219): *** AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
E/GeckoConsole( 1219): AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
I/Gecko   ( 1219): UpdatePrompt: Update error, state: failed, errorCode: 7
I/Gecko   ( 1219): UpdatePrompt: Setting gecko.updateStatus: Install Pending
I/Gecko   ( 1219): *** AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7
E/GeckoConsole( 1219): AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7
If I'm right, errorCode 7 means WRITE_ERROR.
Hey Alex, 

Jason raised this in sprint planning as a priority bug.  Saw that you have already reproduced this.  Would you be the correct person to fix this bug?
Flags: needinfo?(lissyx+mozillians)
Can someone check if FOTA works on 1.3 on Buri?
Keywords: qawanted
(In reply to Candice Serran (:cserran) from comment #13)
> Hey Alex, 
> 
> Jason raised this in sprint planning as a priority bug.  Saw that you have
> already reproduced this.  Would you be the correct person to fix this bug?

Unfortunately no, but as we discussed with Jason, probbaly Dave can help :)
Flags: needinfo?(lissyx+mozillians) → needinfo?(dhylands)
Dave already replied here ...
Flags: needinfo?(dhylands)
If I can reproduce it locally, then I can take a deeper look to figure out what's going on.

Naoki, in your STR you said:

1. flash a 1.3 build on the device
2. change the channel to 1.4
3. force update via the settings

Can you point me to a particular 1.3 build?
How exactly did you change the channel to 1.4?
For step 3, I'm assuming you just meant "Check Now"?

Also, do I need any particular base image on the hamachi?
Flags: needinfo?(nhirata.bugzilla)
I get the very same issue with a buri on 1.4 nightly (https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2g_ril/latest-hamachi-mozilla-central/).

STR:
1. Install nightly from above
2. Search for Update
3. Install
4: Error similar to what's been said above. Except here: https://dpaste-bkero.paas.allizom.org/Ygen/raw

As comment 7 guessed thi was a mount problem, I tried to remount /system as read-write and restarted the process, but no luck. (mount -o remount,rw /system).


(I think I am providing the requested info from nhirata, but I won't clear the needinfo field, in case my problem is not exactly the same one and I am stealing this bug :))
Have we ever successfully updated via FOTA from update.boot2gecko.org?  Wondering if this is an isolated issue or an overall bustage.
Attached file v1.3_logcat.txt — 
Issue also occurs when trying to go from 1.3 yesterday's build to 1.4 today's build.

Gaia      dd94a5ba3f56ce1e8b378feeecc8a2d7a0b30fc7
Gecko     https://hg.mozilla.org/releases/mozilla-b2g28_v1_3/rev/399b31f5a815
BuildID   20140209164003
Version   28.0
ro.build.version.incremental=eng.tclxa.20131223.163538
ro.build.date=Mon Dec 23 16:36:04 CST 2013
Flags: needinfo?(nhirata.bugzilla)
oops. I mean 1.3 from yesterday's build to 1.3 of today's build.
Has anyone successfully updated buri since bug 958202 has gone live?
blocking-b2g: 1.4? → 1.3?
Keywords: qawanted
(In reply to Aki Sasaki [:aki] from comment #22)
> Has anyone successfully updated buri since bug 958202 has gone live?

Nope - haven't seen anyone pull this off yet.
(In reply to Jason Smith [:jsmith] from comment #23)
> (In reply to Aki Sasaki [:aki] from comment #22)
> > Has anyone successfully updated buri since bug 958202 has gone live?
> 
> Nope - haven't seen anyone pull this off yet.

Ok.  Either something is busted in my rollout, or something is busted with the FOTA in general.

If http://update.boot2gecko.org/hamachi/1.4.0/nightly/update.xml looks good and http://update.boot2gecko.org/hamachi/1.4.0/nightly/b2g_update_20140210040202.mar?build_id=20140210040202&version=30.0a1 is a good FOTA mar, we can rule out the former.
Blocking 1.3 (1.3+) as FOTA doesn't work now.
blocking-b2g: 1.3? → 1.3+
Keywords: qablocker
(In reply to Aki Sasaki [:aki] from comment #24)
> (In reply to Jason Smith [:jsmith] from comment #23)
> > (In reply to Aki Sasaki [:aki] from comment #22)
> > > Has anyone successfully updated buri since bug 958202 has gone live?
> > 
> > Nope - haven't seen anyone pull this off yet.
> 
> Ok.  Either something is busted in my rollout, or something is busted with
> the FOTA in general.
> 
> If http://update.boot2gecko.org/hamachi/1.4.0/nightly/update.xml looks good
> and
> http://update.boot2gecko.org/hamachi/1.4.0/nightly/b2g_update_20140210040202.
> mar?build_id=20140210040202&version=30.0a1 is a good FOTA mar, we can rule
> out the former.

Searching for http://mxr.mozilla.org/mozilla-central/search?string=isosupdate , I see that the string is isOSUpdate not isOsUpdate.  I'm going to make that change on update.boot2gecko.org ...
(In reply to Aki Sasaki [:aki] from comment #26)
> (In reply to Aki Sasaki [:aki] from comment #24)
> > (In reply to Jason Smith [:jsmith] from comment #23)
> > > (In reply to Aki Sasaki [:aki] from comment #22)
> > > > Has anyone successfully updated buri since bug 958202 has gone live?
> > > 
> > > Nope - haven't seen anyone pull this off yet.
> > 
> > Ok.  Either something is busted in my rollout, or something is busted with
> > the FOTA in general.
> > 
> > If http://update.boot2gecko.org/hamachi/1.4.0/nightly/update.xml looks good
> > and
> > http://update.boot2gecko.org/hamachi/1.4.0/nightly/b2g_update_20140210040202.
> > mar?build_id=20140210040202&version=30.0a1 is a good FOTA mar, we can rule
> > out the former.
> 
> Searching for
> http://mxr.mozilla.org/mozilla-central/search?string=isosupdate , I see that
> the string is isOSUpdate not isOsUpdate.  I'm going to make that change on
> update.boot2gecko.org ...

Done. Could someone re-test?
Marking qawanted for a retest.
Keywords: qawanted
dave: assigning to you since you are investigating this today
Assignee: nobody → dhylands
Dave,

Please review if this turns out to be a real issue per retest.
Assignee: dhylands → nobody
Flags: needinfo?(dhylands)
Fixing bz conflict
Assignee: nobody → dhylands
Flags: needinfo?(dhylands)
\o/

\o/

that did work on my Buri :)
Attached file buri.fota.log — 
Successfull FOTA recovery log \o/
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
(In reply to Alexandre LISSY :gerard-majax from comment #33)
> Created attachment 8374348 [details]
> buri.fota.log
> 
> Successfull FOTA recovery log \o/

Confirmed I'm not bricked, but there's one problem here - the system update isn't applying the new package, as the old build ID = new build ID. I'm opening a new ticket to track this.
Status: RESOLVED → VERIFIED
Keywords: qawanted
Followup bug filed in bug 971204.
Whiteboard: [systemsfe] → [systemsfe][NPOTB]
You need to log in before you can comment on or make changes to this bug.