965874 - Error occurs when trying to FOTA update due to /system partition as read-only

Status: VERIFIED FIXED

(Firefox OS Graveyard :: General, defect)

Description

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

1. flash a 1.3 build on the device
2. change the channel to 1.4
3. force update via the settings

expected: FOTA downloads, reboots and updates, then reboots again
actual: error:
01-30 13:05:18.298: I/Gecko(1810): *** AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
01-30 13:05:18.298: I/Gecko(1810): UpdatePrompt: Update error, state: failed, errorCode: 7
01-30 13:05:18.328: I/Gecko(1810): UpdatePrompt: Setting gecko.updateStatus: Install Pending
01-30 13:05:18.358: I/Gecko(1810): *** AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7

Comment 1

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: logcat_OTA.txt

1.4 to 1.4 has the same issue for nkot.  I think I may have a different issue when I try to update mine.  nkot's logcat is in the attachments:

01-31 11:17:19.019: I/GonkAutoMounter(1482): Mounted /system partition as read-only
01-31 11:17:19.039: I/Gecko(136): *** AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
01-31 11:17:19.039: I/Gecko(136): UpdatePrompt: Update error, state: failed, errorCode: 7
01-31 11:17:19.069: I/Gecko(136): UpdatePrompt: Setting gecko.updateStatus: Install Pending
01-31 11:17:19.119: I/Gecko(136): *** AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7

Comment 2

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Not sure who's working on FOTA?

Comment 3

[Aki Sasaki (not active)]

I have deployed FOTA for buri on update.boot2gecko.org, but that's the extent of my knowledge.

Comment 4

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Found out that Alexandre is working on FOTA.  Alexandre, could you take a look at this please?

Comment 5

[:gerard-majax]

I have no idea about this, I worked on the lower level, producing the update.zip that works in recovery mode and that is embedded in the MAR package.

Comment 6

[Jason Smith [:jsmith]]

Dave - Any ideas?

Comment 7

[Dave Hylands [:dhylands]]

I suspect that the /system partition is mounted read-only by default and the update script may need to mount /system as writable.

We may need to ask the vendor.

It may also depend on the kernel/boot.img which is flashed onto the phone.

Comment 8

[Dave Hylands [:dhylands]]

Reading comment 1, the updater should only be trying to mount the /system partition as writable when applying an OTA update, not when applying a FOTA update.

The attached logcat also indicates that this is an OTA update versus a FOTA update.

Comment 9

[Dave Hylands [:dhylands]]

This is really strange.

According to the logcat, the update which was downloaded was this one:
http://update.boot2gecko.org/hamachi/1.4.0/nightly/b2g_update_20140131040202.mar?build_id=20140131040202&version=29.0a1

which appears to be a FOTA update containing the following files:
-rw-r--r-- 1 dhylands dhylands       14 Jan 31 23:03 precomplete
-rw-r--r-- 1 dhylands dhylands       67 Jan 31 23:03 update.manifest
-rw-r--r-- 1 dhylands dhylands 50380167 Jan 31 23:03 update.zip
-rw-r--r-- 1 dhylands dhylands       78 Jan 31 23:03 updatev2.manifest

However, the updater prints a progress line (with ===) for each file, and the number of progress lines printed doesn't agree with the .mar file containing 4 files.

The failed 7 also indicates that it was processing an OTA update rather than a FOTA update.

Comment 10

[Gabriele Svelto [:gsvelto]]

(In reply to Dave Hylands [:dhylands] from comment #9)
> The failed 7 also indicates that it was processing an OTA update rather than
> a FOTA update.

Is it possible that the wrong update file was downloaded? I.e. we dowloaded an OTA update and tried to process it as a FOTA one or something along the lines?

Comment 11

[:gerard-majax]

This is still reproducing as of right now, with hamachi/1.4.0/nightly channel:
I/GonkAutoMounter( 1467): Mounted /system partition as read-only
I/Gecko   ( 1219): *** AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
E/GeckoConsole( 1219): AUS:SVC readStatusFile - status: failed: 7, path: /data/local/updates/0/update.status
I/Gecko   ( 1219): UpdatePrompt: Update error, state: failed, errorCode: 7
I/Gecko   ( 1219): UpdatePrompt: Setting gecko.updateStatus: Install Pending
I/Gecko   ( 1219): *** AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7
E/GeckoConsole( 1219): AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: pending, status: failed: 7

Comment 12

[:gerard-majax]

If I'm right, errorCode 7 means WRITE_ERROR.

Comment 13

[Candice Serran (:cserran)]

Hey Alex, 

Jason raised this in sprint planning as a priority bug.  Saw that you have already reproduced this.  Would you be the correct person to fix this bug?

Comment 14

[Jason Smith [:jsmith]]

Can someone check if FOTA works on 1.3 on Buri?

Comment 15

[:gerard-majax]

(In reply to Candice Serran (:cserran) from comment #13)
> Hey Alex, 
> 
> Jason raised this in sprint planning as a priority bug.  Saw that you have
> already reproduced this.  Would you be the correct person to fix this bug?

Unfortunately no, but as we discussed with Jason, probbaly Dave can help :)

Comment 16

[:gerard-majax]

Dave already replied here ...

Comment 17

[Dave Hylands [:dhylands]]

If I can reproduce it locally, then I can take a deeper look to figure out what's going on.

Naoki, in your STR you said:

1. flash a 1.3 build on the device
2. change the channel to 1.4
3. force update via the settings

Can you point me to a particular 1.3 build?
How exactly did you change the channel to 1.4?
For step 3, I'm assuming you just meant "Check Now"?

Also, do I need any particular base image on the hamachi?

Comment 18

[Frederik Braun [:freddy]]

I get the very same issue with a buri on 1.4 nightly (https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2g_ril/latest-hamachi-mozilla-central/).

STR:
1. Install nightly from above
2. Search for Update
3. Install
4: Error similar to what's been said above. Except here: https://dpaste-bkero.paas.allizom.org/Ygen/raw

As comment 7 guessed thi was a mount problem, I tried to remount /system as read-write and restarted the process, but no luck. (mount -o remount,rw /system).


(I think I am providing the requested info from nhirata, but I won't clear the needinfo field, in case my problem is not exactly the same one and I am stealing this bug :))

Comment 19

[Aki Sasaki (not active)]

Have we ever successfully updated via FOTA from update.boot2gecko.org?  Wondering if this is an isolated issue or an overall bustage.

Comment 20

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: v1.3_logcat.txt

Issue also occurs when trying to go from 1.3 yesterday's build to 1.4 today's build.

Gaia      dd94a5ba3f56ce1e8b378feeecc8a2d7a0b30fc7
Gecko     https://hg.mozilla.org/releases/mozilla-b2g28_v1_3/rev/399b31f5a815
BuildID   20140209164003
Version   28.0
ro.build.version.incremental=eng.tclxa.20131223.163538
ro.build.date=Mon Dec 23 16:36:04 CST 2013

Comment 21

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

oops. I mean 1.3 from yesterday's build to 1.3 of today's build.

Comment 22

[Aki Sasaki (not active)]

Has anyone successfully updated buri since bug 958202 has gone live?

Comment 23

[Jason Smith [:jsmith]]

(In reply to Aki Sasaki [:aki] from comment #22)
> Has anyone successfully updated buri since bug 958202 has gone live?

Nope - haven't seen anyone pull this off yet.

Comment 24

[Aki Sasaki (not active)]

(In reply to Jason Smith [:jsmith] from comment #23)
> (In reply to Aki Sasaki [:aki] from comment #22)
> > Has anyone successfully updated buri since bug 958202 has gone live?
> 
> Nope - haven't seen anyone pull this off yet.

Ok.  Either something is busted in my rollout, or something is busted with the FOTA in general.

If http://update.boot2gecko.org/hamachi/1.4.0/nightly/update.xml looks good and http://update.boot2gecko.org/hamachi/1.4.0/nightly/b2g_update_20140210040202.mar?build_id=20140210040202&version=30.0a1 is a good FOTA mar, we can rule out the former.

Comment 25

[Preeti Raghunath(:Preeti)]

Blocking 1.3 (1.3+) as FOTA doesn't work now.

Comment 26

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #24)
> (In reply to Jason Smith [:jsmith] from comment #23)
> > (In reply to Aki Sasaki [:aki] from comment #22)
> > > Has anyone successfully updated buri since bug 958202 has gone live?
> > 
> > Nope - haven't seen anyone pull this off yet.
> 
> Ok.  Either something is busted in my rollout, or something is busted with
> the FOTA in general.
> 
> If http://update.boot2gecko.org/hamachi/1.4.0/nightly/update.xml looks good
> and
> http://update.boot2gecko.org/hamachi/1.4.0/nightly/b2g_update_20140210040202.
> mar?build_id=20140210040202&version=30.0a1 is a good FOTA mar, we can rule
> out the former.

Searching for http://mxr.mozilla.org/mozilla-central/search?string=isosupdate , I see that the string is isOSUpdate not isOsUpdate.  I'm going to make that change on update.boot2gecko.org ...

Comment 27

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #26)
> (In reply to Aki Sasaki [:aki] from comment #24)
> > (In reply to Jason Smith [:jsmith] from comment #23)
> > > (In reply to Aki Sasaki [:aki] from comment #22)
> > > > Has anyone successfully updated buri since bug 958202 has gone live?
> > > 
> > > Nope - haven't seen anyone pull this off yet.
> > 
> > Ok.  Either something is busted in my rollout, or something is busted with
> > the FOTA in general.
> > 
> > If http://update.boot2gecko.org/hamachi/1.4.0/nightly/update.xml looks good
> > and
> > http://update.boot2gecko.org/hamachi/1.4.0/nightly/b2g_update_20140210040202.
> > mar?build_id=20140210040202&version=30.0a1 is a good FOTA mar, we can rule
> > out the former.
> 
> Searching for
> http://mxr.mozilla.org/mozilla-central/search?string=isosupdate , I see that
> the string is isOSUpdate not isOsUpdate.  I'm going to make that change on
> update.boot2gecko.org ...

Done. Could someone re-test?

Comment 28

[Jason Smith [:jsmith]]

Marking qawanted for a retest.

Comment 29

[Hema Koka [:hema]]

dave: assigning to you since you are investigating this today

Comment 30

[Preeti Raghunath(:Preeti)]

Dave,

Please review if this turns out to be a real issue per retest.

Comment 31

[Jason Smith [:jsmith]]

Fixing bz conflict

Comment 32

[:gerard-majax]

\o/

\o/

that did work on my Buri :)

Comment 33

[:gerard-majax]

Attachment: buri.fota.log

Successfull FOTA recovery log \o/

Comment 34

[Jason Smith [:jsmith]]

(In reply to Alexandre LISSY :gerard-majax from comment #33)
> Created attachment 8374348 [details]
> buri.fota.log
> 
> Successfull FOTA recovery log \o/

Confirmed I'm not bricked, but there's one problem here - the system update isn't applying the new package, as the old build ID = new build ID. I'm opening a new ticket to track this.

Comment 35

[Jason Smith [:jsmith]]

Followup bug filed in bug 971204.