[OTA] [Data Migration] [Messages] Lose the original messages after update from v1.0.1 to v1.1.0

VERIFIED FIXED in Firefox 25

Status

()

defect
VERIFIED FIXED
6 years ago
6 years ago

People

(Reporter: askeing, Assigned: gerard-majax)

Tracking

({dataloss})

unspecified
mozilla25
All
Gonk (Firefox OS)
Points:
---
Dependency tree / graph
Bug Flags:
in-testsuite -
in-moztrap +

Firefox Tracking Flags

(blocking-b2g:leo+, firefox23 wontfix, firefox24 wontfix, firefox25 fixed, b2g18 verified, b2g18-v1.0.0 wontfix, b2g18-v1.0.1 wontfix, b2g-v1.1hd fixed)

Details

(Whiteboard: [migration], [u=commsapps-user c=messaging p=0], [LeoVB+])

Attachments

(6 attachments, 2 obsolete attachments)

(Reporter)

Description

6 years ago
### ENV:
Inari
from v1.0.1 (pvt)
    Gaia:     93241eb6c5d6c110710fad8da3ccd4423312b0c9
    Gecko:    http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/9c62297d11b0
    BuildID   20130624070215
    Version   18.0
to v1-train (complete gaia/gecko OTA update)
    Gaia mozillaorg - 16b872bd7d3aa4f1a7ae73bb19806af8b5d23dc0
    Gecko mozillaorg - 914243b75089d42b0672c8b01b7adf33568309f5

Unagi
from v1.0.1 pvt
    Gaia:     93241eb6c5d6c110710fad8da3ccd4423312b0c9
    Gecko:    http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/9c62297d11b0
    BuildID   20130626070217
    Version   18.0
to v1-train
    Gaia mozillaorg - 4d6cbe49fdba32f72ec8bc6280e4782c250ab41e
    Gecko mozillaorg - 914243b75089d42b0672c8b01b7adf33568309f5

PS:
The command of building complete gaia/gecko OTA update package:
B2G_UPDATER=1 B2G_SYSTEM_APPS=1 MOZ_BUILD_DATE=XXX ANDROIDFS_DIR=XXX ./build.sh gecko-update-full
ref: https://wiki.mozilla.org/B2G/Updating


### STR:
1. Flash device to v1.0.1 build from pvt.
2. Change update URL.
3. Open Messages app.
5. Send/receive messages.
6. Install gaia/gecko complete update.
7. After auto-restart, open Messages app.
8. Check the messages in Messages app.

## Actual results:
No original messages after update.

## Expected results:
Still have original messages in Messages app.
(Reporter)

Updated

6 years ago
Blocks: 885114
blocking-b2g: --- → leo?
Component: General → Gaia::SMS
Keywords: dataloss
Whiteboard: [migration]
This is something we'd block the release on if it is reproducible.  Also, I'm seeing something similar myself and will provide some logcat info later today.
blocking-b2g: leo? → leo+
(Reporter)

Comment 2

6 years ago
Posted file adb_logcat
Inari
from v1.0.1 (pvt)
    Gaia:     f000719da5116eca00b78aa8121f38264d3dafee
    Gecko:    http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/9c62297d11b0
    BuildID   20130630070203
    Version   18.0
to v1-train (complete gaia/gecko OTA update)
    Gaia mozillaorg/v1-train - 6ed9316f397dfe09d8fa88506881d1a4e697643a
    Gecko mozillaorg/gecko-18 - 8a1953cec5ba17158bbd715c0647dc5189e4f472

Youtube: http://youtu.be/-9drddDvFZE

Before OTA, send SMS "before ota" to itself by Messages app.

After OTA, lose the SMS "before ota" in Messages app.
Then re-send the SMS "after ota" by Messages app, the original SMS "before ota" appear in Messages app again.
Looking for an engineer on this, David?
Flags: needinfo?(dscravaglieri)
Flags: needinfo?(dscravaglieri)
Whiteboard: [migration] → [migration], [u=commsapps-user c=messaging p=0]
(Assignee)

Comment 4

6 years ago
I'm seeing this in your logcat:
E/GeckoConsole(  585): [JavaScript Error: "[Exception... "Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsIMobileMessageService.createThread]"  nsresult: "0x80070057 (NS_ERROR_ILLEGAL_VALUE)"  location: "JS frame :: jar:file:///system/b2g/omni.ja!/components/MobileMessageDatabaseService.js :: onsuccess :: line 2295"  data: no]" {file: "jar:file:///system/b2g/omni.ja!/components/MobileMessageDatabaseService.js" line: 2295}]

I remember hitting similar issues with some old MMS :)
(Assignee)

Comment 5

6 years ago
I can't reproduce this with the pvt build pointed at. Could you provide the SMS database that contains the faulty SMS so we can investigate this ?

Just adb pull /data/local/indexedDB/chrome/226660312ssm.sqlite is enough.
Flags: needinfo?(fyen)
(Reporter)

Comment 6

6 years ago
Device:  Unage

From v1.0.1
  Gaia:     054cdc27404e2daca91d3065d9783681032b2151
  Gecko:    http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/9c62297d11b0
  BuildID   20130709070211
  Version   18.0

To v1.1.0
(change update URL to http://update.boot2gecko.org/unagi/1.1.0/nightly/update.xml )
  Gaia:     e251ee6bdab13d8620afa8f9c2d5f14e5e6a4f99
  Gecko:    http://hg.mozilla.org/releases/mozilla-b2g18/rev/f917f3fb17ab
  BuildID   20130709070206
  Version   18.0

There are screenshots and db file in the attachment.
Flags: needinfo?(fyen)
(Assignee)

Comment 7

6 years ago
(In reply to Askeing Yen[:askeing] from comment #6)
> Created attachment 773058 [details]
> Update Unagi from v1.0.1 to v1.1.0
> 
> Device:  Unage
> 
> From v1.0.1
>   Gaia:     054cdc27404e2daca91d3065d9783681032b2151
>   Gecko:   
> http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/9c62297d11b0
>   BuildID   20130709070211
>   Version   18.0
> 
> To v1.1.0
> (change update URL to
> http://update.boot2gecko.org/unagi/1.1.0/nightly/update.xml )
>   Gaia:     e251ee6bdab13d8620afa8f9c2d5f14e5e6a4f99
>   Gecko:    http://hg.mozilla.org/releases/mozilla-b2g18/rev/f917f3fb17ab
>   BuildID   20130709070206
>   Version   18.0
> 
> There are screenshots and db file in the attachment.

Thanks, I'll look into this. Can you check if you hit the issue when simply flashing gecko ?

Updated

6 years ago
Flags: in-moztrap?
(Reporter)

Comment 8

6 years ago
It still draft, will be actived after review.
  https://moztrap.mozilla.org/manage/case/8962/

mark as moztrap+
Flags: in-moztrap? → in-moztrap+
(Assignee)

Comment 9

6 years ago
Excellent, after reflashing a v1-train and battling a bit and then pushing your database, I'm reproducing the issue. Let's investigate now!
(Assignee)

Updated

6 years ago
Assignee: nobody → lissyx+mozillians
(Assignee)

Comment 10

6 years ago
So the culprit is that the thread's lastMessageType element was never added during the schema upgrade from v10 to v11. The database you provided has a version set to 11, yet the upgrade path should have added this field.

Debug shows that this field is 'undefined'.
(Assignee)

Comment 11

6 years ago
(In reply to Askeing Yen[:askeing] from comment #6)
> Created attachment 773058 [details]
> Update Unagi from v1.0.1 to v1.1.0
> 
> Device:  Unage
> 
> From v1.0.1
>   Gaia:     054cdc27404e2daca91d3065d9783681032b2151
>   Gecko:   
> http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/9c62297d11b0
>   BuildID   20130709070211
>   Version   18.0
> 
> To v1.1.0
> (change update URL to
> http://update.boot2gecko.org/unagi/1.1.0/nightly/update.xml )
>   Gaia:     e251ee6bdab13d8620afa8f9c2d5f14e5e6a4f99
>   Gecko:    http://hg.mozilla.org/releases/mozilla-b2g18/rev/f917f3fb17ab
>   BuildID   20130709070206
>   Version   18.0
> 
> There are screenshots and db file in the attachment.

Did you performed the copy AFTER updating to 1.1 ?

Could you post database from 1.0.1, BEFORE doing the update ?
Flags: needinfo?(fyen)
(Assignee)

Comment 12

6 years ago
(In reply to Alexandre LISSY :gerard-majax from comment #10)
> So the culprit is that the thread's lastMessageType element was never added
> during the schema upgrade from v10 to v11. The database you provided has a
> version set to 11, yet the upgrade path should have added this field.
> 
> Debug shows that this field is 'undefined'.

If I force by hand the version to be 10, push the database on my v1-train, it correctly updates to 11 and the field is not missing.
(Assignee)

Comment 13

6 years ago
Okay, I've retried to reproduce from scratch, and I still cannot. I really need that you provide the database of 1.0.1 version.
(Reporter)

Comment 14

6 years ago
Posted file v101 sms db
Unagi pvt v1.0.1 build
Gaia:     054cdc27404e2daca91d3065d9783681032b2151
  B-D     2013-07-06 19:39:29
Gecko:    http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/9c62297d11b0
BuildID   20130710070208
Version   18.0
Flags: needinfo?(fyen)
(Reporter)

Comment 15

6 years ago
Posted file v110 sms db
After update to http://update.boot2gecko.org/unagi/1.1.0/nightly/update.xml
Gaia:     680c72ccbd2d5045e590a6ba08de2aac6af71953
  B-D     2013-07-11 05:27:20
Gecko:    http://hg.mozilla.org/releases/mozilla-b2g18/rev/b64372810dd7
BuildID   20130710190458
Version   18.0
(Reporter)

Comment 16

6 years ago
Posted file adb logcat
for 773772 & 773773
(Assignee)

Comment 17

6 years ago
Thanks, I'm having a look at this !
(Assignee)

Comment 18

6 years ago
Funny, the v1.0.1 database's DB_VERSION is at 1. And I can't open it correctly within "indexedDB browser" addon.
(Assignee)

Comment 19

6 years ago
Okay, I collected some log while in the train:

I/Gecko   ( 4727): MobileMessageDatabaseService[1373524660586]: Opening transaction for object stores: sms
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524660587]: Opening transaction for object stores: sms
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663088]: Database needs upgrade: sms 8 11
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663093]: Correct new database version: true
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663094]: Upgrade to version 9. Add transactionId index for incoming MMS.
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663101]: Upgrade to version 10. Upgrade type if it's not existing.
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663102]: upgradeSchema9: IN
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663108]: Upgrade to version 11. Add last message type into threadRecord.
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663109]: upgradeSchema10: IN
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663553]: upgradeSchema9: openCursor().onsuccess
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663553]: upgradeSchema9: openCursor().onsuccess: messageRecord={"deliveryIndex":["sent",1373516526726],"readIndex":[1,1373516526726],"delivery":"sent","deliveryStatus":"success","sender":null,"receiver":"0978725105","body":"Before update.","messageClass":"normal","timestamp":1373516526726,"read":1,"id":1,"threadId":1,"threadIdIndex":[1,1373516526726],"participantIdsIndex":[[1,1373516526726]]}
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663555]: upgradeSchema9: openCursor().onsuccess: messageRecord.type typeof undefined
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663555]: upgradeSchema9: openCursor().onsuccess: messageRecord.type is undefined
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663556]: upgradeSchema9: openCursor().onsuccess: new messageRecord={"deliveryIndex":["sent",1373516526726],"readIndex":[1,1373516526726],"delivery":"sent","deliveryStatus":"success","sender":null,"receiver":"0978725105","body":"Before update.","messageClass":"normal","timestamp":1373516526726,"read":1,"id":1,"threadId":1,"threadIdIndex":[1,1373516526726],"participantIdsIndex":[[1,1373516526726]],"type":"sms"}
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663563]: upgradeSchema10: openCursor().onsuccess
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663567]: upgradeSchema10: threadRecord={"participantIds":[1],"participantAddresses":["0978725105"],"lastMessageId":2,"lastTimestamp":1373516528517,"subject":"Before update.","unreadCount":0,"id":1}
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663678]: upgradeSchema9: openCursor().onsuccess
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663679]: upgradeSchema9: openCursor().onsuccess: messageRecord={"deliveryIndex":["received",1373516528517],"readIndex":[1,1373516528517],"delivery":"received","deliveryStatus":"success","sender":"+886978725105","receiver":null,"body":"Before update.","messageClass":"normal","timestamp":1373516528517,"read":1,"id":2,"threadId":1,"threadIdIndex":[1,1373516528517],"participantIdsIndex":[[1,1373516528517]]}
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663679]: upgradeSchema9: openCursor().onsuccess: messageRecord.type typeof undefined
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663680]: upgradeSchema9: openCursor().onsuccess: messageRecord.type is undefined
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663680]: upgradeSchema9: openCursor().onsuccess: new messageRecord={"deliveryIndex":["received",1373516528517],"readIndex":[1,1373516528517],"delivery":"received","deliveryStatus":"success","sender":"+886978725105","receiver":null,"body":"Before update.","messageClass":"normal","timestamp":1373516528517,"read":1,"id":2,"threadId":1,"threadIdIndex":[1,1373516528517],"participantIdsIndex":[[1,1373516528517]],"type":"sms"}
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663687]: upgradeSchema10: request.onsuccess
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663687]: upgradeSchema10: request.onsuccess: continuing with messageRecord={"deliveryIndex":["received",1373516528517],"readIndex":[1,1373516528517],"delivery":"received","deliveryStatus":"success","sender":"+886978725105","receiver":null,"body":"Before update.","messageClass":"normal","timestamp":1373516528517,"read":1,"id":2,"threadId":1,"threadIdIndex":[1,1373516528517],"participantIdsIndex":[[1,1373516528517]]}
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663688]: upgradeSchema10: request.onsuccess: setting threadRecord.lastMessageType=undefined
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663695]: upgradeSchema10: openCursor().onsuccess
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663746]: upgradeSchema9: openCursor().onsuccess
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524663752]: upgradeSchema9: openCursor().onsuccess: no cursor!
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524648521]: Opened database: sms 11
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524648524]: Started transaction [object IDBTransaction] of type readonly
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524648527]: Retrieving object store sms
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524648569]: Last assigned message ID was 2
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524648598]: Transaction [object IDBTransaction] completed.
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524648599]: Opened database: sms 11
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524648600]: Started transaction [object IDBTransaction] of type readwrite
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524648601]: Retrieving object store sms
I/Gecko   ( 4727): MobileMessageDatabaseService[1373524648672]: Transaction [object IDBTransaction] completed.

So it's like upgradeSchema9 is not committed to the SQLite database !
(Assignee)

Comment 20

6 years ago
I have a fix, inspired by bug 862385, but I'm not very convinced it's the correct way to handle this case.
(Assignee)

Comment 21

6 years ago
Please find attached a first tentative patch that enforces the serialization by moving upgrade code for v9 into v10.
Attachment #773943 - Flags: feedback?(ctai)
(Assignee)

Comment 22

6 years ago
(In reply to Alexandre LISSY :gerard-majax from comment #21)
> Created attachment 773943 [details] [diff] [review]
> Ensuring serialization of adding .type before using it
> 
> Please find attached a first tentative patch that enforces the serialization
> by moving upgrade code for v9 into v10.

Better patch soon with help of :julienw eye's!
(Assignee)

Comment 23

6 years ago
Please find attached a better patch (thanks to :julienw!) that fixes the issue.
Attachment #773943 - Attachment is obsolete: true
Attachment #773943 - Flags: feedback?(ctai)
Attachment #774003 - Flags: review?(gene.lian)
Should the product/component be updated for this? (for the sake of good housekeeping)
yes. (and it will help uplifting as well)
Component: Gaia::SMS → DOM: Device Interfaces
Product: Boot2Gecko → Core
Sorry I don't understand why the original logic:

  while (currentVersion != event.newVersion) {
    // ...
    currentVersion++;
  }

doesn't work? Could you please summarize why/how your patch can fix the upgrading process? I don't see anything different for fixing upgradeSchema9.
Gene: the problem with a synchronous loop is that all cursor codes run concurrently, due to their asynchronous nature.

It means we don't know which cursor callbacks get called when and in which order.

With Alexandre's patch, we ensure the upgrade path for version n is called only when the ugprade for version n-1 has finished.

Is it clearer ?
Oh! Got it! Reviewing.
Btw, I'm wondering that's why sometimes we would have mysterious type errors during the DB initialization. Nice catch, Alexandre!
Comment on attachment 774003 [details] [diff] [review]
Ensuring serialization of upgrades

Review of attachment 774003 [details] [diff] [review]:
-----------------------------------------------------------------

Nice catch!

::: dom/mobilemessage/src/gonk/MobileMessageDatabaseService.js
@@ +173,5 @@
>  
>        let currentVersion = event.oldVersion;
> +
> +      function update(currentVersion) {
> +        var next = update.bind(self, currentVersion + 1);

nit: s/var/let/

@@ +720,4 @@
>      };
>    },
>  
> +  upgradeSchema10: function upgradeSchema10(transaction, next) {

I found there's something wrong in the upgradeSchema10 (not your fault). We should put cursor.continue() into the request.onsuccess and request.onerror so that we can ensure everything is synchronously updated after applying your patch.

That would be nice if you could correct that since you're here. Reopening bug 863241 also works for me.
Attachment #774003 - Flags: review?(gene.lian) → review+
(Assignee)

Comment 31

6 years ago
(In reply to Gene Lian [:gene] from comment #30)
> Comment on attachment 774003 [details] [diff] [review]
> Ensuring serialization of upgrades
> 
> Review of attachment 774003 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Nice catch!
> 
> ::: dom/mobilemessage/src/gonk/MobileMessageDatabaseService.js
> @@ +173,5 @@
> >  
> >        let currentVersion = event.oldVersion;
> > +
> > +      function update(currentVersion) {
> > +        var next = update.bind(self, currentVersion + 1);
> 
> nit: s/var/let/
> 
> @@ +720,4 @@
> >      };
> >    },
> >  
> > +  upgradeSchema10: function upgradeSchema10(transaction, next) {

Thanks.

I'll be uploading a revised patch for this one.

> 
> I found there's something wrong in the upgradeSchema10 (not your fault). We
> should put cursor.continue() into the request.onsuccess and request.onerror
> so that we can ensure everything is synchronously updated after applying
> your patch.
> 
> That would be nice if you could correct that since you're here. Reopening
> bug 863241 also works for me.

Unfortunately, I'm not sure I can do the change in time since this is my week off, and I have to admit that I expected to have time but I won't. If you can wait until next monday, I can address it at the same time and ensure proper testing.
(Assignee)

Comment 32

6 years ago
Please find a revised patch, addressing the s/var/let nit.
Attachment #774003 - Attachment is obsolete: true
Attachment #777312 - Flags: review?(gene.lian)
Comment on attachment 777312 [details] [diff] [review]
Ensuring serialization of upgrades v2

Review of attachment 777312 [details] [diff] [review]:
-----------------------------------------------------------------

No worries. Let's solve this issue first. I think we can reopen that bug instead.
Attachment #777312 - Flags: review?(gene.lian) → review+
(Assignee)

Updated

6 years ago
Keywords: checkin-needed
https://hg.mozilla.org/projects/birch/rev/1b19f45cc922

Should this have tests?
Flags: in-testsuite?
Keywords: checkin-needed
(Assignee)

Comment 35

6 years ago
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #34)
> https://hg.mozilla.org/projects/birch/rev/1b19f45cc922
> 
> Should this have tests?

I'd like to, but we discussed about this in Paris office and we really miss some framework to ensure proper upgrades are running.
https://hg.mozilla.org/mozilla-central/rev/1b19f45cc922
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla25
(Reporter)

Comment 39

6 years ago
Unagi v1.0.1
  Gaia:     054cdc27404e2daca91d3065d9783681032b2151
  Gecko:    http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/9c62297d11b0
  BuildID   20130721070205
  Version   18.0
to v1.1.0 http://update.boot2gecko.org/unagi/1.1.0/nightly/update.xml
  Gaia:     c4bdd3210864938b8ffd252c442ddcd442d09944
  Gecko:    http://hg.mozilla.org/releases/mozilla-b2g18/rev/1dffc79bfac5
  BuildID   20130721070210
  Version   18.0

verified.
(Reporter)

Comment 40

6 years ago
There is no v1.0.1=>v1.1.0hd and v1.0.1=>v1.2+ cases.
Mark as verified.
Status: RESOLVED → VERIFIED
(In reply to Askeing Yen[:askeing] from comment #40)
> There is no v1.0.1=>v1.1.0hd and v1.0.1=>v1.2+ cases.

should still work though

Updated

6 years ago
Whiteboard: [migration], [u=commsapps-user c=messaging p=0] → [migration], [u=commsapps-user c=messaging p=0], [LeoVB+]
(Reporter)

Updated

6 years ago
Duplicate of this bug: 897432
You need to log in before you can comment on or make changes to this bug.