886723 - [SMS][MMS] Numbers of Messages Exposed to Gaia Should be Normalized.

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: General, defect)

Comment 1

[Gene Lian (I already quit Mozilla)]

Just having a discussion with Julien and Gregor, we're hoping to normalize the sender/receiver of the messages exposed to the Gaia end, so that the (normalized) numbers can be correctly associated with the (normalized) numbers in the Contacts API.

Comment 2

[Gene Lian (I already quit Mozilla)]

This bug is cloned from bug 877718 and should be marked as leo+.

Comment 3

[Vicamo Yang [:vicamo][:vyang]]

Please don't go back to the old ways.  There is no such thing "normalized" phone number.

Comment 4

[Julien Wajsberg [:julienw]]

Vicamo, could you please explain what you mean ?

In Bug 877718 we need to be able to compare the number we get from the GSM network (as message.sender) with the number we get from contacts.

In this specific case, we could probably compare the 7 last digits (if the contacts API send us the normalized phon enumber) but in the general case it would be quite useful to have it from the SMS API too.

So please explain why this should not be done. Maybe "normalize" is not the right word, but to me it means having a comparable number.

Comment 5

[Gene Lian (I already quit Mozilla)]

Attachment: Patch

Comment 6

[Gene Lian (I already quit Mozilla)]

Attachment: Patch, V2

Comment 7

[Gene Lian (I already quit Mozilla)]

Hi Vicamo, I only normalize the numbers that are going to be exposed to Gaia. The mechanism of finding participants are still doing as it is.

Comment 8

[Vicamo Yang [:vicamo][:vyang]]

(In reply to Julien Wajsberg [:julienw] from comment #4)
> Vicamo, could you please explain what you mean ?

Before bug 833291, we have normalized phone number in B2G SMS implementation.  Then we found:

1). Normalization doesn't always work because it depends on MCC.  In PhoneNumberUtils, MCC id is determined as MCC of current voice network, or MCC of the SIM card, depending on the network status.  However, we store the outgoing message before actually trying to send it.  As a result, the recipient(s) of that outgoing message may not be normalized when voice network is not available.  So in some cases you may get two threads, one with a normalized phone number, the other with an unnormalized one.

2). We need to search messages by non-normalized phone numbers, and that's why we have bug 833291.

Returing normalized numbers is indeed a little bit different from the case we had.  I think we both agree that we should not store normalized phone number, but:

A). It doesn't really solve the problem.  The problem is Contact API doesn't have similar matching mechanism like SMS db has.  Please solve the root cause.

B). As I mentioned above, phone number normalization depends on MCC of either voice network or your SIM card at the time, so this process only works when you have either one.  Adopting this solution means opening bugs for the same phenomenon with bug 877718 but under airplane mode or booting without a SIM card inserted.

C). The number to be normalized won't necessarily be of the same MCC with your SIM card, so PhoneNumberUtils may still either fail to normalize or normalize to a wrong number.

D). You just shouldn't normalize any phone numbers in SMS API.  Output format is also part of API, so from web API's point of view, we just don't have to define the output format for phone numbers got from SMS API calls.  We ensure all phone numbers stored to and retrieved from the SMS database are exactly the same with the ones you feed in.  SMS API doesn't involve any process of phone number handling.  That's an implementation detail.

> In Bug 877718 we need to be able to compare the number we get from the GSM
> network (as message.sender) with the number we get from contacts.

So fix Contacts API instead.

> In this specific case, we could probably compare the 7 last digits (if the
> contacts API send us the normalized phon enumber) but in the general case it
> would be quite useful to have it from the SMS API too.

SMS API has already similar matching algorithm built-in. 

> So please explain why this should not be done. Maybe "normalize" is not the
> right word, but to me it means having a comparable number.

Judging from Gene's patch, yes, your're normalizing phone numbers.  Phone numbers CAN NOT be easily compared.  That's the first thing you should keep in mind before discussing any solution here.  Any solution that tries to compare two simple strings will fail in some cases.  Bug 877718 is just a living example.

Comment 9

[Vicamo Yang [:vicamo][:vyang]]

(In reply to Gene Lian [:gene] from comment #7)
> Hi Vicamo, I only normalize the numbers that are going to be exposed to
> Gaia.

Data exposed to content is ALSO part of API.  Please don't forget this.

Comment 10

[Gregor Wagner [:gwagner]]

(In reply to Vicamo Yang [:vicamo][:vyang] from comment #8)
> (In reply to Julien Wajsberg [:julienw] from comment #4)
> > Vicamo, could you please explain what you mean ?
> 
> Before bug 833291, we have normalized phone number in B2G SMS
> implementation.  Then we found:
> 
> 1). Normalization doesn't always work because it depends on MCC.  In
> PhoneNumberUtils, MCC id is determined as MCC of current voice network, or
> MCC of the SIM card, depending on the network status.  However, we store the
> outgoing message before actually trying to send it.  As a result, the
> recipient(s) of that outgoing message may not be normalized when voice
> network is not available.  So in some cases you may get two threads, one
> with a normalized phone number, the other with an unnormalized one.

We are not talking about transforming it into an international format here. We just want a common base for comparison. The normalization we talk about is just removing non-dialable characters and replacing valid alpha patterns...: http://mxr.mozilla.org/mozilla-central/source/dom/phonenumberutils/PhoneNumber.jsm#242

Comment 11

[Gregor Wagner [:gwagner]]

(In reply to Vicamo Yang [:vicamo][:vyang] from comment #8)
> (In reply to Julien Wajsberg [:julienw] from comment #4)
> 
> Returing normalized numbers is indeed a little bit different from the case
> we had.  I think we both agree that we should not store normalized phone
> number, but:
> 
> A). It doesn't really solve the problem.  The problem is Contact API doesn't
> have similar matching mechanism like SMS db has.  Please solve the root
> cause.

We have now a 'match' feature. The idea is that when we insert a contact we transform it to all known variants: local format, international format, the format the user enters, normalized version... And when we do a 'match' search we match with all these numbers.

Comment 12

[Vicamo Yang [:vicamo][:vyang]]

(In reply to Gregor Wagner [:gwagner] from comment #10)
> (In reply to Vicamo Yang [:vicamo][:vyang] from comment #8)
> > 1). Normalization doesn't always work because it depends on MCC.
> We are not talking about transforming it into an international format here.

But I think Contacts API always normalized filter values, doesn't it?
http://mxr.mozilla.org/mozilla-central/source/dom/contacts/fallback/ContactDB.jsm#462
http://mxr.mozilla.org/mozilla-central/source/dom/contacts/fallback/ContactDB.jsm#870

> We just want a common base for comparison. The normalization we talk about
> is just removing non-dialable characters and replacing valid alpha
> patterns...:
> http://mxr.mozilla.org/mozilla-central/source/dom/phonenumberutils/
> PhoneNumber.jsm#242

Comment 13

[Julien Wajsberg [:julienw]]

as I understand, at the moment we receive a message, we have the correct SIM card. Therefore we could normalize at that moment, and store both numbers. This would always be ok.

Anyway, I think that for our specific bug here we can try to do without this.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Vicamo Yang [:vicamo][:vyang] from comment #8)
> outgoing message before actually trying to send it.  As a result, the
> recipient(s) of that outgoing message may not be normalized when voice
> network is not available.  So in some cases you may get two threads, one
> with a normalized phone number, the other with an unnormalized one.

That would explain why I see bug 871433, right?

Comment 15

[Julien Wajsberg [:julienw]]

Likely not, Vicamo was explaining something that's not implemented.