944422 - [Wap push] SI messages which contain '&' symbol in the URL are not received

Status: RESOLVED INVALID

(Firefox OS Graveyard :: Gaia, defect)

Description

[isabelrios]

Seen on v1.2 11/28 build:
Gecko-d77ba08
Gaia-1b67b55

1. From a valid tool, send a wap push SI message to the DuT containing:
Tex: 120 characters
URL: 120 characters

EXPETED
The message is received and displayed correctly. 
This test will be part of the certification process.

ACTUAL
The message is not received. 
For more info, this scenarios has been checked:
-Message 120 characters, short url: OK
-Short message, url 120 characters: NOK
-Message and url 120 characters each: NOK

Comment 1

[Preeti Raghunath(:Preeti)]

Product team,

Please chime in about what the acceptable number of characters should be.

Comment 2

[Peter Dolanjski [:pdol]]

Bruce, what is the requirement for number of characters?

Comment 3

[Enpei]

Hi Bruce,

I cannot reproduce the bug on Buri.
Gaia      8d762f3376318fd6be390432db750ae4904c9ab6
Gecko     http://hg.mozilla.org/releases/mozilla-b2g26_v1_2/rev/758f3fb32dda
BuildID   20131204004003
Version   26.0

It can receive SI with 150 characters in URL. And able to receive 500 characters in text.

-Message 120 characters, short url: OK
-Short message, url 120 characters: OK
-Message and url 120 characters each: OK

But if you know the spec, please let me know.

Comment 4

[Jason Smith [:jsmith]]

Isabel - Are you able to reproduce this on the latest build available? comment 3 seems to imply this doesn't reproduce on most recent builds.

Comment 5

[Bruce Huang [:bhuang] (inactive)]

There's actually no limitation that I know of for SI messages (in fact that's one of the reasons it's used over SMS).  Isabel, it would be helpful to look into whether there is any specific formatting that is causing this.

Comment 6

[isabelrios]

Hi,

After some more tests, with latest v1.2 available build:
Gecko-492afac
Gaia-e226f3d

Here what I could see:

- Using an almost valid url, corresponding to a bugzilla bug, but removing the two las digits to have 120 chars: https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&namedcmd=Gecko%2Fcore%20bugs%20Basecamp%2B%20Open&list_id=8716
The message is not received

- Using an URL like: www.fagadsfdadfafdfadffadfadfeefedfasdfdfdadfadfefaefaedfadfadfadfaefdfdfadfadfadfadfafdadfaafawefafeawfeafeaefafe.cadfd
The message is recevied successfully

In both cases the text message has 120 chars.

What kind of URL are you using to test, first or second case? 
Thanks,
Isabel

Comment 7

[Enpei]

(In reply to Isabel Rios [:isabel_rios] from comment #6)
> Hi,
> 
> After some more tests, with latest v1.2 available build:
> Gecko-492afac
> Gaia-e226f3d
> 
> Here what I could see:
> 
> - Using an almost valid url, corresponding to a bugzilla bug, but removing
> the two las digits to have 120 chars:
> https://bugzilla.mozilla.org/buglist.
> cgi?cmdtype=runnamed&namedcmd=Gecko%2Fcore%20bugs%20Basecamp%2B%20Open&list_i
> d=8716
> The message is not received
> 
> - Using an URL like:
> www.
> fagadsfdadfafdfadffadfadfeefedfasdfdfdadfadfefaefaedfadfadfadfaefdfdfadfadfad
> fadfafdadfaafawefafeawfeafeaefafe.cadfd
> The message is recevied successfully
> 
> In both cases the text message has 120 chars.
> 
> What kind of URL are you using to test, first or second case? 
> Thanks,
> Isabel
Hi Isabel,

After some test, I found that the main problem is "&" character. Buri can receive https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamedtestnamedcmd=Gecko%2Fcore%20bugs%20Basecamp%2B%20Openlist_id=8716 which is 122 characters, but it cannot receive https://bugillz&.com.tw

If you have the same result, let's update summary. Thanks for bring up this problem.

Comment 8

[Enpei]

will set in-moztrap again once root cause if found.

Comment 9

[isabelrios]

(In reply to Enpei from comment #8)
> will set in-moztrap again once root cause if found.

Hi Enpei, 

You are right, seems to be a problem with the '&' symbol.

Will update the summary then.

Thanks for your help on this.

Comment 10

[Enpei]

(In reply to Isabel Rios [:isabel_rios] from comment #9)
> (In reply to Enpei from comment #8)
> > will set in-moztrap again once root cause if found.
> 
> Hi Enpei, 
> 
> You are right, seems to be a problem with the '&' symbol.
> 
> Will update the summary then.
> 
> Thanks for your help on this.

Thank you for confirming this again!

Comment 11

[Jason Smith [:jsmith]]

Comments for 12/5 triage:

I think this is a blocker. & is realistic to include in a URL. In this case, the user doesn't even receive the WAP push message. This also blocks the Pre-IOT tracker bug.

Comment 12

[Preeti Raghunath(:Preeti)]

For TEF IOT blocker

Comment 13

[David Scravaglieri [:scravag]]

Gabriele could you take a look at this ?

Comment 14

[Dietrich Ayala (:dietrich)]

Assigning to Gabriele for now, if not right person, reassign.

Comment 15

[Gabriele Svelto [:gsvelto]]

I've analyzed this problem and I'm closing it as INVALID, let me explain why.

The issue is caused by the messages being refused by the DOMParser as invalid once they enter the application. As it turns out this is correct and expected behavior: the WAP Push message is an XML file and as such it cannot contain unescaped '&' symbols ('<' and '>' must also be escaped). So the message being sent is really invalid XML and thus should be dropped, which we currently do. If you escape the symbols as per the XML rules you'll see that this works, so in the case of comment 6 the correct message 'href' entry should be:

https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&amp;namedcmd=Gecko%2Fcore%20bugs%20Basecamp%2B%20Open&amp;list_id=8716

Note that '&' symbols have been turned into '&amp;' as per the XML rules. This works as expected with the &amp; strings being turned into '&' symbols in the URL by the DOM parser. What I find surprising is that the platform delivers such messages at all since the XML is invalid. I will open a follow up bug because I think that the platform should drop these messages before delivering them to the application.

Comment 16

[Enpei]

(In reply to Gabriele Svelto [:gsvelto] from comment #15)
> I've analyzed this problem and I'm closing it as INVALID, let me explain why.
> 
> The issue is caused by the messages being refused by the DOMParser as
> invalid once they enter the application. As it turns out this is correct and
> expected behavior: the WAP Push message is an XML file and as such it cannot
> contain unescaped '&' symbols ('<' and '>' must also be escaped). So the
> message being sent is really invalid XML and thus should be dropped, which
> we currently do. If you escape the symbols as per the XML rules you'll see
> that this works, so in the case of comment 6 the correct message 'href'
> entry should be:
> 
> https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&amp;
> namedcmd=Gecko%2Fcore%20bugs%20Basecamp%2B%20Open&amp;list_id=8716
> 
> Note that '&' symbols have been turned into '&amp;' as per the XML rules.
> This works as expected with the &amp; strings being turned into '&' symbols
> in the URL by the DOM parser. What I find surprising is that the platform
> delivers such messages at all since the XML is invalid. I will open a follow
> up bug because I think that the platform should drop these messages before
> delivering them to the application.

Hi Gabriele, I tried sending URLs with or without &amp; on both FxOS and Andorid, here is the result:
1. URL with &
Android: Received. Show exactly what original URL is.
FxOS: No notification.
2. URL with &amp;
Android: Received. Show exactly what original URL is.
FxOS: Received. &amp; will be shown as "&" in WAP push message.

I am not sure whether "&" in URL is a normal case in general live WAP push message, or sender will send with &amp; first, but just base on Android result, could that be a reference or we can say it's an incorrect design?

Meanwhile I found a bug that whenever FxOS receives SI message with "&", it cannot receive any new SI messages even URL has no "&" in it. Filed bug 949963 for tracking.

Comment 17

[Gabriele Svelto [:gsvelto]]

First of all thank you very much for having done this cross-verification, it allowed me to figure out what's going on in more detail:

(In reply to Enpei from comment #16)
> Hi Gabriele, I tried sending URLs with or without & on both FxOS and
> Andorid, here is the result:
> [...]
> 2. URL with &
> Android: Received. Show exactly what original URL is.
> FxOS: Received. & will be shown as "&" in WAP push message.

Does this mean that in the '&' case both FxOS and Android show the same URL?

> I am not sure whether "&" in URL is a normal case in general live WAP push
> message, or sender will send with & first, but just base on Android
> result, could that be a reference or we can say it's an incorrect design?

If the original message was in XML format then this behavior is incorrect as the '&' symbol should have been escaped into '&'. If the original message was in WBXML format however (mime-type application/vnd.wap.sic or application/vnd.wap.slc) then Android's behavior is correct; this is because WBXML allows the '&' character to be encoded as-is if the target encoding allows it (see WAP-192-WBXML-20010725-a section 6.1 here http://is.gd/7QBBJX ).

In this case however it means that our platform layer is receiving a valid WBXML document and converting it into an invalid XML document. It might be a platform bug but I have to verify it. If this is the case I'll re-open this bug and attach a fix.

> Meanwhile I found a bug that whenever FxOS receives SI message with "&", it
> cannot receive any new SI messages even URL has no "&" in it. Filed bug
> 949963 for tracking.

Thanks, I'll have a look at it.

Comment 18

[Enpei]

(In reply to Gabriele Svelto [:gsvelto] from comment #17)
> First of all thank you very much for having done this cross-verification, it
> allowed me to figure out what's going on in more detail:
> 
> (In reply to Enpei from comment #16)
> > Hi Gabriele, I tried sending URLs with or without & on both FxOS and
> > Andorid, here is the result:
> > [...]
> > 2. URL with &
> > Android: Received. Show exactly what original URL is.
> > FxOS: Received. & will be shown as "&" in WAP push message.
> 
> Does this mean that in the '&' case both FxOS and Android show the same
> URL?
No, Android will still display "&" as the URL while FxOS will convert it and display "&" in wap push message.

Comment 19

[Gabriele Svelto [:gsvelto]]

(In reply to Enpei from comment #18)
> No, Android will still display "&" as the URL while FxOS will convert it
> and display "&" in wap push message.

OK, then Android's doing it right and the issue is caused by Gecko not converting the WBXML message into XML properly. I've opened another bug with a detailed description of the problem, it's bug 950105. If somebody will help me run the xpcshell tests I can provide a fix myself.