1149160 - Investigate Marfeel compatibility

Status: RESOLVED FIXED

(Web Compatibility :: Site Reports, defect, P1)

Description

[Mike Taylor [:miketaylr]]

Originally reported in Bug 1147352.

Starting at https://bugzilla.mozilla.org/show_bug.cgi?id=1147352#c3, there is some preliminary investigation into the bug. This bug is a deep dive compat investigation--is it possible to get the site working by spoofing UA, changing prefixes, etc.

Comment 1

[Karl Dubost💡 :karlcow]

Some preliminary findings on Bug 1147352

The gardab.js is trying to match on 
* iOS
  var e = /.*(iPad|iPhone|iPod).*OS ([0-9])_([0-9]).*/g,
* Android Chrome
  var e = /.*Android ([0-9])\.([0-9]).*; (.*) Build.* (Chrome|.+)\/.*? (Mobile|.+ )?.*Safari/g,
* BlackBerry 10
  var e = /.*BB10; ([\d\w]*)\).*Version\/(\d*)\.(\d*)\..*Mobile.*/g,
* Silk
  var e = /\bSilk\b/g;


BUT UA override doesn't solve the issues, the framework might be doing additional detections. 


Some sites using Marfeel and then failing in 
    Windows Phone, 
    Opera (Blink, Presto) 
    Firefox Android
    Firefox OS  
    and probably others
because of the framework:

* http://www.elconfidencial.com/
* http://vozpopuli.com/
* http://www.belleamour.co.uk/
* http://www.confessionsofanover-workedmom.com/
* http://www.the-socialites-closet.com/
* http://www.capital.fr/


In the case of El Confidencial, the script is called by

 window.mrf = {
     host: 'bc.marfeel.com',
     dt: 's',
     blacklistedUrls: ['www.elconfidencial.com/mercados/indice', 'www.elconfidencial.com/mercados/cotizacion', 'www.elconfidencial.com/tags', 'www.elconfidencial.com/autores']
 };
 (function(e, t) {
     var n = new XMLHttpRequest;
     n.open("GET", "//bc.marfeel.com/statics/marfeel/gardab.js", false);
     n.send();
     if (n.status === 200) {
         var r = e.getElementsByTagName("script")[0],
             i = e.createElement("script");
         i.innerHTML = n.responseText;
         r.parentNode.insertBefore(i, r);
     }
 })(document, window);


For Voxpopuli, 1st script.

var serverActual = "vozpopuli.com";
var serverPubli = "ad.vozpopuli.com";
var serverSocial = "social.vozpopuli.com";
var idEdicionActual = 2308;
window.mrfHost = "bc.marfeel.com";
(function(e, t) {
    var n = new XMLHttpRequest;
    n.open("GET", "//bc.marfeel.com/statics/marfeel/gardab.js", false);
    n.send();
    if (n.status === 200) {
        var r = e.getElementsByTagName("script")[0],
            i = e.createElement("script");
        i.innerHTML = n.responseText;
        r.parentNode.insertBefore(i, r)
    }
})(document, window)


The only notable difference in between all the scripts for the above list is

News outlets seem to get 
http://bc.marfeel.com/statics/marfeel/gardab.js
Blogs seem to get
http://b.marfeel.com/statics/marfeel/gardab.js

Some minor differences in between the two scripts:

→ git diff bc.js b.js
diff --git a/bc.js b/b.js
index 02ac869..c2fb68a 100644
--- a/bc.js
+++ b/b.js
@@ -116,6 +116,7 @@
         locationPathname: t.location.pathname === "/" ? "/index" : t.location.pathname,
         navigatorUserAgent: t.navigator.userAgent,
         documentDomain: e.domain,
+        documentCookie: e.cookie,
         windowWidth: t.screen.width,
         windowHeight: t.screen.height,
         getMarfeelHost: function() {
@@ -148,8 +149,8 @@
             var t = "//" + i.getMarfeelHost() + "/" + i.documentDomain + i.locationPathname + i.locationSearch;
             return e && (t = m(t, e)), r.type && (t = m(t, "marfeeldt=" + r.type)), t
         },
-        setLocation: function(e) {
-            t.location.href = e
+        reloadLocation: function() {
+            t.location.reload()
         },
         requestGeo: function() {
             var e = new XMLHttpRequest;
@@ -189,24 +190,23 @@
             return t && t.getAttribute("content") === "false" ? !1 : !0
         },
         comesFromMarfeel: function() {
-            var e = i.locationSearch.indexOf("fromt=yes") > -1 || i.getCookieValue("FromMarfeel") === "YES" || i.getCookieValue("FromMarfeelOnError") === "YES";
-            return i.setCookieValue("FromMarfeel", "NO"), e
+            return /fromt=yes/i.test(i.locationSearch + ";" + i.documentCookie)
         },
         userWantsMarfeel: function() {
             var e = parseInt(i.getCookieValue("MarfeelCreation")) > u;
             return e ? i.getCookieValue("MarfeelGarda") !== "NO" : !0
         },
-        fallbackToClassicVersion: function(t) {
-            i.setLocation(m(t || e.location.href, "fromt=yes"))
+        fallbackToClassicVersion: function(e) {
+            i.setCookieValue("fromt", "YES", e), i.reloadLocation()
         },
         redirectToMarfeel: function(t) {
             var r = i.getMarfeelUrl(t),
                 o = new XMLHttpRequest,
                 u;
             i.documentDomain === "dailycaller.com" && i.requestGeo(), o.open("GET", r), o.timeout = 1e4, o.ontimeout = function() {
-                s.fallbackToClassicVersion()
+                s.fallbackToClassicVersion(6e5)
             }, o.onreadystatechange = function() {
-                o.readyState === n && (o.status > 0 && o.status < 400 ? (u = e.open("text/html", "replace"), u.write(o.responseText), u.close()) : s.fallbackToClassicVe
+                o.readyState === n && (o.status > 0 && o.status < 400 ? (u = e.open("text/html", "replace"), u.write(o.responseText), u.close()) : s.fallbackToClassicVe
             }, o.send()
         },
         showMarfeelBadge: function() {

Comment 2

[Hallvord R. M. Steen [:hallvors]]

On elconfidencial.com the main Marfeel code is delivered here:
http://bc.marfeel.com/statics/www.elconfidencial.com/index/main.s.js?build=6190
There are 62 instances of the string "webkit" and no attempts at supporting engines that require other prefixes, or use standardised equivalents where these are supported. Some details:

In Animation.getTranslateX / Y :

var t = e.style["-webkit-transform"]

This relies on not only the prefix but also the WebKit quirk that gives the style object hyphenated properties. The normal way would be e.style["WebkitTransform"] - camel-cased without hyphens. See bug 970134 for further details.

Several places uses the webkitTransitionEnd event name. Today there is a standardised equivalent:
https://developer.mozilla.org/en-US/docs/Web/Events/transitionend

In _moveContentTo and several other places:

r["-webkit-transform"]
Same issue about hyphenated property names on style. The transform property without prefix has wide engine support by now.
https://developer.mozilla.org/en-US/docs/Web/CSS/transform

updatePosition:
 this.scrollIndicator.style["-webkit-transition-duration"] = n + "s")
https://developer.mozilla.org/en-US/docs/Web/CSS/transition-duration

this.currentImageStyle["-webkit-transform-origin"]
https://developer.mozilla.org/en-US/docs/Web/CSS/transform-origin

Some prefixing regarding full screening video elements:
	function i() {
		this.video.webkitSupportsFullscreen && s.call(this)
	}
	function s() {
		this.video.webkitEnterFullscreen(), this.trackPlay()
	}

(Aside: the hypenated property name style isn't everywhere - I also see some 
this.element.style.webkitTransform )

When loading in Firefox with a spoofed UA (Safari-ish), the site redirects a couple of times, and the redirect seems to come from some JS error handling. I'm not sure exactly what error happens, but it's likely either caused by the old JS environment being blown away by document.write() or the webkit-based code not reading the expected style values from an element.

Comment 3

[Hallvord R. M. Steen [:hallvors]]

The error that causes forwarding to the desktop site is caused by Google Analytics stuff not being defined - so it's about the document.write blowing away the old environment and its variables. Every script in the old document whose variables and methods are expected to be available after document.write, must be included in the newly generated source when document.write() is called after DOMContentReady.

Comment 4

[Hallvord R. M. Steen [:hallvors]]

Something deeper in the JS (main.s.js) is going wrong. This code snippet seems to not do what the author expects:

  i = function (e, t) {
    t = t || n;
    var r = t.querySelectorAll(e);
    return r.__proto__ = i._pluginsArr,
    r
  };

So that's supposed to return a NodeList with __proto__ set to an object with some methods. Some of those methods try to iterate over this with a for() loop, and find nothing.. See
http://jsfiddle.net/hh2em8tm/1/

Comment 5

[Hallvord R. M. Steen [:hallvors]]

(The more compatible way to do this would be to define the required parts from i._pluginsArr on NodeList.prototype instead. I don't know if they avoid that for any particular reason.)

Comment 6

[Hallvord R. M. Steen [:hallvors]]

Attachment: elconfidencial-nowebkit.png

Doing these replacements (with Fiddler) on the source, defining the ga() method Google Analytics was supposed to define and working around errors caused by the __proto__ aproach to iterating nodelists not working gets us so far - the UI shows up just fine, but no content (most likely because nodelist iteration fails). Also, nothing responds to clicks (for the same reason I believe - there's plenty of addEventListener() code that fails to run).

Comment 7

[Hallvord R. M. Steen [:hallvors]]

Attachment: elconfidencial-nowebkit-fixiteration.png

Um, forgot to paste the replacement rules apparently..

oSession.utilReplaceInResponse('-webkit-transform', 'transform');
oSession.utilReplaceInResponse('WebkitTransformOrigin', 'transformOrigin');
oSession.utilReplaceInResponse('WebkitTransform', 'transform');
oSession.utilReplaceInResponse('webkitTransform', 'transform');
oSession.utilReplaceInResponse('webkitTransitionEnd', 'transitionend');
oSession.utilReplaceInResponse('-webkit-transition', 'transition');
oSession.utilReplaceInResponse('-webkit-background-size', 'background-size');

The next two are more hackish - not necessarily recommended:
oSession.utilReplaceInResponse('webkitSupportsFullscreen', 'ownerDocument.mozFullScreenEnabled');
oSession.utilReplaceInResponse('webkitEnterFullscreen', 'mozRequestFullScreen');


Now, the NodeList iterator stuff. I just hacked the code where it tries to set __proto__ - the aforementioned

        i = function(e, t) {
            t = t || n;
            var r = t.querySelectorAll(e);
            for(var prop in i._pluginsArr)NodeList.prototype[prop] = i._pluginsArr[prop];
            return r.__proto__ = i._pluginsArr, r
        };

and made it add references to NodeList.prototype instead:

        nodelistpatched = false,
        i = function(e, t) {
            t = t || n;
            var r = t.querySelectorAll(e);
            if(!nodelistpatched){
                for(var prop in i._pluginsArr)NodeList.prototype[prop] = i._pluginsArr[prop];
                nodelistpatched =  true;
            }
            return /*r.__proto__ = i._pluginsArr, */ r
        };

Et voila - look at this last screenshot! Content appears, interaction (menu, sliding up/down and left/right) seems to work just fine. The only thing missing is a CSS gradient, which I'm sure Marfeel can fix using for example the CSS -webkit-to-standard translation tool I've got here: http://hallvord.com/temp/moz/cssfixme.htm

I plan to do a little device testing to check what the performance is like. But this should be most of the work they need to do to have a cross-browser implementation.

Comment 8

[Hallvord R. M. Steen [:hallvors]]

Marfeel's use of nodelistobject.__proto__ is not expected to work per spec, because length is supposed to be a property of NodeList.prototype. When __proto__ is overwritten, this.length is supposed to return undefined. Not doing this is a bug/quirk in the Blink engine, and they are working on a fix:
https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/H0MGw0jkdn4
A soon to be released Chrome (and presumably Android browser, Safari when they catch up and align with the spec) will render Marfeel's sites without any content unless they take in the fix suggested above or something equivalent.

Comment 9

[Hallvord R. M. Steen [:hallvors]]

(Something equivalent could be Object.create() with NodeList.prototype as its proto. Thanks to Boris Zbarsky for helping me with spec status details.)

Comment 10

[Karl Dubost💡 :karlcow]

Sent an email to Marfeel team about it.

Comment 11

[Karl Dubost💡 :karlcow]

Status about this bug.
The CTO and the engineering Team have been contacted.
They replied explaining their own issues. They designed their system for iOS only using non standard properties, then adjusted to some other WebKit engines. They might make it compatible with the Web in future releases.

We invited them to comment all their technical issues in this bug.
Including test cases and performance benchmarks.
And if they meet with new issues, we can help them to open new bugs about it.

Comment 12

[Hallvord R. M. Steen [:hallvors]]

I'd had a quick look at performance on both Firefox on Android (on a Motorola X) and Firefox OS (on the Flame). Elconfidencial.com (Marfeel edition) is quite usable on both devices, I don't notice any bad lagging or issues. When scrolling sideways, images are not loaded at first but I think this is normal..

Comment 13

[Oliver Fernandez]

In the last months several changes have been made to Marfeel platform in order to get a wider browser compatibility.

1) The redirection SCRIPT no longer uses a blocking XMLHttpRequest in the main thread, since it is being deprecated by several browser vendors

2) Scroll in the main view is no longer based in Webkit custom property -webkit-overflow-scrolling. A massive engineering effort has been made to just use the normal BODY scroll, while conserving at the same time the swipe based navigation

3) The non-standard __proto__ is no longer used


The main pending tasks is to remove -webkit prefixes from CSS. Safari was the last browser that still had important prefixed properties, like transition or transform. They have unprefixed them in Safari 9 https://developer.apple.com/library/prerelease/mac/releasenotes/General/WhatsNewInSafari/Articles/Safari_9.html

We hope that in the coming weeks we will give an ETA of when we'll can give full support to Firefox and other browsers

Comment 14

[Karl Dubost💡 :karlcow]

That's very good news Oliver. 
Impatient to see the next progress. 
I'm pretty sure that would make for a nice technical blog post.

Thanks.

Comment 15

[Karl Dubost💡 :karlcow]

A dependency that I'm adding as a seeAlso https://webcompat.com/issues/1295

Comment 16

[Karl Dubost💡 :karlcow]

We had another bug about Marfeel. :/
https://webcompat.com/issues/1356

Comment 17

[Oliver Fernandez]

Hello!

This is not happening in the current Firefox version available on the Google Play Store

However, we've noticed Firefox Beta has changed its user agent, and now our code confuses it with the Android AOSP browser, that's why it tries to load Marfeel's UX. Since this fails, what we do is mark the browser as not supported, reload the page, and not load Marfeel anymore.

Do you know if this user agent change is definitive?

Comment 18

[Mike Taylor [:miketaylr]]

Hi Oliver!

Yes, version 41 and above will have a different UA (they'll contain the Android version for compatibility reasons). I'm sorry that it's caused problems with Marfeel.

<https://developer.mozilla.org/en-US/docs/Web/HTTP/Gecko_user_agent_string_reference#Android_%28version_41_and_above%29> and <https://miketaylr.com/posts/2015/07/upcoming-changes-to-the-firefox-for-android-ua-string.html> have some more details on the UA string changes.

Comment 19

[Karl Dubost💡 :karlcow]

Olivier, 

could you give us information about the status of your project?

Comment 20

[Mike Taylor [:miketaylr]]

Seems like we still have issues here: https://github.com/webcompat/web-bugs/issues/576

Comment 21

[Karl Dubost💡 :karlcow]

We should look forward the gofaster add-on and study if only a user-agent override for the marfeel domain would be enough, given that marfeel has not moved yet on making the framework standard based. 

Adding a dependency on gofaster

Comment 22

[Karl Dubost💡 :karlcow]

We would like to know what's the status of marfeel effort?

Comment 23

[Mike Taylor [:miketaylr]]

(In reply to Karl Dubost :karlcow from comment #22)
> We would like to know what's the status of marfeel effort?

Who is this question directed at, Karl?

Comment 24

[Karl Dubost💡 :karlcow]

mike, still oliver (who is in needinfo)

oliver
We would like to know what's the status of marfeel effort?

Comment 25

[Mike Taylor [:miketaylr]]

(I think Karl accidentally removed ni?)

Comment 26

[Oliver Fernandez]

Hello,

First of all, please accept my apologies for not answering before. It turns out that my email server decided to mark emails for Bugzilla as spam, and I wasn't receiving them :( I've already whitelisted the domain.

Some months ago we did a quick fix to prevent the blank page bug. We detect the specific case of Firefox for Android User-Agent, and Marfeel does not load. We think that is better to show the normal web version than a complete white page.

We are currently working on our final solution: Marfeel will adopt a "progressive enhancement" architecture, where all mobile browsers will load the mobile version, and features will be turned on if the browser supports it. This includes Firefox, Opera, Edge, Chrome, Safari, ...

I can't still share a timeframe of when this will be available, but I'll keep you posted about our progress.

Comment 27

[Oliver Fernandez]

I forgot to mention that if you are still seeing the white page, please share with me the domain because we'll fix it ASAP

Comment 28

[Karl Dubost💡 :karlcow]

Oliver, 

That's very good news. I'll go through the previous domains we identified as being marfeel-powered and I'll report the ones which are blank. 

Thanks a lot.

Comment 29

[Karl Dubost💡 :karlcow]

Andro4all presents issues.
https://webcompat.com/issues/3605

Comment 30

[Oliver Fernandez]

So we have

- andro4all.com
- muycomputer.com
- xombitgames.com

We are going to take a look at it. Thanks for the report!

Comment 31

[Karl Dubost💡 :karlcow]

Oliver, Another domain to look at: applesencia.com
https://webcompat.com/issues/4672

Thanks.

Comment 32

[Karl Dubost💡 :karlcow]

yet another one. ghacks.net

Comment 33

[Oliver Fernandez]

We've started to roll-out Firefox compatibility (actually, any mobile browser) in some sites powered by Marfeel technology.

You can test it for example here https://misanimales.com or here https://youaremom.com/

Our plan is to activate all Marfeel features and roll it out to all sites during the coming months.

I'd like to apologize to all Firefox community for not having this earlier, but please understand that the work from our side has been to adapt a 6 year (and growing) code base. 6 year ago the edge features we started using (CSS 3D transformations mainly) where Webkit only, and it has been propagated back to our days.

Fortunately the mobile web  is not Webkit-only anymore. Our intention was never to leave Firefox behind, or work for specific browsers, but to make our product great when the features we needed were available.

Any feedback or bugs report will be highly appreciated :)

Comment 34

[Mike Taylor [:miketaylr]]

Wow that's fantastic news Oliver! We'll take some time to test those 2 sites and follow up with any issues.

Comment 35

[Mike Taylor [:miketaylr]]

Have emailed our Compat list with a request for some testing of the 2 mentioned sites. We'll link bugs here, if we find any.

Comment 36

[Mike Taylor [:miketaylr]]

Hi Oliver, did you ever get a chance to look at some of the associated Marfeel bugs? Thanks!

Comment 38

[Karl Dubost💡 :karlcow]

Some websites have abandoned marfeel it seems. A survey would be good.

Comment 39

[Firefox Bug Husbandry Bot]

See bug 1547409. Moving webcompat whiteboard tags to keywords.

Comment 40

[Raul Bucata]

It seems that the issue is not reproducible anymore, as some sites presented in the list show the correct mobile version of the page:

https://prnt.sc/PbN8BFGML6s5
https://prnt.sc/kErZvhEGng4r
https://prnt.sc/GeR7r5RtJAFu

Karl, can we close this as FIXED?

Tested with:

Browser / Version: Firefox Nightly 100.0a1 (2015869419 -🦎100.0a1-20220317092857🦎)
Operating System: Google Pixel 3 (Android 12) -1080 x 2160 pixels, 18:9 ratio (~443 ppi density)

Comment 41

[Karl Dubost💡 :karlcow]

Given these sites are still using Marfeel
https://sdk.mrf.io/statics/marfeel-sdk.js?id=154

It seems this is a good news and they finally fixed their scripts. Yoohoo!