839414 - Twitter hangs with add-on HTML Validator

Status: RESOLVED INCOMPLETE

(Firefox :: Extension Compatibility, defect)

Description

[mgueury]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:18.0) Gecko/20100101 Firefox/18.0
Build ID: 20130201065344

Steps to reproduce:

Note: I am the extension author.

Install Firefox
Install HTML Validator extension 
Go to www.twitter.com
In the seach bar, look for any keyword 
Or click on a #tag
Do this for sometimes. (2/3 times is enough)
The website will hang (a white turning circle will load infinitely)


Actual results:

Twitter hangs with HTML validator enabled


Expected results:

Not hanging.

The extension has not changed basically since several years.
This is a very new issue. I am debugging it, I currently think that the hanging is due or:
- to later version of Firefox
- to a change in Twitter that happens to show a bug that was already there.

This is the 1rst bug I have in several year. Unhappily, it is very annoying since the end-user have no way or clue to know what happens.

Comment 1

[mgueury]

Work-around 
-----------
For extension users, if they
- go to twitter
- right click on the icon: click disable for twitter.
- the problem goes away (it is hidden...)

Comment 2

[Loic]

Are you sure it's buggy only since FF18?
Because I tried with FF17/16 nightlies and it's buggy too.

Comment 3

[mgueury]

I do not know since when it happens. I do not have a lot of reference of older FF version on my machine. The extension code that is hanging does read a channel (further on this). Whatever, this code has not changed since Firefox 3.

In Firefox 3.6.xxx, the same thing works fine. I would need to downgrade to know since when it happens. Like told before, I am quite unsure what is triggering this bug to appear now.

Comment 4

[mgueury]

The code that hangs in Javascript is this one:

    var url = doc.URL;
    var urlCharset = doc.characterSet;
    var ios = Components.classes["@mozilla.org/network/io-service;1"]
                        .getService(Components.interfaces.nsIIOService);
    var channel = ios.newChannel( url, urlCharset, null );
    channel.loadFlags |= Components.interfaces.nsIRequest.VALIDATE_NEVER;
    channel.loadFlags |= Components.interfaces.nsIRequest.LOAD_FROM_CACHE;
    channel.loadFlags |= Components.interfaces.nsICachingChannel.LOAD_ONLY_FROM_CACHE;
        
...
    oTidyUtil.tidy.log( '<DEBUG>getHtmlFromCache: Part2b' );    
    try
    {
      // Use the cache key to distinguish POST entries in the cache 
      // (see nsDocShell.cpp)
      var cacheChannel = channel.QueryInterface(Components.interfaces.nsICachingChannel);
      cacheChannel.cacheKey = shEntry.cacheKey;
    } 
    catch(e) 
    {
      oTidyUtil.tidy.log( 'Exception 4' );    
    }

    oTidyUtil.tidy.log( '<DEBUG>getHtmlFromCache: Part2c' );    
--> var stream = channel.open();

What this does, is more or less this, this open a channel on a URL with the cachekey of the page and with LOAD_FROM_CACHE.

Comment 5

[mgueury]

I tried with FF 12.0 and the problem is already there. So, I guess the change comes from Twitter website.

Comment 6

[Loic]

(In reply to mgueury from comment #5)
> I tried with FF 12.0 and the problem is already there. So, I guess the
> change comes from Twitter website.

That's what I'm thinking too. Because I tried with various old versions of FF and the issue is present.

Comment 7

[mgueury]

Note, about FF 3.6. 
-------------------
It is possible that this bug was already there. When there was such issue, in FF 3.6, nothing happened until that you close the browser. There closing the browser was hanging. But since it was difficult to diagnose and happened nearly never, previous bugs about that were closed without Fix.

Comment 8

[Loic]

I forgot to add Twitter is updating its website a lot since a few weeks: long duration for search history, clickable links on mobile devices, Vine video service, etc.
So maybe the issue is in one of these updates.

NB: I tried with FF12, same issue.
In addition, with FF21, if I close Firefox during Twitter is hanging after running a search, Firefox stays in memory. :[

Comment 9

[mgueury]

The request of Twitter is a little strange, it seems to be just a <a href="/username"> but when clicking on it, I guess that the link click is intercepted by Javascript.

It requests the same URL /username with XHTTPRequest
and also ith a HTTP request header : accept: application/json, text/javascript.

The website answer with a text/javascript response.

Additional Notes
----------------
FF considers this to be a page reload (it is not) and fires an onLocationChange.
The onLocationChange being trapped by the extension.
The 2 requests (normal page and JS page) are get request to the same URL with the single diff seems to be the HTTP request accept header

Comment 10

[mgueury]

Additional Notes
----------------
If I do a view source after the click, I see that the page source is now the Javascript code. 
- I had expected that view source would hang too. But no.
- This mismatch of 2 pages HTML and Javascript with the same URL does
  also corrupt the HTML source who made the same error than the validator.
  It is like the HTML cache was overwritten by javascript of the XHTTPRequest

Comment 11

[mgueury]

Current state
-------------
It seems that the twitter does something uncommon, that set the mozilla cache in a unexpected state. 

Here is what happens is this:
- username go to twitter: GET www.twitter.com/username
- the page appears
- username click on the link "me", it is intercept by Javascript who does: 
  - a GET XHTTPRequest www.twitter.com/username (same URL)
  - but with a HTTP header Accept: application/json, text/javascript
  - a JSON results comes back
- This is seen like a OnLocationChange. 
  - The extension is triggered.
  - The extension takes the same CacheKey of the page than view source
    (see browser.js of FF)
    (see in HTMLValidator tidyBrowser.js getHtmlFromCache)
  - Creates a channel, with LOAD_FROM_CACHE_ONLY
  - Then open a channel to this page
  - When open is called. The extension thread hangs.
    -> This is the bug.

Observations
------------
- This happens with all FF version 3.6 and later (with different issues)
- The problem is probably after a recent change in twitter
- when the XHTTPRequest comes back, the HTML page in the cache
  www.twitter.com/username seems to have been replaced by the javascript 
  XHTTPRequest
- when doing view-source, the javascript page is read from the cache
  (successfully ?). But I had expected to see the HTML page? 
  Anyway, this is not important here.

What I tried already
--------------------
I tried to see before to open the page, 
- to check the HTTP header of the page, the presence of the Accept Header,
  but I do see application/json but text/html...
- several small changes, all unsuccessful,
- the only solution I have is to block all site by default or to 
  block twitter in the extension. 

What I have not tried yet
-------------------------
To open the page with asyncopen, but I am quite sceptic. Since if the problem exist too, the page will seems to work. And there will be hanging threads without that the
user can notice it. This was the behavior of FF 3.6 and what happens is that it was impossible to debug such issue.

My questions
-----------
Q1. Should I change something in the code  "[e-mail] 2013-02-08 
07:32:34 PST" (see in HTMLValidator tidyBrowser.js getHtmlFromCache) ? 

Q2. Would it be possible to change the code of FF so that it does not freeze
    Whatever, is the new behavior. I have not preference.
    a) get the original HTML page 
    b) get the JSON page
    c) get an error

Thanks

Comment 13

[mgueury]

Changing the component. What I really need is a fix in the channel.open. 

There is no work-around available that allows to change the extension to avoid the issue.

Comment 14

[Ioana (away)]

Your steps include installing an extension. If this bug doesn't reproduce without that extension, it's most surely an Extension Compatibility bug. So please check if it's reproducible without the extension.

Moving this bug on JS Engine in hope someone there can confirm if it's the JS bug you mentioned in comment 13.

Comment 15

[mgueury]

Any news ?

Comment 16

[Loic]

Are you sure it's a bug in Firefox?

Comment 17

[Ioana (away)]

Can you reproduce this issue without the HTML Validator extension?

Comment 18

[Johannes]

Can you please read the issue completely before adding standard comments like this? The issue has been opened by the creator of the HTML Validator extension himself, stating that the problem only happens if the extension is loaded, and that a call to a specific Firefox API function causes an infinite loop.

Comment 19

[Loic]

We need a minimal testcase, without the entire extension, maybe a small XPI showing the issue.

Comment 20

[mgueury]

Hi Loic,

Doing a small extension will not help you, what I propose is 2 things:
1) help you to see the code that cause the issue in the extension
2) show a side-effect of the twitter trick that causes this issue

1. How to debug the extension (and see the issue)
-------------------------------------------------
(it is OS independent and happens on all OS)
- Install HTML Validator extension
- Install venkman (Javascript debugger) extension

- Menu tools/javascript debugger
- In the debugger, be sure to disable the menu Debug/Exclude Application files
- In the browser, go to www.twitter.org (just to see the good files)
  - do nothing more
  - The validator icon shows (5) for HTML 5.
- In the debugger
  - Double click on TidyBrower.js  
  - Look for "channel.open"
  - Put a breakpoint there (or better some lines before)
    - please read the code of the function - getHTMLFromCache
    - it is quite straightforward, it is nearly a cut and paste of what view
      source does
- In the browser 
  - Click me
  - It will stop at the breakpoint 
  - When you do step over of the "channel.open" Firefox will hang

2. Cache issue 
--------------
- Go to twitter
- Login
- You will land in www.twitter.com
- Do view source (you see the page source the HTML page **)
- Click "me" on the above toolbar
- Here no HTML link is enabled (just AJAX XHTTPREQUEST is called) 
  (use whatever way to check this)
- Do view source 
  - Here I would expect to see the same as **
  - But you see the JSON answer to the AJAX CALL

Do not hesitate if you need more clarification

Comment 21

[Age Bosma]

In case it is of any use:

I've noticed that user names with upper-case characters cause the issue as well, while user names with all lower-case characters do not.

- Go to a random Twitter profile page like: https://twitter.com/firefox
- Click on the first @<UserName> you see in one of the posts.
- A "Profile Summary" pop-up appears
- Click on "Go to full profile" at the bottom of the pop-up.

If the user name has one or more upper-case characters in its name the loading indication at the top centre will keep spinning.

Comment 22

[Marco Zehe (:MarcoZ)]

No activity in three years. Please reopen if this is still an issue nowadays.