708252 - The XMLRPC API doesn't work with IIS

Status: RESOLVED FIXED

(Bugzilla :: WebService, defect)

Description

[Karl M. Davis]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0
Build ID: 20111104165243

Steps to reproduce:

c:\Program Files (x86)\bugzilla\bugzilla-4.0.1\contrib>bz_webservice_demo.pl --uri http://tasks.us.i2inc.org/bugzilla/xmlrpc.cgi --login myuser --password mypw --bug_id 1

This is running against a Bugzilla installation hosted via IIS 7.


Actual results:

500 read timeout at C:\Program Files (x86)\bugzilla\bugzilla-4.0.1\contrib\bz_webservice_demo.pl line 188


Expected results:

I was expecting it to return the details of Bug #1.

Comment 1

[Karl M. Davis]

This seems to be a common problem:
*http://groups.google.com/group/mozilla.support.bugzilla/browse_thread/thread/3ebfee1e4e6ad4bd/fad9a3973beb5205
*http://groups.google.com/group/mozilla.support.bugzilla/browse_thread/thread/ec5e1aa2c3baca2e/8c6ea3b5c14a573d

Comment 2

[Karl M. Davis]

Attachment: checksetup.pl

The output from checksetup.pl for my Bugzilla installation.

Comment 3

[Frédéric Buclin]

Confirmed. I tested with IIS 7.5 and Bugzilla 4.3.2, and nothing is coming back.

Comment 4

[Frédéric Buclin]

Something is wrong in SOAP::Server::find_target(). This call:

 $self->on_action->(my $action = $self->action, $method_uri, $method_name);

never returns. In my case, arguments were:

 $self->on_action->(undef, 'Bug', 'get');

I lost track from here as I couldn't find how on_action() works.

Here is the traceback:

C:/Perl/site/lib/SOAP/Lite.pm:2867: SOAP::Server::find_target
C:/Perl/site/lib/SOAP/Lite.pm:2834: (eval)
C:/Perl/site/lib/SOAP/Transport/HTTP.pm:438: SOAP::Server::handle
C:/Perl/site/lib/SOAP/Transport/HTTP.pm:603: SOAP::Transport::HTTP::Server::handle
C:\bugzilla\xmlrpc.cgi:39: SOAP::Transport::HTTP::CGI::handle

Comment 5

[Thorsten Schöning]

(In reply to Frédéric Buclin from comment #4)
> I lost track from here as I couldn't find how on_action() works.

I hope I don't say obvious things to you, but on_action just seems to be a getter/setter for a caller defined sub, which defaults to sub { ; }:

    for my $method (qw(on_action on_dispatch)) {
        my $field = '_' . $method;
        *$method = sub {
            my $self = shift->new;
            # my $self = shift;
            return $self->{$field} unless @_;
            local $@;
            # commented out because that 'eval' was unsecure
            # > ref $_[0] eq 'CODE' ? shift : eval shift;
            # Am I paranoid enough?
            $self->{$field} = shift;
            Carp::croak $@ if $@;
            Carp::croak "$method() expects subroutine (CODE) or string that evaluates into subroutine (CODE)"
                unless ref $self->{$field} eq 'CODE';
            return $self;
        }
    }

Looks like per default on_action would just do nothing, but Bugzilla provides a different implementation in xmlrpc.cgi:

# We use a sub for on_action because that gets us the info about what 
# class is being called. Note that this is a hack--this is technically 
# for setting SOAPAction, which isn't used by XML-RPC.
$server->on_action(sub { $server->handle_login(WS_DISPATCH, @_) })
       ->handle();

Looks to me like the problem is in the login process somewhere, but I don't think it's in handle_login itself, as action as the only undef value seems to be ignored, the key Bug is available in WS_DISPATCH and Bugzilla::WebService::Bug should always be requirable if checksetup.pl returns successfully. On the other hand, ThrowCodeError seems to focus on HTML output and may explain error messages on the mailing list about malformed headers during the request.

Comment 6

[Marshall]

I've spent _days_ trying to get XMLRPC to work.  Can we please add a note to http://www.bugzilla.org/docs/4.4/en/html/api/Bugzilla/WebService/Server/XMLRPC.html saying something like:  "NOTE:  The XMLRPC interface will not work if Bugzilla is running on an IIS system.  See bug 708252.  The following alternatives are known to work and should be used for the time being:  option 1, option 2, ..., option N."

This should also be listed under:
  * http://www.bugzilla.org/releases/4.4/release-notes.html#v44_issues

Comment 7

[Pat Thoyts]

Attachment: Proposed fix for xmlrpc.cgi hang under IIS.

I have just spent some time working on this issue using IIS 8.5 running under Windows Server 2012r2 with Bugzilla release 4.4.9. This has exhibited the problem described here with a simple xmlrpc request hanging while jsonrpc requests succeed normally.

I added the Devel::Trace package to the xmlrpc.cgi and obtained a dump of all the lines executed while processing the xmlrpc call. It turns out that the fault occurs when trying to read from STDIN in read_from_client from CGI.pm. This tries to read CONTENT_LENGTH bytes from the POSTed input. However, the SOAP::Transport::HTTP package has already read this data in the 'handle' function. As a result the read call blocks. The cause of this additional read is due to the creation of a CGI object in the XMLRPC::make_response function when copying headers and cookies into the response. A simple fix is to set the CONTENT_LENGTH to 0 to indicate that no POST data remains before this happens. The patch below is sufficent to solve the problem in my case and I now get the correct result when requesting Bugzilla.version via xmlrpc.cgi.

> C:\Bugzilla>curl --verbose -X POST -H "Content-Type: text/xml" --data @z.xml http://localhost:3128/xmlrpc.cgi
> * About to connect() to localhost port 3128 (#0)
> *   Trying ::1...
> * Connection refused
> *   Trying 127.0.0.1...
> * connected
> * Connected to localhost (127.0.0.1) port 3128 (#0)
> > POST /xmlrpc.cgi HTTP/1.1
> > User-Agent: curl/7.28.1
> > Host: localhost:3128
> > Accept: */*
> > Content-Type: text/xml
> > Content-Length: 105
> >
> * upload completely sent off: 105 out of 105 bytes
> < HTTP/1.1 200 OK
> < Content-Length: 209
> < Content-Type: text/xml
> < Server: Microsoft-IIS/8.5
> < Set-Cookie: Bugzilla_login_request_cookie=uWZX2veQts; path=/; HttpOnly
> < SOAPServer: SOAP::Lite/Perl/1.14
> < X-Content-Type-Options: nosniff
> < X-Frame-Options: SAMEORIGIN
> < X-Xss-Protection: 1; mode=block
> < Date: Sat, 25 Apr 2015 22:11:17 GMT
> <
> <?xml version="1.0" encoding="UTF-8"?><methodResponse><params><param><value><struct><member><name>version</name><value><string>4.4.9</string></value></member></struct></value></param></params></methodResponse>

Comment 8

[Pat Thoyts]

Attachment: Updated patch after testing using bz_webservice_demo.pl

After doing some testing with the contrib/bz_webservice_demo.pl script I discovered the same fix needs to be applied for the handle_login function as well. The reason this works under Apache seems to be that XMLRPC.pm selects a different baseclass for apache (XMLRPC::Transport::HTTP::Apache) that that used for other webservers (XMLRPC::Transport::HTTP::CGI).

Any XMLRPC handler function will need to override the content-length or do something else to prevent attempts to read data from STDIN if they create a CGI object for any reason.

Comment 9

[Frédéric Buclin]

Per the SOAP::Lite documentation at http://search.cpan.org/~phred/SOAP-Lite/lib/SOAP/Lite.pm#CGI

 "CGI scripts may not work under IIS unless scripts use the .pl extension, opposed to .cgi."

Comment 10

[Pat Thoyts]

(In reply to Frédéric Buclin from comment #9)
> Per the SOAP::Lite documentation at
> http://search.cpan.org/~phred/SOAP-Lite/lib/SOAP/Lite.pm#CGI
> 
>  "CGI scripts may not work under IIS unless scripts use the .pl extension,
> opposed to .cgi."

The documentation is incorrect in this regard at least for IIS 8.5. IIS assigns no special powers to an file extensions unless the user assigns them so the quote makes little sense. To make bugzilla work under IIS 8 we need to assign *.cgi files to be handled by the CgiModule using an installed version of Perl. In my case this is strawberry perl using 'C:\strawberry\perl\bin\perl.exe -x"C:\Bugzilla" -wT "%s" %s'. IIS does not need to know anything about .pl scripts as these are not part of Bugzilla's web application.

Given the patch I attached the xmlrpc.cgi script works perfectly well when testing using the bz_webservice_demo.pl script and running the CGI under IIS 8.5. The SOAP documentation is presumably out of date and refers to previous versions of IIS which may have handled file extensions differently.

Comment 11

[Frédéric Buclin]

(In reply to Pat Thoyts from comment #10)
> Given the patch I attached the xmlrpc.cgi script works perfectly well when
> testing using the bz_webservice_demo.pl script and running the CGI under IIS
> 8.5.

Did you make sure your patch doesn't break XMLRPC when using Apache as web server?

Comment 12

[Pat Thoyts]

(In reply to Frédéric Buclin from comment #11)
> (In reply to Pat Thoyts from comment #10)
> > Given the patch I attached the xmlrpc.cgi script works perfectly well when
> > testing using the bz_webservice_demo.pl script and running the CGI under IIS
> > 8.5.
> 
> Did you make sure your patch doesn't break XMLRPC when using Apache as web
> server?

Yes. I have used the bz_webservice_demo script to check that under Apache on both Windows and Linux the xmlrpc service continues to operate correctly.

Comment 13

[David Lawrence [:dkl]]

LpSolit, do you have time to review this as you already have IIS running locally? Also do you think this is important enough for 5.0?

dkl

Comment 14

[Frédéric Buclin]

(In reply to David Lawrence [:dkl] from comment #13)
> LpSolit, do you have time to review this as you already have IIS running locally?

I deleted my VM with IIS installed on it a long time ago.


> Also do you think this is important enough for 5.0?

XMLRPC never worked with IIS, so there is nothing new here (aka not a regression). No reason to block 5.0 for it. It can land on the 5.0 branch once this patch is tested and reviewed.

Comment 15

[Simon Green]

Comment on attachment 8597709 [details] [diff] [review]
Updated patch after testing using bz_webservice_demo.pl

Just setting a reviewer.

Comment 16

[Pat Thoyts]

To possibly assist in getting this reviewed I've pushed a vagrant configuration that will setup a vagrant box using Windows Server 2012R2 and install Bugzilla stable running under IIS.

See https://github.com/patthoyts/bugzilla-iis-vagrant

Comment 17

[Frédéric Buclin]

I have reinstalled IIS 7.5 on Windows 7, but I installed ActivePerl 5.20.2, and unfortunately, XMLRPC::Lite is not available for this version of Perl due to https://rt.cpan.org/Public/Bug/Display.html?id=95613.

Comment 18

[Frédéric Buclin]

(In reply to Frédéric Buclin from comment #17)
> XMLRPC::Lite is not available for this version of Perl
> due to https://rt.cpan.org/Public/Bug/Display.html?id=95613.

Actually, that's not the reason. SOAP::Lite has been fixed one month ago (v1.14), but XMLRPC::Lite has not been recompiled on ActivePerl since this release, and so is not available via ppm. No idea how often ActivePerl tries to recompile modules which failed. Shame!

Comment 19

[Frédéric Buclin]

(In reply to Pat Thoyts from comment #8)
> The reason this works under Apache seems to be that XMLRPC.pm selects
> a different baseclass for apache (XMLRPC::Transport::HTTP::Apache) that that
> used for other webservers (XMLRPC::Transport::HTTP::CGI).

::Apache is only loaded with mod_perl, else ::CGI is loaded by default. So the same code is used by Apache with mod_cgi and by IIS.


> Any XMLRPC handler function will need to override the content-length or do
> something else to prevent attempts to read data from STDIN if they create a
> CGI object for any reason.

But reading from STDIN is not supposed to block anything if we request more characters than what is available.


I tested your patch with IIS 7.5 on Windows 7, and your patch only works if returned data contains ASCII characters only. Else it fails because the response is truncated.

Comment 20

[Pat Thoyts]

Attachment: Prevent re-encoding utf-8 output with IIS.

As noted by Frédéric Buclin in comment #19 enabling xmlrpc with IIS reveals a problem dealing with unicode data. This is a separate fault but it gets revealed once xmlrpc can be used with IIS server.

The problem is that after handling the login the STDOUT channel has had `binmode(STDOUT, :utf-8)` set on it. The XML data that makes up the response is explicitly encoded into UTF-8 in the SOAP::Lite SOAP::Transport::HTTP::Server::make_response function. When this response is written to the STDOUT channel it then gets re-encoded resulting in a double-encoding and a mismatch between the provided content-length and the amount of data written. IIS appears to check the content-length and does not send the excess data to the client.

To solve this I have re-asserted the :bytes layer on the Perl handle in the XMLRPC response handler so that this does not happen. I set this to only do this when the server is IIS but I checked unix/apache and it did not affect reading unicode responses there -- but it all works ok on Apache anyway.

This patch was tested on the current development version (5.1) of bugzilla.

Comment 21

[Frédéric Buclin]

(In reply to Pat Thoyts from comment #20)
> This patch was tested on the current development version (5.1) of bugzilla.

Hum, applying this patch has no effect. Must it be combined with your previous patch? If yes, could you attach one single patch and mark the two other ones as obsolete, please?

Comment 22

[Pat Thoyts]

Attachment: Fix XMLRPC usage when served by IIS on Windows

Combined patch to resolve the blocked read of the POSTed request and deal with responses containing Unicode data.

Comment 23

[Pat Thoyts]

Attachment: (For release-4.4-stable) Fix XMLRPC usage when served by IIS on Windows

The same patch to fix XMLRPC under IIS but applicable to the release-4.4-stable branch.

Comment 24

[Frédéric Buclin]

Comment on attachment 8613322 [details] [diff] [review]
Fix XMLRPC usage when served by IIS on Windows

> sub make_response {
>+    $ENV{CONTENT_LENGTH} = 0; # data has been read by SOAP::Transport::HTTP

Thanks for your patch! :) Your patch works fine with IIS 7.5. To not affect other web servers which already work fine, this should also be enclosed inside a |if $ENV{'SERVER_SOFTWARE'} =~ /IIS/| as you did for binmode(). I will upload an updated patch to do that. r=LpSolit

Comment 25

[Frédéric Buclin]

Attachment: patch, v2

Comment 26

[Frédéric Buclin]

Credits go to Pat.

Comment 27

[Frédéric Buclin]

To ssh://[e-mail]/bugzilla/bugzilla.git
   94b44ff..cdcb6f1  master -> master

To ssh://[e-mail]/bugzilla/bugzilla.git
   55e005a..f7de896  5.0 -> 5.0