570793 - workermgmt app is erroring out

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Michael Morgan [:morgamic]]

Right now we're seeing: 
An error has occurred...

    * Application failed during request deserialization: no element found at line 1, column 0, byte -1 at /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/XML/Parser.pm line 187 

The error is likely:
12:05 <skeen> the php code look for ''faultString''  in the response xml and echo's it out

This is the old code, so wondering if Dave can help debug this error on the Bugzilla side?  Another alternative would be to fast-track the new version of this and get that out there (more in bug 564265).

Comment 1

[Stuart Parmenter]

Any update here?

Comment 2

[Michael Morgan [:morgamic]]

< morgamic> mrz-work: can someone look at bug https://bugzilla.mozilla.org/show_bug.cgi?id=570793 ?
< morgamic> nobody can fill out newhire stuff because the auth portion of the bugzilla API is returning an error
< morgamic> Application failed during request deserialization: no element found at line 1, column 0, byte -1 at  
            /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/XML/Parser.pm line 187 
< morgamic> code isn't in SVN either
< morgamic> afaik
< morgamic> so if someone can tar it up and send to me i can look
< morgamic> we also had a replacement for the old one; worse case we could swap them out - https://bugzilla.mozilla.org/show_bug.cgi?id=564265
< morgamic> need update on that bug too 
< morgamic> workermgmt bug re: perl error is https://bugzilla.mozilla.org/show_bug.cgi?id=570793

Comment 3

[Michael Morgan [:morgamic]]

I've got a new version of this whole thing in bug 564265 ready to go too.  I can't debug unless I have the code.  Sam said he'd look at it tonight.

Comment 4

[Dave Miller [:justdave]]

code is in svn, I put it there myself before I handed off to skeen a while back.  There were two branches, one was the current live site (which was trunk) and one was dtran's aborted rewrite (which was branches/jquery-rewrite).

svn+ssh://svn.mozilla.org/moco/webtools/workermgmt/trunk

Comment 5

[Dave Miller [:justdave]]

does this use the bugs-api server or does it use bugzilla's xmlrpc directly?

Comment 6

[Michael Morgan [:morgamic]]

Dunno, but Sam said: 21:47 <skeen> hey mike , just sent an email with what i found on workermgmt
21:48 <skeen> pretty much looks like something changed serverside behind https://bugzilla.mozilla.org/xmlrpc.cgi

Comment 7

[Michael Morgan [:morgamic]]

Sam's email had:

have there been any recent upgrades to bugzilla?


# The login page of the app posts to:

https://bugzilla.mozilla.org/xmlrpc.cgi

# with the post data (with my pw stared out):

<?xml version="1.0" encoding="iso-8859-1"?>
<methodCall>
<methodName>User.login</methodName>
<params>
 <param>
  <value>
   <struct>
    <member>
     <name>login</name>

     <value>
      <string>[e-mail]</string>
     </value>
    </member>
    <member>
     <name>password</name>
     <value>
      <string>***************</string>

     </value>
    </member>
   </struct>
  </value>
 </param>
</params>
</methodCall>

# and is currently getting a resp:

HTTP/1.1 200 OK
Date: Wed, 09 Jun 2010 04:27:14 GMT
Server: Apache
SOAPServer: SOAP::Lite/Perl/0.711
Content-Length: 453
X-Backend-Server: pm-app-bugs05
Content-Type: text/xml

<?xml version="1.0" encoding="UTF-8"?><methodResponse><fault><value><struct><member><name>faultString</name><value><string>Application failed during request deserialization: 
no element found at line 1, column 0, byte -1 at /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/XML/Parser.pm line 187
</string></value></member><member><name>faultCode</name><value><string>Client</string></value></member></struct></value></fault></methodResponse>

Comment 8

[Reed Loden [:reed]]

When exactly did this start? As far as I know, there haven't been any recent bmo upgrades.

http://groups.google.com/group/mozilla.support.bugzilla/browse_thread/thread/8e08f20532a8c5c6

Maybe related: https://bugzilla.gnome.org/show_bug.cgi?id=447102

Comment 9

[Michael Morgan [:morgamic]]

First saw it yesterday.

Comment 10

[Michael Morgan [:morgamic]]

Second seems x64 related.  Did we move bugzilla to a 64-bit box?

Comment 11

[Reed Loden [:reed]]

skeen: does your bugzilla password contain any weird characters that could be causing issues when being sent via xmlrpc?

Comment 12

[Reed Loden [:reed]]

Has workermgmt worked since bmo was moved behind zeus a few weeks ago?

Comment 13

[Shyam Mani [:fox2mike]]

Yeah, we've seen a number of new hire requests after..so it has been working. Mike, Dave says we moved Bugzilla to 64 bit boxes a few months back.

Comment 14

[Michael Morgan [:morgamic]]

Well... it fails now and no code has changed.  Do we know what modules have changed over the last week?  Server software updates?

Comment 15

[Michael Morgan [:morgamic]]

Also, if I just use "user" and "pass" as the combination we get the same error.  So it's gotta be some bad redirect that causes the loss of the POSTDATA then sends a post with no data in it (or maybe even a bad req).  Prod form's action is: 		<form method="post" action="">

Is that right?

Comment 16

[Michael Morgan [:morgamic]]

I don't know what else to test except to watch the net traffic on the box to catch what bugzilla is receiving (if anything) and what the intranet box is sending.

Comment 17

[Dave Miller [:justdave]]

This appears to have been broken by an update to the SOAP::Lite perl module.

One of the three webservers picked up the new version on June 4th, the other two got it on the 8th.

new version: perl-SOAP-Lite-0.711-1.el5.rf
reverted to: perl-SOAP-Lite-0.710.08-1.el5.rf

Can you try it again?

Comment 18

[Michael Morgan [:morgamic]]

Works now.  Yay!

Comment 19

[Michael Morgan [:morgamic]]

(also, this explains why the errors were intermittent for me on Monday)

Comment 20

[Dave Miller [:justdave]]

We have a winner.  It's even a known issue upstream:
https://bugzilla.mozilla.org/show_bug.cgi?id=569385