652359 - Bugzilla is rejecting long URLs again

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Jesse Ruderman]

Bugzilla is rejecting URLs longer than 4096 characters again. Per bug 445413, it should be 35000 characters.

I'm guessing this is a regression from the move to Phoenix (bug 628372) but it could be the Bugzilla software upgrade (bug 652325).

The error message is "Request-URI Too Large: The requested URL's length exceeds the capacity limit for this server."

Comment 1

[Dave Miller [:justdave]]

Hmmm...  That's an apache-level setting, not within Bugzilla, and puppet's distributing exactly the same apache config to the servers in PHX that it did to the ones in SJC.  I'll have to take a poke and see what's up.  maybe we lost that from the puppet config or something.

Comment 2

[Dave Miller [:justdave]]

This has been gone at least since we move into the cluster in San Jose.  That's when we put all of this stuff into puppet, and the initial commit into puppet doesn't have this.

I just added it back, should be live already.

Comment 3

[Jesse Ruderman]

Nope, still busted :(

Comment 4

[Dave Miller [:justdave]]

maybe mod_perl or CGI.pm overrides it?  Or maybe Zeus is throwing the error (though I'd suspect red text instead of black if it were from Zeus)

Comment 5

[David Lawrence [:dkl]]

(In reply to comment #4)
> maybe mod_perl or CGI.pm overrides it?  Or maybe Zeus is throwing the error
> (though I'd suspect red text instead of black if it were from Zeus)

As I remember, this is a compile time setting which with  RHEL the settingis 4096 bytes. If you had higher limit before you may have been using custom built rpms. With Red Hat's Bugzilla we blocked any request to raise the value and instead encouraged use of POST, Webservices, or using regexes instead when searching woth large list of componets foe example.

dkl

Comment 6

[Jesse Ruderman]

POST is a pain to work with in the browser, and the Bugzilla REST API doesn't seem to support POST at all.  So that's not a great option for me.

Comment 7

[Gervase Markham [:gerv]]

Using POST (UPDATE) for a GET (READ) operation is not very RESTful, so I am against changing the BzAPI to support POST. If the BzAPI's URL length limits are too low, I would look at raising them.

Is there any reason the default setting is so tiny? Presumably it's compile-time because Apache is written in C...

Gerv

Comment 8

[David Lawrence [:dkl]]

(In reply to comment #7)
> Using POST (UPDATE) for a GET (READ) operation is not very RESTful, so I am
> against changing the BzAPI to support POST. If the BzAPI's URL length limits
> are too low, I would look at raising them.

Definitely wasn't suggesting that.

> Is there any reason the default setting is so tiny? Presumably it's
> compile-time because Apache is written in C...

It is a compile time setting and currently 4096 is what is set by the Apache shipped by Red Hat. There is a config directive but it only works if you want to set the value lower than what the compile time setting is for whatever reason.

dkl

Comment 9

[Dave Miller [:justdave]]

We haven't run a custom apache on the Bugzilla servers since we upgraded from RHEL 4 (which was a couple years ago now).  That it took people this long to notice the problem had recurred makes me wonder about the importance.  Building a custom apache is doable, but will be a bit of work (more work involved to set up an mrepo repository specific to Bugzilla and adding that repo to the bugzilla servers so we don't accidentally deploy it everywhere than there will be involved in actually building it).  How badly do we need this?

Comment 10

[Jesse Ruderman]

> That it took people this long to notice the problem

A query I was running every Monday broke in April.  It did not take me long to notice.

Comment 11

[Dave Miller [:justdave]]

(In reply to comment #10)
> > That it took people this long to notice the problem
> 
> A query I was running every Monday broke in April.  It did not take me long
> to notice.

That would imply that the web servers in San Jose are still running the custom apache in question, which seems unlikely and makes me wonder if it's really not a compile time setting after all...

OK, I just checked, the exact same build of apache is in use on all of the webheads in both San Jose and Phoenix, and puppet is pushing identical apache config in both locations as well (the entire /etc/httpd directory for that matter).

The only other variable in this equation is the load balancer.  So there must be a setting on the load balancer somewhere that's different.

Comment 12

[Dave Miller [:justdave]]

OK, I just did a comparison of the HTTP protection settings on both load balancers, and both of them have a maximum url length setting, which is set to "unlimited" on both of them.

Comment 13

[Dave Miller [:justdave]]

Just to make sure if this is the issue or not, can you drop 63.245.209.72 into your hosts file for bugzilla.mozilla.org and see if it works again with you pointing at that IP?

Comment 14

[Jesse Ruderman]

Sorry, I was wrong.  It works fine now when I access BMO directly.  It's only BzAPI that's still busted.  I'll file a new bug for that.

Comment 15

[Jesse Ruderman]

Filed bug 660132 against BzAPI.

Comment 16

[Jesse Ruderman]

{
u'bugzilla_url': <URL that contains my password and works fine if accessed directly in Firefox>,
u'bugs': [], 
u'suggestion': u'Try the bugzilla_url in a browser. If it returns an error, the problem is probably with your Bugzilla rather than with the BzAPI.', 
u'error': 1, 
u'message': u'HTTP Error: 500 Line too long (limit is 4096)', 
u'http_code': 500
}

Comment 17

[Jesse Ruderman]

Oops, meant that comment for the other bug.