1036802 - Requests to the native rest/bzapi endpoints with gzip encoding always result in HTTP/200 responses

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, defect)

Description

[Karl Dubost💡 :karlcow]

+++ This bug was initially created as a clone of Bug #1036268 +++

Following a comment by Byron 1036268#c2

The difference in between these two requests (below) is:

   Accept-Encoding:

It seems the server requires to have "gzip" on the client side.

The correct way to deal with this for the server is either 

* to accept to return a representation which is plain json, without gzip. 
OR
* to send a "406 Not Acceptable" 
   http://tools.ietf.org/html/rfc7231#section-6.5.6

In addition in the payload, it is possible to inform the client about what's available:

>   The server SHOULD generate a payload containing a list of available
>   representation characteristics and corresponding resource identifiers
>   from which the user or user agent can choose the one most
>   appropriate.  A user agent MAY automatically select the most
>   appropriate choice from that list.  
 


→ http --print hH HEAD https://bugzilla.mozilla.org/rest/ "Accept-Encoding:compress"

HEAD /rest/ HTTP/1.1
Accept: */*
Accept-Encoding: compress
Host: bugzilla.mozilla.org
User-Agent: HTTPie/0.8.0

HTTP/1.1 400 
Access-control-allow-headers: origin, content-type, accept
Access-control-allow-origin: *
Connection: close
Content-Type: application/json; charset=UTF-8
Date: Thu, 10 Jul 2014 06:35:25 GMT
Etag: a7Gf340uZLdFVa9R/kfaEg
Server: Apache
Strict-transport-security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Backend-Server: web3.bugs.scl3.mozilla.com
X-content-type-options: nosniff
X-frame-options: SAMEORIGIN
X-xss-protection: 1; mode=block


→ http --print hH HEAD https://bugzilla.mozilla.org/rest/ "Accept-Encoding: gzip"

HEAD /rest/ HTTP/1.1
Accept: */*
Accept-Encoding:  gzip
Host: bugzilla.mozilla.org
User-Agent: HTTPie/0.8.0

HTTP/1.1 200 OK
Access-control-allow-headers: origin, content-type, accept
Access-control-allow-origin: *
Content-Encoding: gzip
Content-Length: 104
Content-Type: application/json; charset=UTF-8
Date: Thu, 10 Jul 2014 06:36:30 GMT
Etag: a7Gf340uZLdFVa9R/kfaEg
Server: Apache
Strict-transport-security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Backend-Server: web3.bugs.scl3.mozilla.com
X-content-type-options: nosniff
X-frame-options: SAMEORIGIN
X-xss-protection: 1; mode=block

Comment 1

[:glob ✱]

this looks like an infra/deployment issue -- bugzilla doesn't care about the encoding, and i can't reproduce this on environments other than bmo:

# GET request with gzip encoding incorrectly returns HTTP/200
~$ curl -s -o /dev/null -w '%{http_code}\n' -H "Accept-Encoding: gzip"  https://bugzilla.mozilla.org/rest/
200

# same request against my dev environment returns HTTP/400 as expected
~$ curl -s -o /dev/null -w '%{http_code}\n' -H "Accept-Encoding: gzip"  http://bz/bmo_4.2/rest/
400

# likewise for landfill
~$ curl -s -o /dev/null -w '%{http_code}\n' -H "Accept-Encoding: gzip"  https://landfill.bugzilla.org/bugzilla-tip/rest/
400

# a normal 404 has the correct status code
~$ curl -s -o /dev/null -w '%{http_code}\n' -H "Accept-Encoding: gzip"  https://bugzilla.mozilla.org/blah
404

# HEAD requests also return HTTP/200
~$ curl -I -s -o /dev/null -w '%{http_code}\n' -H "Accept-Encoding: gzip"  https://bugzilla.mozilla.org/rest/
200

Comment 2

[Kendall Libby [:fubar] (he/him)]

Running it on a webhead, returns 200:

web1.bugs.scl3# curl -s -o /dev/null -w '%{http_code}\n' -H "Accept-Encoding: gzip" -H "Host: bugzilla.mozilla.org" http://localhost:443/rest/
200

But I don't see anything in the apache configs that would cause a difference between the two.

Comment 3

[Ed Morley [:emorley]]

This affects the BzAPI compatibility layer too - see bug 1089805 comment 7 onwards.

Good:
$ curl -s -i -X PUT https://bugzilla.mozilla.org/bzapi/bug/1069823 | head -n 1
HTTP/1.1 401 Authorization Required

Bad:
$ curl -s -i -X PUT -H 'Accept-Encoding: gzip' https://bugzilla.mozilla.org/bzapi/bug/1069823 | head -n 1
HTTP/1.1 200 OK

Comment 4

[Ed Morley [:emorley]]

Transferring needinfo from bug 1089805:

(In reply to David Lawrence [:dkl] from bug 1089805 comment #16)
> Could I see what the current apache configuration is from the webhead to see
> if I can reproduce locally?
> dkl

Comment 5

[Kendall Libby [:fubar] (he/him)]

dkl has access to the webheads and has been poking around on dev. current theory is it being related to etag support: https://issues.apache.org/bugzilla/show_bug.cgi?id=45023

RHN doesn't appear to have an httpd newer than 2.2.15, which we're running, so I think we're left with workarounds. do we want to try https://issues.apache.org/bugzilla/show_bug.cgi?id=45023#c15 on dev? do you want to drop it in .htaccess, or have me put it in puppet?

Comment 6

[David Lawrence [:dkl]]

(In reply to Kendall Libby [:fubar] from comment #5)
> dkl has access to the webheads and has been poking around on dev. current
> theory is it being related to etag support:
> https://issues.apache.org/bugzilla/show_bug.cgi?id=45023
> 
> RHN doesn't appear to have an httpd newer than 2.2.15, which we're running,
> so I think we're left with workarounds. do we want to try
> https://issues.apache.org/bugzilla/show_bug.cgi?id=45023#c15 on dev? do you
> want to drop it in .htaccess, or have me put it in puppet?

So what we have currently in .htaccess is:

<IfModule mod_deflate.c>
  AddOutputFilterByType DEFLATE text/javascript application/json application/javascript text/css
</IfModule>

By commenting out the above, the issue is resolved in that the proper status code is returned. But adding in the workaround stated in https://issues.apache.org/bugzilla/show_bug.cgi?id=45023 to htaccess does not seem to work for me currently.

<IfModule mod_headers.c>
  RequestHeader edit "If-None-Match" "^(.*)-gzip$" "$1"
  Header edit "ETag" "^(.*[^g][^z][^i][^p])$" "$1-gzip"
</IfModule>

Reading word for word the docs at http://httpd.apache.org/docs/2.2/mod/mod_headers.html, this not working may be related to the version of Apache we are currently using. 

"Compatibility:	The merge argument is available in version 2.2.9 and later. The edit argument is available in version 2.2.4 and later."

So the fact that we are running 2.2.15 doesn't help.

To make it more interesting, I updated the code to not return ETags under any circumstances, and it still did not return the proper status code (always 200). 

So the only change that has changed the outcome at all in testing, is to disable gzip for application/json. Very strange.

dkl

Comment 7

[Ed Morley [:emorley]]

Could we disable gzip on the web head and let Zeus handle it instead?

Comment 8

[David Lawrence [:dkl]]

(In reply to Ed Morley [:edmorley] from comment #7)
> Could we disable gzip on the web head and let Zeus handle it instead?

That is something fubar is going to try. We can also test that on bugzilla-dev first I believe since it uses a load balancer.

Comment 9

[Kendall Libby [:fubar] (he/him)]

I managed to get the header modification working, but it didn't fix the issue.

in defaults.conf:
    <Location />
        RequestHeader  edit "If-None-Match" "^(.*)-gzip$" "$1"
        Header  edit "ETag" "^(.*[^g][^z][^i][^p])$" "$1-gzip"
    </Location>

sekrit$ curl -s -o /dev/null -w '%{http_code}\n' -H "Accept-Encoding: gzip" https://bugzilla-dev.allizom.org/rest/
200
sekrit$ curl -v -s -o /dev/null -w '%{http_code}\n' -H "Accept-Encoding: gzip" https://bugzilla-dev.allizom.org/rest/ 2>&1 | grep -i etag
< Etag: OT2PsqfQba5tiD/okvMX/Q

Reverted to normal, and the etag doesn't change. Red herring?

I've updated zeus to compress the following types (actually only had to add text/js and app/json):
 application/json      	
 text/javascript      	
 application/javascript      	
 application/x-javascript      	
 text/css      	
 text/html      	
 text/plain

I'll leave .htaccess changes to #bteam.

Comment 10

[David Lawrence [:dkl]]

Attachment: 1036802_1.patch

Fubar and I have confirmed that allowing Zeus to handle gzip compression and not Apache, that the issue with the status codes are resolved and the proper codes are returned.

This change removes the DEFLATE line from .htaccess and can be pushed live once the Zeus changes have been done in all the proper places.

dkl

Comment 11

[:glob ✱]

Comment on attachment 8522514 [details] [diff] [review]
1036802_1.patch

Review of attachment 8522514 [details] [diff] [review]:
-----------------------------------------------------------------

r=glob

Comment 12

[:glob ✱]

To ssh://[e-mail]/webtools/bmo/bugzilla.git
   9876111..e15f335  master -> master

i'll remind ashish to action bug 1099215 after this week's push.

Comment 13

[:glob ✱]

~$ curl -s -o /dev/null -w '%{http_code}\n' -H "Accept-Encoding: gzip"  https://bugzilla.mozilla.org/rest/
404

\o/