585194 - Have tinderbox serve static logs with "Content-Type: text/plain" and "Content-Encoding: application/gzip" response headers

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Right now they're served as "Content-Type: application/gzip", which means that if one wants to read the raw log, one has to download, gunzip it, then load it in a text editor.  This is a fair amount of overhead, but oftentimes still faster than waiting for showlog.cgi.

If the headers were set correctly, one could just remove the "showlog.cgi" bits from the tinderbox URLs linked from everywhere, and directly load the textual log in web browsers.  I wrote a hacky patch to Firefox's HTTP code to munge these headers just for tinderbox.mozilla.org, and it works great.

This is part of a bigger plan to have TBPL do all its parsing client-side.

Comment 1

[[e-mail] (:bhearsum)]

Chris told me that this could make a significant difference in load, any chance someone can look at this soon?

Comment 2

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

I asked Ben about pushing on this because I've gotten more frequeunt ISEs on loading logs in the last day.

To be clear about load different, this bug alone would allow developers to conveniently bypass showlog.cgi to fetch logs, and fetching a static log should be close to free in terms of CPU cycles (at least compared to showlog.cgi).  The long term goal is to parse logs on the client side, in TBPL.  When we have that, there should definitely be a significant drop in load.  With just this bug fixed, the drop in load will be proportional to the number of developers who choose to step around showlog.cgi, which I can't estimate offhand though I know of maybe 4 people who've wanted this for a long time.  There may be others.

Comment 3

[Robert Helmer [:rhelmer]]

I tested this locally and I think the Apache settings are along the lines of:

AddType text/plain gz
AddCharset ASCII gz
AddEncoding x-gzip gz

I *think* that the only static .gz files inside the tree directories right now are the logs (they will look something like 1281421185.1281234504.1281239003.gz) so doing this for all of /tinderbox/*/ should be safe.

Comment 4

[Dave Miller [:justdave]]

all of the static cached files (.txt, .js, .html panel, quickparse, etc) are in that directory, too, so we'll still have to limit it to gz files.

Comment 6

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to comment #3)
> AddCharset ASCII gz

many of the logs actually contain UTF-8, though.

Comment 7

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I filed bug 601313 for the same issue (plus one more) on the logs being served on ftp.mozilla.org.

Comment 8

[Shyam Mani [:fox2mike]]

I just added this to tinderbox-stage.mozilla.org, can someone verify and then I'll push it to prod?

Comment 9

[Shyam Mani [:fox2mike]]

Alright, it works now : 

Before - 

trinity:~ shyam$ curl -I -L http://tinderbox-stage.mozilla.org/Firefox/1289373624.1289376709.405.gz
HTTP/1.1 200 OK
Date: Wed, 10 Nov 2010 08:35:00 GMT
Server: Apache/2.2.3 (Red Hat)
Last-Modified: Wed, 10 Nov 2010 08:12:05 GMT
ETag: "18d74ae-1345e8-494ae69acdf40"
Accept-Ranges: bytes
Content-Length: 1263080
Content-Type: application/x-gzip
Connection: close

After :

trinity:~ shyam$ curl -I -L http://tinderbox-stage.mozilla.org/Firefox/1289373624.1289376709.405.gz
HTTP/1.1 200 OK
Date: Wed, 10 Nov 2010 08:35:41 GMT
Server: Apache/2.2.3 (Red Hat)
Last-Modified: Wed, 10 Nov 2010 08:12:05 GMT
ETag: "18d74ae-1345e8-494ae69acdf40"
Accept-Ranges: bytes
Content-Length: 1263080
Content-Type: text/plain; charset=ascii
Content-Encoding: x-gzip
Connection: close

I can see logs in clear text in the browser (vs being asked to download the logs).

Verify that other functionality isn't hosed and I'd be happy to push this fix over to production.

Comment 10

[:Gavin Sharp [email: [e-mail]]]

I tried a few things on stage (loading other logs, starring, showlog.cgi) and it looks fine to me.

Comment 11

[Shyam Mani [:fox2mike]]

Alright, based on Gavin's comment..I've pushed this to tinderbox.m.o and verified on http://tinderbox.mozilla.org/Firefox/1289815404.1289819896.15016.gz