566093 - showlog.cgi is quite dumb, should be smarter

Status: RESOLVED DUPLICATE of bug 390341

(Webtools Graveyard :: Tinderbox, defect)

Description

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

We use showlog.cgi to serve logs, and it's (I believe) used by things such as the pushlog to pull out log data and the like.  This script seems really dumb in two major ways:

1) It does not set content-length, even when a full log is requested;

2) It does not do gzip encoding.

Both of these can be verified with wget:

[vladimir@dm-peep01 ~]$ wget 'http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1273782808.1273795801.23338.gz&fulltext=1'
--2010-05-14 20:19:12--  http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1273782808.1273795801.23338.gz&fulltext=1
Resolving tinderbox.mozilla.org... 10.2.74.14
Connecting to tinderbox.mozilla.org|10.2.74.14|:80... connected.
HTTP request sent, awaiting response...
  * lots of waiting -- why? *
Length: unspecified [text/html]
Saving to: `showlog.cgi?log=MozillaTry%2F1273782808.1273795801.23338.gz&fulltext=1'
    [                      <=>              ] 44,167,139  2.03M/s   in 19s
2010-05-14 20:19:47 (2.18 MB/s) - `showlog.cgi?log=MozillaTry%2F1273782808.1273795801.23338.gz&fulltext=1' saved [44167139]


The same file is served with --header="Accept-Encoding: gzip"

(There's a third issue here -- not sure what the 'lots of waiting' period is about.)

If this file had content-length, the browser could deal with it much more sanely.  Also, if the data were compressed before sending, instead of transmitting 44 MB, we'd transmit 2MB:

-rw-r--r-- 1 vladimir users 44167139 May 13 19:36 log.txt
-rw-r--r-- 1 vladimir users  2415180 May 13 19:36 log.txt.gz

gzipping the entire log file takes about 1s on people; I understand that this will likely lead to more cpu usage, but it seems like we should be able to get a handle on that for static log files, especially since the log file looks like it's already stored as a .gz file!

Does showlog.cgi actually do some parsing?  If so, sounds like we need to either get a cache in place there, or we need to move that parsing client-side.

Comment 1

[:Gavin Sharp [email: [e-mail]]]

See bug 390341 / bug 542874.

(It does do processing: http://mxr.mozilla.org/mozilla/source/webtools/tinderbox/showlog.cgi )

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to comment #0)
> gzipping the entire log file takes about 1s on people; I understand that this
> will likely lead to more cpu usage, but it seems like we should be able to get
> a handle on that for static log files, especially since the log file looks like
> it's already stored as a .gz file!
> 
> Does showlog.cgi actually do some parsing?  If so, sounds like we need to
> either get a cache in place there, or we need to move that parsing client-side.

In bug 390341 I suggested creating the two reports that can be generate at runtime by showlog.cgi (the "brief" and "complete" HTML reports) at the time that the log is sent to the server, and serving those directly if the client supports gzip encoding.

Overall this should reduce server CPU time, since right now the log is uncompressed and processing is done for every request.

Setting content-length would be trivial in this case, since the three files that tinderbox could serve (brief or complete report, or full raw log) would already be gzipped so the file size could be used and the file written directly to the client.

Comment 3

[Robert Helmer [:rhelmer]]

duplicate of bug 390341 (with expanded summary) per irc w/ vlad