731090 - Make http://builddata.pub.build.mozilla.org/buildjson/ the main source of builds-4hr.js.gz and builds-YYYY-MM-DD.js.gz

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We generate these much more efficiently on buildapi01 than on cruncher, so lets do away with them on the latter and get people to use the former, via
  http://builddata.pub.build.mozilla.org/buildjson/

tbpl is already switched over, but I heard Axel isn't for his l10n systems.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I've disabled the thee cronjobs that catlee@cruncher runs, and blogged about using builddata.pub.b.m.o at http://blog.mozilla.com/nthomas/2012/02/28/new-home-for-build-data/.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Just to be clear, this is just about accelerating an existing process (bug 690577) to mitigate load issues on cruncher that are impacting on buildapi and self-serve. Load looks much better now.

We could redirect requests for '/builds/builds.*\.js(\.gz)?$' to builddata.pub.b.m.o if people think that would be useful. More complicated than you might first imagine to exclude some jquery files ending in .js in the same dir. Comments ?

Comment 3

[Axel Hecht [:Pike]]

The new host doesn't send the same http headers as the old one, that is, you can't just click on one of the js.gz and see the json.

Also, is the historic data going to be migrated over, too?

Also, when are the json files published on the new host? Right now, it's 3am on the 29th in PST, and I don't see a json file for today yet.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Axel Hecht [:Pike] from comment #3)
> The new host doesn't send the same http headers as the old one, that is, you
> can't just click on one of the js.gz and see the json.

Lets fix that up here. We're sending
 Content-Encoding: x-gzip
 Content-Type: text/plain; charset=UTF-8
on build.m.o, and 
 Content-Type: application/octet-stream
on builddata.pub.build.m.o. Need to figure out how configure nginx to do the right thing.

> Also, is the historic data going to be migrated over, too?

Yes, I can copy it over for you.

> Also, when are the json files published on the new host? Right now, it's 3am
> on the 29th in PST, and I don't see a json file for today yet.

Hmm, I see builds-2012-02-28.js.gz with a mtime of 21 past midnight on the 29th. It's only generating them once a day, where build.m.o was doing it every 30 minutes (but taking longer) and once at cron's @daily. Does that matter to you ?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

http://hg.mozilla.org/build/puppet-manifests/file/063a1aba7648/modules/buildapi/templates/buildapi-nginx.conf.erb is the file to modify for the http conf.

Comment 6

[Axel Hecht [:Pike]]

The cronjob I have to post-process the files runs every 15 minutes for some time of the day, actually, and is just taking the builds-foo.js.gz files. I'd wouldn't want to invest in making that talk to a variety of different files.

So yeah.

Comment 7

[Mike Taylor [:bear]]

(In reply to Nick Thomas [:nthomas] from comment #5)
> http://hg.mozilla.org/build/puppet-manifests/file/063a1aba7648/modules/
> buildapi/templates/buildapi-nginx.conf.erb is the file to modify for the
> http conf.

  gzip  on;
  gzip_http_version 1.1;
  gzip_types text/plain text/html text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

the above is what i've used with nginx - obviously the list of mime types can be shortened for moz's use

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [puppet-manifests] send .js.gz as text/plain with gzip encoding

First time touching nginx, but AIUI the existing gzip directives affect the builds-{running,pending}.js files, compressing them on the fly. There's no type assigned to gz in mime.types, so nothing happens to our js.gz files other than sending them on as application/octet-stream.

The new block forces js.gz requests to text/plain, which the gzip module is going to leave alone, and forces the Content-Type header. I think we're assuming the request will accept a compressed response, but already do this on build.m.o anyway.

I've tested this by duplicating the existing config with the patch (and incrementing the port etc).

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 601908 [details] [diff] [review]
[puppet-manifests] send .js.gz as text/plain with gzip encoding

http://hg.mozilla.org/build/puppet-manifests/rev/15a6237f3d81
and master-puppet1 updated, so this'll be on the public server shortly.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

All the historical data going back to 2009/09/30 has been copied over.

Perhaps builds-4hr.js.gz would be a better file to pull, if you're interested in looking for recent l10n builds.

Comment 11

[Axel Hecht [:Pike]]

Looking at builds-4hr would require a rewrite of the script of the script, basically, as right now, it's built on the premises that all builds for the day are in that one file. Incremental processing would need to tucked in.

I've fixed the script and the cron job to just run once a day for yesterday's nightlies now.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Let us know if we can adjust the cron job.