Make http://builddata.pub.build.mozilla.org/buildjson/ the main source of builds-4hr.js.gz and builds-YYYY-MM-DD.js.gz

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: nthomas, Assigned: nthomas)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

6 years ago
We generate these much more efficiently on buildapi01 than on cruncher, so lets do away with them on the latter and get people to use the former, via
  http://builddata.pub.build.mozilla.org/buildjson/

tbpl is already switched over, but I heard Axel isn't for his l10n systems.
(Assignee)

Comment 1

6 years ago
I've disabled the thee cronjobs that catlee@cruncher runs, and blogged about using builddata.pub.b.m.o at http://blog.mozilla.com/nthomas/2012/02/28/new-home-for-build-data/.
(Assignee)

Comment 2

6 years ago
Just to be clear, this is just about accelerating an existing process (bug 690577) to mitigate load issues on cruncher that are impacting on buildapi and self-serve. Load looks much better now.

We could redirect requests for '/builds/builds.*\.js(\.gz)?$' to builddata.pub.b.m.o if people think that would be useful. More complicated than you might first imagine to exclude some jquery files ending in .js in the same dir. Comments ?
Assignee: nobody → nrthomas
Blocks: 690577
OS: Mac OS X → All
Priority: -- → P2

Comment 3

6 years ago
The new host doesn't send the same http headers as the old one, that is, you can't just click on one of the js.gz and see the json.

Also, is the historic data going to be migrated over, too?

Also, when are the json files published on the new host? Right now, it's 3am on the 29th in PST, and I don't see a json file for today yet.
(Assignee)

Comment 4

6 years ago
(In reply to Axel Hecht [:Pike] from comment #3)
> The new host doesn't send the same http headers as the old one, that is, you
> can't just click on one of the js.gz and see the json.

Lets fix that up here. We're sending
 Content-Encoding: x-gzip
 Content-Type: text/plain; charset=UTF-8
on build.m.o, and 
 Content-Type: application/octet-stream
on builddata.pub.build.m.o. Need to figure out how configure nginx to do the right thing.

> Also, is the historic data going to be migrated over, too?

Yes, I can copy it over for you.

> Also, when are the json files published on the new host? Right now, it's 3am
> on the 29th in PST, and I don't see a json file for today yet.

Hmm, I see builds-2012-02-28.js.gz with a mtime of 21 past midnight on the 29th. It's only generating them once a day, where build.m.o was doing it every 30 minutes (but taking longer) and once at cron's @daily. Does that matter to you ?

Comment 6

6 years ago
The cronjob I have to post-process the files runs every 15 minutes for some time of the day, actually, and is just taking the builds-foo.js.gz files. I'd wouldn't want to invest in making that talk to a variety of different files.

So yeah.

Comment 7

6 years ago
(In reply to Nick Thomas [:nthomas] from comment #5)
> http://hg.mozilla.org/build/puppet-manifests/file/063a1aba7648/modules/
> buildapi/templates/buildapi-nginx.conf.erb is the file to modify for the
> http conf.

  gzip  on;
  gzip_http_version 1.1;
  gzip_types text/plain text/html text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

the above is what i've used with nginx - obviously the list of mime types can be shortened for moz's use
(Assignee)

Comment 8

6 years ago
Created attachment 601908 [details] [diff] [review]
[puppet-manifests] send .js.gz as text/plain with gzip encoding

First time touching nginx, but AIUI the existing gzip directives affect the builds-{running,pending}.js files, compressing them on the fly. There's no type assigned to gz in mime.types, so nothing happens to our js.gz files other than sending them on as application/octet-stream.

The new block forces js.gz requests to text/plain, which the gzip module is going to leave alone, and forces the Content-Type header. I think we're assuming the request will accept a compressed response, but already do this on build.m.o anyway.

I've tested this by duplicating the existing config with the patch (and incrementing the port etc).
Attachment #601908 - Flags: review?(catlee)
(Assignee)

Updated

6 years ago
Attachment #601908 - Flags: review?(bear)

Updated

6 years ago
Attachment #601908 - Flags: review?(catlee) → review+

Updated

6 years ago
Attachment #601908 - Flags: review?(bear) → review+
(Assignee)

Comment 9

6 years ago
Comment on attachment 601908 [details] [diff] [review]
[puppet-manifests] send .js.gz as text/plain with gzip encoding

http://hg.mozilla.org/build/puppet-manifests/rev/15a6237f3d81
and master-puppet1 updated, so this'll be on the public server shortly.
Attachment #601908 - Flags: checked-in+
(Assignee)

Comment 10

6 years ago
All the historical data going back to 2009/09/30 has been copied over.

Perhaps builds-4hr.js.gz would be a better file to pull, if you're interested in looking for recent l10n builds.

Comment 11

6 years ago
Looking at builds-4hr would require a rewrite of the script of the script, basically, as right now, it's built on the premises that all builds for the day are in that one file. Incremental processing would need to tucked in.

I've fixed the script and the cron job to just run once a day for yesterday's nightlies now.
(Assignee)

Comment 12

6 years ago
Let us know if we can adjust the cron job.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.