1355961 - Include some sccache stats in build_metrics

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[(not currently active) Ted Mielczarek]

I'd like to get some stats out of sccache into Perfherder. As a first cut I'm adding cache hit rate, cache write errors, and non-cacheable requests (compiler invocations that sccache can't handle). This should let us monitor some things that will hamper build times.

Comment 1

[(not currently active) Ted Mielczarek]

Attachment: bug 1355961 - add some sccache stats to build_metrics.

This commit makes sccache dump JSON stats at the end of the build, and then
reads them in `BuildScript.generate_build_stats` and adds them to the
build_metrics we submit to Perfherder. The stats dumping is done in
Makefile.in where we currently dump verbose sccache stats because sccache
doesn't persist stats to disk right now and it will also shut down its server
process after 5 minutes, so when the post-build automation steps take more
than 5 minutes the server shuts down and the stats are lost.

Currently it's collecting:
* Cache hit rate
* Cache write errors
* Non-cacheable requests (compiler invocations that sccache can't cache)

We can always grow this list later.

Review commit: https://reviewboard.mozilla.org/r/129580/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/129580/

Comment 2

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=33586777235fbaddf2b5c2e6885cc84f50c736cf

Comment 3

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=33586777235fbaddf2b5c2e6885cc84f50c736cf

That didn't work. I think I'm putting the stats JSON in the wrong place.

Comment 4

[Gregory Szorc [:gps]]

Comment on attachment 8857620 [details]
bug 1355961 - add some sccache stats to build_metrics.

https://reviewboard.mozilla.org/r/129580/#review132210

The only potential issue I see with this is what happens in case of failure loading the JSON file. If that raises an exception, that could fail the build. I like failing fast. But if there are any cases where the stats file isn't produced, this could bite us.

Comment 5

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #3)
> That didn't work. I think I'm putting the stats JSON in the wrong place.

Oh, hah! In Makefile.in I wrote "sccache-stats.json", but in buildbase.py I wrote "sccache_stats.json".

Comment 6

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f89685fd34dd424e81f695e355f742a59e7b65d2

Comment 7

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cfb76275fa9c45d95b803629ced8ef3078f8bf1d

Comment 8

[(not currently active) Ted Mielczarek]

OK, that's better:
    sccache cache_write_errors summary opt taskcluster-m4.4xlarge: 0
    sccache hit rate summary opt taskcluster-m4.4xlarge: 1
    sccache requests_not_cacheable summary opt taskcluster-m4.4xlarge: 49


    sccache cache_write_errors summary debug taskcluster-m4.4xlarge: 2
    sccache hit rate summary debug taskcluster-m4.4xlarge: 1
    sccache requests_not_cacheable summary debug taskcluster-m4.4xlarge: 49

Comment 9

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=67a5a35d88f28c7103a648870bfc602ea08c2939

Comment 10

[(not currently active) Ted Mielczarek]

https://hg.mozilla.org/integration/mozilla-inbound/rev/b5552791f6c8da640cfcb248ba4681d4d31594a5
bug 1355961 - add some sccache stats to build_metrics. r=gps

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/b5552791f6c8