947256 - Report ccache statistics at the end of the build

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement, P3)

Description

[Gregory Szorc [:gps]]

I think it would be useful if |mach build| printed ccache hit/miss rates at the end of the build.

Implementation:

1) Run and parse `ccache -s` at the beginning of the build
2) Run and parse `ccache -s` at the end of the build
3) Compute some simple hit/miss rates and print them.

e.g.

ccache hit rate: 32%; miss rate: 68%; files in cache: 159693 (+145)

This would probably live in the |mach build| command, which is implemented in python/mozbuild/mozbuild/mach_commands.py.

FWIW, I have a theory hit rates in the wild aren't as high as people suspect they are and this print out should shed some light on the data (whatever it is).

Comment 1

[Gregory Szorc [:gps]]

Attachment: Report ccache statistics at the end of the build

I banged this one out real quick. Not the most robust code in the world.
Needs refinement before landing. Perhaps someone can take it the last
mile.

Comment 2

[Sylvestre Ledru [:Sylvestre]]

> Needs refinement before landing.
Gregory, it is not very clear what you expect from a potential contributor here. Could you explain a bit more?

Comment 3

[Gregory Szorc [:gps]]

The patch needs a few tests and some documentation.

Comment 4

[Ting-Yu Lin [:TYLin] (PDT, UTC-7)]

Attachment: Report ccache hit rate at the end of the build

This patch add a line at the end of the build like follows:
ccache (direct) hit rate: 98.7%; (preprocessed) hit rate: 0.5%; miss rate: 0.8%

To run the test_ccachestats.py:
./mach python-test python/mozbuild/mozbuild/test/controller/

Gregory, if the documentation or tests are still incomplete, please give some
suggestions. Thanks.

Comment 5

[Gregory Szorc [:gps]]

Comment on attachment 8464573 [details] [diff] [review]
Report ccache hit rate at the end of the build

Review of attachment 8464573 [details] [diff] [review]:
-----------------------------------------------------------------

This looks sane. I'm going to send this to Try then land it for you if it looks good.

::: python/mozbuild/mozbuild/test/controller/test_ccachestats.py
@@ +1,3 @@
> +# This Source Code Form is subject to the terms of the Mozilla Public
> +# License, v. 2.0. If a copy of the MPL was not distributed with this
> +# file, You can obtain one at http://mozilla.org/MPL/2.0/.

This new file will need to be added to python/moz.build.

Comment 6

[Gregory Szorc [:gps]]

https://tbpl.mozilla.org/?tree=Try&rev=9c814745516d

Comment 7

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/fx-team/rev/56b1d9be395b

Comment 8

[Ting-Yu Lin [:TYLin] (PDT, UTC-7)]

Gregory, thank you for helping landing this patch. Excuse me for forgetting to mention your credit in the commit message since this patch is derived from your wip.

Comment 9

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/56b1d9be395b