Add monitoring for number of try heads
Categories
(Developer Services :: Mercurial: hg.mozilla.org, task)
Tracking
(Not tracked)
People
(Reporter: sheehan, Assigned: sheehan)
Details
Attachments
(2 files)
When the number of heads on try grows too large, performance of pushes and other operations is affected. We should add some simple monitoring through InfluxDB that can tell us when we reach some number of heads on try, so we know when to prune them. The last time I pruned try, we had reached over 50k heads, which is far too many.
|
Assignee | |
Comment 1•5 years ago
|
||
This commit adds a Telegraf exec input plugin to monitor the
number of heads on the try repo on hgssh. The command is a call
to hg log, taking a call to the revset template function and
passing it through the count template filter to get the number
of revisions in the specified revset (in this case, head()).
The data is sent as a single integer to InfluxDB and a follow
up commit will introduce a TICKscript to send alerts when the
number of heads grows too high.
Despite using the head() revset from within the template for
the command, we specify -r 0 as the revset for the log command.
Mercurial's templater applies the given template to each revision
in the specified revset, and since we only want to run the revset
a single time and then count the number off revisions, we specify
a single revision in the -r argument.
|
|
Assignee | |
Comment 2•5 years ago
|
||
In the previous commit we added reporting for the number of
heads on try, and in this commit we tell Kapacitor to send a
Slack alert in the event we cross 40000/50000 heads (as a warning
and critical alert, respectively). This script is checked in to
version-control-tools with the other alerting scripts for tracking
purposes but will need to be manually added to Kapacitor through
Chronograf once the previous commit is deployed.
Depends on D33551
Pushed by cosheehan@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/3ae4efdfac68
telegraf: add an exec plugin to report the number of heads on try r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b793a0142828
TICKscripts: add alert for too many try heads r=glob
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/0c3c8497e990 TICKscript: fix time unit in `try_heads.tick`
Description
•