taskclusteretl - Investigate the difference in CI vs Firefox Cloud AWS costs
Categories
(Data Platform and Tools :: General, task, P1)
Tracking
(Not tracked)
People
(Reporter: trink, Assigned: trink)
Details
The CI dashboard is reporting about 10K more a month. This could be due to post analysis credits and/or a different summation as they have independently maintained project lists.
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 1•5 years ago
•
|
||
It appears the Firefox Cost filter contains moz-fx-c-community-workers (which is not included in the Firefox CI analysis, that is in the community analysis). The Firefox cost analysis does not include the cloudops-taskcluster-aws-prod project. Another very small difference is the CI dashboard uses the unblended cost. So we aren't comparing the same thing at least on the main page. If you go to Page 4 on the Firefox cost dashboard and select Application = Taskcluster, Project = Taskcluster Platform - 8100 and cloudops-taskcluster-aws-prod, Environment = prod and undefined, Provider = AWS the filter set will be closer to aligning the two.
But you get
Mar 314K vs 297K (If I run the CI cost query again I get 314K, so clearly the data has changed since it was first calculated)
Apr 291K vs 274K (CI query re-run produces 290K)
May 243K vs 275K (242K after re-running and having the 28K credit come out)
The strange part is all the cost data was backfilled in May so I would have expected Mar and Apr to have been finalized by then. In normal operation the cost is only updated in a five day window and never revisited.
Note: the underlying derived_daily_costs_per_workertype table matches the raw queries off the aws billing data so at least that is good... it is pointing to an issue with the derived_workertype_costs rollup
|
|
Assignee | |
Comment 2•5 years ago
|
||
The derived_workertype_costs query ignores workerPoolIds with no hours in the cost_per_workertype table. This is fallout from the detail cost breakout work as there are rows with no hours that validly have to be ignored, adding an additional query to pick up rows with no hours and no cost_per_ms should capture the additional overhead
|
|
Assignee | |
Comment 3•5 years ago
|
||
https://github.com/mozilla-services/lua_sandbox_extensions/pull/517
Applied the changes and the comparison with the filters described in comment one are now (amortized vs unblended cost so they will not be an exact match):
Mar 314,042 vs 313,678
Apr 290,870 vs 290,289K
May 243,135 vs 281,489 (there was an after the fact credit of 28K)
So that is looking good, whether we should switch to amortized cost and do we want to apply credits a month after the fact are a different discussions. Please open a bug for either if you have strong opinions.
Description
•