taskclusteretl - include the raptor perftest-output in the ETL process
Categories
(Data Platform and Tools :: General, enhancement, P1)
Tracking
(Not tracked)
People
(Reporter: trink, Assigned: trink)
References
Details
currently only raptor-output is included
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 1•5 years ago
|
||
They reworked this code and renamed the logging components causing issues for ETL process. https://hg.mozilla.org/mozilla-central/rev/67e5d2425c755a8d8d3a3f369768c8bceebb02e8. Some of the talos data will be missing too as there was a bug fix to guard against excessively long log lines on Dec 9. https://bugzilla.mozilla.org/show_bug.cgi?id=1602651.
There is a general bug to refactor the ETL process which is in progress: https://bugzilla.mozilla.org/show_bug.cgi?id=1607793. However, with the arbitrary breaking changes we are seeing to the logs on an almost monthly basis and no communication about the changes I am not sure how much it will help.
There was a unrelated change (renaming of some GCP TC workerTypes) which broke the billing analysis last week which I am fixing now (schema and query changes). I most likely won't have enough time to get this fixed/tested/deployed before the work week and my PTO; pushing the production deployment out to early March.
AIUI, this issue blocks the completion of the Fenix performance release criteria dashboard. If that understanding is correct, I'd like to push for a rapid resolution. If a fix is not possible in the next week, I can try to find a workaround using the Treeherder REST API similar to how health.graphics pulls in data, but that approach add complexity and is somewhat fragile.
Assignee | ||
Comment 3•5 years ago
|
||
This is in test today (new refactored ingestion including this change), with a bit of luck it will be packaged and deployed to dev before the work week. During work week we can make the go/no go decision for production.
Assignee | ||
Comment 4•5 years ago
|
||
The refactor was deployed today. Feb 1 will be the first day with a full dataset.
Updated•3 years ago
|
Description
•