Closed Bug 1610609 Opened 5 years ago Closed 5 years ago

taskclusteretl - include the raptor perftest-output in the ETL process

Categories

(Data Platform and Tools :: General, enhancement, P1)

enhancement
Points:
3

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: trink, Assigned: trink)

References

Details

currently only raptor-output is included

Assignee: nobody → mtrinkala
Points: --- → 1
Priority: -- → P1

They reworked this code and renamed the logging components causing issues for ETL process. https://hg.mozilla.org/mozilla-central/rev/67e5d2425c755a8d8d3a3f369768c8bceebb02e8. Some of the talos data will be missing too as there was a bug fix to guard against excessively long log lines on Dec 9. https://bugzilla.mozilla.org/show_bug.cgi?id=1602651.

There is a general bug to refactor the ETL process which is in progress: https://bugzilla.mozilla.org/show_bug.cgi?id=1607793. However, with the arbitrary breaking changes we are seeing to the logs on an almost monthly basis and no communication about the changes I am not sure how much it will help.

There was a unrelated change (renaming of some GCP TC workerTypes) which broke the billing analysis last week which I am fixing now (schema and query changes). I most likely won't have enough time to get this fixed/tested/deployed before the work week and my PTO; pushing the production deployment out to early March.

Points: 1 → 3
Priority: P1 → P2

AIUI, this issue blocks the completion of the Fenix performance release criteria dashboard. If that understanding is correct, I'd like to push for a rapid resolution. If a fix is not possible in the next week, I can try to find a workaround using the Treeherder REST API similar to how health.graphics pulls in data, but that approach add complexity and is somewhat fragile.

This is in test today (new refactored ingestion including this change), with a bit of luck it will be packaged and deployed to dev before the work week. During work week we can make the go/no go decision for production.

Priority: P2 → P1

The refactor was deployed today. Feb 1 will be the first day with a full dataset.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Blocks: 1613439
Component: Pipeline Ingestion → General
You need to log in before you can comment on or make changes to this bug.