Closed Bug 1283463 Opened 8 years ago Closed 7 years ago

Support realtime logging of EMR clusters in Airflow

Categories

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P4)

Product:
Cloud Services Graveyard
Component:
Metrics: Pipeline
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rvitillo, Assigned: whd)

References

Details

(Whiteboard: [SvcOps])

User Story

Airflow doesn't currently show logs in real-time. Since the logs are uploaded in semi real-time to S3 while a job is running, Airflow could fetch those logs and display them in the GUI.
      No description provided.
Points: --- → 3
Priority: -- → P4
To clarify, Airflow does show real time logging of EMR cluster status via our EMR operator, but not the EMR logs themselves. The task here would be to modify our EMR operator to also fetch the EMR logs that are uploaded in semi real-time and add them to the logging output already available.

Depending on whether we move to the upstream EMR operator (bug #1325393), this would require maintaining a separate version of that operator, or making such logic generically useful.
Assignee: nobody → whd
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Whiteboard: [SvcOps]
Product: Cloud Services → Cloud Services Graveyard
You need to log in before you can comment on or make changes to this bug.