Closed Bug 1479132 Opened 7 years ago Closed 5 years ago

Test Databricks operator against main_summary DAG

Categories

(Data Platform and Tools :: General, enhancement, P2)

Product:
Data Platform and Tools
Component:
General
Type:
enhancement
Points:
2

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: amiyaguchi, Assigned: amiyaguchi)

References

Details

(Whiteboard: [DataPlatform])

The Databricks operator is available in airflow. There are a few attractive benefits here: * Ephemeral instances boostrap in minutes * Compute costs are charged by the minute * Auto-scaling can be enabled to avoid guessing cluster sizes It can be wrapped underneath the EMROperator to run our current jobs in an opaque way. The result of this should be the measured time of the smoke test and the total normalized compute hours (or cost).
Depends on: 1481274
Priority: -- → P2
Blocks: 1484337
Depends on: 1484343
Depends on: 1484346
Assignee: nobody → amiyaguchi
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX
Component: Scheduling → General
You need to log in before you can comment on or make changes to this bug.